1. (brick) Do a yum update as described above
2. (brick) Turn off puppet agent: service puppet stop
3. (uclhcpup) Bump hiera osg version and optionally condor and gwms in the relevant node yaml file in /etc/puppetlabs/code/environments/production/hieradata/nodes:

1. (brick) puppet agent --test errors are harmless, this is just to get the latest osg-release
2. (brick) yum clean all
3. (brick) puppet agent --test a second time
4. (brick) Do a second yum update as described above
5. (brick) Turn on puppet agent: service puppet start

GIT Change Management

The master branch corresponds to the production puppet environment, and is cloned at /etc/uclhc-puppet/production. The development branch corresponds to the development puppet environment and is cloned at /etc/uclhc-puppet/development. Changes should be made in /etc/uclhc-puppet/development and tested on the uclhctest.t2.ucsd.edu host before pushed to production.

Assuming changes are committed and pushed back to github from development, the following steps merge the changes to production:

Upgrade OSG Version

1. (all bricks) Do a yum update as described above
2. (all bricks) Turn off puppet agents: service puppet stop
3. (uclhcpup) Bump hiera osg version and optionally condor and gwms in /etc/puppetlabs/code/environments/production/hieradata/global.yaml:

puppet resource service puppet ensure=running enable=true

Puppetize brick

Open firewall in uclhcpup; see /etc/sysconfig/iptables

rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

yum install puppet

Modify /etc/puppet/puppet.conf and fill in the relevant hostname:

[main]
vardir = /var/lib/puppet
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl


[agent]
pluginsync      = true
report          = true  
ignoreschedules = true
daemon          = false
ca_server       = uclhcpup.t2.ucsd.edu
certname        = hostname
environment     = production
server          = uclhcpup.t2.ucsd.edu
configtimeout   = 10m

Start puppet:

puppet resource service ensure=running enable=true

On uclhcpup:

Look for and sign hostname:

puppet cert list
puppet cert sign hostname

Create relevant yaml in /etc/puppet/hiera/nodes/

Generate a node specific ssh auth keys file from /etc/puppet/environments/production/modules/uclhc-admin/files/root.ssh

Ensure host certs are copied and named appropriately into /etc/puppet/environments/production/modules/host_cert/files/ and check permissions

In foreman, assign relevant host group (UCLHC/ATLAS / UCLHC/CMS):
hosts -> click host -> edit -> hostgroup pulldown

click submit.

To apply classes immediately, run:

puppet agent --test

Admin Documentation

Brick Yum Update

We pin some packages to specific version numbers and only upgrade through puppet, these should be excluded in a normal yum update:

yum update --exclude=condor* --exclude=glideinwms* 

Upgrade OSG Version

1. (all bricks) Do a yum update as described above
2. (all bricks) Turn off puppet agents: service puppet stop
3. (uclhcpup) Merge dev to master:

   cd /etc/uclhc-puppet/production
   git fetch
   git merge origin/development
   git push

4. (uclhcpup) Bump hiera osg version and optionally condor and gwms in /etc/puppet/hiera/global.yaml:

   osg_repos::release: x
   condor::condor_version: y
   condor::gwms_version: z
   

5. (all bricks) puppet agent --test errors are harmless, this is just to get the latest osg-release
6. (all bricks) yum clean all
7. (all bricks) puppet agent --test a second time
8. (all bricks) Do a second yum update as described above
9. (all bricks) Turn on puppet agent: service puppet start

-- JeffreyDost - 2016/02/09