Installation Walkthrough and Notes on the GlideinWMS System

DISCLAIMER: Any information you read on this page is a lie. Actually, it isn't but you should treat it as such. These are my personal notes so YMMV! I do not want to be held responsible for your destruction of the world. Having said all of that, enjoy! :)

STEP 1: Obtain, install, and update a Linux distribution

I used Scientific Linux 5.3 from: http://www.scientificlinux.org/

You can also use Redhat Enterprise 5.3 or CentOS 5.3. My installation was on two VMWare Workstation 6.5.3 virtual machines named SL#1 and SL#2. I selected that they would have 512 MiB? of RAM and up to 16 GiB? hard disks. To use GlideinWMS, there is no requirement to use a 64-bit system. You can use a 32-bit and all of the software will be compatible. The reverse is not necessarily true, but I have not confirmed this yet. It's also a good idea to update the system with the latest recommended patches and security updates before continuing any futher. I also reconfigured /etc/ssh/sshd_config to use a different port and adjusted the VM to acquire an IP address from my router in order to make my virtual machines accessible remotely through SSH. I had to change /etc/hosts to list the names of my systems as FQDNs, aliases, and their local IP addresses. Also, I had to edit /etc/resolv.conf to setup DNS after switching to a static IP in the GUI since it wasn't getting it from DHCP anymore.

STEP 2: Install GlideinWMS

Part 1: Check and Install Prerequisite Software

First, we need to verify all the required software is loaded on our system.

• A reasonably recent Linux OS (RH/SL4 nad RH/SL5 tested at press time).

cat /etc/redhat-release
Scientific Linux SL release 5.3 (Boron)

• The Python interpreter (v2.3.4 or above)

python -V
Python 2.4.3

• The perl-Time-HiRes rpm.

perl -e 'use Time::HiRes;'
# NO ERROR - MEANS THAT IT IS INSTALLED

• The OSG client software.
  • Installed from the GlideinWMS installer by selecting "Components" and then "OSG VDT client".
• A HTTP server, like Apache or TUX.
  • Installed from the GlideinWMS installer by selecting "Components" and then "Web server".
• The Condor distribution.
  • This is NOT (?) required if you install: pool Collector AND User Schedd
  • Installed from the GlideinWMS installer by selecting "Components" and then "Base Condor installation".
• The RRDTool package (v1.2.18 or later)
  • The GlideinWMS installer tells us what to install by selecting "Components" and then "rrdtool graphics package".
  • Since no installation takes place, you have to go find the files on the web. The website that lists the installation RPMS points to broken links. I found the RPMs for my 64-bit installation here:
  • http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/

yum install ruby

# This is installed with --nodeps because it has a cross-dependency with rrdtool
rpm -ivh --nodeps perl-rrdtool-1.3.8-2.el5.rf.x86_64.rpm
rpm -ivh rrdtool-1.3.8-2.el5.rf.x86_64.rpm
rpm -ivh rrdtool-devel-1.3.8-2.el5.rf.x86_64.rpm
rpm -ivh python-rrdtool-1.3.8-2.el5.rf.x86_64.rpm

• The M2Crypto python library (v0.17 or later)

python -c "import M2Crypto? "
# NO ERROR - MEANS THAT IT IS INSTALLED

• The javascriptRRD package (0.4 or later)
  • This is also not installed by the GlideinWMS installer.
• The glideinWMS software.
  • Installed from the GlideinWMS installer.

Part 2: Install GlideinWMS Collector (my SL1)

Ok. Let's proceed with installing the collector on SL1. First we need a new user to own the installation. Let's call them gfactory.

su - root
groupadd -g 5001 gfactory
useradd -c "GFactory Daemon" -g 5001 -m -s /bin/bash -u 5001 gfactory

Let's download and copy condor to the gfactory's home directory:

mv condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz /home/gfactory/
cd /home/gfactory/
chown -R gfactory:gfactory /home/gfactory/condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz

We want to begin the installation with our new user, MAKE SURE TO DO THIS AS THE gfactory USER. The installation adds a cron job for gfactory, but the linux OS may not support cron jobs from the root user. Here is how to start the installation:

mv glideinWMS/ /home/gfactory/
chown -R gfactory:gfactory /home/gfactory/glideinWMS
su - gfactory
cd glideinWMS/install
./glideinWMS_install

Choose the option "glideinWMS Collector" and then let's answer the questions for the installer interactively. Ignore warnings/errors about CA certificates not being installed. The VDT installer doesn't do it, but the glideWMS installer does. Here are the installer answers:

Do you have already a VDT installation?: (y/n) n
Do you want to install the full OSG VDT client?: (y/n) n
Do you want to install a minimal Grid VDT client?: (y/n) y
Where do you want the VDT installed?: [/home/gfactory/vdt] /home/gfactory/vdt
Directory '/home/gfactory/vdt' does not exist, should I create it?: (y/n) y
What pacman version should I use?: [pacman-3.26]

This next question is a trick question, don't type anything here, just press enter or you'll get an error.

Which platform do you want to use (leave empty for autodetect):
WARNING: It appears that SELinux is enabled on this computer. ... Please press enter to continue the installation, or control-c to cancel.
Do you agree to the licenses? [y/n] y
Where would you like to install CA files? l
Where should I fetch the CAs from?: [http://software.grid.iu.edu/pacman/cadist/ca-certs-version]
Where do you have the Condor tarball? /home/gfactory/condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz
Where do you want to install it?: [/home/gfactory/glidecondor]
Directory '/home/gfactory/glidecondor' does not exist, should I create it?: (y/n) y
If something goes wrong with Condor, who should get email about it?: me@mydomain.com
Do you want to split the config files between condor_config and condor_config.local?: (y/n) [y]

In order for the Factory to submit to the grid, we need a proxy on all of the servers. However, we needed VDT to be installed before this is possible. Well, at least the minimal VDT is now installed, so we can use VOMS (Virtual Organization Membership Service) to create the proxy. Let's pause our installation in this window and open up another terminal window. In this new terminal, we will generate the proxy:

Using root, I put my certificate file cert.p12 in the gfactory directory, then I switched back to gfactory.

su - root
cd <the directory where the certificate is stored>
cp cert.p12 /home/gfactory
exit
su - gfactory

Now we need to source the VDT setup script in order to work with the VDT software.

. /home/gfactory/vdt/setup.sh

In order to initialize the voms proxy, I need the private key out of my cert. I was able to get my private key using this command and I saved it to a file called cert.key:

openssl pkcs12 -in cert.p12 -info

Now the permissions need to be adjusted on these files in order to generate the proxy:

chmod 644 /home/gfactory/cert.p12
chmod 400 /home/gfactory/cert.key

Run this command to generate the proxy for 500 hours:

/home/gfactory/vdt/glite/bin/voms-proxy-init -cert /home/gfactory/cert.p12 -key /home/gfactory/cert.key -out /home/gfactory/.globus/x509_service_proxy -valid 500:0.0

Now, let's make the cert and key owned by root to prevent any bad stuff from happening:

su - root
chown root:root /home/gfactory/cert.p12
chown root:root /home/gfactory/cert.key

Back in the previous terminal window with our Collector installation, let's continue where we left off.

Will you be using a proxy or a cert? (proxy/cert) proxy
Where is your proxy located?: /home/gfactory/.globus/x509_service_proxy
What name would you like to use for this pool?: [My glideinWMS pool] ChrisB? Pool
How many secondary schedds do you want?: [9] 3

******************************************
WMS collector successfully installed
******************************************

Part 3: Install GlideinWMS Factory (my SL1)

First, we must install some additional prerequisite software. We need to get the latest flot tarball, move it into the gfactory's home and extract it. Here is the website to get it from:

http://code.google.com/p/flot/

We will be using the same proxy used in Part 2, so no additional configuration will be necessary for the proxy. This file is:

/home/gfactory/.globus/x509_service_proxy

We also need to add a web directory for glidein submission:

su - root
mkdir /var/www/html/glidefactory
chown gfactory:gfactory /var/www/html/glidefactory

We are now installing the factory. Select option 2 from the glideinWMS installation script.

su - gfactory
cd /home/gfactory/glideinWMS/install
./glideinWMS_install

Here are my answers:

Do you have already a javascriptRRD installation?: (y/n) y
Where is javascriptRRD installed?: /home/gfactory/javascriptrrd-0.4.2
Do you have already a Flot installation?: (y/n) y
Where is Flot installed?: /home/gfactory/flot
Where is your proxy located?: /home/gfactory/.globus/x509_service_proxy
Where will you host your config and log files?: [/home/gfactory/glideinsubmit][/home/gfactory/glideinsubmit]
Directory '/home/gfactory/glideinsubmit' does not exist, should I create it?: (y/n) y
Where will the web data be hosted?: [/var/www/html/glidefactory]
What Web URL will you use?: [http://myhost.mydomain/glidefactory/] http://sl1/glidefactory/
Give a name to this Glidein Factory?: [mySites-sl1] gfactory-sl1
Give a name to this Glidein instance?: [v1_0]
What is the Condor base directory?: [/home/gfactory/glidecondor]
The following glidein schedds have been found: ... Do you want to use all of them?: (y/n) y
Do you want to use CCB (requires Condor 7.3.0 or better)?: (y/n) n
Please list all the GCB servers you will be using ... Leave an empty line when finished ... GCB node:
Do you want to use gLExec?: (y/n) y
Force VO frontend to provide its own proxy?: (y/n) [y] y
Do you want to fetch entries from RESS?: (y/n) [n] n
Do you want to fetch entries from BDII?: (y/n) [n] n
Please list all additional glidein entry points, Entry name (leave empty when finished): myentry
Gatekeeper for 'myentry': devg-6.t2.ucsd.edu
RSL for 'myentry':
Work dir for 'myentry': .
Site name for 'myentry': [myentry] mysite
gLExec path for 'myentry': [OSG]
Should glideins use the more efficient Match authentication (works for Condor v7.1.3 and later)?: (y/n) y
Do you want to create the glidein (as opposed to just the config file)?: (y/n) [n]n
To create the glidein, you need to run
/home/gfactory/glideinWMS/creation/create_glidein /home/gfactory/glideinsubmit/glidein_v1_0.cfg/glideinWMS.xml
Configuration files are located in /home/gfactory/glideinsubmit/glidein_v1_0.cfg
Remember to set X509_USER_PROXY before starting the daemon

Ok, now add this line to .bashrc:

export X509_USER_PROXY=/home/gfactory/.globus/x509_service_proxy

Let's create the glidein:

/home/gfactory/glideinWMS/creation/create_glidein /home/gfactory/glideinsubmit/glidein_v1_0.cfg/glideinWMS.xml

Let's restart Condor:

killall condor_master
/home/gfactory/glidecondor/start_condor.sh

Let's also restart the Factory:

/home/gfactory/glideinsubmit/glidein_v1_0/factory_startup stop
/home/gfactory/glideinsubmit/glidein_v1_0/factory_startup start

If the factory fails to start, consult the factory's log directory:

ls /home/gfactory/glideinsubmit/glidein_v1_0/log

This concludes the glideinWMS collector and factory installation. I hope it was fun.

Part 4: Install GlideinWMS Frontend System (my SL2)

The next system needed is the GlideinWMS frontend system. We will install this on my system named SL2. This system needs most of the pre-requisites of SL1.

• Any Condor-supported OS
• The OSG client software
• The Condor distribution
• A HTTP server, like Apache or TUX
• The RRDTool package (v1.2.18 or later)
• The M2Crypto? python library (v0.17 or later)
• The javascriptRRD package (0.4 or later)
• The perl-Time-HiRes rpm
• The glideinWMS software

Let's assume we've already installed these components. If not, follow the steps in part 1. First, we need to create a user account to host the frontend software installation:

su - root
groupadd -g 5002 frontend
useradd -c "Frontend Daemon" -g 5002 -m -s /bin/bash -u 5002 frontend

Make sure to get a copy of condor into the frontend's home directory:

cp condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz /home/frontend/
chown frontend:frontend /home/frontend/condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz

Copy the glideinWMS software into the frontend user's directory from CVS:

su - frontend
cvs -d :pserver:anonymous@cdcvs.fnal.gov:/cvs/cd_read_only co -r v2_1 glideinWMS

Now, let's being by installing Condor, select option 6 in the installer's menu:

/home/frontend/glideinWMS/install/glideinWMS_install
Do you have already a VDT installation?: (y/n) n
Do you want to install the full OSG VDT client?: (y/n) n
Do you want to install a minimal Grid VDT client?: (y/n) y
Where do you want the VDT installed?: [/home/frontend/vdt]
Directory '/home/frontend/vdt' does not exist, should I create it?: (y/n) y
What pacman version should I use?: [pacman-3.26]
What VDT cache should I use?: [http://vdt.cs.wisc.edu/vdt_1101_cache]
Which platform do you want to use (leave empty for autodetect):
Do you agree to the licenses? [y/n] y
Where would you like to install CA files? l
Where should I fetch the CAs from?: [http://software.grid.iu.edu/pacman/cadist/ca-certs-version]
Where do you have the Condor tarball? /home/frontend/condor-7.3.1-linux-x86_64-rhel5-dynamic.tar.gz
Where do you want to install it?: [/home/frontend/glidecondor]
Directory '/home/frontend/glidecondor' does not exist, should I create it?: (y/n) y
If something goes wrong with Condor, who should get email about it?: me@mydomain.ext
Do you want to split the config files between condor_config and condor_config.local?: (y/n) [y]
What node is the collector running (i.e. CONDOR_HOST)?: sl1

Condor for VO Frontend is installed

Now we need the HTTP server installation:

su - root
/home/frontend/glideinWMS/install/glideinWMS_install
Please select: 8
Please select: c
What port do you want to use?: [80]
Select install method: ... select: 1

HTTPD is installed

Copy over the JavaScriptRRD? package into the frontend's home directory:

cp javascriptrrd-0.4.2.zip /home/frontend/
chown frontend:frontend /home/frontend/javascriptrrd-0.4.2.zip
su - frontend
unzip javascriptrrd-0.4.2.zip

JavaScriptRRD is installed

We also need our proxy certificate to identify ourselves (this could be the same as the one generated on SL1):

su - frontend
mkdir .globus
scp -c blowfish -P 4221 gfactory@sl1:/home/gfactory/cert.p12 /home/frontend/
. /home/frontend/vdt/setup.sh
openssl pkcs12 -in cert.p12 -out cert.key

Now remove the public keys from the private key file:

vi cert.key

Now the permissions need to be adjusted on these files in order to generate the proxy:

chmod 644 /home/frontend/cert.p12
chmod 400 /home/frontend/cert.key

Run this command to generate the proxy for 500 hours:

/home/frontend/vdt/glite/bin/voms-proxy-init -cert /home/frontend/cert.p12 -key /home/frontend/cert.key -out /home/frontend/.globus/x509_service_proxy -valid 500:0.0

Now, let's make the cert and key owned by root to prevent any bad stuff from happening:

su - root
chown root:root /home/frontend/cert.p12
chown root:root /home/frontend/cert.key

Proxy was created and installed successfully!

Let's also install Flot:

wget http://flot.googlecode.com/files/flot-0.5.tar.gz
gunzip -c flot-0.5.tar.gz | tar xvf -

Flot was installed successfully!

Let's create a directory for the host and config log files:

mkdir /home/frontend/frontstage

Let's also create a directory for the web files:

su - root
mkdir /var/www/html/vofrontend
chown frontend:frontend /var/www/html/vofrontend

Finished with directory setup!

Now comes time to install the frontend. Let's begin:

. /home/frontend/vdt/setup.sh /home/frontend/glideinWMS/install/glideinWMS_install
Please select: 7
Do you have already a javascriptRRD installation?: (y/n) y
Where is javascriptRRD installed?: /home/frontend/javascriptrrd-0.4.2
Do you have already a Flot installation?: (y/n) y
Where is Flot installed?: /home/frontend/flot
Where is your proxy located?: /home/frontend/.globus/x509_service_proxy
Where will you host your config and log files?: [/home/frontend/frontstage]
Where will the web data be hosted?: [/var/www/html/vofrontend]
What Web URL will you use?: [http://sl2.minialienship.homeip.net/vofrontend/] http://sl2/vofrontend/
Give a name to this VO Frontend?: [myVO-sl2] frontend-sl2
Give a name to this VO Frontend instance?: [v1_0]
What node is the WMS collector running?: sl1
Collector name(s): [sl1]
Do you want to monitor all of them?: (y/n) y
What kind of jobs do you want to monitor?: [(JobUniverse? ==5)&&(GLIDEIN_Is_Monitor ! TRUE)&&(JOB_Is_Monitor ! TRUE)]
Give a name to the main group: [main]
Match string: [True]
Job attributes: []
Factory attributes: []
Do you want to use is to submit glideins: (y/n) [y] y
An empty entry means you are done. ... proxy fname:
DN (leave empty when finished):
Do you want to use the more efficient Match authentication (works for Condor v7.1.3 and later)?: (y/n) y
Do you want to expose the Grid env. to the user jobs?: (y/n) y
Do you want to create the VO Frontend instance (as opposed to just the config file)?: (y/n) [n] n

Ok, now add these lines to .bashrc:

vi ~/.bashrc
. /home/frontend/vdt/setup.sh
export X509_USER_PROXY=/home/frontend/.globus/x509_service_proxy

Let's create the VO Frontend instance:

/home/frontend/glideinWMS/creation/create_frontend /home/frontend/frontstage/instance_v1_0.cfg/frontend.xml

Let's restart Condor:

killall condor_master
/home/frontend/glidecondor/start_condor.sh

Let's also restart the VO Frontend:

/home/frontend/frontstage/frontend_frontend-sl2-v1_0/frontend_startup stop
/home/frontend/frontstage/frontend_frontend-sl2-v1_0/frontend_startup start

If the frontend fails to start, consult the frontend's log directory:

ls /home/frontend/frontstage/frontend_frontend-sl2-v1_0/log