Nodes

• pnfs-2.t2.ucsd.edu used for gfactory
• gftp-6b.t2.ucsd.edu used for job submission

cd ~fkw/CMS/CMSSW_1_5_4
source source_me
cd ../submitDir

Email about restarting gfactory

If things still do not work, you need to login to
gfactory@pnfs-2

kill the glidein factory
('killall python' will work)

and create a new factory.

You can go to 
glideinWMS/creation
and modify
glideinWMS.xml

Look for "Expose_Grid_environment" (or similar) and put all values to True...
I thought I set it right, but there must be a bug somewhere :(

The change the name (the one there is v3.1, make it v3.2 or v4)
and create the new factory
./create_glidein glideinWMS.xml

at this point, go to $HOME
cd $HOME

modify start_factory.sh to point to the correct new one
(replace v3.1 with whatever you called it)
and you can now restart it
./start_factory.sh

Restarting the submission machine

I managed to crash condor on gftp-6b by running out of disk space. After cleaning up all the disk space, I logged in as root and did a /etc/init.d/condor restart Checked what this actually does. It appears to start up the 9 schedd's.

The gfactory web monitoring

The most useful part is in the total directory.

condor_history -name schedd_jobs7@ -completedsince "2007-11-12 06:14"

Setting priority factors in condor

condor_userprio -all
condor_userprio -setprio group_cms.uscms1586@osg-gw-4.t2.ucsd.edu 0.5
condor_userprio -setfactor group_cms.uscms1586@osg-gw-4.t2.ucsd.edu 1
condor_userprio -all

Refreshing the proxy for gfactory

All you need to do is:
On gftp-6b:
----------
voms-proxy-init -voms cms -valid 500:0.0
 This creates /tmp/x509up_u502
the copy that over to pnfs-2.

On pnfs-2:
----------
copy the file into ~gfactory/.globus/x509_service_proxy

Then be patient for about 30min or an hour or so. Eventually, the system starts back up.
If you kill and restart, it might go a little faster. However, Igor claims you need not kill and restart the gfactory.