This confused me because I once succeeded with the second, and then never again.

To see that you are succeeding, login to glidein-1 and do:

condor_q -global -globus

This will show you the status of the glideins at the remote host.

In general, gfactory is pretty quick. It picks up the submissions form glidein-2 within a minute, and submits glideins accordingly. If you see any significant delay then something's wrong.

gfactory details

The gfactory is deployed on glidein-1 in the account "gfactory". According to the ~gfactory/start_factory, the present version is in:
/home/gfactory/glideinsubmit/glidein_CCRC08_2

The master xml file that describes the system seems to be in:
~gfactory/glideinWMS/creation/glideinWMS.xml

logfiles

Glidein logs are in ~/glideinsubmit/glidein/entry_/log

In this case, it is ~/glideinsubmit/glidein_CCRC08_2/entry_CIEMAT-LCG2-LCG02-CMS/log

They are in job.*.err

manage gfactory

In /home/gfactory/start_factory.sh look up which version of the gfactory configuration is presently running. Then copy the xml for that version into the creation directory, modify it, and reconfigure.

ps -auwx |grep python
killall python
killall -9 python
cd glideinWMS/creation
cp ~/glideinsubmit/glidein_CCRC08_2/glideinWMS.xml

./reconfig_glidein glideinWMS.xml
cd
./start_factory.sh

Location of the script that is run by glidein on worker nodes

[1012] gfactory@glidein-1 ~/glideinsubmit/glidein_CCRC08_2$ pwd
/home/gfactory/glideinsubmit/glidein_CCRC08_2
[1016] gfactory@glidein-1 ~/glideinsubmit/glidein_CCRC08_2$ less glidein_startup.sh 

Useful commands on glidein-1

condor_status -any -constraint 'GlideinMyType =?= "glidefactory"' -format 'Entry: %s ' EntryName -format "Site: %s " GLIDEIN_Site -format "Gatekeeper: %s\n" GLIDEIN_Gatekeeper

Next some useful bdii querries:

ldapsearch -LLL -x -h lcg-bdii.cern.ch -p 2170 -b 'mds-vo-name=local,o=grid'

ldapsearch -LLL -x -h is.grid.iu.edu -p 2170 -b 'mds-vo-name=local,o=grid'

Finally, a query to show the status of the remote glideins at the remote sites:

condor_q -global -globus

For this last query, the glidien at the remote site can be in one of the following states:

• ACTIVE = well it's running
• PENDING = its queued up in the remote batch system
• UNKNOWN = this is what happens when a glidein gets lost for whatever reason. It's basically the same as the "hold" state.
• UNSUBMITTED = in the schedd queue on gfactory but not yet sent to the remote site
• STAGE_OUT = no idea what this means

Whenever you want to know more about a specific job, do:

condor_q -global -globus -l 333.2

Here 333.2 is the condor id for that job. There's a variety of classAd attributes that give you important information on the state of the job.

Igor says that frontend does a condor_release on the held jobs in gfactory. He says furthermore, that the held jobs should then be taken care of by condor_g if they don't exist any more at the remote site. He says that doing a condor_rm is really not useful, nor advisable. Given that Igor expects condor_g to take care of the UNKNOWN once released by frontend, there is no counter in frontend that would check how often a job was released. Given that we have presently (May11 21:37) 1487 UNKNOWN, we got plenty of opportunity to watch if this is really happening. An initial wild guess indicates that this is indeed working because this afternoon (May 11 15:32) we had 2099 UNKNOWN. However, the UNSUBMITTED seem stuck forever if they are stuck. We have 874 of these at hera since earlier in the day, and this count hasn't changed.

From Igor:

Another useful command on glidein-1
cd glideinWMS/tools/
python wmsXMLView.py

This gives you the same information as:
http://glidein-1.t2.ucsd.edu:8319/glidefactory//monitor/glidein_CCRC08_2/schedd_status.xml

Igor

Here's the monitoring

http://glidein-1.t2.ucsd.edu:8319/glidefactory/monitor/glidein_CCRC08_2/

things not to do

don't change x509 stuff while condor has anything in the queue.

frontend

To restart the frontend:

su - frontend
ps -auwx |grep python
killall python
killall -9 python
./start_frontend.sh
ps -auwx |grep python

The shell script "start_frontend.sh" points to the location of the config file. The configs are pretty obvious. They include such things as maximum jobs expected to run. Maximum number of idle glideins per entrypoint, etc.

The configuration script is at:

/home/frontend/glideinfrontend/v2/etc/vofrontend.cfg

This is very CMS CRAB specific (it assumes you are running CRAB). The config lines that do this are

# specify the appropriate constraint
job_constraint='(JobUniverse==5) && ((DESIRED_Sites=!=UNDEFINED) ||
(DESIRED_Gatekeepers=!=UNDEFINED))'

# String for matching jobs to glideins
match_string='(job.has_key("DESIRED_Sites") and
(glidein["attrs"]["GLIDEIN_Site"] in job["DESIRED_Sites"].split(",")))
or (job.has_key("DESIRED_Gatekeepers") and
(glidein["attrs"]["GLIDEIN_Gatekeeper"] in
job["DESIRED_Gatekeepers"].split(",")))'

GCB

The GCB logs are in ~/gcbcondor/condor_local/log/

We have two of them gcbuser@t2data0 and gcbuser@gftp-3

Most of the interesting stuff is in GCB_Broker_Log. But ocasionally you want to look at the others, too.

Commands worth knowing on lxplus

source /afs/cern.ch/cms/LCG/LCG-2/UI/cms_ui_env.sh
lcg-info --list-se --vo cms --query 'SE=*osg-se.sprace.org.br*' --attrs Path,Root
lcg-ls -l -b -D srmv2 srm://osg-se.sprace.org.br:8443/srm/managerv2?SFN=/pnfs/sprace.org.br/data/

lcg-cp file://`pwd`/site.txt srm://t2data2.t2.ucsd.edu:8443/srm/managerv2?SFN=/pnfs/sdsc.edu/data3/cms/phedex/store/user/spadhi/abc.txt

UCSD cluster condor system

While I'm at it, I migth as well document how one changes the priority of a user on the UCSD tier-2 cluster. I had to do that on Sunday evening to get some traction on getting jobs through at UCSD.

First of all, find out where the negotiator is actually runnning:

condor_status -negotiator

At present, the negotiator is run on osg-gw-1 .

Then login to that node as root, and do:

condor_userprio -all |more
condor_userprio -all |grep uscms1586
condor_userprio -setfactor group_cms.uscms1586@osg-gw-2.t2.ucsd.edu 1
condor_userprio -setfactor group_cms.uscms1586@osg-gw-4.t2.ucsd.edu 1
condor_userprio -all |grep uscms1586
condor_userprio -setprio  group_cms.uscms1586@osg-gw-2.t2.ucsd.edu 0.5
condor_userprio -setprio  group_cms.uscms1586@osg-gw-4.t2.ucsd.edu 0.5

The setprio command basically resets your effective priority to 0.5, i.e you start over near zero, as if you hadn't run here in ages. The setfactor command sets the mu;tiplier for your wallclock time to get the effective priority. This is an integer. Setting it to 1 is thus the best you can do.

Managing the certificate

To manage a long lived certificate from a crontab without breaking out a sweat, the following is useful:

• First created a very long lived proxy:

[1600] fkw@uaf-1 ~/CMS/condor$  voms-proxy-init -hours 9876543:0 -out=/tmp/testForDave.file
Cannot find file or dir: /home/condor/execute/dir_11128/userdir/glite/etc/vomses
Enter GRID pass phrase:
Your identity: /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373
Cannot find file or dir: /home/condor/execute/dir_11128/userdir/glite/etc/vomses
Creating proxy ............................................... Done

Warning: your certificate and proxy will expire Sat Feb  7 19:51:06 2009
which is within the requested lifetime of the proxy

• Then use that very long lived proxy to et a short lived proxy with a CMS attribute attached to it:

[1602] fkw@uaf-1 ~/CMS/condor$  voms-proxy-init -voms cms -hours 120 -valid=96:0 -cert=/tmp/testForDave.file 
Cannot find file or dir: /home/condor/execute/dir_11128/userdir/glite/etc/vomses
Your identity: /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373/CN=proxy
Cannot find file or dir: /home/condor/execute/dir_11128/userdir/glite/etc/vomses
Creating temporary proxy .................................................... Done
Contacting  voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "cms" Done
Creating proxy ........................................ Done
Your proxy is valid until Sat Jun  7 16:02:59 2008

• Finally, verify that this got me a proxy with attribute that is short lived, and by default in the right place:

[1603] fkw@uaf-1 ~/CMS/condor$ voms-proxy-info -all
WARNING: Unable to verify signature! Server certificate possibly not installed.
Error: Cannot find certificate of AC issuer for vo cms
subject   : /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373/CN=proxy/CN=proxy
issuer    : /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373/CN=proxy
identity  : /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373/CN=proxy
type      : unknown
strength  : 512 bits
path      : /tmp/x509up_u10179
timeleft  : 95:59:39
=== VO cms extension information ===
VO        : cms
subject   : /DC=org/DC=doegrids/OU=People/CN=Frank Wuerthwein 699373
issuer    : /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch
attribute : /cms/Role=NULL/Capability=NULL
attribute : /cms/uscms/Role=NULL/Capability=NULL
timeleft  : 95:59:39

-- FkW - 25 Apr 2008 -- SanjayPadhi - 06 Aug 2008