Install and Configure BOSCO for Glidein-Based Submission

1. Finally, add the following elements to your frontend configuration file, frontend.xml. Note, you may add them to either the group or global credential definition. Note: All paths should be absolute, not relative.

    <credentials>
      <credential absfname="/path/to/grid_proxy" security_class="frontend" trust_domain="grid" type="grid_proxy"/>
      <credential absfname="/home/frontend/.ssh/bosco_key.rsa.pub" keyabsfname="/home/frontend/.ssh/bosco_key.rsa" pilotabsfname="/path/to/grid_proxy" security_class="frontend" trust_domain="bosco" type="key_pair"/>
   </credentials>
   

2. Please stop, reconfig, and restart your frontend. If successful, the FRONTEND_HOST is now properly configured.

   [root@FRONTEND_HOST ~]$ service gwms-frontend stop
   [root@FRONTEND_HOST ~]$ service gwms-frontend reconfig
   [root@FRONTEND_HOST ~]$ service gwms-frontend start
   

1. Remove and retouch the 60-campus_factory.config file.

   [root@FACTORY_HOST ~]$ rm /etc/condor/config.d/60-campus_factory.config
   [root@FACTORY_HOST ~]$ touch /etc/condor/config.d/60-campus_factory.config

2. Now, add the entry for the BOSCO_HOST to factory configuration file, glideinWMS.xml.

   <entry name="CMS_TX_US_XXXXX_BOSCO" auth_method="key_pair" enabled="True" gatekeeper="BOSCO_USER@BOSCO_HOST" gridtype="batch BATCH_TYPE" rsl="" trust_domain="bosco" verbosity="std" work_dir="~/">
      <config>
         <max_jobs>
            <default_per_frontend glideins="256" held="50" idle="50"/>
            <per_entry glideins="256" held="50" idle="50"/>
            <per_frontends>
            </per_frontends>
         </max_jobs>
         <release max_per_cycle="20" sleep="0.2"/>
         <remove max_per_cycle="5" sleep="0.2"/>
         <restrictions require_glidein_glexec_use="False" require_voms_proxy="False"/>
         <submit cluster_size="10" max_per_cycle="100" sleep="0.2" slots_layout="fixed">
            <submit_attrs>
            </submit_attrs>
         </submit>
      </config>
      <allow_frontends>
      </allow_frontends>
      <attrs>
         <attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="default"/> <attr name="GLEXEC_JOB" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="False"/>
         <attr name="GLIDEIN_CMSSite" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>
         <attr name="GLIDEIN_CPUS" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="8"/>
         <attr name="GLIDEIN_Country" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="US"/>
         <attr name="GLIDEIN_Glexec_Use" comment="This has been REQUIRED for historical reasons, OPTIONAL/NONE alt values" const="False" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="NONE"/>
         <attr name="GLIDEIN_MaxMemMBs" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="int" value="49152"/>
         <attr name="GLIDEIN_Max_Walltime" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="int" value="171000"/>
         <attr name="GLIDEIN_ResourceName" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>
         <attr name="GLIDEIN_Site" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>
         <attr name="GLIDEIN_Supported_VOs" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="CMS,MIS"/>
         <attr name="USE_CCB" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="True"/> <attr name="X509_CERT_DIR" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="/cvmfs/oasis.opensciencegrid.org/mis/certificates"/>
      </attrs>
      <files>
      </files>
      <infosys_refs>
      </infosys_refs>
      <monitorgroups>
      </monitorgroups>
   </entry>
   

3. Finally, build up a global ssh fingerprint list so that the FACTORY_HOST trusts the keys of both the BOSCO_HOST and the FRONTEND_HOST.

   [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa BOSCO_HOST >> /etc/ssh/ssh_known_hosts
   [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa FRONTEND_HOST >> /etc/ssh/ssh_known_hosts
   

4. Stop, reconfigure and restart your factory. If successful, the FACTORY_HOST is now properly configured. You may now submit user jobs to the BOSCO_HOST via the FRONTEND_HOST.

   [root@FRONTEND_HOST ~]$ service gwms-factory stop
   [root@FRONTEND_HOST ~]$ service gwms-factory reconfig
   [root@FRONTEND_HOST ~]$ service gwms-factory start 
   

Troubleshooting

If glideins and/or direct bosco user jobs fail to be successfully submitted into a local pbs/slurm batch system, it may be useful to modify the ~/bosco/glite/bin/pbs_submit.sh submission script on the BOSCO_HOST to see the qsub/sbatch error messages directly.

Before:

jobID=`${pbs_binpath}/qsub $bls_tmp_file` # actual submission
retcode=$?
if [ "$retcode" != "0" ] ; then
       rm -f $bls_tmp_file
       exit 1
fi

After:

jobID=`${pbs_binpath}/qsub $bls_tmp_file` # actual submission
retcode=$?
echo “Full qsub output: $jobID” 1>&2
if [ "$retcode" != "0" ] ; then
       rm -f $bls_tmp_file
       exit 1
fi

1. Finally, add the following elements to your frontend configuration file, frontend.xml. Note, you may add them to either the group or global credential definition. Note: All paths should be absolute, not relative.

    <credentials> <credential absfname="/path/to/grid_proxy" security_class="frontend" trust_domain="grid" type="grid_proxy"/> <credential absfname="/home/frontend/.ssh/bosco_key.rsa.pub" keyabsfname="/home/frontend/.ssh/bosco_key.rsa" pilotabsfname="/path/to/grid_proxy" security_class="frontend" trust_domain="bosco" type="key_pair"/> </credentials> 

2. Please stop, reconfig, and restart your frontend. If successful, the FRONTEND_HOST is now properly configured.

   [root@FRONTEND_HOST ~]$ service gwms-frontend stop [root@FRONTEND_HOST ~]$ service gwms-frontend reconfig [root@FRONTEND_HOST ~]$ service gwms-frontend start 

1. Now, add the entry for the BOSCO_HOST to factory configuration file, glideinWMS.xml.

    <entry name="CMS_TX_US_XXXXX_BOSCO" auth_method="key_pair" enabled="True" gatekeeper="BOSCO_USER@BOSCO_HOST" gridtype="batch BATCH_TYPE" rsl="" trust_domain="bosco" verbosity="std" work_dir="~/"> <config> <max_jobs> <default_per_frontend glideins="256" held="50" idle="50"/> <per_entry glideins="256" held="50" idle="50"/> <per_frontends> </per_frontends> </max_jobs> <release max_per_cycle="20" sleep="0.2"/> <remove max_per_cycle="5" sleep="0.2"/> <restrictions require_glidein_glexec_use="False" require_voms_proxy="False"/> <submit cluster_size="10" max_per_cycle="100" sleep="0.2" slots_layout="fixed"> <submit_attrs> </submit_attrs> </submit> </config> <allow_frontends></allow_frontends> <attrs> <attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="default"/> <attr name="GLEXEC_JOB" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="False"/> <attr name="GLIDEIN_CMSSite" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/> <attr name="GLIDEIN_CPUS" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="8"/> <attr name="GLIDEIN_Country" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="US"/> <attr name="GLIDEIN_Glexec_Use" comment="This has been REQUIRED for historical reasons, OPTIONAL/NONE alt values" const="False" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="NONE"/> <attr name="GLIDEIN_MaxMemMBs" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="int" value="49152"/> <attr name="GLIDEIN_Max_Walltime" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="int" value="171000"/> <attr name="GLIDEIN_ResourceName" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/> <attr name="GLIDEIN_Site" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/> <attr name="GLIDEIN_Supported_VOs" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="CMS,MIS"/> <attr name="USE_CCB" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="True"/> <attr name="X509_CERT_DIR" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="/cvmfs/oasis.opensciencegrid.org/mis/certificates"/> </attrs> <files> </files> <infosys_refs> </infosys_refs> <monitorgroups> </monitorgroups> </entry> 

2. Finally, build up a global ssh fingerprint list so that the FACTORY_HOST trusts the keys of both the BOSCO_HOST and the FRONTEND_HOST.

   [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa BOSCO_HOST >> /etc/ssh/ssh_known_hosts [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa FRONTENT_HOST >> /etc/ssh/ssh_known_hosts 

3. Stop, reconfigure and restart your factory. If successful, the FACTORY_HOST is now properly configured. You may now submit user jobs to the BOSCO_HOST via the FRONTEND_HOST.

   [root@FRONTEND_HOST ~]$ service gwms-factory stop [root@FRONTEND_HOST ~]$ service gwms-factory reconfig [root@FRONTEND_HOST ~]$ service gwms-factory start 

• Already have an ssh login to the target BOSCO_HOST

1. Login to the FRONTEND_HOST via ssh as the FRONTEND_USER. NOTE it is important to log in with -A, this assumes you already have your personal public key access to the BOSCO_HOST. The bosco_cluster --add command will use this login to copy the bosco credentials over to the node.

    [user@client ~]$ ssh -A FRONTEND_USER@FRONTEND_HOST

1. Since BOSCO is not installed in your FRONTEND_HOST path, we must (at least temporarily) source its environment configuration file, bosco_setenv. Please run the following:

    [FRONTEND_USER@FRONTEND_HOST ~]$ source ~/bosco/bosco_setenv 

1. Add the BOSCO_HOST by running the bosco_cluster script with the following parameters, this will forward the passwordless bosco ssh key, and install bosco on the remote side:

    [FRONTEND_USER@FRONTEND_HOST ~]$ bosco_cluster --add BOSCO_USER@BOSCO_HOST BATCH_TYPE 

   where BATCH_TYPE = pbs, condor, etc.
2. Run a BOSCO test job to check the connection between the FRONTEND_HOST and the BOSCO_HOST and its worker nodes.

    [FRONTEND_USER@FRONTEND_HOST ~]$ bosco_cluster --test BOSCO_USER@BOSCO_HOST 

1. Download the BOSCO installer tarball in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ wget ftp://ftp.cs.wisc.edu/condor/bosco/1.2/boscoinstaller.tar.gz 

2. Unzip and untar the BOSCO installer in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ tar -xzf boscoinstaller.tar.gz 

3. Run the boscoinstaller script to install BOSCO on the FRONTEND_HOST.

   [FRONTEND_USER@FRONTEND_HOST ~]$ python boscoinstaller 

4. Generate a passwordless rsa key, just press enter twice with no password when it prompts for one. Note it is important to name the key bosco_key.rsa:

    [FRONTEND_USER@FRONTEND_HOST ~]$ ssh-keygen -t rsa -f ~/.ssh/bosco_key.rsa

1. For whatever reason the installer doesn't create the .bosco dir so create it manually:

    [FRONTEND_USER@FRONTEND_HOST ~]$ mkdir ~/.bosco 

2. Start up BOSCO:

    [FRONTEND_USER@FRONTEND_HOST ~]$ bosco_start

3. Add the BOSCO_HOST by running the bosco_cluster script with the following parameters, this will forward the passwordless bosco ssh key, and install bosco on the remote side:

    [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_cluster --add BOSCO_USER@BOSCO_HOST BATCH_TYPE 

   where BATCH_TYPE = pbs, condor, etc.

1. Find the bosco_cluster script in ~/bosco/bin. Copy all lines from line #9 to line #13, line #36 to line #40, and line #811 to line #828 from bosco_cluster and paste them (in numerical order) into a separate bash script. This script will be used to generate the FRONTEND_USER ssh-key pair and passphrase for BOSCO. The resulting script (which we'll refer to here in this document as bosco_keygen.sh) should look like the following:

   #!/bin/bash
   
   # Bosco key location
   bosco_key=$HOME/.ssh/bosco_key.rsa
   
   # Bosco password location
   PASSPHRASE_LOCATION=$HOME/.bosco/.pass
   
   if [ `uname` = Darwin ] ; then
       MD5SUM=md5
   else
       MD5SUM=md5sum
   fi
   
   # If the key doesn't exist, create it
   if [ ! -e $bosco_key ]; then
       # Generate a md5sum password
       PASSPHRASE=`echo $RANDOM$RANDOM$RANDOM$RANDOM | $MD5SUM | awk '{print $1}'`
   
       # Output the password to a specially crafted file
       mkdir -p `dirname $PASSPHRASE_LOCATION`
       echo $PASSPHRASE > $PASSPHRASE_LOCATION
       chmod go-rwx $PASSPHRASE_LOCATION
   
       # Check if the passphrase is empty
       ssh-keygen -q -t rsa -f $bosco_key -P $PASSPHRASE > /dev/null
   
       if [ $? -ne 0 ]; then
           echo "Error running keygen" >&2
           exit 1
       fi
   fi
   
   

1. Find the bosco_quickstart script and comment all lines starting from line #164 to the end of the file at line #242. As you'll see, this modification removes the automated procedure for connecting and testing your first BOSCO resource. We will do this manually.

   ## Start Bosco
   #echo "************** Starting Bosco: ***********"
   #bosco_start
   #
   ## Connect one cluster
   #REMOTE_HOST=""
   #REMOTE_USER=""
   #REMOTE_TYPE=""
   #echo "************** Connect one cluster (resource) to BOSCO: ***********"
   #echo "At any time hit [CTRL+C] to interrupt."
   #echo 
   #q_tmp=""
   #read -p "Type the submit host name for the BOSCO resource and press [ENTER]: " q_tmp
   #while [ "x$q_tmp" = "x" ]; do
   #  read -p "No default, please type the name and press [ENTER]: " q_tmp
   #done
   #REMOTE_HOST=$q_tmp
   #
   #q_tmp=""
   #read -p "Type your username on $REMOTE_HOST (default $USER) and press [ENTER]: " q_tmp
   #if [ "x$q_tmp" = "x" ]; then 
   #  REMOTE_USER=$USER
   #else
   #  REMOTE_USER=$q_tmp
   #fi
   #
   #q_tmp=""
   #read -p "Type the queue manager for $REMOTE_HOST (pbs, condor, lsf, sge, slurm) and press [ENTER]: " q_tmp
   #while [ "x$q_tmp" = "x" ]; do
   #  read -p "No default, please type the queue manager and press [ENTER]: " q_tmp
   #done
   #if [ "x$q_tmp" = "xslurm" ]; then
   #  # SLURM is using PBS emulation
   #  q_tmp="pbs"
   #fi
   #REMOTE_TYPE=$q_tmp
   #
   #echo "Connecting $REMOTE_HOST, user: $REMOTE_USER, queue manager: $REMOTE_TYPE"
   #show_progress "-20-" bosco_cluster --add $REMOTE_USER@$REMOTE_HOST $REMOTE_TYPE 
   #if [ $? -ne 0 ]; then
   #  echo "Failed to connect the cluster $REMOTE_HOST. Please check your data and retry."
   #  exit 1
   #fi
   #
   #echo "$REMOTE_HOST connected"
   #
   #echo "************** Testing the cluster (resource): ***********"
   ##echo "This may take up to 2 minutes... please wait."
   #show_progress "This may take up to 2 minutes... please wait." bosco_cluster --test $REMOTE_USER@$REMOTE_HOST 
   ## MMDB move this underneath 
   #echo "BOSCO on $REMOTE_HOST Tested"
   #if [ $? -ne 0 ]; then
   #  echo "Failed to test the cluster $REMOTE_HOST. Please check your data and retry."
   #  echo "The BOSCO install document contains some troubleshooting information:"
   #  echo "https://twiki.grid.iu.edu/bin/view/CampusGrids/BoscoInstall"
   #  exit 1
   #fi
   #
   #
   #echo "************** Congratulations, Bosco is now setup to work with $REMOTE_HOST! ***********"
   #cat << EOF
   #You are ready to submit jobs with the "condor_submit" command.
   #Remember to setup the environment all the time you want to use Bosco:
   #source ~/bosco/bosco_setenv
   #
   #Here is a quickstart guide about BOSCO:
   #https://twiki.grid.iu.edu/bin/view/CampusGrids/BoscoQuickStart
   #
   #To remove Bosco you can run:
   #source ~/bosco/bosco_setenv; bosco_uninstall --all
   #
   #Here is a submit file example (supposing you want to run "myjob.sh"):
   #universe = grid
   #Executable = myjob.sh
   #arguments = 
   #output = myjob.output.txt
   #error = myjob.error.txt
   #log = myjob.log
   #transfer_output_files = 
   #should_transfer_files = YES
   #when_to_transfer_output = ON_EXIT
   #queue 1
   #EOF
   

1. A User Command Line is illustrated by a green box that displays a prompt:

    [user@client ~]$ 

2. A Root Command Line is illustrated by a red box that displays the root prompt:

    [root@client ~]$ 

3. Lines in a file are illustrated by a yellow box that displays the desired lines in a file:

    priorities=1 

1. Login to the FRONTEND_HOST via ssh as the FRONTEND_USER.

    [user@client ~]$ ssh FRONTEND_USER@FRONTEND_HOST

2. Download the BOSCO quickstart installer tarball in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ wget ftp://ftp.cs.wisc.edu/condor/bosco/1.2/bosco_quickstart.tar.gz 

3. Unzip and untar the BOSCO quickstart installer in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ tar -xzf bosco_quickstart.tar.gz 

4. Find the bosco_quickstart script and comment all lines starting from line #164 to the end of the file at line #242. As you'll see, this modification removes the automated procedure for connecting and testing your first BOSCO resource. We will do this manually.

   # Start Bosco echo "************** Starting Bosco: ***********" bosco_start # Connect one cluster <<<-------------------- line #164 #REMOTE_HOST="" #REMOTE_USER="" #REMOTE_TYPE="" #echo "************** Connect one cluster (resource) to BOSCO: ***********" #echo "At any time hit [CTRL+C] to interrupt." # ... # ... # ... #Executable = myjob.sh #arguments = #output = myjob.output.txt #error = myjob.error.txt #log = myjob.log #transfer_output_files = #should_transfer_files = YES #when_to_transfer_output = ON_EXIT #queue 1 #EOF <<<-------------------- line #242 

5. Run the bosco_quickstart script to install and start BOSCO on the FRONTEND_HOST.

   [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_quickstart 
   Bosco Quickstart Detailed logging of this run is in ./bosco_quickstart.log 
   Bosco is not installed. 
   You need Bosco to run this quickstart. Do you want to install Bosco? Select y/n and press [ENTER]): y 
   ************** Downloading and Installing Bosco 
   *********** Installing BOSCO................ BOSCO Installed 
   
   

1. Since BOSCO is not installed in your FRONTEND_HOST path, we must (at least temporarily) source its environment configuration file, bosco_setenv. Please append the following to your .bash_profile:

    source ~/bosco/bosco_setenv 

2. Find the bosco_cluster script in ~/bosco/bin. Copy all lines from line #9 to line #13, line #36 to line #40, and line #811 to line #828 from bosco_cluster and paste them (in numerical order) into a separate bash script. This script will be used to generate the FRONTEND_USER ssh-key pair and passphrase for BOSCO. The resulting script (which we'll refer to here in this document as bosco_keygen.sh) should look like the following:

   #!/bin/bash 
   # Bosco key location bosco_key=$HOME/.ssh/bosco_key.rsa 
   # Bosco password location PASSPHRASE_LOCATION=$HOME/.bosco/.pass if [ `uname` = Darwin ] ; then MD5SUM=md5 else MD5SUM=md5sum fi # If the key doesn't exist, create it if [ ! -e $bosco_key ]; then # Generate a md5sum password PASSPHRASE=`echo $RANDOM$RANDOM$RANDOM$RANDOM | $MD5SUM | awk '{print $1}'` # Output the password to a specially crafted file mkdir -p `dirname $PASSPHRASE_LOCATION` echo $PASSPHRASE > $PASSPHRASE_LOCATION chmod go-rwx $PASSPHRASE_LOCATION # Check if the passphrase is empty ssh-keygen -q -t rsa -f $bosco_key -P $PASSPHRASE > /dev/null if [ $? -ne 0 ]; then echo "Error running keygen" >&2 exit 1 fi fi 
   
   

1. Run the bosco_keygen.sh script to generate the ssh key pair for the FRONTEND_USER. You will find the keys, bosco_key.rsa and bosco_key.rsa.pub, in the FRONTEND_USER .ssh directory.

   [FRONTEND_USER@FRONTEND_HOST ~]$ chmod +x bosco_keygen.sh 
   
   

1. Next, we need to strip the passphrase from the private key. To do so, you will need to know the passphrase generated by the bosco_keygen.sh script. You can find the passphrase in the ~/.bosco/.pass file. Once you have the passphrase, run the following

   [FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ openssl rsa -in bosco_key.rsa -out bosco_key.rsa_nopass 
   
   

[FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ mv bosco_key.rsa_nopass bosco_key.rsa 

 [FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ chmod 600 bosco_key.rsa 

1. Copy the FRONTEND_USER 's public BOSCO key on the FRONTEND_HOST, bosco_key.rsa.pub, and append it to the authorized_keys file in the .ssh directory of the BOSCO_USER on the BOSCO_HOST.
2. Again, find the ~/bosco/bin/bosco_cluster bash script on the FRONTEND_HOST and comment out all lines between #811 to #837.

   if [ $? -eq 0 ]; then
   
   

1. Add the BOSCO_HOST by running the bosco_cluster script with the following parameters:

    [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_cluster --add BOSCO_USER@BOSCO_HOST BATCH_TYPE 

   where BATCH_TYPE = pbs, condor, etc.
2. Run a BOSCO test job to check the connection between the FRONTEND_HOST and the BOSCO_HOST and its worker nodes.

    [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_cluster --test BOSCO_USER@BOSCO_HOST 

3. If successful, run bosco_stop on the FRONTEND_HOST.

    [FRONTEND_USER@FRONTEND_HOST ~]$ bosco_stop 

4. Remove source ~/bosco/bosco_setenv from .bash_profile.
5. Finally, add the following elements to your frontend configuration file, frontend.xml. Note, you may add them to either the group or global credential definition.

    <credentials>
      <credential absfname="/path/to/grid_proxy" security_class="frontend" trust_domain="grid" type="grid_proxy"/>
      <credential absfname="~/.ssh/bosco_key.rsa.pub" keyabsfname="~/.ssh/bosco_key.rsa" pilotabsfname="/path/to/grid_proxy" security_class="frontend" trust_domain="bosco" type="key_pair"/>
   </credentials>
   

6. Please stop, reconfig, and restart your frontend. If successful, the FRONTEND_HOST is now properly configured.

   [root@FRONTEND_HOST ~]$ service gwms-frontend stop
   [root@FRONTEND_HOST ~]$ service gwms-frontend reconfig
   [root@FRONTEND_HOST ~]$ service gwms-frontend start  
   

7. Next, login to the FACTORY_HOST via ssh as root.

    [user@client ~]$ ssh root@FACTORY_HOST

8. Install condor-bosco on the FACTORY_HOST from root.

    [root@FACTORY_HOST ~]$ yum install condor-bosco 

9. Remove and retouch the 60-campus_factory.config file.

    [root@FACTORY_HOST ~]$ rm /etc/condor/config.d/60-campus_factory.config [root@FACTORY_HOST ~]$ touch /etc/condor/config.d/60-campus_factory.config

10. Now, add the entry for the BOSCO_HOST to factory configuration file, glideinWMS.xml.

     <entry name="CMS_TX_US_XXXXX_BOSCO" auth_method="key_pair" enabled="True" gatekeeper="BOSCO_USER@BOSCO_HOST" gridtype="batch BATCH_TYPE" rsl="" trust_domain="bosco" verbosity="std" work_dir="~/">
             <config>
                <max_jobs>
                   <default_per_frontend glideins="256" held="50" idle="50"/>
                   <per_entry glideins="256" held="50" idle="50"/>
                   <per_frontends>
                   </per_frontends>
                </max_jobs>
                <release max_per_cycle="20" sleep="0.2"/>
                <remove max_per_cycle="5" sleep="0.2"/>
                <restrictions require_glidein_glexec_use="False" require_voms_proxy="False"/>
                <submit cluster_size="10" max_per_cycle="100" sleep="0.2" slots_layout="fixed">
                   <submit_attrs>
                   </submit_attrs>
                </submit>
             </config>
             <allow_frontends></allow_frontends>
             <attrs>
                <attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="default"/>
                <attr name="GLEXEC_JOB" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="False"/>
                <attr name="GLIDEIN_CMSSite" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>
                <attr name="GLIDEIN_CPUS" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="8"/>
                <attr name="GLIDEIN_Country" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="US"/>
                <attr name="GLIDEIN_Glexec_Use" comment="This has been REQUIRED for historical reasons, OPTIONAL/NONE alt values" const="False" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="NONE"/>
                <attr name="GLIDEIN_MaxMemMBs" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="int" value="49152"/>
                <attr name="GLIDEIN_Max_Walltime" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="int" value="171000"/>
                <attr name="GLIDEIN_ResourceName" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>            
                <attr name="GLIDEIN_Site" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="TX_US_XXXXX"/>
                <attr name="GLIDEIN_Supported_VOs" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="CMS,MIS"/>
                <attr name="USE_CCB" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="True"/>
                <attr name="X509_CERT_DIR" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="/cvmfs/oasis.opensciencegrid.org/mis/certificates"/>
             </attrs>
             <files>
             </files>
             <infosys_refs>
             </infosys_refs>
             <monitorgroups>
             </monitorgroups>
    </entry>
    

11. Finally, build up a global ssh fingerprint list so that the FACTORY_HOST trusts the keys of both the BOSCO_HOST and the FRONTEND_HOST.

    [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa BOSCO_HOST >> /etc/ssh/ssh_known_hosts
    [root@FACTORY_HOST ~]$ ssh-keyscan -t rsa,dsa FRONTENT_HOST >> /etc/ssh/ssh_known_hosts
    

12. Stop, reconfigure and restart your factory. If successful, the FACTORY_HOST is now properly configured. You may now submit user jobs to the BOSCO_HOST via the FRONTEND_HOST.

    [root@FRONTEND_HOST ~]$ service gwms-factory stop
    [root@FRONTEND_HOST ~]$ service gwms-factory reconfig
    [root@FRONTEND_HOST ~]$ service gwms-factory start
    
    

• http://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS/doc.prd/recipes/batch.html
• https://twiki.grid.iu.edu/bin/view/CampusGrids/BoscoQuickStart
• https://twiki.grid.iu.edu/bin/view/CampusGrids/BoSCO
• https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallGlideinWMSFactory
• https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallGlideinWMSFrontend

1. Now, add the entry for the BOSCO_HOST to factory configuration file, glideinWMS.xml.

1. Run the bosco_keygen.sh script to generate the ssh key pair for the FRONTEND_USER. You will find the keys, bosco_key.rsa and bosco_key.rsa.pub, in the FRONTEND_USER .ssh directory.

   [FRONTEND_USER@FRONTEND_HOST ~]$ chmod +x bosco_keygen.sh
   [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_keygen.sh
   

2. Next, we need to strip the passphrase from the private key. To do so, you will need to know the passphrase generated by the bosco_keygen.sh script. You can find the passphrase in the ~/.bosco/.pass file. Once you have the passphrase, run the following

     [FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ openssl rsa -in bosco_key.rsa -out bosco_key.rsa_nopass
   
   

 [FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ mv bosco_key.rsa_nopass bosco_key.rsa 

 [FRONTEND_USER@FRONTEND_HOST ~/.ssh]$ chmod 600 bosco_key.rsa 

1. Copy the FRONTEND_USER 's public BOSCO key on the FRONTEND_HOST, bosco_key.rsa.pub, and append it to the authorized_keys file in the .ssh directory of the BOSCO_USER on the BOSCO_HOST.
2. Again, find the ~/bosco/bin/bosco_cluster bash script on the FRONTEND_HOST and comment out all lines between #811 to #837.

   if [ $? -eq 0 ]; then 
       echo "Cluster $remote_host already installed"
       echo "Reinstalling on $remote_host"
       reinstall=1
   else
       reinstall=0
   fi
   
   

1. Add the BOSCO_HOST by running the bosco_cluster script with the following parameters:

    [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_cluster --add BOSCO_USER@BOSCO_HOST BATCH_TYPE 

   where BATCH_TYPE = pbs, condor, etc.
2. Run a BOSCO test job to check the connection between the FRONTEND_HOST and the BOSCO_HOST and its worker nodes.

    [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_cluster --test BOSCO_USER@BOSCO_HOST 

3. If successful, run bosco_stop on the FRONTEND_HOST.

    [FRONTEND_USER@FRONTEND_HOST ~]$ bosco_stop 

4. Remove source ~/bosco/bosco_setenv from .bash_profile. Your FRONTEND_HOST is now properly configured.
5. Next, login to the FACTORY_HOST via ssh as root.

     [user@client ~]$ ssh root@FACTORY_HOST

6. Install condor-bosco on the FACTORY_HOST from root.

     [root@FACTORY_HOST ~]$ yum install condor-bosco 

7. Remove and retouch the 60-campus_factory.config file.

     [root@FACTORY_HOST ~]$ rm /etc/condor/config.d/60-campus_factory.config 
     [root@FACTORY_HOST ~]$ touch /etc/condor/config.d/60-campus_factory.config

Additional Documentation

HOWTO Install and Configure BOSCO for Glidein-Based Submission

This document follows the general OSG documentation conventions:

1. A User Command Line is illustrated by a green box that displays a prompt:

     [user@client ~]$ 

2. A Root Command Line is illustrated by a red box that displays the root prompt:

     [root@client ~]$ 

3. Lines in a file are illustrated by a yellow box that displays the desired lines in a file:

   priorities=1 

• BOSCO_HOST is the hostname of the host from which glideins will be submitted to the BOSCO resource's local batch queue.

• BOSCO_USER is the username of the user on the BOSCO_HOST that has access to the BOSCO resource's local batch queue; e.g., cmsbosco
• FACTORY_ADMIN_USER is the username of the user on the FACTORY_HOST used for all non-root administrative tasks; e.g., gfactory
• FACTORY_VO_USER is the username of the user on the FACTORY_HOST from which glideins are submitted to the BOST_HOST; e.g., fecmsglobal
• FRONTEND_USER is the username of the user on the FRONTEND_HOST that submits requests for glideins to the FACTORY_HOST; e.g., frontend

• glideinWMS Factory 3.2.6 or later (3.2.8 recommended)
• HTCondor 8.2.4 or later
• condor-bosco

1. Login to the FRONTEND_HOST via ssh as the FRONTEND_USER.

     [user@client ~]$ ssh FRONTEND_USER@FRONTEND_HOST

2. Download the BOSCO quickstart installer tarball in the FRONTEND_USER home directory.

     [FRONTEND_USER@FRONTEND_HOST ~]$ wget ftp://ftp.cs.wisc.edu/condor/bosco/1.2/bosco_quickstart.tar.gz 

3. Unzip and untar the BOSCO quickstart installer in the FRONTEND_USER home directory.

     [FRONTEND_USER@FRONTEND_HOST ~]$ tar -xzf bosco_quickstart.tar.gz 

4. Find the bosco_quickstart script and comment all lines starting from line #164 to the end of the file at line #242. As you'll see, this modification removes the automated procedure for connecting and testing your first BOSCO resource. We will do this manually.

   # Start Bosco
   echo "************** Starting Bosco: ***********"
   bosco_start
   
   # Connect one cluster                                        <<<-------------------- line #164
   #REMOTE_HOST=""
   #REMOTE_USER=""
   #REMOTE_TYPE=""
   #echo "************** Connect one cluster (resource) to BOSCO: ***********"
   #echo "At any time hit [CTRL+C] to interrupt."
   # ...
   # ...
   # ...
   #Executable = myjob.sh
   #arguments = 
   #output = myjob.output.txt
   #error = myjob.error.txt
   #log = myjob.log
   #transfer_output_files = 
   #should_transfer_files = YES
   #when_to_transfer_output = ON_EXIT
   #queue 1
   #EOF                                                          <<<-------------------- line #242
   

5. Run the bosco_quickstart script to install and start BOSCO on the FRONTEND_HOST.

   [FRONTEND_USER@FRONTEND_HOST ~]$ ./bosco_quickstart 
   
   

1. Since BOSCO is not installed in your FRONTEND_HOST path, we must (at least temporarily) source its environment configuration file, bosco_setenv. Please append the following to your .bash_profile:

    source ~/bosco/bosco_setenv 

2. Find the bosco_cluster script in ~/bosco/bin. Copy all lines from line #9 to line #13, line #36 to line #40, and line #811 to line #828 from bosco_cluster and paste them (in numerical order) into a separate bash script. This script will be used to generate the FRONTEND_USER ssh-key pair and passphrase for BOSCO. The resulting script (which we'll refer to here in this document as bosco_keygen.sh) should look like the following:

   #!/bin/bash
   
   

Additional Documentation

HOWTO Install and Configure Bosco for Glidein-Based Submission

About this Document

Definitions

Hostnames:

• BOSCO_HOST is the hostname of the host where glideins will be submitted to the BOSCO resource's local batch queue by the glideinWMS factory on behalf of the VO frontend under the BOSCO_USER.
• FACTORY_HOST is the hostname of the host where you've installed and configured your glideinWMS factory.
• FRONTEND_HOST is the hostname of the host where you've installed and configured your VO's glideinWMS frontend.

Usernames:

• FACTORY_VO_USER is the user on FACTORY_HOST that submits glideins to

Requirements

• Installed and configured on FRONTED_HOST: glideinWMS Frontend X.X.X or later
• Installed and configured on FACTORY_HOST: glideinWMS Factory 3.2.6 or later (3.2.8 recommended); HTCondor 8.2.4 or later

Installation and Configuration

Additional Documentation

[1522] frontend@test-frontend-1 ~$ ./bosco_generate_keys.sh

[1522] frontend@test-frontend-1 ~$ cd .ssh

[1523] frontend@test-frontend-1 ~/.ssh$ ls

authorized_keys bosco_key.rsa bosco_key.rsa.pub known_hosts

[1523] frontend@test-frontend-1 ~/.ssh$ openssl rsa -in bosco_key.rsa -out bosco_key.rsa_nopass

Enter pass phrase for bosco_key.rsa:

writing RSA key

[1524] frontend@test-frontend-1 ~/.ssh$ ls

authorized_keys bosco_key.rsa bosco_key.rsa_nopass bosco_key.rsa.pub known_hosts

[1524] frontend@test-frontend-1 ~/.ssh$ mv bosco_key.rsa_nopass bosco_key.rsa

[1524] frontend@test-frontend-1 ~/.ssh$ ls

authorized_keys bosco_key.rsa bosco_key.rsa.pub known_hosts

[1524] frontend@test-frontend-1 ~/.ssh$

chmod 600 bosco_key.rsa

How to remove bosco ...

- from bosco resource login node: (1) rm ~/bosco from BOSCO_USER's home directory. (2) rm FRONTEND_USER's public key from authorized_keys file on BOSCO_LOGIN_HOST

- from frontend: (1)

15. recognize frontend keys on factory

The BOSCO submission from the Factory uses SSH. Before being able to do ssh in batch mode for the glidein submission you must trust the keys of the BOSCO resource, the one added with "bosco_cluster --add" on the Frontend, in our example carvergrid.nersc.gov. You have a couple of options depending on how you want to configure ssh:

• Build up a global fingerprint list. Collect the keys in /etc/ssh/ssh_known_hosts (or ~/.ssh/known_hosts of the users submitting glideins). Note that you'll have to update the fingerprint list if the BOSCO recource key will change or all the glidein submisison attempts will fail

  ssh-keyscan -t rsa,dsa carvergrid.nersc.gov >> /etc/ssh/ssh_known_hosts 

• Alternatively disable strict host key checking in ssh for the BOSCO resource adding these lines to /etc/ssh/ssh_config (or ~/.ssh/config of the users submitting glideins). The syntax below can accept also IP numbers or wildcards (to include more hosts).

  Host carvergrid.nersc.gov StrictHostKeyChecking no 

  The setting above will add the key the first time and give a warning if the key changes subsequently. To avoid host key verification, and not use known_hosts file you can do the following (not recommended unless on a local network). In this example the BOSCO resources have IPs 192.168.0.XXX.

  Host 192.168.0.* StrictHostKeyChecking no UserKnownHostsFile=/dev/null 

If you have access to the ssh key on the factory, it is recommended also to manually ssh to the host to see if the ssh connection works correctly.

16.

1. Login as the FRONTEND_USER on the FRONTEND_HOST.

ssh FRONTEND_USER@FRONTEND_HOST

2. Download the bosco quickstart installer in the FRONTEND_USER's home directory.

wget ftp://ftp.cs.wisc.edu/condor/bosco/1.2/bosco_quickstart.tar.gz

3. Untar the bosco quickstart installer in the FRONTEND_USER's home directory.

tar -xzf bosco_quickstart.tar.gz

4. Comment out all lines starting from line #164 to the end of the file (line #242) in the the bosco_quickstart bash script.

5. Run the bosco_quickstart installer.

./bosco_quickstart

5.5. Add to .bash_profile.

source ~/bosco/bosco_setenv

6. Select 'y' to download and install bosco in the FRONTEND_USER's home directory.

[1441] frontend@test-frontend-1 ~$ ./bosco_quickstart

Bosco Quickstart

Detailed logging of this run is in ./bosco_quickstart.log

Bosco is not installed. You need Bosco to run this quickstart.

Do you want to install Bosco? Select y/n and press [ENTER]): y

************ Downloading and Installing Bosco *********

Installing BOSCO.................

BOSCO Installed

************ Starting Bosco: *********

BOSCO Started

7. Copy all lines from line #9 to line #13, line #36 to line #40, and line #811 to line #828 in the ~/bosco/bin/bosco_cluster bash script and paste them (in numberical order) in a separate, temporary bash script. This script will be used to generate the FRONTEND_USER's bosco ssh key pair and passphrase. The result should look like the following:

#!/bin/bash

# Bosco key location

bosco_key=$HOME/.ssh/bosco_key.rsa

# Bosco password location

PASSPHRASE_LOCATION=$HOME/.bosco/.pass

# Compute md5 hash

if [ `uname` = Darwin ] ; then

MD5SUM? =md5

else

MD5SUM? =md5sum

fi

# If the key doesn't exist, create it

if [ ! -e $bosco_key ]; then

# Generate a md5sum password

PASSPHRASE=`echo $RANDOM$RANDOM$RANDOM$RANDOM | $MD5SUM | awk '{print $1}'`

# Output the password to a specially crafted file

mkdir -p `dirname $PASSPHRASE_LOCATION`

echo $PASSPHRASE > $PASSPHRASE_LOCATION

chmod go-rwx $PASSPHRASE_LOCATION

# Check if the passphrase is empty

ssh-keygen -q -t rsa -f $bosco_key -P $PASSPHRASE > /dev/null

if [ $? -ne 0 ]; then

echo "Error running keygen" >&2

exit 1

fi

fi

8. Generate the FRONTEND_USER's bosco ssh key pair and passphrase by running the bash script created in Step 7.

[1507] frontend@test-frontend-1 ~$ chmod +x bosco_generate_keys.sh

[1508] frontend@test-frontend-1 ~$ ./bosco_generate_keys.sh

[1508] frontend@test-frontend-1 ~$ cd .ssh

[1509] frontend@test-frontend-1 ~/.ssh$ ls

authorized_keys bosco_key.rsa bosco_key.rsa.pub known_hosts

9. Copy the FRONTEND_USER's public bosco key on the FRONTEND_HOST, bosco_key.rsa.pub, and append it to the authorized_keys file in the .ssh directory of the BOSCO_USER on the BOSCO_LOGIN_HOST.

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAlLh6nD? +5uYmXxG4UFdHWYcokIDT7Tm+/pRnxyGseZMzG61gJU26HwCy1wvilaj3PZ+yws2896BTi4mx4IPQ+1iev2GXjWnyTTz1T3SldS4pgwYqM9tVjkkz/BI5ddVXAlurkH3YAHYrdkPamVNJg7lUcqFnFZZEN8y0gE6l2kf5PdFMkQBVo6OzFLbLp3bTAaAeTfnEcT/VEs3xqgY4SlMtHcejbnP+GKio1mwEamz2EVoCyyVFNIHnFmY1U+ze8FMPfmkB3Yzrz1B669LlP/1t2iG1weir7Z75bP3d8/nEN2T9b15bq73QfhzybWjzfdQLT8pS3Su8LZ+7z5RyabQ== frontend@test-frontend-1.t2.ucsd.edu

10. Next, comment out all lines between #811 to #837 in the ~/bosco/bin/bosco_cluster bash script.

11. Add the BOSCO_LOGIN_HOST by running the bosco_cluster bash script with the following parameters:

./bosco_cluster --add $BOSCO_USER@$BOSCO_LOGIN_HOST $BATCH_TYPE

where $BATCH_TYPE = pbs, condor, ...

[1529] frontend@test-frontend-1 ~$ bosco_cluster --add bosco@osg-gw-7.t2.ucsd.edu condor

Downloading for bosco@osg-gw-7.t2.ucsd.edu......

Unpacking..

Sending libraries to bosco@osg-gw-7.t2.ucsd.edu.

Creating BOSCO for the WN's................................................

Installing on cluster bosco@osg-gw-7.t2.ucsd.edu.

Installation complete

The cluster bosco@osg-gw-7.t2.ucsd.edu has been added to BOSCO

It is available to run jobs submitted with the following values:

> universe = grid

> grid_resource = batch condor bosco@osg-gw-7.t2.ucsd.edu

OR to run load balanced job on all available clusters with:

> universe = vanilla

[1530] frontend@test-frontend-1 ~$

12. Run a bosco test job to check the connection between the FRONTEND_HOST and the BOSCO_RESOURCE and it's worker nodes ...

[1534] frontend@test-frontend-1 ~$ ./bosco_cluster --test bosco@osg-gw-7.t2.ucsd.edu

-bash: ./bosco_cluster: No such file or directory

[1534] frontend@test-frontend-1 ~$ bosco_cluster --test bosco@osg-gw-7.t2.ucsd.edu

Testing ssh to bosco@osg-gw-7.t2.ucsd.edu...Passed!

Testing bosco submission...Passed!

Submission and log files for this job are in /home/frontend/bosco/local.bosco/bosco-test/boscotest.13683

Waiting for jobmanager to accept job...Passed

Checking for submission to remote condor cluster (could take ~30 seconds)...Passed!

Waiting for job to exit... this could take a while (waiting 60 seconds)...Failed

The job did not end in 60 seconds. This is not always a bad thing...

Maybe condor is waiting longer to poll for job completion?

Here is the current status of the job:"

-- Submitter: test-frontend-1.t2.ucsd.edu : <127.0.0.1:11000?sock=6597_e220_3> : test-frontend-1.t2.ucsd.edu

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

1.0 frontend 5/14 15:34 0+00:00:00 I 0 0.0 echo Hello

[1535] frontend@test-frontend-1 ~$

12.5 Run bosco_stop on FRONTEND_HOST.

12.75. Remove source ~/bosco/bosco_setenv from .bash_profile.

13. Install condor-bosco on the FACTORY_HOST from ROOT.

Install HTCondor-BOSCO

[root@client ~]$ yum install condor-bosco [root@client ~]$ rm /etc/condor/config.d/60-campus_factory.config [root@client ~]$ touch /etc/condor/config.d/60-campus_factory.config

There is an Advanced configuration section in the BOSCO documentation that can be useful to know, specially how to deal with multi homed BOSCO resources and the specification of custom submit properties.

14. Add entry to factory config ...

<entry name="T2_US_UCSD_BOSCO" auth_method="key_pair" enabled="True" gatekeeper="bosco@osg-gw-7.t2.ucsd.edu" gridtype="batch condor" rsl="" schedd_name="schedd_glideins3@cabinet-10-10-4.t2.ucsd.edu" trust_domain="bosco" verbosity="std" work_dir="~/">

<config>

<max_jobs>

<default_per_frontend glideins="256" held="5" idle="10"/>

<per_entry glideins="256" held="5" idle="10"/>

<per_frontends>

</per_frontends>

</max_jobs>

<release max_per_cycle="20" sleep="0.2"/>

<remove max_per_cycle="5" sleep="0.2"/>

<restrictions require_glidein_glexec_use="False" require_voms_proxy="False"/>

<submit cluster_size="10" max_per_cycle="100" sleep="0.2" slots_layout="fixed">

<submit_attrs>

</submit_attrs>

</submit>

</config>

<allow_frontends>

</allow_frontends>

<attrs>

<attr name="CONDOR_OS" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="default"/>

<attr name="CONDOR_VERSION" const="False" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="default"/>

<attr name="GLEXEC_JOB" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="False"/>

<attr name="GLIDEIN_CMSSite" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="T2_US_UCSD"/>

<attr name="GLIDEIN_CPUS" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="1"/>

<attr name="GLIDEIN_Country" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="US"/>

<attr name="GLIDEIN_Glexec_Use" comment="This has been REQUIRED for historical reasons, OPTIONAL/NONE alt values" const="False" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="NONE"/>

<attr name="GLIDEIN_MaxMemMBs" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="int" value="49152"/>

<attr name="GLIDEIN_Max_Walltime" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="int" value="172000"/>

<attr name="GLIDEIN_REQUIRED_OS" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="default"/>

<attr name="GLIDEIN_ResourceName" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="T2_US_UCSD"/>

<attr name="GLIDEIN_Site" const="True" glidein_publish="True" job_publish="True" parameter="True" publish="True" type="string" value="T2_US_UCSD"/>

<attr name="GLIDEIN_Supported_VOs" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="True" type="string" value="CMS,MIS"/>

<attr name="USE_CCB" const="True" glidein_publish="True" job_publish="False" parameter="True" publish="True" type="string" value="True"/>

<attr name="X509_CERT_DIR" const="True" glidein_publish="False" job_publish="True" parameter="True" publish="True" type="string" value="/cvmfs/oasis.opensciencegrid.org/mis/certificates"/>

</attrs>

</attrs>

<files>

</files>

<infosys_refs>

</infosys_refs>

<monitorgroups>

</monitorgroups>

</entry>

2.