(r14) Condor_annex < UCSDTier2

UCSDTier2 Web>Condor_annex (revision 14) (raw view)~~EditAttach~~
<noautolink>

---+!! Install and Configure condor_annex

%TOC%

---++ About this Document

*condor_annex* is a Perl-based script that utilizes the Amazon Web Services (AWS) command-line interface (CLI) and other AWS services to orchestrate the delivery of HTCondor execute nodes running on AWS Elastic Compute Cloud (EC2) instances to an HTCondor pool. This document describes how to install, configure, and run condor_annex successfully from your own local HTCondor pool.

This document follows the general Open Science Grid (OSG) documentation conventions:

   1 A _User Command Line_ is illustrated by a green box that displays a prompt:<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #e0f0e0; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [user@client ~]$ </pre>
   1 A _Root Command Line_ is illustrated by a red box that displays the _root_ prompt: <pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@client ~]$ </pre>
   1 _Lines in a file_ are illustrated by a yellow box that displays the desired lines in a file: <pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffffcc; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> priorities=1 </pre>

---++ Definitions

   * *SUBMIT* is the hostname of an HTCondor submit node, where users submit their jobs to your local pool.
   * *CENTRAL_MANAGER* is the hostname of your HTCondor central manager, where job and machine class ads are matched.
   * *EXECUTE* is the hostname of an HTCondor execute node in your local pool.
   * *ANNEX* is the hostname of an EC2 instance configured as a condor_annex execute node. 

---++ Requirements

   * An HTCondor pool
   * An Amazon Web Services Account

---++ Step 1: Install and Configure an HTCondor Pool 

If you do not already have your own HTCondor Pool, you may want to first start by installing your own _personal_ HTCondor pool to experiment with condor_annex. Please consult the [[http://research.cs.wisc.edu/htcondor/manual/v8.4/ref.html][HTCondor Manual]] and/or [[https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki][Wiki]] for more information.

---++ Step 2: Obtain an Amazon Web Services Account

In order to use condor_annex, you must already have an AWS account. You may establish an AWS account under the UC-wide agreement by following the [[https://blink.ucsd.edu/technology/cloud/aws/][instructions provided by Blink]].

---++ Step 3: Obtain AWS Account Credentials

condor_annex issues programmatic requests to AWS services via the AWS command-line interface (CLI). In order to issue these requests, the AWS CLI must sign them using your AWS account credentials. These credentials consist of an *Access Key ID* and a *Secret Access Key*. If you do not have these access keys, you may create them using the AWS Management Console. AWS recommends that you use Identity and Access Management (IAM) access keys instead of your root account access keys.

To create access keys, you must have permissions to perform the required IAM actions.  

   1 Open the [[https://console.aws.amazon.com/iam][IAM console]].
   1 In the navigation pane, choose _Users_.
   1 If you do not already have an IAM username, then select *Add User*. Each new user is issued credentials. 
   1 If you already have an IAM username, then choose your IAM username (not the check box). 
   1 Next, select the _Security Credentials_ tab and then choose *Create Access Key*.
   1 Your credentials will look something like this: 
      * Access Key ID: AKIAIOSFODNN7EXAMPLE
      * Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   1 Choose *Download .csv file*, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

Save your Access Key ID and Secret Access Key. You will need to provide them later when configuring the AWS CLI. If you need more information about AWS Security Credentials, please consult the [[http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html][AWS documentation]].

---++ Step 4: Select a Region for the Annex

Amazon Elastic Compute Cloud (EC2) instances are hosted in multiple locations world-wide. These locations are composed of *Regions* and *Availability Zones*. Each Region is a separate geographic area. However, each Region also has multiple, isolated locations known as Availability Zones (AZs). However, *not all AWS Regions are created equal*. Each Region may offer only a subset of AWS services. You can find out what services are offered in each Region from [[https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/][the table provided here]].

When selecting a Region for your annex, you must select a region that offers all of the AWS services required by condor_annex to function properly. These services are:

   * [[https://aws.amazon.com/ebs/][Amazon Elastic Block Store (EBS)]]
   * [[https://aws.amazon.com/ec2/][Amazon Elastic Compute Cloud (EC2)]]
   * [[https://aws.amazon.com/cloudwatch/][Amazon CloudWatch]]
   * [[https://aws.amazon.com/sns/][Amazon Simple Notification Service (SNS)]]
   * [[https://aws.amazon.com/s3/][Amazon Simple Storage Service (S3)]]
   * [[https://aws.amazon.com/vpc/][Amazon Virtual Private Cloud (VPC)]]
   * [[https://aws.amazon.com/autoscaling/][AWS Auto Scaling Groups (ASG)]]
   * [[https://aws.amazon.com/cloudformation/][AWS CloudFormation]]
   * [[https://aws.amazon.com/iam/][AWS Identity and Access Management (IAM)]]
   * [[https://aws.amazon.com/lambda/][AWS Lambda]]

__AWS Lambda__ currently has the most limited deployment of any AWS service required by condor_annex. For example, AWS Lambda is only available in the following Regions within the United States at this time:

   * Northern Virginia (us-east-1)
   * Ohio (us-east-2)
   * Oregon (us-west-2)

Select your desired Region accordingly from the drop-down menu in the upper-right-hand side of the AWS Management Console. 

---++ Step 5: Generate an EC2 Key Pair

After selecting a Region for your annex, you will need to generate an SSH key pair that will allow you to login to your EC2 instances in that Region. You can create a key pair using the EC2 console or the command line. You will specify this key pair when launching your instances with condor_annex.  

To create your key pair using the Amazon EC2 console

   1 Open the [[https://console.aws.amazon.com/ec2/][EC2 console]].
   1 In the navigation pane, under _NETWORK & SECURITY_, choose _Key Pairs_.
   1 Choose *Create Key Pair*.
   1 Enter a name for the new key pair in the *Key pair name field* of the Create Key Pair dialog box, and then choose *Create*.
   1 The private key file is automatically downloaded by your browser. The base file name is the name you specified as the name of your key pair, and the file name extension is .pem. Save the private key file in a safe place. *This is the only chance for you to save the private key file*. You'll need to provide the name of your key pair when you launch an instance and the corresponding private key each time you connect to the instance.
   1 Use the following command to set the permissions of your private key file so that only you can read it. <pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #e0f0e0; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [user@SUBMIT ~]$ chmod 400 my-key-pair.pem </pre>

If you would like to create your SSH key pair using the AWS CLI or import your own key pair, please consult the [[http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html][AWS documentation]].

---++ Step 6: Configure Default VPC Security Group

condor_annex will automatically create and configure an AWS Security Group (i.e., a virtual firewall) around all of the instances within an annex. However, depending on your HTCondor pool configuration, it may also been useful to place some on-demand resources in AWS. For example, you may want to a separate HTCondor central manager instance located in AWS in order to flock user jobs over to the annex instead of connecting the annex instances back to your local central manager.  

Any such on-demand resources may be placed in your AWS Region's default Virtual Private Cloud (VPC) Security Group. To configure the default VPC Security Group:

   1 Open the [[https://console.aws.amazon.com/vpc/][VPC console]].
   1 In the navigation pane, under _Security_, choose _Security Groups_.
   1 Select the Security Group in the list that has *Group Name* _default_ and *Description* _default VPC security group_. 
   1 Next, select the _Inbound Rules_ tab and then click on the *Edit* button.

By default, the only inbound rule should be one allowing all traffic from instances assigned to the default VPC Security Group. 

      |  *Type*      | *Protocol* | *Port Range*  | *Source*                                             |
      | ALL Traffic | ALL          | ALL                 | The security group ID (sg-xxxxxxxx)  |

 We recommend the following set of inbound rules be used for the default VPC Security Group:

   1 Keep the default rule allowing all traffic from instances assigned to the default VPC Security Group.
   1 Allow all inbound traffic from instances within your AWS Region's default VPC's private network IP address space.
   1 Allow inbound SSH traffic on port 22. 
   1 Allow inbound ICMP traffic.
   1 Allow inbound HTCondor UDP traffic on port 9618
   1 Allow inbound HTCondor TCP traffic on port 9618.
   
In their most permissive form, these inbound rules for the default VPC security group will look something like this:

   | *Type*                    | *Protocol* |  *Port Range* | * Source*                     |
   | All traffic                 | All             | All                   | sg-5437332d (default) |
   | All traffic                 | All             | All                   | 172.31.0.0/16              |
   | SSH                       | TCP          | 22                   | 0.0.0.0/0                      |
   | All ICMP                 | All             | N/A                 | 0.0.0.0/0                      |
   | Custom UDP Rule | UDP          | 9618               | 0.0.0.0/0                      |
   | Custom TCP Rule | TCP          | 9618               | 0.0.0.0/0                      |

Of course, you should try to restrict the *Source* IP address space for these rules as much as possible. For example, you may want to limit them to inbound traffic from your home institution's public IP address space.

By default, each Security Group, including the default VPC Security Group, allows ALL outbound traffic. 

   |  *Type*      | *Protocol* | *Port Range*  | *Source* |
   | ALL Traffic | ALL          | ALL                 | 0.0.0.0/0 |

If you would like to restrict outbound traffic from the default VPC Security Group, select the _Outbound Rules_ tab, click on the *Edit* button, and then configure the outbound rules accordingly. 

---++ Step 7: Create an condor_annex-compatible Amazon Machine Image

Each HTCondor execute instance within your annex must run a condor_annex-compatible Amazon Machine Image (AMI). By default, condor_annex will attempt to use one of the publicly available Amazon Linux AMIs with HTCondor 8.4.2 pre-installed currently provided by the HTCondor team. These condor_annex-compatible AMIs are available in the following AWS Regions within the United States:

   | *Region*  | *AMI ID*          |
   | us-east-1 | ami-91e1a3fb  |
   | us-west-1 | ami-7f06731f  |
   | us-west-2 | ami-ac8890cd |

If these preconfigured AMIs cannot be successfully modified to suit your needs, you will need to create your own condor_annex-compatible AMI. We have done so for our own purposes by building a condor_annex-compatible CentOS 6-based AMI. 

To build your own condor_annex-compatible AMI, open the Elastic Compute Cloud (EC2) dashboard in the Region where you will run your annex. Click on the *Launch Instance* button. This will open the instance launch configuration wizard. Follow these steps. 

   1 __Choose an Amazon Machine Image (AMI)__: We configured our annex's execute instances to use CentOS 6. To find a suitable CentOS 6 AMI to start from, select the _AWS Marketplace_ tab and then enter "CentOS 6" in the search box. Your search will return multiple results. However, the most up-to-date AMI should be the first one in the list. Unless you have special requirements for your configuration, select this AMI by clicking on the *Select* button.
   1 __Choose an Instance Type__: Once you have selected an AMI, the launch configuration wizard will prompt you to select an instance type on which to build your condor_annex execute node. Choose one that suits your needs. Once you have selected your instance type, click on the *Next: Configure Instance Details* button.
   1 __Configure Instance Details__:  Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the *Number of Instances* at _1_. Next, select one of your *Network* VPCs. In general, you should choose the default VPC whose Security Group was pre-configured in the previous step. Once you have determined which VPC will host this instance, select a specific *Subnet* in which to place it. The other networking options *Auto-assign Public IP* and *Placement group* may be left set to their default settings of _Use subnet setting (Enabled)_ and _No placement group_, respectively. After configuring the networking details, if you would like to apply a specific *IAM role* to the instance, then select an appropriate role for it. Otherwise, leave IAM role set to its default value of _None_. All other instance details may be configured with their default values. Once you have completed configuring your instance details, click on the *Next: Add Storage* button.
   1 __Add Storage__: In general, you will not have to modify the configuration of your root storage volume for the instance. However, the launch wizard may still default to a _Magnetic_ volume type, even though the _General Purpose SSD_ option is now becoming AWS' recommended default. Our instance launch wizard still defaults to _Magnetic_. As such, we changed our root *Volume Type* from an 8GiB _Magnetic_ volume to an 8 GiB _General Purpose SSD_ volume and selected *Delete on Termination.* Once you have completed the configuration of your root volume, click on the *Next: Tag Instance* button.
   1 __Tag Instance__: Add a _Name_ to your instance and then click on the *Next: Configure Security Group* button.
   1 __Configure Security Group__: _Select an *existing* security group_ and choose your _default VPC security group_. Once you have selected a security group, click on the *Review and Launch* button.
   1 __Review Instance Launch__: Review the configuration of your instance and make any necessary changes. Once done, click on the *Launch* button. You will be prompted to *Select an existing key pair or create a new key pair*, which will enable your SSH access to the instance. Select one of your existing key pairs (or create a new one) and then agree to the acknowledgement statement by clicking on the checkbox next to it. Once you have selected your key pair, click on the *Launch Instances* button. After launching your instance, the wizard will display the __Launch Status__ page. To return to the main EC2 dashboard, scroll down the click on the *View Instances* button.

Once the instance has started up and enters a _running_ *Instance State*, you will install and configure the software required to create a condor_annex-compatible AMI on the instance. To begin, open a terminal and login into the instance via SSH.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #e0f0e0; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [user@client ~]$ ssh -i ~/.ssh/HTCondorAnnex.pem centos@ANNEX.PUBLIC.IP </pre>

Then switch to root.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #e0f0e0; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [centos@ANNEX-PRIVATE-IP ~]$ sudo -i </pre>

First, update the instance's base OS configuration.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum update</pre>

Then install the Extra Packages for Enterprise Linux (EPEL) repository.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install epel-release</pre>

Next, install the yum priorities package

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install yum-plugin-priorities </pre>

and the appropriate [[https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories][Open Science Grid (OSG) repositories]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ rpm -Uvh https://repo.grid.iu.edu/osg/3.3/osg-3.3-el6-release-latest.rpm</pre>

Once the OSG repositories are available on the instance, [[https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallCertAuth][install the CA certificates and fetch-crl]]. <pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install osg-ca-certs
 [root@ANNEX-PRIVATE-IP ~]$ yum install fetch-crl </pre>

Next, [[https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallWNClient][install the OSG Worker Node Client]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install osg-wn-client </pre>

After the client software is installed, manually create both a _condor_ group and user and then [[https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallCondor][install HTCondor]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ groupadd condor
 [root@ANNEX-PRIVATE-IP ~]$ useradd condor -g condor
 [root@ANNEX-PRIVATE-IP ~]$ yum install condor.x86_64 </pre>

This would complete the typical software installation of a standard OSG HTCondor execute node, except for [[https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallCvmfs][CVMFS]]. However, in order to support condor_annex, several other software packages must be properly installed on the instance.

In addition to the standard OSG software, you must also install [[https://cloudinit.readthedocs.io/en/latest/][cloud-init]]

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install cloud-init</pre>

and several other python packages, including [[https://pypi.python.org/pypi/pip][pip]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ yum install pystache
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-argparse
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-daemon
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-requests 
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-pip </pre>

You may also want to make sure these packages are up-to-date.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ pip install --upgrade pip </pre>

Once these packages are installed, [[http://docs.aws.amazon.com/cli/latest/userguide/installing.html][install the AWS CLI]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ pip install awscli </pre>

and the [[http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-helper-scripts-reference.html][AWS CloudFormation Helper Scripts]].

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz </pre>

Several standard directories and symbolic links found on Amazon Linux AMIs must be created to successfully use the CloudFormation Helper Scripts on CentOS 6. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-hup /etc/init.d/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ chmod 775 /usr/bin/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ mkdir /opt/aws
 [root@ANNEX-PRIVATE-IP ~]$ mkdir /opt/aws/bin
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-hup /opt/aws/bin/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-init /opt/aws/bin/cfn-init
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-signal /opt/aws/bin/cfn-signal
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-get-metadata /opt/aws/bin/cfn-get-metadata </pre>

If you are using a different base OS AMI, please see [[http://www.danieleyer.com/blog/2016/07/15/adding-aws-cfn-bootstrap-cloudformation-helper-scripts-to-custom-linux-amis/][this link]] for some possible changes to the CloudFormation Helper Script configuration. 

Activate (or deactivate) the following services as indicated and then logout from the instance. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@ANNEX-PRIVATE-IP ~]$ chkconfig iptables off
 [root@ANNEX-PRIVATE-IP ~]$ service iptables stop
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig fetch-crl-boot on
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig fetch-crl-cron on
 [root@ANNEX-PRIVATE-IP ~]$ service fetch-crl-boot start
 [root@ANNEX-PRIVATE-IP ~]$ service fetch-crl-cron start
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig condor on
 [root@ANNEX-PRIVATE-IP ~]$ service condor start
 [root@ANNEX-PRIVATE-IP ~]$ exit </pre>

Now that you have prepared a condor_annex-compatible AMI on this instance, you'll need to save it for future use on other instances. To do so:

   1 Return to your web browser and go to the [[https://console.aws.amazon.com/ec2/][EC2 console]]. 
   1 In the navigation pane, under _INSTANCES_, choose _Instances_. 
   1 There you will see a list of each individual instance available in the Region. Select the instance you've just configured your condor_annex-compatible AMI on.
   1 From the dropdown menu *Actions*, go to _Image_ and select _Create Image_. 
   1 You will be prompted to make changes to the AMI before its creation. You'll likely want to add an *Image name* and check the *Delete on Termination* box. Make any other adjustments you find necessary and then click on the *Create Image* button. This will create an AMI from your instance that can be used with condor_annex. 
   1 Go ahead and *Close* the _Create Image request received_ dialog box to return to the EC2 Dashboard. 
   1 In the navigation pain, under _IMAGES_, click on _AMIs_. There you will see a list of the your custom AMIs, including the condor_annex-compatible AMI that was just created from your instance. Note the *AMI ID* for this image as it will be one of the required inputs when calling condor_annex.

---++ Step 8: Configure HTCondor Pool for Password Authentication

condor_annex currently assumes that your local HTCondor pool allows daemon-to-daemon communication via Password Authentication. If your local pool is not yet configured to use a pool password, you must first [[http://research.cs.wisc.edu/htcondor/manual/v8.4/3_6Security.html#SECTION00463400000000000000][generate and store a password file]] on both the  *SUBMIT* node and *CENTRAL_MANAGER* by running the following command on each:

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ condor_store_cred -c add </pre>

This command will prompt you to enter a pool password. Once entered, a password file will be stored on the local machine. By default, the password file created on each machine is /etc/condor/condor_pool_password. Make sure that you run this pool password command --- entering the same password --- on both the SUBMIT node and the CENTRAL_MANAGER of your local pool. You may also use Password Authentication with your local *EXECUTE* nodes. However, this is not required by condor_annex. Only the *ANNEX* instances require the use of Password Authentication. 

Once your SUBMIT node and CENTRAL_MANAGER have the pool password file, you must configure their HTCondor daemons to use Password Authentication. On both the machines, login as root and go to the HTCondor config.d directory.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@CENTRAL_MANAGER ~]$ cd /etc/condor/config.d </pre>

In this directory, create the following HTCondor configuration file (99_condor_annex_passwd.config)
 
<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffffcc; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> ALLOW_DAEMON = $(ALLOW_DAEMON), condor_pool@*
 SEC_DEFAULT_AUTHENTICATION = REQUIRED 
 SEC_DEFAULT_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS), PASSWORD
 SEC_DEFAULT_ENCRYPTION = OPTIONAL
 SEC_DEFAULT_INTEGRITY = REQUIRED
 SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE
 SEC_PASSWORD_FILE = /etc/condor/condor_pool_password </pre>

and then restart condor. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@CENTRAL_MANAGER ~]$ service condor restart </pre>

Your local HTCondor pool should now be ready to use Password Authentication with condor_annex. 

---++ Step 9. Install and configure AWS CLI

The [[https://aws.amazon.com/cli/][AWS Command Line Interface (CLI)]] is a tool to manage your AWS resources and services from the command line as well as automate your interaction with them via scripting. Remember, condor_annex itself is a Perl-based script that relies on the AWS CLI to automate the construction of an annex given the inputs provided by a user. As such, the AWS CLI must be installed and configured on any host that will run condor_annex.

If you plan to let your users run condor_annex for themselves when they need additional resources, then you should install the AWS CLI on your HTCondor pool's SUBMIT node. To install the AWS CLI, login as root to your SUBMIT node and run the following commands.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ yum install python-pip
 [root@SUBMIT ~]$ pip install awscli </pre>

Once the AWS CLI is installed, each user who wants to run condor_annex will have to configure the CLI using their AWS Security Credentials. To configure the CLI, they must run the following command and enter the requested information.

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #e0f0e0; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [user@SUBMIT ~]$ aws configure
 AWS Access Key ID [None]: ****************4FSQ
 AWS Secret Access Key [None]: ****************RbV6
 Default region name [None]: us-east-1
 Default output format [None]: json
</pre>

For the _Default region name_ and _Default output format_, please make sure to instruct your users to enter (1) the codename for the AWS Region that contains your pre-configured condor_annex-compatible AMI and (2) json, respectively. Once a user completes this AWS CLI configuration process, they should find the settings stored in the new .aws directory that has been created in their home directory.

---++ Step 10. Install and configure condor annex

Now that the AWS CLI is installed and configured on you local pool's SUBMIT node, you can also install condor_annex on it. Since condor_annex is not currently distributed via RPMs, you will have to clone one of the git repositories where it is stored. As such, you must begin by installing git. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ yum install git </pre>

Once git is installed, you should clone one of the repositories containing condor_annex into /opt. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ cd /opt </pre>

The current development version of condor_annex from the HTCondor team is available in:

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ git clone https://github.com/htcondor/htcondor.git -b V8_5-condor_annex-branch </pre>

Note, however, this is the __entire__ HTCondor project's development branch for condor_annex. The only components necessary to run condor_annex are actually self-contained within the directory /htcondor/src/condor_annex. A forked repository that only contains these condor_annex components as well as a few minor modifications to them is also available at:

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ [root@SUBMIT ~]$ git clone https://github.com/mkandes/condor_annex.git </pre>

This repository is intended to remain more stable while the HTCondor team continues to develop condor_annex into a HTCondor daemon. It would be our recommendation to use this repository while evaluating condor_annex. 

Finally, condor_annex requires the perl-JSON module. Don't forget to install it after you've cloned condor_annex from one of the repositories. 

<pre style="width: auto; margin: 1em 0px; font-size: 13px; color: #666699; background-color: #ffddbb; border: 1px solid gray; padding: 0.5em 2px; white-space: pre; overflow: auto !important"> [root@SUBMIT ~]$ yum install perl-JSON </pre>

---++ Step 11. Launch a condor_annex

</noautolink>
Topic revision: r14 - 2016/11/22 - 00:49:14 - MartinKandes
UCSDTier2
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback