Install and Configure condor_annex

About this Document

condor_annex is a Perl-based script that utilizes the Amazon Web Services (AWS) command-line interface (CLI) and other AWS services to orchestrate the delivery of HTCondor execute nodes running on AWS Elastic Compute Cloud (EC2) instances to an HTCondor pool. This document describes how to install, configure, and run condor_annex successfully from your own local HTCondor pool.

This document follows the general Open Science Grid (OSG) documentation conventions:

1. A User Command Line is illustrated by a green box that displays a prompt:

    [user@client ~]$ 

2. A Root Command Line is illustrated by a red box that displays the root prompt:

    [root@client ~]$ 

3. Lines in a file are illustrated by a yellow box that displays the desired lines in a file:

    priorities=1 

Definitions

Hostnames:

• SUBMIT is the hostname of the HTCondor submit node, where users submit their jobs to your local pool.
• CENTRAL_MANAGER is the hostname of the HTCondor central manager of your local pool, which collects and matches job and machine class ads.
• EXECUTE is the hostname of an HTCondor execute node in your local pool.
• ANNEX is the hostname of an EC2 instance configured as a condor_annex execute node.

Usernames:

• BOSCO_USER is the username of the user on the BOSCO_HOST that has access to the BOSCO resource's local batch queue; e.g., cmsbosco
• FACTORY_ADMIN_USER is the username of the user on the FACTORY_HOST used for all non-root administrative tasks; e.g., gfactory
• FACTORY_VO_USER is the username of the user on the FACTORY_HOST from which glideins are submitted to the BOST_HOST; e.g., fecmsglobal
• FRONTEND_USER is the username of the user on the FRONTEND_HOST that submits requests for glideins to the FACTORY_HOST; e.g., frontend

Requirements

• An HTCondor pool
• An Amazon Web Services Account

Step 1: Install and Configure an HTCondor Pool

If you do not already have your own HTCondor Pool, you may want to first start by installing your own personal HTCondor pool to experiment with condor_annex. Please consult the HTCondor Manual and/or Wiki for more information:

• http://research.cs.wisc.edu/htcondor/manual/v8.4/ref.html
• https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki

Step 2: Obtain an Amazon Web Services Account

In order to use condor_annex, you must already have an AWS account. You may establish an AWS account under the UC-wide agreement by following the instructions provided by Blink:

• https://blink.ucsd.edu/technology/cloud/aws/

Step 3: Obtain AWS Account Credentials

condor_annex issues programmatic requests to AWS services via the AWS command-line interface (CLI). In order to issue these requests, the AWS CLI must sign them using your AWS account credentials. These credentials consist of an Access Key ID and a Secret Access Key. If you do not have these access keys, you may create them using the AWS Management Console. AWS recommends that you use Identity and Access Management (IAM) access keys instead of your root account access keys.

To create access keys, you must have permissions to perform the required IAM actions.

1. Open the IAM console.
2. In the navigation pane, choose Users.
3. If you do not already have an IAM username, then select Create New Users. Each new user is issued credentials.
4. If you already have an IAM username, then choose your IAM username (not the check box).
5. Next, select the Security Credentials tab and then choose Create Access Key.
6. To see your access key, choose Show User Security Credentials. Your credentials will look something like this:
   • Access Key ID: AKIAIOSFODNN7EXAMPLE?
   • Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
7. Choose Download Credentials, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

If you need more information about AWS Security Credentials, please consult the AWS documentation at:

• http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html

Step 4: Select an Amazon Region for the Annex

Amazon Elastic Compute Cloud (EC2) instances are hosted in multiple locations world-wide. These locations are composed of Regions and Availability Zones. Each Region is a separate geographic area. However, each Region also has multiple, isolated locations known as Availability Zones (AZs), which are simply high-availability data centers. However, not all AWS Regions are created equal. Each Region may offer only a subset of AWS services. You can find out what services are offered in each Region from the table provided here:

• https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/

When selecting a Region for your HTCondor annex, you must select a region that offers all of the AWS services required by condor_annex to function properly. These services are:

• Amazon Elastic Block Store (EBS): https://aws.amazon.com/ebs/
• Amazon Elastic Compute Cloud (EC2): https://aws.amazon.com/ec2/
• Amazon CloudWatch? : https://aws.amazon.com/cloudwatch/
• Amazon Simple Notification Service (SNS): https://aws.amazon.com/sns/
• Amazon Simple Storage Service (S3): https://aws.amazon.com/s3/
• Amazon Virtual Private Cloud (VPC): https://aws.amazon.com/vpc/
• AWS Auto Scaling Groups (ASG): https://aws.amazon.com/autoscaling/
• AWS CloudFormation? : https://aws.amazon.com/cloudformation/
• AWS Identity and Access Management (IAM): https://aws.amazon.com/iam/
• AWS Lambda: https://aws.amazon.com/lambda/

AWS Lambda currently has the most limited deployment of any AWS service required by condor_annex. For example, in the US, AWS Lambda is only available in the following Regions at this time:

• Northern Virginia (us-east-1)
• Ohio (us-east-2)
• Oregon (us-west-2)

Please select your desired Region accordingly from the drop-down menu in the upper-right-hand side of the AWS Management Console.

Step 5: Generate an Amazon EC2 Key Pair

After selecting a Region for your HTCondor annex, you will need to generate an SSH key pair that will allow you to login to your EC2 instances. You can create a key pair using the Amazon EC2 console or the command line. After you create a key pair, you will specify it when launching your instances from condor_annex.

To create your key pair using the Amazon EC2 console

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
3. Choose Create Key Pair.
4. Enter a name for the new key pair in the Key pair name field of the Create Key Pair dialog box, and then choose Create.
5. The private key file is automatically downloaded by your browser. The base file name is the name you specified as the name of your key pair, and the file name extension is .pem. Save the private key file in a safe place. This is the only chance for you to save the private key file. You'll need to provide the name of your key pair when you launch an instance and the corresponding private key each time you connect to the instance.
6. Use the following command to set the permissions of your private key file so that only you can read it.

    [user@client ~]$ chmod 400 my-key-pair.pem 

If you would like to create your SSH key pair using the AWS CLI or import your own key pair, please consult the AWS documentation at:

• http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

Step 6: Configure Default VPC Security Group

condor_annex will create and configure an AWS Security Group (i.e., a virtual firewall) around each annex it builds. However, depending on your HTCondor pool configuration, it may also been useful to place some (static, non-preemptable) on-demand resources in AWS. For example, you may want to a separate HTCondor central manager instance in AWS in order to flock user jobs over to the annex instead of connecting the annex instances all the way back to your local central manager.

Any such resources may be placed in your AWS Region's default Virtual Private Cloud (VPC) Security Group. To configure this default VPC Security Group:

1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
2. In the navigation pane, under Security, choose Security Groups.
3. Select the Security Group in the list that has Group Name default and Description default VPC security group.
4. Next, select the Inbound Rules tab and then click on the Edit button.

By default, only inbound rule for the default VPC Security Group should be

(sg-xxxxxxxx

Source Protocol Port Range Comments The security group ID (sg-xxxxxxxx) All All Allow inbound traffic from instances assigned to the same security group

Type Protocol Port Range Source ---- -------- ---------- ------ All traffic All All sg-5437332d (default) All traffic All All 172.31.0.0/20 SSH TCP 22 0.0.0.0/0 Custom UDP Rule UDP 9618 0.0.0.0/0 Custom TCP Rule TCP 9618 0.0.0.0/0 All ICMP All N/A 0.0.0.0/0

Step 7: Create an condor_annex-compatible Amazon Machine Image

Step 8: Configure HTCondor Pool for Password Authentication

Step 9. Install and configure AWS CLI on HTCondor submit node

Step 10. Install and configure condor annex on HTCondor submit node

Step 11. Launch a condor_annex

-- MartinKandes - 2016/11/01