[root@SUBMIT ~]$ yum install git 

Once git is installed, you should clone one of the repositories containing condor_annex into /opt.

 [root@SUBMIT ~]$ cd /opt 

The current development version of condor_annex from the HTCondor team is available in:

 [root@SUBMIT ~]$ git clone https://github.com/htcondor/htcondor.git -b V8_5-condor_annex-branch 

Note, however, this is the entire HTCondor project's development branch for condor_annex. The only components necessary to run condor_annex are actually self-contained within the directory /htcondor/src/condor_annex. A forked repository that only contains these condor_annex components as well as a few minor modifications to them is also available at:

 [root@SUBMIT ~]$ [root@SUBMIT ~]$ git clone https://github.com/mkandes/condor_annex.git 

This repository is intended to remain more stable while the HTCondor team continues to develop condor_annex into a HTCondor daemon. It would be our recommendation to use this repository while evaluating condor_annex.

Finally, condor_annex requires the perl-JSON module. Don't forget to install it after you've cloned condor_annex from one of the repositories.

 [root@SUBMIT ~]$ yum install perl-JSON 

1. Open the IAM console.

1. Your credentials will look something like this:

1. Choose Download .csv file, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

• Amazon Elastic Block Store (EBS)
• Amazon Elastic Compute Cloud (EC2)
• Amazon CloudWatch
• Amazon Simple Notification Service (SNS)
• Amazon Simple Storage Service (S3)
• Amazon Virtual Private Cloud (VPC)
• AWS Auto Scaling Groups (ASG)
• AWS CloudFormation
• AWS Identity and Access Management (IAM)
• AWS Lambda

1. Open the EC2 console.

1. Open the VPC console.

1. Configure Instance Details: Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the Number of Instances at 1. Next, select one of your Network VPCs. In general, you should choose the default VPC whose Security Group was pre-configured in the previous step. Once you have determined which VPC will host this instance, select a specific Subnet in which to place it. The other networking options Auto-assign Public IP and Placement group may be left set to their default settings of Use subnet setting (Enabled) and No placement group, respectively. After configuring the networking details, if you would like to apply a specific IAM role to the instance, then select an appropriate role for it. Otherwise, leave IAM role set to its default value of None. All other instance details may be configured with their default values. Once you have completed configuring your instance details, click on the Next: Add Storage button.

1. Return to your web browser and go to the EC2 console.
2. In the navigation pane, under INSTANCES, choose Instances.
3. There you will see a list of each individual instance available in the Region. Select the instance you've just configured your condor_annex-compatible AMI on.
4. From the dropdown menu Actions, go to Image and select Create Image.
5. You will be prompted to make changes to the AMI before its creation. You'll likely want to add an Image name and check the Delete on Termination box. Make any other adjustments you find necessary and then click on the Create Image button. This will create an AMI from your instance that can be used with condor_annex.
6. Go ahead and Close the Create Image request received dialog box to return to the EC2 Dashboard.
7. In the navigation pain, under IMAGES, click on AMIs. There you will see a list of the your custom AMIs, including the condor_annex-compatible AMI that was just created from your instance. Note the AMI ID for this image as it will be one of the required inputs when calling condor_annex.

• SUBMIT is the hostname of an HTCondor submit node, where users submit their jobs to your local pool.
• CENTRAL_MANAGER is the hostname of your HTCondor central manager, where job and machine class ads are matched.

1. In the navigation pane, choose Users.
2. If you do not already have an IAM username, then select Add User. Each new user is issued credentials.

1. Next, select the Security Credentials tab and then choose Create Access Key.

Step 4: Select a Region for the Annex

Step 5: Generate an EC2 Key Pair

1. Open the EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
3. Choose Create Key Pair.
4. Enter a name for the new key pair in the Key pair name field of the Create Key Pair dialog box, and then choose Create.

1. Use the following command to set the permissions of your private key file so that only you can read it.

    [user@SUBMIT ~]$ chmod 400 my-key-pair.pem 

1. Open the VPC console at https://console.aws.amazon.com/vpc/.

If you plan to let your users run condor_annex for themselves when they need additional resources, then you should install the AWS CLI on your HTCondor pool's SUBMIT node. To install the AWS CLI, login as root to your SUBMIT node and run the following commands.

 [root@SUBMIT ~]$ yum install python-pip
 [root@SUBMIT ~]$ pip install awscli 

Once the AWS CLI is installed, each user who wants to run condor_annex will have to configure the CLI using their AWS Security Credentials. To configure the CLI, they must run the following command and enter the requested information.

 [user@SUBMIT ~]$ aws configure
 AWS Access Key ID [None]: ****************4FSQ
 AWS Secret Access Key [None]: ****************RbV6
 Default region name [None]: us-east-1
 Default output format [None]: json

For the Default region name and Default output format, please make sure to instruct your users to enter (1) the codename for the AWS Region that contains your pre-configured condor_annex-compatible AMI and (2) json, respectively. Once a user completes this AWS CLI configuration process, they should find the settings stored in the new .aws directory that has been created in their home directory.

• SUBMIT is the hostname of the HTCondor submit node, where users submit their jobs to your local pool.
• CENTRAL_MANAGER is the hostname of the HTCondor central manager of your local pool, which collects and matches job and machine class ads.
• EXECUTE is the hostname of an HTCondor execute node in your local pool.
• ANNEX is the hostname of an EC2 instance configured as a condor_annex execute node.

1. Choose an Amazon Machine Image (AMI): We configured our annex's execute instances to use CentOS? 6. To find a suitable CentOS? 6 AMI to start from, select the AWS Marketplace tab and then enter "CentOS 6" in the search box. Your search will return multiple results. However, the most up-to-date AMI should be the first one in the list. Unless you have special requirements for your configuration, select this AMI by clicking on the Select button.
2. Choose an Instance Type: Once you have selected an AMI, the launch configuration wizard will prompt you to select an instance type on which to build your condor_annex execute node. Choose one that suits your needs. Once you have selected your instance type, click on the Next: Configure Instance Details button.
3. Configure Instance Details: Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the Number of Instances at 1. Next, select one of your Network VPCs. In general, you should choose the default VPC whose Security Group was pre-configured in the previous step. Once you have determined which VPC will host this instance, select a specific Subnet in which to place it. All other networking options Auto-assign Public IP and Placement group may be left set to their default settings of Use subnet setting (Enabled) and No placement group, respectively. After configuring the networking details, if you would like to apply a specific AWS IAM role to the instance, then select an appropriate role for it. Otherwise, leave IAM role set to its default value of None. All other instance details may be configured with their default values. Once you have completed configuring your instance details, click on the Next: Add Storage button.

1. Tag Instance: Add a Name to your instance and then click on the Next: Configure Security Group button.

1. Review Instance Launch: Review the configuration of your instance and make any necessary changes. Once done, click on the Launch button. You will be prompted to Select an existing key pair or create a new key pair, which will enable your SSH access to the instance. Select one of your existing key pairs (or create a new one) and then agree to the acknowledgement statement by clicking on the checkbox next to it. Once you have selected your key pair, click on the Launch Instances button. After launching your instance, the wizard will display the Launch Status page. To return to the main EC2 dashboard, scroll down the click on the View Instances button.

 [user@client ~]$ ssh -i ~/.ssh/HTCondorAnnex.pem centos@ANNEX.PUBLIC.IP 

 [centos@ANNEX-PRIVATE-IP ~]$ sudo -i 

 [root@ANNEX-PRIVATE-IP ~]$ yum update

 [root@ANNEX-PRIVATE-IP ~]$ yum install epel-release

 [root@ANNEX-PRIVATE-IP ~]$ yum install yum-plugin-priorities 

 [root@ANNEX-PRIVATE-IP ~]$ rpm -Uvh https://repo.grid.iu.edu/osg/3.3/osg-3.3-el6-release-latest.rpm

 [root@ANNEX-PRIVATE-IP ~]$ yum install osg-ca-certs
 [root@ANNEX-PRIVATE-IP ~]$ yum install fetch-crl 

 [root@ANNEX-PRIVATE-IP ~]$ yum install osg-wn-client 

 [root@ANNEX-PRIVATE-IP ~]$ groupadd condor
 [root@ANNEX-PRIVATE-IP ~]$ useradd condor -g condor
 [root@ANNEX-PRIVATE-IP ~]$ yum install condor.x86_64 

 [root@ANNEX-PRIVATE-IP ~]$ yum install cloud-init

 [root@ANNEX-PRIVATE-IP ~]$ yum install pystache
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-argparse
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-daemon
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-requests 
 [root@ANNEX-PRIVATE-IP ~]$ yum install python-pip 

 [root@ANNEX-PRIVATE-IP ~]$ pip install --upgrade pip 

Once these packages are installed, install the AWS CLI.

 [root@ANNEX-PRIVATE-IP ~]$ pip install awscli 

and the AWS CloudFormation Helper Scripts.

 [root@ANNEX-PRIVATE-IP ~]$ easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz 

Several standard directories and symbolic links found on Amazon Linux AMIs must be created to successfully use the CloudFormation? Helper Scripts on CentOS? 6.

 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-hup /etc/init.d/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ chmod 775 /usr/bin/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ mkdir /opt/aws
 [root@ANNEX-PRIVATE-IP ~]$ mkdir /opt/aws/bin
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-hup /opt/aws/bin/cfn-hup
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-init /opt/aws/bin/cfn-init
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-signal /opt/aws/bin/cfn-signal
 [root@ANNEX-PRIVATE-IP ~]$ ln -s /usr/bin/cfn-get-metadata /opt/aws/bin/cfn-get-metadata 

 [root@ANNEX-PRIVATE-IP ~]$ chkconfig iptables off
 [root@ANNEX-PRIVATE-IP ~]$ service iptables stop
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig fetch-crl-boot on
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig fetch-crl-cron on
 [root@ANNEX-PRIVATE-IP ~]$ service fetch-crl-boot start
 [root@ANNEX-PRIVATE-IP ~]$ service fetch-crl-cron start
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig condor on
 [root@ANNEX-PRIVATE-IP ~]$ service condor start
 [root@ANNEX-PRIVATE-IP ~]$ exit 

Return now to the AWS Management Console in your web browser and then go to the EC2 dashboard. In the navigation pane, under INSTANCES, choose Instances. There you will see a list of each individual instance available in the AWS Region. Select the instance you've just configured your condor_annex-compatible AMI on. Then from the dropdown menu Actions, go to Image and select Create Image. You will be prompted to make changes to the AMI before its creation. You'll likely want to add an Image name and check the Delete on Termination box. Make any other adjustments you find necessary and then click on the Create Image button. This will create an AMI from your instance that can be used with condor_annex.

Go ahead and Close the Create Image request received dialog box to return to the EC2 Dashboard. In the navigation pain, under IMAGES, click on AMIs. There you will see a list of the your custom AMIs, including the condor_annex-compatible AMI that was just created from your instance. Note the AMI ID for this image as it will be one of the required inputs when calling condor_annex.

 [root@SUBMIT ~]$ condor_store_cred -c add 

This command will prompt you to enter a pool password. Once entered, a password file will be stored on the local machine. By default, the password file created on each machine is /etc/condor/condor_pool_password. Make sure that you run this pool password command --- entering the same password --- on both the SUBMIT node and the CENTRAL_MANAGER of your local pool. You may also use Password Authentication with your local EXECUTE nodes. However, this is not required by condor_annex. Only the ANNEX instances require the use of Password Authentication.

Once your SUBMIT node and CENTRAL_MANAGER have the pool password file, you must configure their HTCondor daemons to use Password Authentication. On both the machines, login as root and go to the HTCondor config.d directory.

 [root@CENTRAL_MANAGER ~]$ cd /etc/condor/config.d 

In this directory, create the following HTCondor configuration file (99_condor_annex.config)

 ALLOW_DAEMON = $(ALLOW_DAEMON), condor_pool@*
 SEC_DEFAULT_AUTHENTICATION = REQUIRED
 SEC_DEFAULT_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS), PASSWORD
 SEC_DEFAULT_ENCRYPTION = OPTIONAL
 SEC_DEFAULT_INTEGRITY = REQUIRED
 SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE
 SEC_PASSWORD_FILE = /etc/condor/condor_pool_password 

and then restart condor.

 [root@CENTRAL_MANAGER ~]$ service condor restart 

Your local HTCondor pool should now be ready to use Password Authentication with condor_annex.

Step 9. Install and configure AWS CLI

Step 10. Install and configure condor annex

To build your own condor_annex-compatible AMI, open the Elastic Compute Cloud (EC2) dashboard in the AWS Region where you will run your annex. Click on the Launch Instance button. This will open the instance launch configuration wizard. Follow its steps.

1. Choose an Amazon Machine Image (AMI): We configured our annex's execute instances to use CentOS? 6. To find a suitable CentOS? 6 AMI to start from, select the AWS Marketplace tab on the left-hand side of the page and then enter "CentOS 6" in the search box. Your search will return multiple results. However, the most up-to-date AMI should be the first one in the list. Unless you have special requirements for your configuration, select this AMI. Here, we selected the latest official CentOS? 6 AMI available that still supports paravirtualization (pv).
2. Choose an Instance Type: Next, the launch configuration wizard will prompt you to select an instance type on which to build your condor_annex execute node. Choose one that suits your needs. For this example, a t1.micro instance was chosen. Once you have selected your instance type, click on the Next: Configure Instance Details button.
3. Configure Instance Details: Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the Number of Instances at 1. Next, select one of your Network VPCs for the AWS Region in which you have chosen to locate the annex. Once you have determined which VPC will host this instance, select a specific Subnet in which to place it. For example, we chose our default VPC for the us-east-1 Region and its default subnet for the us-east-1a Availability Zone. All other networking options Auto-assign Public IP and Placement group were left set to their default settings of Use subnet setting (Enabled) and No placement group, respectively. After configuring the networking details, if you would like to apply a specific AWS IAM role to the instance, then select an appropriate IAM role for it. Otherwise, leave IAM role set to its default value of None. All other instance details were configured with their default values. Once you have completed configuring your instance details, click on the Next: Add Storage button.
4. Add Storage: In general, you will not have to modify the configuration of your root storage volume for the instance. However, the launch wizard may still default to a Magnetic volume type, even though the General Purpose SSD option is now becoming AWS' recommended default. Our instance launch wizard still defaults to Magnetic. As such, we changed our root Volume Type from an 8GiB Magnetic volume to an 8 GiB? General Purpose SSD volume and selected Delete on Termination. Once you have completed the configuration of your root volume, click on the Next: Tag Instance button.
5. Tag Instance: Add a Name to your instance. Once you have finished tagging your instance, click on the Next: Configure Security Group button.
6. Configure Security Group: Select an existing security group and choose your default VPC security group. Once you have selected a security group, click on the Review and Launch button.
7. Review Instance Launch: Review the configuration of your instance and make any necessary changes. Once done, click on the Launch button. You will be prompted to Select an existing key pair or create a new key pair, which will enable your SSH access to the instance. Select one of your existing key pairs (or create a new one) and then agree to the acknowledgement statement by clicking on the checkbox next to it. Once you have selected your key pair, click on the Launch Instances button. After launching your instance, the wizard will display the Launch Status page. To return to the main EC2 dashboard, scroll down the click on the View Instances button.

Once the instance has started up and enters a running Instance State, you will install and configure the software required to create a condor_annex-compatible AMI on the instance. To begin, log into the instance via SSH.

 [user@client ~]$ ssh -i ~/.ssh/HTCondorAnnex.pem centos@107.22.156.79 

Then switch to root.

 [centos@ip-107-22-156-79 ~]$ sudo -i 

First, update the instance's base OS configuration.

 [root@ip-107-22-156-79 ~]$ yum update

Then install the Extra Packages for Enterprise Linux (EPEL) repository.

 [root@ip-107-22-156-79 ~]$ yum install epel-release

Next, install the yum priorities package

 [root@ip-107-22-156-79 ~]$ yum install yum-plugin-priorities 

and the appropriate Open Science Grid (OSG) repositories.

 [root@ip-107-22-156-79 ~]$ rpm -Uvh https://repo.grid.iu.edu/osg/3.3/osg-3.3-el6-release-latest.rpm

Once the OSG repositories are available on the instance, install the CA certificates and fetch-crl.

 [root@ip-107-22-156-79 ~]$ yum install osg-ca-certs
 [root@ip-107-22-156-79 ~]$ yum install fetch-crl 

Next, install the OSG Worker Node Client.

 [root@ip-107-22-156-79 ~]$ yum install osg-wn-client 

After the client software is installed, manually create both a condor group and user and then install HTCondor.

 [root@ip-107-22-156-79 ~]$ groupadd condor
 [root@ip-107-22-156-79 ~]$ useradd condor -g condor
 [root@ip-107-22-156-79 ~]$ yum install condor.x86_64 

This would complete the typical software installation of a standard OSG HTCondor execute node, except for CVMFS. However, in order to support condor_annex, several other software packages must be properly installed on the instance.

In addition to the standard OSG software, you must also install cloud-init

 [root@ip-107-22-156-79 ~]$ yum install yum install cloud-init

and several other python packages, including pip.

 [root@ip-107-22-156-79 ~]$ yum install pystache
 [root@ip-107-22-156-79 ~]$ yum install python-argparse
 [root@ip-107-22-156-79 ~]$ yum install python-daemon
 [root@ip-107-22-156-79 ~]$ yum install python-requests 
 [root@ip-107-22-156-79 ~]$ yum install python-pip 

You may also want to make sure these packages are up-to-date.

 [root@ip-107-22-156-79 ~]$ pip install --upgrade pip 

pip install awscli easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz ln -s /usr/bin/cfn-hup /etc/init.d/cfn-hup chmod 775 /usr/bin/cfn-hup mkdir /opt/aws mkdir /opt/aws/bin ln -s /usr/bin/cfn-hup /opt/aws/bin/cfn-hup ln -s /usr/bin/cfn-init /opt/aws/bin/cfn-init ln -s /usr/bin/cfn-signal /opt/aws/bin/cfn-signal ln -s /usr/bin/cfn-get-metadata /opt/aws/bin/cfn-get-metadata

1. Keep the default rule allowing all traffic from instances assigned to the default VPC Security Group.
2. Allow all inbound traffic from instances within your AWS Region's default VPC's private network IP address space.
3. Allow inbound SSH traffic on port 22.
4. Allow inbound ICMP traffic.
5. Allow inbound HTCondor UDP traffic on port 9618
6. Allow inbound HTCondor TCP traffic on port 9618.

Type	Protocol	Port Range	Source <-- -->
SSH	TCP	22	0.0.0.0/0
All ICMP	All	N/A	0.0.0.0/0
Custom UDP Rule	UDP	9618	0.0.0.0/0
Custom TCP Rule	TCP	9618	0.0.0.0/0
All traffic	All	All	172.31.0.0/16
All traffic	All	All	sg-5437332d (default)

Of course, you should try to restrict the size of the Source IP address space for these rules as much as possible. For example, you may want to limit them to inbound traffic from your home institution's public IP address space.

By default, each Security Group, including the default VPC Security Group, allows ALL outbound traffic.

If you would like to restrict outbound traffic from the default VPC Security Group, select the Outbound Rules tab, click on the Edit button, and configure the outbound rules accordingly.

1. In the navigation pane, under Security, choose Security Groups.

1. Next, select the Inbound Rules tab and then click on the Edit button.

Source Protocol Port Range Comments The security group ID (sg-xxxxxxxx) All All Allow inbound traffic from instances assigned to the same security group

These static, on-demand resources may be placed in your Region's default Virtual Private Cloud (VPC) Security Group. To configure this default VPC Security Group:

1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
2. In the navigation pane, under SECURITY, choose Security Groups.
3. Select the Security Group in the list that has Group Name default and Description default VPC security group.
4. Next, select the Inbound Rules tab and then choose Edit.

By default, each Security Group, including the default VPC Security Group,

Type Protocol Port Range Source ---- -------- ---------- ------ All traffic All All sg-5437332d (default) All traffic All All 172.31.0.0/20 SSH TCP 22 0.0.0.0/0 Custom UDP Rule UDP 9618 0.0.0.0/0 Custom TCP Rule TCP 9618 0.0.0.0/0 All ICMP All N/A 0.0.0.0/0

Step 7: Create an condor_annex-compatible Amazon Machine Image

• ANNEX is the hostname of an EC2 instance configured as a condor_annex execute node.

Step 1: Install and Configure an HTCondor Pool

Step 2: Obtain an Amazon Web Services Account

Step 3: Obtain AWS Account Credentials

1. If you do not already have an IAM username, then select Create New Users. Each new user is issued credentials.

Step 4: Select an Amazon Region for the Annex

• https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/

When selecting a Region for your HTCondor annex, you must select a region that offers all of the AWS services required by condor_annex to function properly. These services are:

• Amazon Elastic Block Store (EBS): https://aws.amazon.com/ebs/
• Amazon Elastic Compute Cloud (EC2): https://aws.amazon.com/ec2/
• Amazon CloudWatch? : https://aws.amazon.com/cloudwatch/
• Amazon Simple Notification Service (SNS): https://aws.amazon.com/sns/
• Amazon Simple Storage Service (S3): https://aws.amazon.com/s3/
• Amazon Virtual Private Cloud (VPC): https://aws.amazon.com/vpc/
• AWS Auto Scaling Groups (ASG): https://aws.amazon.com/autoscaling/
• AWS CloudFormation? : https://aws.amazon.com/cloudformation/
• AWS Identity and Access Management (IAM): https://aws.amazon.com/iam/
• AWS Lambda: https://aws.amazon.com/lambda/

AWS Lambda currently has the most limited deployment of any AWS service required by condor_annex. For example, in the US, AWS Lambda is only available in the following Regions at this time:

• Northern Virginia (us-east-1)
• Ohio (us-east-2)
• Oregon (us-west-2)

Please select your desired Region accordingly from the drop-down menu in the upper-right-hand side of the AWS Management Console.

Step 5: Generate an Amazon EC2 Key Pair

After selecting a Region for your HTCondor annex, you will need to generate an SSH key pair that will allow you to login to your EC2 instances. You can create a key pair using the Amazon EC2 console or the command line. After you create a key pair, you will specify it when launching your instances from condor_annex.

To create your key pair using the Amazon EC2 console

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
3. Choose Create Key Pair.
4. Enter a name for the new key pair in the Key pair name field of the Create Key Pair dialog box, and then choose Create.
5. The private key file is automatically downloaded by your browser. The base file name is the name you specified as the name of your key pair, and the file name extension is .pem. Save the private key file in a safe place. This is the only chance for you to save the private key file. You'll need to provide the name of your key pair when you launch an instance and the corresponding private key each time you connect to the instance.
6. Use the following command to set the permissions of your private key file so that only you can read it.

    [user@client ~]$ chmod 400 my-key-pair.pem 

If you would like to create your SSH key pair using the AWS CLI or import your own key pair, please consult the AWS documentation at:

• http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

Step 6: Configure Default VPC Security Group

Step 7: Create an condor_annex-compatible Amazon Machine Image (AMI)

Step 8: Configure HTCondor Pool for Password Authentication

Step 9. Install and configure AWS CLI on HTCondor submit node

Step 10. Install and configure condor annex on HTCondor submit node

Step 11. Launch a condor_annex

Install and Configure condor_annex

About this Document

condor_annex is a Perl-based script that utilizes the Amazon Web Services (AWS) command-line interface (CLI) and other AWS services to orchestrate the delivery of HTCondor execute nodes to an HTCondor pool. This document describes how to install, configure, and run condor_annex successfully from your own local HTCondor pool.

This document follows the general Open Science Grid (OSG) documentation conventions:

1. A User Command Line is illustrated by a green box that displays a prompt:

    [user@client ~]$ 

2. A Root Command Line is illustrated by a red box that displays the root prompt:

    [root@client ~]$ 

3. Lines in a file are illustrated by a yellow box that displays the desired lines in a file:

    priorities=1 

Definitions

Hostnames:

• SUBMIT is the hostname of the HTCondor submit node, where users submit their jobs to your local pool.
• CENTRAL_MANAGER is the hostname of the HTCondor central manager of your local pool, which collects and matches job and machine class ads.
• EXECUTE is the hostname of an HTCondor execute node in your local pool.
• ANNEX is the hostname (or IP address) of an AWS-based instance configured as an condor_annex execute node.

Usernames:

• BOSCO_USER is the username of the user on the BOSCO_HOST that has access to the BOSCO resource's local batch queue; e.g., cmsbosco
• FACTORY_ADMIN_USER is the username of the user on the FACTORY_HOST used for all non-root administrative tasks; e.g., gfactory
• FACTORY_VO_USER is the username of the user on the FACTORY_HOST from which glideins are submitted to the BOST_HOST; e.g., fecmsglobal
• FRONTEND_USER is the username of the user on the FRONTEND_HOST that submits requests for glideins to the FACTORY_HOST; e.g., frontend

Requirements

• An HTCondor pool
• An Amazon Web Services Account

Install and Configure an HTCondor Pool

If you do not already have your own HTCondor Pool, you may want to first start by installing your own personal HTCondor pool to experiment with condor_annex. Please consult the HTCondor Manual and/or Wiki for more information:

• http://research.cs.wisc.edu/htcondor/manual/v8.4/ref.html
• https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki

Obtain an Amazon Web Services Account

In order to use condor_annex, you must already have an AWS account. You may establish an AWS account under the UC-wide agreement by following the instructions provided by Blink:

• https://blink.ucsd.edu/technology/cloud/aws/

Obtain Your AWS Account Credentials

condor_annex issues programmatic requests to AWS services via the AWS command-line interface (CLI). In order to issues these requests, the AWS CLI must sign them using your AWS account credentials. These credentials consist of an Access Key ID and a Secret Access Key. If you do not have these access keys, you may create them using the AWS Management Console. AWS recommends that you use Identity and Access Management (IAM) access keys instead of your root account access keys.

To create access keys, you must have permissions to perform the required IAM actions.

1. Open the IAM console.
2. In the navigation pane, choose Users.
3. If you do not already have an IAM username, then select Create New Users. Each new user is issued Security Credentials when their IAM username is created.
4. If you already have an IAM username, then choose your IAM username (not the check box).
5. Next, select the Security Credentials tab and then choose Create Access Key.
6. To see your access key, choose Show User Security Credentials. Your credentials will look something like this:
   • Access Key ID: AKIAIOSFODNN7EXAMPLE?
   • Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
7. Choose Download Credentials, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

If you need more information about AWS Security Credentials, please consult the AWS documentation at:

• http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html

Select an Amazon Region for Your Annex

Generate an Amazon EC2 Key Pair

Create an condor_annex-compatible Amazon Machine Image (AMI)

Configure your HTCondor Pool to Use a Password Authentication

Install and configure AWS CLI on HTCondor submit node.

Install and configure condor annex on HTCondor submit node.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

1. Login to the FRONTEND_HOST via ssh as the FRONTEND_USER. NOTE it is important to log in with -A, this assumes you already have your personal public key access to the BOSCO_HOST. The bosco_cluster --add command will use this login to copy the bosco credentials over to the node.

    [user@client ~]$ ssh -A FRONTEND_USER@FRONTEND_HOST

2. Download the BOSCO installer tarball in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ wget ftp://ftp.cs.wisc.edu/condor/bosco/1.2/boscoinstaller.tar.gz 

3. Unzip and untar the BOSCO installer in the FRONTEND_USER home directory.

    [FRONTEND_USER@FRONTEND_HOST ~]$ tar -xzf boscoinstaller.tar.gz 

4. Run the boscoinstaller script to install BOSCO on the FRONTEND_HOST.

   [FRONTEND_USER@FRONTEND_HOST ~]$ python boscoinstaller 

5. Generate a passwordless rsa key, just press enter twice with no password when it prompts for one. Note it is important to name the key bosco_key.rsa:

    [FRONTEND_USER@FRONTEND_HOST ~]$ ssh-keygen -t rsa -f ~/.ssh/bosco_key.rsa

-- MartinKandes - 2016/11/01