Difference: Condor_annex (12 vs. 13)

Revision 132016/11/21 - Main.MartinKandes

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Line: 30 to 30
 

Step 1: Install and Configure an HTCondor Pool

Changed:
<
<
If you do not already have your own HTCondor Pool, you may want to first start by installing your own personal HTCondor pool to experiment with condor_annex. Please consult the HTCondor Manual and/or Wiki for more information:

>
>
If you do not already have your own HTCondor Pool, you may want to first start by installing your own personal HTCondor pool to experiment with condor_annex. Please consult the HTCondor Manual and/or Wiki for more information.
 

Step 2: Obtain an Amazon Web Services Account

Changed:
<
<
In order to use condor_annex, you must already have an AWS account. You may establish an AWS account under the UC-wide agreement by following the instructions provided by Blink:

>
>
In order to use condor_annex, you must already have an AWS account. You may establish an AWS account under the UC-wide agreement by following the instructions provided by Blink.
 

Step 3: Obtain AWS Account Credentials

Line: 47 to 42
  To create access keys, you must have permissions to perform the required IAM actions.
Changed:
<
<
  1. Open the IAM console.
>
>
  1. Open the IAM console.
 
  1. In the navigation pane, choose Users.
  2. If you do not already have an IAM username, then select Add User. Each new user is issued credentials.
  3. If you already have an IAM username, then choose your IAM username (not the check box).
  4. Next, select the Security Credentials tab and then choose Create Access Key.
Changed:
<
<
  1. To see your access key, choose Show User Security Credentials. Your credentials will look something like this:
>
>
  1. Your credentials will look something like this:
 
    • Access Key ID: AKIAIOSFODNN7EXAMPLE?
    • Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Changed:
<
<
  1. Choose Download Credentials, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

Save your Access Key ID and Secret Access Key. You will need to provide them later when configuring the AWS CLI. If you need more information about AWS Security Credentials, please consult the AWS documentation at:

>
>
  1. Choose Download .csv file, and store the keys in a secure location. Your secret key will no longer be available through the AWS Management Console; you will have the only copy. Keep it confidential in order to protect your account, and never email it. Do not share it outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.
 
Changed:
<
<
>
>
Save your Access Key ID and Secret Access Key. You will need to provide them later when configuring the AWS CLI. If you need more information about AWS Security Credentials, please consult the AWS documentation.
 

Step 4: Select a Region for the Annex

Changed:
<
<
Amazon Elastic Compute Cloud (EC2) instances are hosted in multiple locations world-wide. These locations are composed of Regions and Availability Zones. Each Region is a separate geographic area. However, each Region also has multiple, isolated locations known as Availability Zones (AZs). However, not all AWS Regions are created equal. Each Region may offer only a subset of AWS services. You can find out what services are offered in each Region from the table provided here:

>
>
Amazon Elastic Compute Cloud (EC2) instances are hosted in multiple locations world-wide. These locations are composed of Regions and Availability Zones. Each Region is a separate geographic area. However, each Region also has multiple, isolated locations known as Availability Zones (AZs). However, not all AWS Regions are created equal. Each Region may offer only a subset of AWS services. You can find out what services are offered in each Region from the table provided here.
  When selecting a Region for your annex, you must select a region that offers all of the AWS services required by condor_annex to function properly. These services are:
Changed:
<
<
>
>
 
Changed:
<
<
AWS Lambda currently has the most limited deployment of any AWS service required by condor_annex. For example, AWS Lambda is only available in the following Regions within the United States at this time:
>
>
AWS Lambda currently has the most limited deployment of any AWS service required by condor_annex. For example, AWS Lambda is only available in the following Regions within the United States at this time:
 
  • Northern Virginia (us-east-1)
  • Ohio (us-east-2)
Line: 94 to 85
  To create your key pair using the Amazon EC2 console
Changed:
<
<
  1. Open the EC2 console at https://console.aws.amazon.com/ec2/.
>
>
  1. Open the EC2 console.
 
  1. In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
  2. Choose Create Key Pair.
  3. Enter a name for the new key pair in the Key pair name field of the Create Key Pair dialog box, and then choose Create.
  4. The private key file is automatically downloaded by your browser. The base file name is the name you specified as the name of your key pair, and the file name extension is .pem. Save the private key file in a safe place. This is the only chance for you to save the private key file. You'll need to provide the name of your key pair when you launch an instance and the corresponding private key each time you connect to the instance.
  5. Use the following command to set the permissions of your private key file so that only you can read it.
     [user@SUBMIT ~]$ chmod 400 my-key-pair.pem 
Changed:
<
<
If you would like to create your SSH key pair using the AWS CLI or import your own key pair, please consult the AWS documentation at:

>
>
If you would like to create your SSH key pair using the AWS CLI or import your own key pair, please consult the AWS documentation.
 

Step 6: Configure Default VPC Security Group

condor_annex will automatically create and configure an AWS Security Group (i.e., a virtual firewall) around all of the instances within an annex. However, depending on your HTCondor pool configuration, it may also been useful to place some on-demand resources in AWS. For example, you may want to a separate HTCondor central manager instance located in AWS in order to flock user jobs over to the annex instead of connecting the annex instances back to your local central manager.

Changed:
<
<
These on-demand resources may be placed in your AWS Region's default Virtual Private Cloud (VPC) Security Group. To configure the default VPC Security Group:
>
>
Any such on-demand resources may be placed in your AWS Region's default Virtual Private Cloud (VPC) Security Group. To configure the default VPC Security Group:
 
Changed:
<
<
  1. Open the VPC console at https://console.aws.amazon.com/vpc/.
>
>
  1. Open the VPC console.
 
  1. In the navigation pane, under Security, choose Security Groups.
  2. Select the Security Group in the list that has Group Name default and Description default VPC security group.
  3. Next, select the Inbound Rules tab and then click on the Edit button.
Line: 140 to 129
 
Custom UDP Rule UDP 9618 0.0.0.0/0
Custom TCP Rule TCP 9618 0.0.0.0/0
Changed:
<
<
Of course, you should try to restrict the size of the Source IP address space for these rules as much as possible. For example, you may want to limit them to inbound traffic from your home institution's public IP address space.
>
>
Of course, you should try to restrict the Source IP address space for these rules as much as possible. For example, you may want to limit them to inbound traffic from your home institution's public IP address space.
  By default, each Security Group, including the default VPC Security Group, allows ALL outbound traffic.
Line: 158 to 147
 
us-west-1 ami-7f06731f
us-west-2 ami-ac8890cd
Changed:
<
<
If these preconfigured AMIs cannot be successfully modified to suit your needs, you will need to create your own condor_annex-compatible AMI. We have do so for our own purposes by building a condor_annex-compatible CentOS? 6-based AMI.
>
>
If these preconfigured AMIs cannot be successfully modified to suit your needs, you will need to create your own condor_annex-compatible AMI. We have done so for our own purposes by building a condor_annex-compatible CentOS? 6-based AMI.
 
Changed:
<
<
To build your own condor_annex-compatible AMI, open the Elastic Compute Cloud (EC2) dashboard in the AWS Region where you will run your annex. Click on the Launch Instance button. This will open the instance launch configuration wizard. Follow these steps.
>
>
To build your own condor_annex-compatible AMI, open the Elastic Compute Cloud (EC2) dashboard in the Region where you will run your annex. Click on the Launch Instance button. This will open the instance launch configuration wizard. Follow these steps.
 
  1. Choose an Amazon Machine Image (AMI): We configured our annex's execute instances to use CentOS? 6. To find a suitable CentOS? 6 AMI to start from, select the AWS Marketplace tab and then enter "CentOS 6" in the search box. Your search will return multiple results. However, the most up-to-date AMI should be the first one in the list. Unless you have special requirements for your configuration, select this AMI by clicking on the Select button.
  2. Choose an Instance Type: Once you have selected an AMI, the launch configuration wizard will prompt you to select an instance type on which to build your condor_annex execute node. Choose one that suits your needs. Once you have selected your instance type, click on the Next: Configure Instance Details button.
Changed:
<
<
  1. Configure Instance Details: Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the Number of Instances at 1. Next, select one of your Network VPCs. In general, you should choose the default VPC whose Security Group was pre-configured in the previous step. Once you have determined which VPC will host this instance, select a specific Subnet in which to place it. All other networking options Auto-assign Public IP and Placement group may be left set to their default settings of Use subnet setting (Enabled) and No placement group, respectively. After configuring the networking details, if you would like to apply a specific AWS IAM role to the instance, then select an appropriate role for it. Otherwise, leave IAM role set to its default value of None. All other instance details may be configured with their default values. Once you have completed configuring your instance details, click on the Next: Add Storage button.
>
>
  1. Configure Instance Details: Only one instance is required to configure your condor_annex-compatible AMI. Therefore, you may leave the Number of Instances at 1. Next, select one of your Network VPCs. In general, you should choose the default VPC whose Security Group was pre-configured in the previous step. Once you have determined which VPC will host this instance, select a specific Subnet in which to place it. The other networking options Auto-assign Public IP and Placement group may be left set to their default settings of Use subnet setting (Enabled) and No placement group, respectively. After configuring the networking details, if you would like to apply a specific IAM role to the instance, then select an appropriate role for it. Otherwise, leave IAM role set to its default value of None. All other instance details may be configured with their default values. Once you have completed configuring your instance details, click on the Next: Add Storage button.
 
  1. Add Storage: In general, you will not have to modify the configuration of your root storage volume for the instance. However, the launch wizard may still default to a Magnetic volume type, even though the General Purpose SSD option is now becoming AWS' recommended default. Our instance launch wizard still defaults to Magnetic. As such, we changed our root Volume Type from an 8GiB Magnetic volume to an 8 GiB? General Purpose SSD volume and selected Delete on Termination. Once you have completed the configuration of your root volume, click on the Next: Tag Instance button.
  2. Tag Instance: Add a Name to your instance and then click on the Next: Configure Security Group button.
  3. Configure Security Group: Select an existing security group and choose your default VPC security group. Once you have selected a security group, click on the Review and Launch button.
Line: 246 to 235
  If you are using a different base OS AMI, please see this link for some possible changes to the CloudFormation? Helper Script configuration.
Changed:
<
<
Activate (or deactivate) the following services as indicated and then and then logout from the instance.
>
>
Activate (or deactivate) the following services as indicated and then logout from the instance.
 
 [root@ANNEX-PRIVATE-IP ~]$ chkconfig iptables off
 [root@ANNEX-PRIVATE-IP ~]$ service iptables stop

Line: 258 to 247
  [root@ANNEX-PRIVATE-IP ~]$ service condor start [root@ANNEX-PRIVATE-IP ~]$ exit
Changed:
<
<
Return now to the AWS Management Console in your web browser and then go to the EC2 dashboard. In the navigation pane, under INSTANCES, choose Instances. There you will see a list of each individual instance available in the AWS Region. Select the instance you've just configured your condor_annex-compatible AMI on. Then from the dropdown menu Actions, go to Image and select Create Image. You will be prompted to make changes to the AMI before its creation. You'll likely want to add an Image name and check the Delete on Termination box. Make any other adjustments you find necessary and then click on the Create Image button. This will create an AMI from your instance that can be used with condor_annex.
>
>
Now that you have prepared a condor_annex-compatible AMI on this instance, you'll need to save it for future use on other instances. To do so:
 
Changed:
<
<
Go ahead and Close the Create Image request received dialog box to return to the EC2 Dashboard. In the navigation pain, under IMAGES, click on AMIs. There you will see a list of the your custom AMIs, including the condor_annex-compatible AMI that was just created from your instance. Note the AMI ID for this image as it will be one of the required inputs when calling condor_annex.
>
>
  1. Return to your web browser and go to the EC2 console.
  2. In the navigation pane, under INSTANCES, choose Instances.
  3. There you will see a list of each individual instance available in the Region. Select the instance you've just configured your condor_annex-compatible AMI on.
  4. From the dropdown menu Actions, go to Image and select Create Image.
  5. You will be prompted to make changes to the AMI before its creation. You'll likely want to add an Image name and check the Delete on Termination box. Make any other adjustments you find necessary and then click on the Create Image button. This will create an AMI from your instance that can be used with condor_annex.
  6. Go ahead and Close the Create Image request received dialog box to return to the EC2 Dashboard.
  7. In the navigation pain, under IMAGES, click on AMIs. There you will see a list of the your custom AMIs, including the condor_annex-compatible AMI that was just created from your instance. Note the AMI ID for this image as it will be one of the required inputs when calling condor_annex.
 

Step 8: Configure HTCondor Pool for Password Authentication

Line: 274 to 269
 
 [root@CENTRAL_MANAGER ~]$ cd /etc/condor/config.d 
Changed:
<
<
In this directory, create the following HTCondor configuration file (99_condor_annex.config)
>
>
In this directory, create the following HTCondor configuration file (99_condor_annex_passwd.config)
 
 ALLOW_DAEMON = $(ALLOW_DAEMON), condor_pool@*

Changed:
<
<
SEC_DEFAULT_AUTHENTICATION = REQUIRED SEC_DEFAULT_AUTHENTICATION_METHODS = $(SEC_DEFAULT_AUTHENTICATION_METHODS), PASSWORD SEC_DEFAULT_ENCRYPTION = OPTIONAL SEC_DEFAULT_INTEGRITY = REQUIRED
>
>
SEC_CLIENT_AUTHENTICATION = REQUIRED SEC_CLIENT_AUTHENTICATION_METHODS = $(SEC_CLIENT_AUTHENTICATION_METHODS), PASSWORD SEC_CLIENT_ENCRYPTION = OPTIONAL SEC_CLIENT_INTEGRITY = REQUIRED SEC_DAEMON_AUTHENTICATION = REQUIRED SEC_DAEMON_AUTHENTICATION_METHODS = $(SEC_DAEMON_AUTHENTICATION_METHODS), PASSWORD SEC_DAEMON_ENCRYPTION = OPTIONAL SEC_DAEMON_INTEGRITY = REQUIRED SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED SEC_NEGOTIATOR_AUTHENTICATION_METHODS = $(SEC_NEGOTIATOR_AUTHENTICATION_METHODS), PASSWORD SEC_NEGOTIATOR_ENCRYPTION = OPTIONAL SEC_NEGOTIATOR_INTEGRITY = REQUIRED
  SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE SEC_PASSWORD_FILE = /etc/condor/condor_pool_password
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback