UCSD Condor Group Configuration

Contents

Introduction

This document is meant to describe the condor configuration at UCSD CMS T2 as it pertains to condor groups. A full description of groups and quotas in condor is out of the scope of this document. Please see the current condor documentation for your condor version for more detailed information. There are some links to the condor 6.8 documentation in the reference section below.

Credit goes to Frank Wuerthwein and Igo Sfiligoi who put together the initial group configuration for UCSD. Terrence Martin deployed, adapted and maintains the current group configuration at UCSD.

Glossary

• Job slot or slot refers to an individual vm on a condor worker node.
• Node refers to an individual worker node which may contain one or more vm or job slots.
• UCSD is used to refer to the University of California at San Diego Physics Department Particle Physics Compact Muon Solenoid Project Tier 2 Center. It does not imply the University of California at San Diego as a whole.

Why Condor Groups?

Condor groups are a feature of condor in use at UCSD (version 6.7.18 as of this writing) that allows you to assign users to groups to provide a more flexible priority system that allows you to treat collections of users as a single unit when it comes to job priority. Condor groups also allow the groups to be given quotas which is a very useful feature for helping to ensure that certain groups are a minimum set of resources at a site based on that sites policy.

UCSD decided to implement quotas to solve the issue of poor sharing between VOs that use a single account for access to the cluster versus those that use an account per VO user. For example CDF always authenticates to UCSD using a single userid called cdf while CMS users are each assigned a userid/username. This resulted in all CDF users having to compete on an equal basis with each CMS user resulting in CDF always coming out on the bottom against any given CMS user since CDF as a whole used many more resources than even a busy single CMS user. Unfortunately the standard priorites and factors were not seen to be effective in trying balance out this inequity.

In addition to balancing priorities it was recognized that it would be desirable to be able to provide a quota for each group that submits jobs and ensure that if a VO owns the resources at UCSD that they will get some reasonable portion of all resources at UCSD based on their contribution.

Groups also allow UCSD to give preferential treatment to VOs for which we have made arrangements for access to resources of short, medium or long terms.

What are Condor Group Quotas?

Unlike some other quota systems group quotas in condor does not necessarily require that you limit the maximum number of nodes that a group can have access to. If you do want that type of maximum slot quota per VO that can be done via groups as well, or as some sites do through the condor START expressions or RANK expressions.

Condor groups quotas act instead like a minimum for the number of job slots for which a VO like CDF or CMS will get preferrential access regardless of their current individual group or user priority or queue position.

For example say you have 200 job slots in your condor queue. You assign 50 job slots to CDF and 50 to CMS groups with the remaining 100 slots unassigned to any group (condor treats this as group none). In the job queue you could have 75 CDF jobs, 25 CMS jobs and 200 ATLAS jobs. Regardless of the existing priority of each group CDF will at least get 50 slots, CMS will get their 25. For the remaining 125 slots, including the 25 CMS is not using Atlas and CDF will compete for those slots based on their existing condor priority.

Now if CMS were to schedule an additional 50 jobs to the cluster what would happen is that as soon as any slot became free CMS, regardless of position in the queue or their existing priority would be given that slot. CMS would continue be given all the slots that become free regardless of the CMS condor priority until they have used up their 50 slot quota. After they have gotten their 50 slots they compete with CDF and Atlas based on condor priority for any slots that are not required to fill another groups quota.

How do Condor Groups Compete with Each Other for Slots if their Own Quotas are Not Filled?

The case may occur where two or more condor groups have not yet been assigned their defined quota of job slots. For example say you have 10 slots free and 50 idle CMS and 50 idle CDF jobs. Since there are not enough free slots to fill CMS and CDFs 50 slot quotas CDF and CMS compete based on their groups priority just as if they were two users competing over those same 10 slots if there were no quotas. Of course groups without quotas, groups that have filled their quota or non grouped users are not eligible for any slots until both CDF and CMS have filled their quota in this example.

How Do Members of a Condor Group Compete with Each Other if their Quota is Filled?

In many respects condor groups act just like users when it comes to priorities after they have gotten their quota. The difference is that they compete with other groups and users not in their group as a group with their group priority rather than as individuals. There are some additional features with groups like default priorities and default priority factors that are described later that can alter how competitive a particular group or groups as a whole are versus non-grouped users.

How Do Users Compete Withing a Group?

Within a group users compete based on their own individual priority.

How Do I get Condor Groups and Quotas at my Site?

It is fairly straight forward to enable condor groups and quotas. There are two parts to configuring groups with an OSG CE. The first is the condor configuration on the negotiator or central manager. The second is configuring the condor jobmanager to assign jobs to groups based on the owner of the submitted job. This assignment is done by adding the parameter +AccountingGroup with the correct group string to the jobmanagers condor submit script.

Condor Group Configuration

The condor group configuration should be placed in the configuration file for your central manager (Collector/Negotiator) if you keep that on a seperate system from your Schedds.

Defining the Groups

# Group settings
GROUP_NAMES = group_cms, group_cdf, group_cmsprod, group_lcgadmin

Assigning the Group Quotas

GROUP_QUOTA_group_cms = 90
GROUP_QUOTA_group_cmsprod = 90
GROUP_QUOTA_group_cdf = 100
GROUP_QUOTA_group_lcgadmin = 1
GROUP_AUTOREGROUP = True

(Optional) Group Default Priority

As an option you can configure group priority factors to be different than the default priority factor when they exceed their quota.

GROUP_PRIO_FACTOR_group_cms = 10
GROUP_PRIO_FACTOR_group_cmsprod = 10
GROUP_PRIO_FACTOR_group_cdf = 10
GROUP_PRIO_FACTOR_group_lcgadmin = 10

Complete Group Configuration

Here is the complete group configuration we use. We also set some overall default priorities and factors to allow us to adjust priorities for our grouped users versus non-grouped users.

# Group settings
GROUP_NAMES = group_cms, group_cdf, group_cmsprod, group_lcgadmin

GROUP_QUOTA_group_cms = 90
GROUP_QUOTA_group_cmsprod = 90
GROUP_QUOTA_group_cdf = 100
GROUP_QUOTA_group_lcgadmin = 1
GROUP_AUTOREGROUP = True

GROUP_PRIO_FACTOR_group_cms = 10
GROUP_PRIO_FACTOR_group_cmsprod = 10
GROUP_PRIO_FACTOR_group_cdf = 10
GROUP_PRIO_FACTOR_group_lcgadmin = 10

DEFAULT_PRIO = 100
REMOTE_PRIO_FACTOR = 1000

Condor Jobmanager Modifications

UCSD uses pattern matching in the condor.pm jobmanager. This is effective at UCSD because of how we handle assigment of usernames for OSG users. Each VO that is mapped as a single user is given a username that matches their VO name. For VOs like CMS that allow a userid mapping for each CMS user we use a distinctive pattern in the username that we can match. eg. uscms1546 is a uscms user and is a cms user.

If you cannot easily determine the VO based on the userid you will have to implement some other solution for determining which group each job should belong to. If you come up with another solution for your own sites configuration I would be happy to add it to this document.

The edits below should be made to condor.pm which can be found in $VDT_LOCATION/globus/lib/perl/Globus/GRAM/JobManager/condor.pm

Jobmanager Pattern Match Base on Username

This pattern match should be placed before the condor.pm script starts to write the condor submit script.

    map {
        if ($_->[0] eq "LOGNAME") {$logname =  $_->[1]; }
    } @environment;

    if ($logname =~ /.*cms.*/) { $AccountingGroup = "group_cms." . $logname; }
    if ($logname =~ /.*lcgadmin.*/) { $AccountingGroup = "group_lcgadmin." . $logname; }
    if ($logname =~ /.*cdf.*/) { $AccountingGroup = "group_cdf." . $logname; }
    if ($logname =~ /.*caf.*/) { $AccountingGroup = "group_cdf." . $logname; }

Telling Condor the Group for the Current Job

The following fragment should be placed somewhere in the middle of the condor submit script output section of condor.pm

    print SCRIPT_FILE "+AccountingGroup = \"$AccountingGroup\"\n";

What the Jobmanager Output Should Look Like

For a CMS user

Log = /osglocal/osgcore/globus/tmp/gram_job_state/gram_condor_log.1733.1158777890
log_xml = True
+AccountingGroup = "group_cms.uscms1197"
should_transfer_files = YES
when_to_transfer_output = ON_EXIT

For a CDF user

Log = /osglocal/osgcore/globus/tmp/gram_job_state/gram_condor_log.7887.1158708975
log_xml = True
+AccountingGroup = "group_cdf.cdf"
should_transfer_files = YES
when_to_transfer_output = ON_EXIT

For a user not in a group

Log = /osglocal/osgcore/globus/tmp/gram_job_state/gram_condor_log.18371.1158776918
log_xml = True
+AccountingGroup = ""
should_transfer_files = YES
when_to_transfer_output = ON_EXIT

Final Comments

Short Jobs

In the abscence of an aggressive pre-emption mechanism where lower priority VOs are kicked if a high priority VO job is queued some mechanism has to be used to ensure that low priority jobs do not dominate the cluster. The approach UCSD has taken is to limit the overall time a job can execute to around 48 hours and to set their site policy accordingly. In addition to providing the churn to keep the condor priority system putting the right jobs on the cluster shorter jobs are also less risky for users as the price of a failed or pre-empted job is much less.

No Max Quotas at UCSD

At UCSD our overall goal is to fill the cluster with whoever is asking for it if the slots are available for use. Maximum quotas work against this goal due to their inflexibility and potential waste of resources. For example if you assign 50 of 150 total jobs slots only for CDF you are making good use of your cluster only if the other 100 nodes are used by other VOs. If however you have 100 idle nodes and 100 idle CDF jobs waiting for their 50 slots you are wasting a lot of potential CPU time as there is no reason CDF should not get to run on those idle nodes if they are not otherwise being used.

Maximum quotas are suitable for systems like disk where you are attempting to prevent the entire storage system from being overwhelmed by a single user. However in a cluster it is perfectly fine for a single user to consume 100% of the resources for a short period of time, say 12 hours. The cost for that user is that their ability to compete with others will degrade when they go to submit more work to the cluster. The dynamic priority system takes care of making sure that over time no one VO or user can dominate the cluster. Condor groups allow a site to ensure that individual VOs compete with each other as a group, and the condor group quotas provide the flexible mechanism to give certain cluster users an advantage when it is time to hand out resources.

Contributors to the UCSD T2 Condor Group configuration and Implementation

• Terrence Martin
• Igor Sfiligoi
• Frank Wuerthwein

References

• Group Accounting in Condor 6.8: Group Accounting
• Condor Negotiatior Configuration Parameters: condor_ negotiator Configuration File Entries

Authors

-- TerrenceMartin - 20 Sep 2006