Banning Users

We now have a tool to parse this log, courtesy of I. Sfiligoi. To use the tool (currently only installed on submit-4):

To query a particular user:

condor_userlog -rotated -const 'Owner=="uscms2330"' -fullname -attr Owner,JOB_GLIDEIN_Site /opt/glidecondor/condor_local/log/EventLog
Job      Host            Start Time  Evict Time  Wall Time Good Time CPU Usage
31869.4  uscms2330,IFCA   8/29 00:20  8/29 00:21   0+00:01   0+00:01   0+00:00
31869.8  uscms2330,IFCA   8/29 00:20  8/29 00:21   0+00:01   0+00:01   0+00:00
31869.7  uscms2330,IFCA   8/29 00:20  8/29 00:21   0+00:01   0+00:01   0+00:00
31869.5  uscms2330,IFCA   8/29 00:20  8/29 00:21   0+00:01   0+00:01   0+00:00
31869.6  uscms2330,IFCA   8/29 00:20  8/29 00:21   0+00:01   0+00:01   0+00:00
...

• Notify sites where jobs ran. (Note that individual jobs could have run on more than one site! The new EventLog gives you this information, since it tracks condor events, not clusters.)
• Report the results to CMS Security Contacts (Ian and Mine, and the cms-comp-security mailing list).

Compromised Pilot Certificate

• Remove the particular pilot proxy from the rotation in the glideinWMS frontend and replace it with another of the 50 we have available. (N.B. As of Wednesday August 29, 2012 the additional proxies are not yet registered with the CMS VO.)
• Ask Factory Ops to kill any running pilots with the banned proxy and remove any queued pilots.
• Ban the compromised pilot DN at on the condor collector

• Notify the sites where jobs ran with pilots with a compromised credential (effectively all sites).
• Report the results to CMS Security Contacts (Ian and Mine, and the cms-comp-security mailing list)

• which sites jobs from the banned user ran on
• names of pilots which ran jobs from the banned user
• incoming IP address from which jobs were submitted

• ComponentLog says that there was e.g. a request to submit a new task.
• FrontendLog says that IP n connected at time t.

FrontendLog is written by $CRABSERVER_ROOT/src/python/CommandManager/server_side/server2.c. S.B. looked a bit if it was obvious how to change to add the task name to the IP connection message (user's DN does not seem there, but task name would do), but it looks too complicted for understanding in a short time and we should not make very extensive changes to CRAB2 at this time.

• find out which sites pilot jobs ran on using this proxy (above) and notify them

General Observation

Note that if a compromise is thought to spread from pilot to user DN and vice-versa, the entire system could be considered compromised on short order, given that user tasks have of order O(1000) jobs and there are only 10 pilot proxies. The probability that any task of 1000 jobs that have already run or started avoided a particular pilot proxy is very very small (1.7 x 10^-46).

To consider (Igor, Stefano, Lola, James):

• Step to renew all the pilot proxies?
• Are the pilot credentials themselves compromised or just the proxy hijacked (simpler situation)?

• incoming IP address from which jobs were submitted (Do we have this information in the CRAB SERVER, and if so, where?)

letts@submit-4 /opt/glidecondor/condor_local/log$ tail -100 /opt/glidecondor/condor_local/log/EventLog |egrep '^Cluster|^JOB_Site'
Cluster = 31675
JOB_Site = "CERN"
Cluster = 31675
JOB_Site = "CERN"
Cluster = 31607
JOB_Site = "JINR"

IP address from which jobs were submitted? - STEFANO

• Report the results to CMS Security Contacts (Ian and Mine)

• Report the results to CMS Security Contacts (Ian and Mine)

1. Implement condor logging level changes and document how the information should be used or extracted. (James/Igor?) - IGOR AGREED TO DO IT. DONE on submit-4.

1. Who do we report incidents to? Oli? Ian? Any designated CMS Security contact person? Sites/users? - ASKED OLI: Ans. Ian and Mine.

condor_q -format '%s ' Owner -format '%s\n' x509userproxysubject | sort | uniq -c

The local UNIX userid typically has the form uscmsxxx.

• Remove the particular pilot proxy from the rotation in the glideinWMS frontend and replace it with another of the 50 we have available.
• Kill any running pilots with the banned proxy
• BAN COMPROMISED PILOT DN AT THE COLLECTOR
• Contact Factory Ops to kill any queued pilots

1. Document to be REVIEWED BY IGOR - DONE

1. Implement condor logging level changes and document how the information should be used or extracted. (James/Igor?) - IGOR AGREED TO DO IT.

1. Who do we report incidents to? Oli? Ian? Any designated CMS Security contact person? Sites/users? - ASKED OLI

PROCEDURES FOR GLIDEINWMS CRAB SERVER DURING THE CMS SECURITY CHALLENGE SSC6

Banning Users

During the CMS Security Challenge, glideinWMS CRAB SERVER operators may be asked to ban a particular DN and provide certain information about the "attack". In particular, given a particular user DN, admins may be asked to take action to:

Initial Actions

• hold any running jobs
• hold any queued jobs
• block the user from further submissions

Detailed Procedures

The procedure for banning a user starts with mapping the certificate DN to a local userid on the UCSD CRAB SERVERS. This can be done by looking at the list of mappings. The local UNIX userid typically has the form uscmsxxx.

On each of the submitter nodes (glidein-2.t2.ucsd.edu, submit-[1-4].t2.ucsd.edu), HOLD any pending or running jobs from this user by running:

condor_hold uscmsxxx

Remove the local userid from the /etc/passwd file on all submitter nodes. This will block any further submissions.

Collecting Information

Information an operator should collect:

• find out which sites jobs ran on
• incoming IP address from which jobs were submitted (do we even have this information in the CRAB SERVER, and if so, where?)

Detailed Procedures

Information on which sites jobs ran at is (or soon will be) in the condor logs on the submission nodes. NEED DETAILS, AND TO IMPLEMENT IGOR'S CONFIG MODIFICATIONS

Other actions based on information collected:

• Notify sites where jobs ran. Note that individual jobs could have run on more than one site!
• Report the results to CMS Computing Management?

Compromised Pilot Certificate

The compromise of a pilot certificate is much more complicated than the case of a compromised user certificate, since there are only O(10) pilot certificates which are cycled round-robin to run glideinWMS pilots. User jobs will then connect to startd's run by these pilots for executing the user jobs. If a pilot certificate is compromised, then potentially every site and every user of glideinWMS for CMS analysis during the time since the compromise can be affected.

Initial Actions

If a glideinWMS pilot DN is compromised, admins will have to:

• remove the particular pilot proxy from the rotation in the glideinWMS frontend and replace it with another of the 50 we have available.
• kill any running pilots with the banned proxy

Detailed Procedures

There are two frontend instances running on glidein-frontend.t2.ucsd.edu under user frontend, instance_v5_4 for general usage and instance_o5_4 for xrootd overflow. These procedures could apply to either frontend.

Pilot certificates are removed from the configuration file frontend.xml in the CMS frontend, in ~/frontstage/instance_[ov]5_4.cfg in the section under security. For example, there is a list of pilot certificates used:

         <security>
            <proxies>
               <proxy absfname="/home/frontend/.globus/x509_pilot05_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot06_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot07_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot08_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot09_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot10_cms_prio.proxy" security_class="cmsprio"/>
            </proxies>
         </security>

Remove the compromised proxy from the list and replace it with another that is not being used already in this frontend or in any other running frontend on the machine. Other certificates can be found in ~/.globus.

Reconfigure the frontend:

./frontend_startup reconfig ../instance_v5_4.cfg/frontend.xml

how to kill all pilots with DN=X

Collecting Information

• find out which sites pilot jobs ran on using this proxy and notify them
• find out which users had jobs which ran on pilots with a compromised proxy

Detailed Procedures

Given the large number of pilots running at any given time O(10000) and the small number of proxies O(10), every site and every user who ran a job in the glideinWMS analysis system since the time of the compromise of a pilot certificate will have been affected. To make this point, look at every site where pilots are currently running using one certificate:

letts@submit-4 ~$ condor_status -const '(GLIDEIN_X509_GRIDMAP_DNS=?="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu,/DC=org/DC=doegrids/OU=Services/CN=glidein-frontend.t2.ucsd.edu,/DC=org/DC=doegrids/OU=Services/CN=uscmspilot05/glidein-1.t2.ucsd.edu")' -l | grep ^GLIDEIN_CMSSite | sort | uniq -c
     20 GLIDEIN_CMSSite = "T1_CH_CERN"
      3 GLIDEIN_CMSSite = "T1_US_FNAL"
      5 GLIDEIN_CMSSite = "T2_BE_IIHE"
     36 GLIDEIN_CMSSite = "T2_BE_UCL"
      5 GLIDEIN_CMSSite = "T2_BR_SPRACE"
      4 GLIDEIN_CMSSite = "T2_BR_UERJ"
     11 GLIDEIN_CMSSite = "T2_CH_CERN"
      2 GLIDEIN_CMSSite = "T2_CH_CSCS"
     39 GLIDEIN_CMSSite = "T2_DE_DESY"
      6 GLIDEIN_CMSSite = "T2_DE_RWTH"
      3 GLIDEIN_CMSSite = "T2_ES_IFCA"
     30 GLIDEIN_CMSSite = "T2_FR_GRIF_LLR"
      4 GLIDEIN_CMSSite = "T2_HU_Budapest"
     37 GLIDEIN_CMSSite = "T2_IT_Bari"
     67 GLIDEIN_CMSSite = "T2_IT_Legnaro"
      9 GLIDEIN_CMSSite = "T2_IT_Pisa"
     18 GLIDEIN_CMSSite = "T2_RU_JINR"
      2 GLIDEIN_CMSSite = "T2_UA_KIPT"
      8 GLIDEIN_CMSSite = "T2_UK_London_Brunel"
      7 GLIDEIN_CMSSite = "T2_UK_London_IC"
     18 GLIDEIN_CMSSite = "T2_UK_SGrid_RALPP"
      1 GLIDEIN_CMSSite = "T2_US_Caltech"
    130 GLIDEIN_CMSSite = "T2_US_Florida"
     13 GLIDEIN_CMSSite = "T2_US_MIT"
     26 GLIDEIN_CMSSite = "T2_US_Nebraska"
      3 GLIDEIN_CMSSite = "T2_US_Purdue"
     41 GLIDEIN_CMSSite = "T2_US_UCSD"
     26 GLIDEIN_CMSSite = "T2_US_Wisconsin"
     64 GLIDEIN_CMSSite = "T3_US_Colorado"
     50 GLIDEIN_CMSSite = "T3_US_Omaha"
     10 GLIDEIN_CMSSite = "T3_US_OSU"
      1 GLIDEIN_CMSSite = "T3_US_TTU"
      7 GLIDEIN_CMSSite = "T3_US_UMD"

WRITE HOW TO GET THIS LIST OF SITES, USERS.

Other Actions

• Notify the sites and users whose jobs ran with pilots with a compromised credential
• Report the results to CMS Computing Management?

Action Items

1. Document to be REVIEWED BY IGOR
2. Does CRAB log IP addresses where submissions come from? (Stefano)
3. Implement condor logging level changes and document how the information should be used or extracted. (James/Igor?)
4. Document how to get a list of sites and users that ran on a pilot with a particular pilot certificate since a particular time (JAMES) - probably a complicated looking condor command.
5. Who do we report incidents to? Oli? Ian? Any designated CMS Security contact person? Sites/users?

-- JamesLetts - 2012/08/27