PROCEDURES FOR GLIDEINWMS CRAB SERVER DURING THE CMS SECURITY CHALLENGE SSC6

Banning Users

During the CMS Security Challenge, glideinWMS CRAB SERVER operators may be asked to ban a particular DN and provide certain information about the "attack". In particular, given a particular user DN, admins may be asked to take action to:

Initial Actions

  • hold any running jobs
  • hold any queued jobs
  • block the user from further submissions

Detailed Procedures

The procedure for banning a user starts with mapping the certificate DN to a local userid on the UCSD CRAB Servers. This can be done by looking at the list of mappings. The command condor_q can also give you the same information, but only if the user still has jobs pending or running. The local UNIX userid typically has the form uscmsxxx.

condor_q -format '%s ' Owner -format '%s\n' x509userproxysubject | sort | uniq -c

On each of the submitter nodes (glidein-2.t2.ucsd.edu, submit-[1-4].t2.ucsd.edu), HOLD any pending or running jobs from this user by running:

condor_hold uscmsxxx

Remove the local userid from the /etc/passwd file on all submitter nodes. This will block any further submissions.

Collecting Information

Information an operator should collect:

  • which sites jobs from the banned user ran on
  • names of pilots which ran jobs from the banned user
  • incoming IP address from which jobs were submitted

Detailed Procedures

Information on which sites jobs ran at is (or soon will be) in the condor logs on the submission nodes in the file /opt/glidecondor/condor_local/log/EventLog. Eventually we will have a tool to parse this log, but the information is available in terms of condor Cluster ID (the same information you get from condor_q or condor_history) and JOB_Site:

letts@submit-4 /opt/glidecondor/condor_local/log$ tail -100 /opt/glidecondor/condor_local/log/EventLog |egrep '^Cluster|^JOB_Site'
Cluster = 31675
JOB_Site = "CERN"
Cluster = 31675
JOB_Site = "CERN"
Cluster = 31607
JOB_Site = "JINR"
Pilot names are also available in the EventLog. From this information it should be possible to determine which other jobs ran on the same pilots that may have been compromised, if any.

IP address from which jobs were submitted are more difficult to determine. In principle, this info is in two logs in $PRODAGENT_WORKDIR/CommandManager

  • ComponentLog says that there was e.g. a request to submit a new task.
  • FrontendLog says that IP n connected at time t.
However, there is no guaranteed relationship.

FrontendLog is written by $CRABSERVER_ROOT/src/python/CommandManager/server_side/server2.c. S.B. looked a bit if it was obvious how to change to add the task name to the IP connection message (user's DN does not seem there, but task name would do), but it looks too complicted for understanding in a short time and we should not make very extensive changes to CRAB2 at this time.

Other actions based on information collected:

  • Notify sites where jobs ran. Note that individual jobs could have run on more than one site!
  • Report the results to CMS Security Contacts (Ian and Mine)

Compromised Pilot Certificate

The compromise of a pilot certificate is much more complicated than the case of a compromised user certificate, since there are only O(10) pilot certificates which are cycled round-robin to run glideinWMS pilots. User jobs will then connect to startd's run by these pilots for executing the user jobs. If a pilot certificate is compromised, then potentially every site and every user of glideinWMS for CMS analysis during the time since the compromise can be affected.

How do you know a pilot proxy was compromised? GOOD QUESTION!

Initial Actions

If a glideinWMS pilot DN is compromised, admins will have to:

  • Remove the particular pilot proxy from the rotation in the glideinWMS frontend and replace it with another of the 50 we have available.
  • Kill any running pilots with the banned proxy
  • BAN COMPROMISED PILOT DN AT THE COLLECTOR
  • Contact Factory Ops to kill any queued pilots

Detailed Procedures

There are two frontend instances running on glidein-frontend.t2.ucsd.edu under user frontend, instance_v5_4 for general usage and instance_o5_4 for xrootd overflow. These procedures could apply to either frontend.

Pilot certificates are removed from the configuration file frontend.xml in the CMS frontend, in ~/frontstage/instance_[ov]5_4.cfg in the section under security. For example, there is a list of pilot certificates used:

         <security>
            <proxies>
               <proxy absfname="/home/frontend/.globus/x509_pilot05_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot06_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot07_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot08_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot09_cms_prio.proxy" security_class="cmsprio"/>
               <proxy absfname="/home/frontend/.globus/x509_pilot10_cms_prio.proxy" security_class="cmsprio"/>
            </proxies>
         </security>

Remove the compromised proxy from the list and replace it with another that is not being used already in this frontend or in any other running frontend on the machine. Other certificates can be found in ~/.globus.

Reconfigure the frontend:

./frontend_startup reconfig ../instance_v5_4.cfg/frontend.xml

To remove all running and queued pilots with a particular DN, it is necessary to contact the Factory Operators (osg-gfactory-support@physics.ucsd.edu). Also ask them for a history of pilots that ran at sites with that DN since the time of the incident (pilot name of the form "glidein_15640@node20-9.wn.iihe.ac.be", site).

Collecting Information

  • find out which sites pilot jobs ran on using this proxy (above) and notify them
  • find out which users had jobs which ran on pilots with a compromised proxy

Detailed Procedures

Given the large number of pilots running at any given time O(10000) and the small number of proxies O(10), every site and every user who ran a job in the glideinWMS analysis system since the time of the compromise of a pilot certificate will have been affected. To make this point, look at every site where pilots are currently running using one certificate:

letts@submit-4 ~$ condor_status -const '(GLIDEIN_X509_GRIDMAP_DNS=?="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu,/DC=org/DC=doegrids/OU=Services/CN=glidein-frontend.t2.ucsd.edu,/DC=org/DC=doegrids/OU=Services/CN=uscmspilot05/glidein-1.t2.ucsd.edu")' -l | grep ^GLIDEIN_CMSSite | sort | uniq -c
     20 GLIDEIN_CMSSite = "T1_CH_CERN"
      3 GLIDEIN_CMSSite = "T1_US_FNAL"
      5 GLIDEIN_CMSSite = "T2_BE_IIHE"
     36 GLIDEIN_CMSSite = "T2_BE_UCL"
      5 GLIDEIN_CMSSite = "T2_BR_SPRACE"
      4 GLIDEIN_CMSSite = "T2_BR_UERJ"
     11 GLIDEIN_CMSSite = "T2_CH_CERN"
      2 GLIDEIN_CMSSite = "T2_CH_CSCS"
     39 GLIDEIN_CMSSite = "T2_DE_DESY"
      6 GLIDEIN_CMSSite = "T2_DE_RWTH"
      3 GLIDEIN_CMSSite = "T2_ES_IFCA"
     30 GLIDEIN_CMSSite = "T2_FR_GRIF_LLR"
      4 GLIDEIN_CMSSite = "T2_HU_Budapest"
     37 GLIDEIN_CMSSite = "T2_IT_Bari"
     67 GLIDEIN_CMSSite = "T2_IT_Legnaro"
      9 GLIDEIN_CMSSite = "T2_IT_Pisa"
     18 GLIDEIN_CMSSite = "T2_RU_JINR"
      2 GLIDEIN_CMSSite = "T2_UA_KIPT"
      8 GLIDEIN_CMSSite = "T2_UK_London_Brunel"
      7 GLIDEIN_CMSSite = "T2_UK_London_IC"
     18 GLIDEIN_CMSSite = "T2_UK_SGrid_RALPP"
      1 GLIDEIN_CMSSite = "T2_US_Caltech"
    130 GLIDEIN_CMSSite = "T2_US_Florida"
     13 GLIDEIN_CMSSite = "T2_US_MIT"
     26 GLIDEIN_CMSSite = "T2_US_Nebraska"
      3 GLIDEIN_CMSSite = "T2_US_Purdue"
     41 GLIDEIN_CMSSite = "T2_US_UCSD"
     26 GLIDEIN_CMSSite = "T2_US_Wisconsin"
     64 GLIDEIN_CMSSite = "T3_US_Colorado"
     50 GLIDEIN_CMSSite = "T3_US_Omaha"
     10 GLIDEIN_CMSSite = "T3_US_OSU"
      1 GLIDEIN_CMSSite = "T3_US_TTU"
      7 GLIDEIN_CMSSite = "T3_US_UMD"
This is 33 out of 39 sites running glideins at this time.

To get detailed information about pilot history, ask Factory Ops for a history of pilots that ran at sites with that DN since the time of the incident (pilot name of the form "glidein_15640@node20-9.wn.iihe.ac.be", site). Then you can cross-reference the list of pilots against the condor_history (potentially a lot of work).

Other Actions

  • Notify the sites and users whose jobs ran with pilots with a compromised credential
  • Report the results to CMS Security Contacts (Ian and Mine)

General Observation

Note that if a compromise is thought to spread from pilot to user DN and vice-versa, the entire system could be considered compromised on short order, given that user tasks have of order O(1000) jobs and there are only 10 pilot proxies. The probability that any task of 1000 jobs that have already run or started avoided a particular pilot proxy is very very small (1.7 x 10^-46).

To consider (Igor, Stefano, Lola, James):

  • Step to renew all the pilot proxies?
  • Are the pilot credentials themselves compromised or just the proxy hijacked (simpler situation)?

-- JamesLetts - 2012/08/27

Edit | Attach | Print version | History: r7 < r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2012/08/28 - 23:29:23 - JamesLetts
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback