The script GlideinFactoryStatus.py was written to analyze the data in each client's Status_Attributes.rrd file. It is currently located in /home/gfactory/cwm/GlideinFactoryStatus.py

Documentation

This script analyzes the most recent 6 hours worth of glidein data in various ways. To see the data in graphical form, visit the website http://glidein-1.t2.ucsd.edu:8319/glidefactory/monitor/glidein_Production_v3_1/factoryStatus.html.

LEGEND:
time = the amount of time during the past 6 hours that the site was above critical
avg = the average value of the quantity during the time it was past critical
max(min) = the maximum(minimum) value of the quntity during the time it was past critical
recent = the most recent local time the quantity was past critical

Usage: python GlideinFactoryStatus.py [option] [argument]

Options:
-h, --help return this message
-d enter the debugger
-o ..., --offset=... This option offsets the range of data to be analyzed by this many hours. The default is zero. Offset must be between 0 and (61 - 6 = 55) hours.

How it works

If an element of a client is above a certain critical number for more than a certain fraction of the time checked, then that element is flagged as supscious. The critical values are listed in theoutput.

typical output (the formatting looks much better on the command line stdout)

16 sites had a unmatched-to-running glidein ratio of greater than 0.5 for more than 0.2 of the time

site time avg max recent
CMS_T2_IT_Legnaro_ce04 5:15 2.56 4.85 08:40
CMS_T2_UK_SGrid_RALPP_hep206_Grid1000 4:00 2.00 3.23 09:55
CMS_T2_UK_SGrid_RALPP_hep207_Grid1000 3:00 4.35 7.69 09:50
CMS_T2_IN_TIFR 2:00 0.50 0.50 07:50
CMS_T2_IT_Legnaro_ce03 4:30 6.77 11.36 08:45
CMS_T2_UK_SGrid_RALPP_hep207_Grid2000 4:30 2.52 5.76 09:55
CMS_T2_UK_SGrid_RALPP_hep206_Grid2000 4:00 1.49 2.38 09:55
CMS_T2_ES_CIEMAT_lcg02 1:30 1.26 3.56 09:45
CMS_T2_AT_Vienna_lcgce 4:09 0.99 2.05 10:45
CMS_T2_FR_GRIF_IRFU_node07 6:00 7.29 25.70 10:45
CMS_T2_HU_Budapest_grid107 5:15 8.50 22.00 10:45
CMS_T2_CH_CSCS_ce02 1:50 0.58 0.70 09:20
CMS_T2_CH_CSCS_ce01 1:12 0.62 0.71 09:15
CMS_T2_RU_JINR_ce01 5:00 0.90 1.00 10:45
CMS_T2_RU_JINR_ce02 6:00 0.91 1.00 10:45
CMS_T2_IT_Legnaro_ce01 5:15 2.65 5.00 08:20

----------------

10 sites had a registered-to-running glidein ratio of less than 0.5 for more than 0.2 of the time

site time avg min recent
CMS_T2_US_Nebraska_Husker 6:00 0.01 0.01 10:50
CMS_T2_UK_SGrid_RALPP_hep206_Grid1000 2:00 0.05 0.05 10:00
CMS_T2_IT_Legnaro_ce02 6:00 0.28 0.03 08:00
CMS_T2_IT_Legnaro_ce03 1:30 0.11 0.11 08:50
CMS_T2_UK_SGrid_RALPP_hep206_Grid2000 2:00 0.04 0.04 10:00
CMS_T2_UK_SGrid_RALPP_hep207_Grid2000 1:30 0.01 0.01 10:00
CMS_T3_US_Omaha_ff 6:00 0.01 0.01 13:40
CMS_T2_AT_Vienna_lcgce 1:50 0.22 0.07 10:35
CMS_T2_IN_TIFR 4:00 0.08 0.02 08:00
CMS_T2_UK_SGrid_RALPP_hep207_Grid1000 3:00 0.25 0.02 10:00

----------------

7 sites had a claimed-to-running glidein ratio of less than 0.5 for more than 0.2 of the time

site time avg min recent
CMS_T2_US_Nebraska_Husker 6:00 0.01 0.01 10:50
CMS_T2_ES_IFCA_ce01 3:00 0.39 0.04 10:45
CMS_T2_ES_IFCA_ce02 3:00 0.38 0.14 10:40
CMS_T2_ES_CIEMAT_lcg02 2:10 0.36 0.29 10:45
CMS_T3_US_Omaha_ff 6:00 0.01 0.01 13:40
CMS_T2_CH_CSCS_ce02 2:40 0.40 0.27 09:25
CMS_T2_CH_CSCS_ce01 2:40 0.40 0.27 09:25