TWiki> UCSDTier2 Web>DaveEvansSTEP09Results (2009/06/22, FkW)EditAttach

STEP09 Results

Definition of STEP09 time period used 

# week one
#START_STEP09=`date -u -d "Jun 8 00:00:00 UTC 2009" +%s`
#END_STEP09=`date -u -d "Jun 14 23:59:59 UTC 2009" +%s`

# week two
START_STEP09=`date -u -d "Jun 15 00:00:00 UTC 2009" +%s`
END_STEP09=`date -u -d "Jun 21 23:59:59 UTC 2009" +%s`

Summary

Over all sites during both weeks: 

No jobs at: T2_FR_CCIN2P3 - OK
No jobs at: T2_RU_RRC_KI - OK
No jobs at: T2_UK_SGrid_Bristol - OK
No jobs at: T2_FI_HIP - OK
No jobs at: T2_PT_LIP_Coimbra - Dashboard claims 59 jobs
No jobs at: T2_RU_PNPI - OK
No jobs at: T2_TR_ULAKBIM - OK

TOTAL STATISTICS
undef:  0
0 :  187999
1 :  8
6 :  21
7 :  2
9 :  9
11 :  72
143 :  1
8001 :  748
8020 :  34426
50115 :  26
60307 :  35

allstep09 njobs_allstep09

week1 only 

TOTAL STATISTICS
undef:  0
0 :  65588
1 :  8
6 :  2
7 :  2
9 :  6
8001 :  365
8020 :  11299
50115 :  3

week1

week2 only 

TOTAL STATISTICS
undef:  0
0 :  122411
6 :  19
9 :  3
11 :  72
143 :  1
8001 :  383
8020 :  23127
50115 :  23
60307 :  35

week2

Details

Week 1 

99.5551318716
ALERT FAILED JOBS AT:  T2_AT_Vienna
88.137755102
ALERT FAILED JOBS AT:  T2_BE_UCL
100.0
Site is OK:  T2_BR_UERJ
100.0
Site is OK:  T2_CH_CSCS
99.8509687034
ALERT FAILED JOBS AT:  T2_DE_DESY
100.0
Site is OK:  T2_EE_Estonia
84.5041322314
ALERT FAILED JOBS AT:  T2_ES_IFCA
No jobs at:  T2_FR_CCIN2P3
26.7918088737
ALERT FAILED JOBS AT:  T2_FR_GRIF_IRFU
100.0
Site is OK:  T2_HU_Budapest
100.0
Site is OK:  T2_IT_Bari
100.0
Site is OK:  T2_IT_Pisa
99.8717948718
ALERT FAILED JOBS AT:  T2_KR_KNU
100.0
Site is OK:  T2_PL_Warsaw
100.0
Site is OK:  T2_PT_LIP_Lisbon
29.3203883495
ALERT FAILED JOBS AT:  T2_RU_INR
85.1656626506
ALERT FAILED JOBS AT:  T2_RU_JINR
No jobs at:  T2_RU_RRC_KI
84.9572649573
ALERT FAILED JOBS AT:  T2_TR_METU
100.0
Site is OK:  T2_TW_Taiwan
100.0
Site is OK:  T2_UK_London_Brunel
No jobs at:  T2_UK_SGrid_Bristol
76.7419276523
ALERT FAILED JOBS AT:  T2_US_Caltech
81.7528735632
ALERT FAILED JOBS AT:  T2_US_MIT
94.7012987013
ALERT FAILED JOBS AT:  T2_US_Purdue
99.6262317363
ALERT FAILED JOBS AT:  T2_US_Wisconsin
100.0
Site is OK:  T2_BE_IIHE
98.3114446529
ALERT FAILED JOBS AT:  T2_BR_SPRACE
100.0
Site is OK:  T2_CN_Beijing
100.0
Site is OK:  T2_DE_RWTH
100.0
Site is OK:  T2_ES_CIEMAT
No jobs at:  T2_FI_HIP
26.7918088737
ALERT FAILED JOBS AT:  T2_FR_GRIF_LLR
99.398249453
ALERT FAILED JOBS AT:  T2_FR_IPHC
28.2211324119
ALERT FAILED JOBS AT:  T2_IN_TIFR
100.0
Site is OK:  T2_IT_Legnaro
100.0
Site is OK:  T2_IT_Rome
No jobs at:  T2_PK_NCP
No jobs at:  T2_PT_LIP_Coimbra
No jobs at:  T2_RU_IHEP
100.0
Site is OK:  T2_RU_ITEP
No jobs at:  T2_RU_PNPI
100.0
Site is OK:  T2_RU_SINP
No jobs at:  T2_TR_ULAKBIM
21.3211249182
ALERT FAILED JOBS AT:  T2_UA_KIPT
100.0
Site is OK:  T2_UK_London_IC
92.350103377
ALERT FAILED JOBS AT:  T2_UK_SGrid_RALPP
76.0617760618
ALERT FAILED JOBS AT:  T2_US_Florida
62.6717334113
ALERT FAILED JOBS AT:  T2_US_Nebraska
99.685138539
ALERT FAILED JOBS AT:  T2_US_UCSD
TOTAL STATISTICS
undef:  0
0 :  65588
1 :  8
6 :  2
7 :  2
9 :  6
8001 :  365
8020 :  11299
50115 :  3

Week 2 

98.0189184366
ALERT FAILED JOBS AT:  T2_AT_Vienna
99.2225017151
ALERT FAILED JOBS AT:  T2_BE_UCL
90.5547226387
ALERT FAILED JOBS AT:  T2_BR_UERJ
99.0282937982
ALERT FAILED JOBS AT:  T2_CH_CSCS
99.9876604146
ALERT FAILED JOBS AT:  T2_DE_DESY
99.7532894737
ALERT FAILED JOBS AT:  T2_EE_Estonia
55.3399034594
ALERT FAILED JOBS AT:  T2_ES_IFCA
No jobs at:  T2_FR_CCIN2P3
16.5113182423
ALERT FAILED JOBS AT:  T2_FR_GRIF_IRFU
90.3861308117
ALERT FAILED JOBS AT:  T2_HU_Budapest
98.9703532753
ALERT FAILED JOBS AT:  T2_IT_Bari
99.9764373233
ALERT FAILED JOBS AT:  T2_IT_Pisa
99.3435448578
ALERT FAILED JOBS AT:  T2_KR_KNU
99.2405063291
ALERT FAILED JOBS AT:  T2_PL_Warsaw
99.8623537509
ALERT FAILED JOBS AT:  T2_PT_LIP_Lisbon
48.2242990654
ALERT FAILED JOBS AT:  T2_RU_INR
100.0
Site is OK:  T2_RU_JINR
No jobs at:  T2_RU_RRC_KI
31.5846994536
ALERT FAILED JOBS AT:  T2_TR_METU
100.0
Site is OK:  T2_TW_Taiwan
92.0454545455
ALERT FAILED JOBS AT:  T2_UK_London_Brunel
No jobs at:  T2_UK_SGrid_Bristol
99.4567611217
ALERT FAILED JOBS AT:  T2_US_Caltech
54.1891748755
ALERT FAILED JOBS AT:  T2_US_MIT
97.8160158835
ALERT FAILED JOBS AT:  T2_US_Purdue
99.5139732685
ALERT FAILED JOBS AT:  T2_US_Wisconsin
91.2878787879
ALERT FAILED JOBS AT:  T2_BE_IIHE
87.4529485571
ALERT FAILED JOBS AT:  T2_BR_SPRACE
99.3860845839
ALERT FAILED JOBS AT:  T2_CN_Beijing
98.888766641
ALERT FAILED JOBS AT:  T2_DE_RWTH
98.4453689512
ALERT FAILED JOBS AT:  T2_ES_CIEMAT
No jobs at:  T2_FI_HIP
16.5113182423
ALERT FAILED JOBS AT:  T2_FR_GRIF_LLR
99.7245179063
ALERT FAILED JOBS AT:  T2_FR_IPHC
66.1177644711
ALERT FAILED JOBS AT:  T2_IN_TIFR
99.1495747874
ALERT FAILED JOBS AT:  T2_IT_Legnaro
99.9624483665
ALERT FAILED JOBS AT:  T2_IT_Rome
0.0
ALERT FAILED JOBS AT:  T2_PK_NCP
No jobs at:  T2_PT_LIP_Coimbra
0.0
ALERT FAILED JOBS AT:  T2_RU_IHEP
100.0
Site is OK:  T2_RU_ITEP
No jobs at:  T2_RU_PNPI
39.4004282655
ALERT FAILED JOBS AT:  T2_RU_SINP
No jobs at:  T2_TR_ULAKBIM
7.12765957447
ALERT FAILED JOBS AT:  T2_UA_KIPT
99.6721311475
ALERT FAILED JOBS AT:  T2_UK_London_IC
95.2655150352
ALERT FAILED JOBS AT:  T2_UK_SGrid_RALPP
37.4408742456
ALERT FAILED JOBS AT:  T2_US_Florida
99.1995731057
ALERT FAILED JOBS AT:  T2_US_Nebraska
99.1722506898
ALERT FAILED JOBS AT:  T2_US_UCSD
TOTAL STATISTICS
undef:  0
0 :  122411
6 :  19
9 :  3
11 :  72
143 :  1
8001 :  383
8020 :  23127
50115 :  23
60307 :  35

Both Weeks 

98.5714285714
ALERT FAILED JOBS AT:  T2_AT_Vienna
94.5931548808
ALERT FAILED JOBS AT:  T2_BE_UCL
94.4639718805
ALERT FAILED JOBS AT:  T2_BR_UERJ
99.2379244649
ALERT FAILED JOBS AT:  T2_CH_CSCS
99.9604625877
ALERT FAILED JOBS AT:  T2_DE_DESY
99.9042451325
ALERT FAILED JOBS AT:  T2_EE_Estonia
60.0925925926
ALERT FAILED JOBS AT:  T2_ES_IFCA
No jobs at:  T2_FR_CCIN2P3
22.0552147239
ALERT FAILED JOBS AT:  T2_FR_GRIF_IRFU
96.0039305601
ALERT FAILED JOBS AT:  T2_HU_Budapest
99.2585964464
ALERT FAILED JOBS AT:  T2_IT_Bari
99.9815191277
ALERT FAILED JOBS AT:  T2_IT_Pisa
99.5350999535
ALERT FAILED JOBS AT:  T2_KR_KNU
99.6815286624
ALERT FAILED JOBS AT:  T2_PL_Warsaw
99.9090771329
ALERT FAILED JOBS AT:  T2_PT_LIP_Lisbon
34.1826923077
ALERT FAILED JOBS AT:  T2_RU_INR
88.5797101449
ALERT FAILED JOBS AT:  T2_RU_JINR
No jobs at:  T2_RU_RRC_KI
61.5347721823
ALERT FAILED JOBS AT:  T2_TR_METU
100.0
Site is OK:  T2_TW_Taiwan
95.3145917001
ALERT FAILED JOBS AT:  T2_UK_London_Brunel
No jobs at:  T2_UK_SGrid_Bristol
90.8966148216
61.6128782602
ALERT FAILED JOBS AT:  T2_US_MIT
96.8875812945
ALERT FAILED JOBS AT:  T2_US_Purdue
99.6016994158
ALERT FAILED JOBS AT:  T2_US_Wisconsin
98.3848314607
ALERT FAILED JOBS AT:  T2_BE_IIHE
91.8045112782
ALERT FAILED JOBS AT:  T2_BR_SPRACE
99.7634691196
ALERT FAILED JOBS AT:  T2_CN_Beijing
99.0928686905
ALERT FAILED JOBS AT:  T2_DE_RWTH
98.8921828678
ALERT FAILED JOBS AT:  T2_ES_CIEMAT
No jobs at:  T2_FI_HIP
22.0552147239
ALERT FAILED JOBS AT:  T2_FR_GRIF_LLR
99.6220580656
ALERT FAILED JOBS AT:  T2_FR_IPHC
52.5195968645
ALERT FAILED JOBS AT:  T2_IN_TIFR
99.4946492271
ALERT FAILED JOBS AT:  T2_IT_Legnaro
99.9733546496
ALERT FAILED JOBS AT:  T2_IT_Rome
0.0
ALERT FAILED JOBS AT:  T2_PK_NCP
No jobs at:  T2_PT_LIP_Coimbra
0.0
ALERT FAILED JOBS AT:  T2_RU_IHEP
100.0
Site is OK:  T2_RU_ITEP
No jobs at:  T2_RU_PNPI
80.7744565217
ALERT FAILED JOBS AT:  T2_RU_SINP
No jobs at:  T2_TR_ULAKBIM
15.9173754557
ALERT FAILED JOBS AT:  T2_UA_KIPT
99.7012696042
ALERT FAILED JOBS AT:  T2_UK_London_IC
94.5765472313
ALERT FAILED JOBS AT:  T2_UK_SGrid_RALPP
46.7331392828
ALERT FAILED JOBS AT:  T2_US_Florida
85.3809576468
ALERT FAILED JOBS AT:  T2_US_Nebraska
99.3696969697
ALERT FAILED JOBS AT:  T2_US_UCSD
TOTAL STATISTICS
undef:  0
0 :  187999
1 :  8
6 :  21
7 :  2
9 :  9
11 :  72
143 :  1
8001 :  748
8020 :  34426
50115 :  26
60307 :  35

Attempts at making dashboard accounting consistent with our logs

There are 50 sites in James' master list: https://twiki.cern.ch/twiki/bin/view/CMS/Step09AnalysisMetrics

1 of them has no pledge recorded: T2_PK_NCP 7 of them have no jobs submitted from glidein system that actually came back with logfiles, and the logfile shows an end date within the 2 weeks STEP09 period: No jobs at: T2_FR_CCIN2P3 No jobs at: T2_RU_RRC_KI No jobs at: T2_UK_SGrid_Bristol No jobs at: T2_FI_HIP No jobs at: T2_PT_LIP_Coimbra No jobs at: T2_RU_PNPI No jobs at: T2_TR_ULAKBIM

The total number of jobs across the 50-7=43 sites with jobs we have records for is: 223347

Now let's compare this with dashboard info: 296256

out of this we are entitled to subtract sites listed in dashboard that we don't analyze logs for: T3_UK_London_QMUL 2449 T3_UK_London_RHUL 376 T3_US_Colorado 984 T3_US_UMD 628 T1_TW_ASGC 2 unknown 51

Total: 4490

So the dashboard says Sanjay ran: 291766

We are missing logs for 68419 jobs.

We know that we had a problem of the crabserver submitting duplicates of some jobs for some period of time during STEP09. So next we count the duplicates using the CMSSW_#.log files. Those logfiles record what was done is terms of condor submissions for the job with this number in this task. We thus know if the task was duplicated, or not.

-- BristolDave - 2009/06/22

Attachments Attachments
Topic attachments
I Attachment Action Size Date Who Comment
pngpng allstep09.png manage 13.2 K 2009/06/22 - 15:51 BristolDave  
pngpng njobs_allstep09.png manage 12.8 K 2009/06/22 - 15:51 BristolDave  
pngpng week1.png manage 13.0 K 2009/06/22 - 15:51 BristolDave  
pngpng week2.png manage 13.0 K 2009/06/22 - 15:51 BristolDave  
Edit | Attach | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2009/06/22 - 17:11:58 - FkW
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback