OSG CE And Submitter Testing And Monitoring

Mission statement

This page is meant to gather information about the OSG CE scalability activity.

Tools

Two software packages have been developped:

1. a load generating engine
   • The first version was developed by Toni Coarasa.
     Follow the links for installation instructions and usage instructions.
   • The code was refactored by Igor Sfiligoi to make it more user friendly, flexible and able to stress test non-Grid resources.
     The package can be downloaded here: loadtest_condor.v1_0_0.tgz
     The programs are in the bin directory; it only relies on command line options that are documented in the binary help screen.
2. a load monitoring package
   • The first version was developed by Toni Coarasa.
     Follow the links for a description.

Test hardware

Results

First round of tests were performed in Fall 2008 (by Toni Coarasa):

• GT2 at UCSD:
  Submitted more than 30k jobs.
  The observed limit was 0.44Hz (i.e. 26.4 jobs/min).
  See the following presentation.

Second round of tests were performed in Summer 2009 (by Igor Sfiligoi):

• GT2 at UCSD:
  Submitted 5k jobs.
  Job rate limited to 7.5 jobs/min under heavy load.
  Can sustain 33 jobs/min if monitoring not important.
  More details at: rates_gt2_ucsd.pdf
• GT2 between FNAL and UCSD :
  Submitted 5k jobs - from UCSD to FNAL, and from FNAL to UCSD.
  Job rate limited to 6.9 jobs/min under heavy load.
  Can sustain 25-28 jobs/min if monitoring not important (CPU speed seems a factor here).
  Network latency seems to be a factor.
  More details for UCSD to FNAL at: rates_gt2_fnal_r.pdf
  and for FNAL to UCSD at: rates_gt2_ucsd_r.pdf
• GT2 between Italy and UCSD :
  Submitted 5k jobs - from Italy to UCSD.
  Job rate limited to 3.5 jobs/min under heavy load.
  Can sustain 7 jobs/min if monitoring not important.
  Network latency is definitely a factor.
  More details: rates_gt2_ucsd_r2.pdf
• Network latency is definitely a factor with GT2; below you can see it graphically:
  
  The following shows only the limiting rate:
  

A third round of tests, using multiple users, were performed in Summer 2009 (by Igor Sfiligoi):

• GT2 at UCSD:
  Submitted 5k jobs per user.
  With 4 parallel users the job rate limit is 26 jobs/min (compare this to 7.5 jobs/min using a single user).
  Can sustain 47 jobs/min if monitoring not important (compare this to 33 jobs/min using a single user).
  More details at: rates_gt2_single_vs_multi_30min.ods or rates_gt2_single_vs_multi_30min.pdf
• GT2 between Italy and UCSD"
  Submitted 5k jobs per user. Due to limited client resources (single CPU, 1G of memory) -maxidle 1k was used when using multiple DNs.
  With 4 parallel users the job rate limit is 21 jobs/min (compare this to 5.2 jobs/min using a single user).
  Can sustain 43 jobs/min if monitoring not important (compare this to 14 jobs/min using a single user).
  More details at: rates_gt2_r2_single_vs_multi_30min.ods or rates_gt2_r2_single_vs_multi_30min.pdf
• Network latency is much more noticeable when a single DN is used. Using multiple users, the networking latencies don't seem to be a major issue.
• GT2 submissions from a resource constrained client
  Submitting 5k jobs for 4 users from a 1 CPU, 1GB machine.
  By trying to submit all jobs as fast as possible (vs using -maxidle 1k), the average job rate drops from 20 job/min to 11 jobs/min.
  More details at: rates_gt2_r2_resource_constraint.ods or rates_gt2_r2_resource_constraint.pdf

-- IgorSfiligoi - 2009/08/28

Deprecated information... left only for historical reference

Monitoring

The monitoring has been done using the package described in "Description of the osgmonitoring.rpm package". and the cacti installed in t2sentry0.t2.ucsd.edu.

Installation Instructions for dummys

Running the tests for dummys

Things left to do

1. create the client side tarball, and document it on this twiki, and attach the tarball to the twiki page.
2. communicate with Terrence to make sure that we have the client monitoring online on uaf-2, and uaf-1.
3. document the process of putting monitoring via t2sentry0 and cacti into place

Goal to be finished: Monday April 6th

-- ToniCoarasa - 19 Sep 2008