TWiki> UCSDTier2 Web>BDIItests (revision 4)EditAttach

This page documents bdii loadtests done by Sanjay Padhi for OSG.

Description of tools used

We use the python interface to root in order to plot results in realtime while the test is running, and then store the results histograms in rootfile. The logic of the test is as follows:

  • Run N python threads
  • Each thread queries the bdii with:
    • os.popen("ldapsearch -xLLL -p2170 -h is-dev.grid.iu.edu -b o=grid","r")
  • Each thread collects the return, and records the time it took to complete the query.
  • The average return time for the N threads is logged in hprof histogram, after all N threads have returned.
  • Run the next N threads. The N threads are launched at most once per second. However, given that the return time of the ldapsearch is several seconds, the launch of the N threads is really much less often than once per second.
  • Continue doing this for a fixed amount of time dt

We then record a few different things:

  • The average time it takes for a query to return is plotted versus epoch time (hprof1 histogram)
  • The number of queries executed is plotted versus epoch time (htime histogram)
  • This is done for all queries as well as separately for those that fail only. A query is defined as failure if it does not come back with content.

Final Results

For the final results, we ran this as follows:

  • Once at CERN for 1h with N=15.
  • 8 instances of the test program run in parallel on our 8core desktop at CERN
  • Submitted 50 instances of the test program to the UCSD cluster.
    • As those 50 instances don't start all at the same time, we then add the histograms up from all the jobs in such a way as to get an appropriate time history.

One instance run from CERN

8 instances run in parallel from CERN

50 instances run as jobs submitted to various clusters

Results from miscellaneous initial testruns

Sunday September 14th

Ran a few different short tests, then one longer test of a few hours. For the longer run we picked: N = 15 and dt = 12000 seconds = 200 minutes = 3h 20min, and 18000 seconds = 300min = 5h respectively.

We then ran this test simultaneously from CERN (12000 seconds) and UCSD (18000 seconds). The CERN test ended at 2:35 Monday September 15th CERN time, while the UCSD one ended at 19:39 pacific on the 14th, i.e. 2h and 4min later.

  • Response time for the bdii querries from CERN:
    bdii-from-cern.gif

  • Response time for the bdii queries from UCSD:
    bdii-from-ucsd.gif

  • bdii host system monitoring: network traffic:
    riley-if_eth0-day.png

  • bdii host system monitoring: netstat:
    riley-netstat-day.png

  • bdii host system monitoring: processes:
    riley-processes-day.png

  • bdii host system monitoring: loadavg:
    riley-load-day.png

Understanding the client profile better (Monday September 15th)

To understand the client profile better, we did a series of tests where we varied N first on just one machine, and then having the same N but running the test program 4 times in parallel on 4 different (but identical hardware) hosts.

We find the the time per query depends significantly on the number of parallel python threads, but not significantly on whether we run one or 4 simultaneously.

N time for 4 in parallel time for one by itself
1 1.7-2.2s 2.2
10 3.2-3.5s 3.2
20 4.8-5.0s 5.0

-- FkW - 10 Sep 2008

Topic attachments
I Attachment Action Size Date Who Comment
gifgif bdii-from-cern.gif manage 12.5 K 2008/09/15 - 10:53 FkW Response time for the bdii querries from CERN
gifgif bdii-from-ucsd.gif manage 29.4 K 2008/09/15 - 11:00 FkW bdii response fro UCSD
pngpng riley-if_eth0-day.png manage 26.4 K 2008/09/15 - 10:54 FkW bdii host system monitoring: network traffic
pngpng riley-load-day.png manage 29.0 K 2008/09/15 - 10:55 FkW bdii host system monitoring: loadavg
pngpng riley-netstat-day.png manage 26.8 K 2008/09/15 - 10:54 FkW bdii host system monitoring: netstat
pngpng riley-processes-day.png manage 22.6 K 2008/09/15 - 10:55 FkW bdii host system monitoring: processes
Edit | Attach | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2008/09/20 - 09:18:18 - FkW
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback