dCache Scalability Tests

This page contains the results of tests ran against dCache as part of the OSG scalability and reliability area activity.

FNAL test instance

The FNAL team, composed mainly by Tanya Levshina and Neha Sharma, operates a test instance of dCache.

The test instance is composed of 6 nodes:

• dCache admin node
• chimera/pnfs node
• srm node
• a dCache door, and
• 2 pool nodes.

The SRM machine is a single 3.6GHz Xeon, 4GB of RAM and GigE? Ethernet, running a 32-bit version of SL4.

FNAL lcg-ls tests

The tests reported in this section were performed against that instance, using a glideTester instance using FNAL sleeper pool resources.

All glideTester jobs were running lcg-ls in a tight loop for a specified amount of time.

 lcg-ls -b -D srmv2 $mytestdir

Run 1

This run ran against dCache v1.5.9-17 with Chimera. Default parameters were used.

The glideTester jobs were configured to run for 20 minutes (1200s).

Complete results can be seen below:

The tested system peaked at about 9Hz with 50 clients. After that level it became painfully slow, basically unusable, although no user visible errors could be noticed.

It should be noted that once the unstable situation was reached, the system remained very slow even with a small number of clients, until a human operator fixed the problem on the server side.

Run 2

This run ran against dCache v1.5.9-12 with PNFS. Default parameters were used.

The glideTester jobs were configured to run for 20 minutes (1200s) up to 200 jobs, and for 40 minutes (2400s) for higher concurrencies.

Complete results can be seen below:

The tested system was pretty consistent at 2Hz up to 800 clients. However, starting at 700 clients users start to see errors.

Run 3

This run ran against dCache v1.9.5-19 with Chimera. The following parameters have been tuned:

1. modify max_connections from default 100 to 250 in postgresql.conf of srm database
2. in dCacheSetup, make sure:
   srmAsynchronousLs=true
3. in /opt/d-cache/libexec/apache-tomcat-5.5.20/conf/server.xml,
   find element: <Connector className="org.globus.tomcat.coyote.net.HTTPSConnector"
   set these parameters:

• maxThreads="1000"
• minSpareThreads="25"
• maxSpareThreads="200"
• maxProcessors="1000"
• minProcessors="25"
• maxSpareProcessors="200"
• enableLookups="false"
• disableUploadTimeout="true"
• acceptCount="1024"

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

The tested system peaked at ~4.3Hz with 50 clients, and was slowly degrading to just below 2Hz at 1000 client. The error rates started to show up with 600 clients, but were really problematic after 800 clients.

Run 4

-- IgorSfiligoi - 2010/06/17

This test used postgres and kwp, with the following paramaters:

max_connections - 250  shared_buffers - 512MB work_mem - 16MB  max_fsm_pages - 1000000  

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

The tested system was peaked at 2.7Hz with 50 clients, stays in the 2Hz range until abount 400 clients and then starts to degrate. Errors start to appear with 400 clients, but get really problematic at around 800.

Run 5

-- IgorSfiligoi - 2010/06/17

This test was similar to Run 4, but using GUMS.

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

At low concurrencies, the tested system gave similar results as Run 4. It peaked at 2.7Hz with 50 clients, stays in the 2Hz range until abount 400 clients and then starts to degrate.

However, once reached 800 clients, the system seems to misbehave badly.

Run 6

-- IgorSfiligoi - 2010/12/22

This test was similar to Run 5, but using XACML GUMS and dcache version 1.9.5-23.

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

The tested system performs significantly better at low concurrencies ( 4.7Hz vs 2.7Hz), but it steadily declines and is only marginally better at higher concurrencies.

The concurrency limit is still around the 600 mark.

Run 7

-- IgorSfiligoi - 2011/04/19

This test was similar to Run 6, just the OS was upgraded to SL5 and the dcache version is dcache-server-1.9.5-25.The clients were running on the UCSD sleeper pool.

The glideTester jobs were configured to run for 20 minutes (1200s) for concurrencies up to 150, and 40 minutes (2400s) up to 400 and 80 minutes (4800s) after that.

Complete results can be seen below:

The tested system performs similarly to the previous test, although it is marginally better. The concurrency limit seems to have improved to about the 800 mark.

The server hung up during the 1.2k run, and had to be manually restarted.

Run 8

-- IgorSfiligoi - 2011/05/23

This test was similar to Run 7, just the the dcache version is now dcache-server-1.9.5-26.The clients were running on the UCSD sleeper pool.

The glideTester jobs were configured to run for 20 minutes (1200s) for concurrencies up to 150, and 40 minutes (2400s) up to 400 and 80 minutes (4800s) after that.

Complete results can be seen below:

The system performed slightly worse than before, hitting the wall at 1k. But it did recover by itself once the jobs stopped.

FNAL lcg-cp tests

The tests reported in this section were performed against that instance, using a glideTester instance using FNAL sleeper pool resources.

All glideTester jobs were running lcg-cp from the SE to the local disk of a 10Mbyte file in a tight loop for a specified amount of time.

 lcg-cp -b -D srmv2 $mytestdir/igors_file.dat file:$PWD/igors_file.dat 

Run 1

This run ran against dCache v1.9.5-19 with Chimera. The following parameters have been tuned:

1. modify max_connections from default 100 to 250 in postgresql.conf of srm database
2. in dCacheSetup, make sure:
   srmAsynchronousLs=true
3. in /opt/d-cache/libexec/apache-tomcat-5.5.20/conf/server.xml,
   find element: <Connector className="org.globus.tomcat.coyote.net.HTTPSConnector"
   set these parameters:

• maxThreads="1000"
• minSpareThreads="25"
• maxSpareThreads="200"
• maxProcessors="1000"
• minProcessors="25"
• maxSpareProcessors="200"
• enableLookups="false"
• disableUploadTimeout="true"
• acceptCount="1024"

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

* - Two jobs got stuck and did not finish for over 1 hour and had to be hard killed.
^ - Twenty jobs got stuck and did not finish for over 1 hour and had to be hard killed.

The tested system peaked at 50 clients, delivering files at 2.2Hz, or 200Mbit/s, and then declining to ~1.4Hz
The first error appear with 150 clients, but are still bearable up to about 600 client.
With 800 clients, more than half of all attempts failed, while with 1000 clients all the attempts failed.

Run 2

-- IgorSfiligoi - 2010/06/17

This test used postgres and kwp, with the following paramaters:

max_connections - 250  shared_buffers - 512MB work_mem - 16MB  max_fsm_pages - 1000000  

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

The tested system was peaked at 2.2Hz with 50 clients and 200Mbit/s, and then rapidly deteriorates.

By 400 clients the system was practically unusable.

Run 3

-- IgorSfiligoi - 2010/06/17

This test was similar to Run 2, but using GUMS.

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

Like with Run 2, the tested system peaked at 2.2Hz with 50 clients and 200Mbit/s, and then rapidly deteriorated.

The deterioration rate was much faster, though. At 200 clients the system was already unusable.

Run 4

-- IgorSfiligoi - 2010/12/22

This test was similar to Run 3, but using XACML GUMS and dcache version 1.9.5-23.

The glideTester jobs were configured to run for 40 minutes (2400s).

Complete results can be seen below:

Like with Run 3, the tested system peaked at 2.2Hz with 50 clients and 200Mbit/s. But the deteriorarion is much slower; while errors start to appear around the 150 mark, the system is still usable(with retries) up to about 500 concurrent clients.

Run 5

-- IgorSfiligoi - 2011/04/20

This test was similar to Run 4, just the OS was upgraded to SL5 and the dcache version is dcache-server-1.9.5-25.The clients were running on the UCSD sleeper pool.

The glideTester jobs were configured to run for 20 minutes (1200s) for concurrencies up to 150, and 40 minutes (2400s) up to 400 and 80 minutes (4800s) after that.

Complete results can be seen below:

The tested system performs similarly to the previous test, although it is marginally better. The concurrency limit seems to have improved to about the 800 mark.

The server hung up during the 1.2k run, and had to be manually restarted.

Run 5

-- IgorSfiligoi - 2011/05/23

This test was similar to Run 5, just the the dcache version is now dcache-server-1.9.5-26.The clients were running on the UCSD sleeper pool.

The glideTester jobs were configured to run for 20 minutes (1200s) for concurrencies up to 150, and 80 minutes (4800s) after that.

Complete results can be seen below:

The system performed slightly worse than before, severely degrading already at 600. But it never hang.