Difference: GridFTPLVSTests2016 (8 vs. 9)

Revision 92017/02/07 - Main.CliftonPotter

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Changed:
<
<

GridFTP? -LVS Tests (gfal-copy writing to /dev/null

>
>

GridFTP? -LVS Tests (gfal-copy writing to /dev/null to test total throughput)

  This page details scalability tests on the GridFTP? server/LVS run over the course of November 2016-January 2017

Overview

Line: 12 to 12
 
  • Upon completion of gfal-copy the script sleeps for 1 second and then executes gfal-copy again. (Total Time: 60 minutes)
  • The total load on each individual gftp-x.t2.ucsd.edu server and total number of active jobs in the condor_q were recorded every 30 seconds
  • The total throughput (sum of all individual gftp-x loads) was calculated.
Added:
>
>
  • The total throughput per job was also calculated.
  • Fast approximation method splitting batch throughput into subintervals identifying 4 distinct phases of the overall Condor Queuing process.
  • 4th order approximation functions for both throughput and throughput/job and verification of optimum number of jobs (~2100) as well as an average expected throughput (on the order of 29 Gbit/s)
 

Data

Changed:
<
<
1000 Jobs:
1000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484711637329&to=1484717359888
1000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484711637329&to=1484717359888
>
>
1000 Jobs:
Throughput:
1000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484711637329&to=1484717359888
Number Jobs:
1000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484711637329&to=1484717359888
 
Changed:
<
<
2000 Jobs:
2000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484719120593&to=1484724632263
2000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484719120593&to=1484724632263
>
>
2000 Jobs:
Throughput:
2000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484719120593&to=1484724632263
Number Jobs:
2000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484719120593&to=1484724632263
 
Changed:
<
<
3000 Jobs:
3000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484725398613&to=1484730470759
3000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484725398613&to=1484730470759
>
>
3000 Jobs:
Throughput:
3000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484725398613&to=1484730470759
Number Jobs:
3000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484725398613&to=1484730470759
 
Changed:
<
<
4000 Jobs:
4000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484735972258&to=1484741688843
4000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484735972258&to=1484741688843
>
>
4000 Jobs:
Throughput:
4000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484735972258&to=1484741688843
Number Jobs:
4000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484735972258&to=1484741688843
 
Changed:
<
<
5000 Jobs:
5000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484742602465&to=1484747749937
5000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484742602465&to=1484747749937
>
>
5000 Jobs:
Throughput:
5000jTEST.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/caltech-network-tests?from=1484742602465&to=1484747749937
Number Jobs:
5000j_active.csv http://condorflux.t2.ucsd.edu:3000/dashboard/db/condor-metrics-test-001?from=1484742602465&to=1484747749937
  Overall:
FIGURE 1:
Line: 47 to 50
 
4000 95 191 48 26.263 Gbit/s 36.328 Gbit/s
5000 85 172 43 23.334 Gbit/s 37.076 Gbit/s
Changed:
<
<

Looking at the overall behavior of the each batch's individual job submission, execution, and completion (Figure 1), an immediate problem arises. The individual jobs do not execute simultaneously. This would lead to skewed overall averages. To compensate for this, each of the batches were split into four equally divided subintervals:


First Subinterval: J obs Submitting (Ramp Up)
Second Subinterval : All Jobs Active (Steady State)
Third Subinterval : All Jobs Active (Steady State)
Fourth Subinterval: Jobs Completing (Cool Down)
<--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}-->
>
>

Looking at the overall behavior of the each batch's individual job submission, execution, and completion (Figure 1), an immediate problem arises. The individual jobs do not execute simultaneously. This would lead to skewed overall averages. To compensate for this, each of the batches were split into four equally divided subintervals:


First Subinterval: Jobs Submitting (Ramp Up)
Second Subinterval : All Jobs Active (Steady State)
Third Subinterval : All Jobs Active (Steady State)
Fourth Subinterval: Jobs Completing (Cool Down)
<--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}-->
 
<--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}-->
1000 Jobs: Avg Throughput:(Gbits/sec)Avg # Jobs: Avg Throughput per Job:
S1:24.48367459564.050.04340692241
S2:24.0918667710000.02409186677
S3:24.93353163969.90.02570732202
S4:18.4091187269.66666670.06826620037
2000 Jobs: Avg Throughput: (Gbits/sec)Avg # Jobs: Avg Throughput per Job:
S1:28.227300831249.950.02258274398
S2:28.1361569220000.01406807846
S3:29.755994961992.3571430.01493507079
S4:19.22643441808.640.02377625941
3000 Jobs:
Line: 124 to 126
 
Changed:
<
<








Initial Attempts *Only leaving for Archival Purposes
>
>








Initial Attempts *Archiving initial attempts
 

This page details scalability tests on the GridFTP? server/LVS run over the course of November 2016

Overview

 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback