GridFTP? -LVS Tests

This page details scalability tests on the GridFTP? server/LVS run over the course of November 2016

Overview

To test the throughput and client load limits of the LVS and the GridFTP? servers

  • A variable (range: 100-8k) number of 30-40 minute jobs were submitted via condor_submit and put into the condor_q(ueue)
  • Each job proceeds by selecting a file from a list at random, gfal-copying that file (writing it to /dev/null) and then sleeping for a variable amount of time (10 sec) before repeating the process and continuing for 30-40 minutes.
  • The throughput (Gbps) of each individual gftp-X.t2.ucsd.edu server as well as the number of active jobs was recorded at 30 second time intervals through each client's execution
  • Objective is to test 6 GridFTP? Servers on the LVS setup and to see if the system performs as expected
  • !!!Number of clients active when maximum throughput occurs
  • !!!Total Throughput/Client (Gbps/Client)

Example Code

gfcTest.sh

#!/bin/bash
sleepTime=$1 #seconds to sleep after each gfal-copy
totalTime=$2*60 #minutes to execute script for
while "$SECONDS" -lt "$totalTime"?
do
file=$(cat fileList.txt | sort -R | head -1)
home=$(echo "gsiftp://gftp.t2.ucsd.edu/hadoop")
path=$home$file
gfal-copy -f -v $path file:/dev/null
sleep "$sleepTime"s
done

fileList.txt

/Path/To/File/test_1.file
/Path/To/File/test_2.file
...
/Path/To/File/test_n.file

100_30.submit

executable = gfcTest.sh

error = out/output_100_30/test-$(Cluster).$(Process).error

log = out/output_100_30/test-$(Cluster).$(Process).log

output = out/output_100_30/test-$(Cluster).$(Process).out

transfer_input_files = fileList.txt

RequestMemory? = 1000

arguments = 10 30

queue 100

Data

# Submitted JobsTest Duration (min)# Active Jobs @ Max ThroughputStacked Max Throughput (Gbps)Total Throughput/Client (Gbps)
100301005.5320.06
10003096614.260.02
15004026917.20.07
200040158614.360.01
22504048114.20.03
25004079526.430.04
40003051620.030.04
800030219518.390.01
40004089613.650.02
80004039719.830.05

Edit | Attach | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2016/12/09 - 05:59:01 - CliftonPotter
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback