Difference: WSGramTests (1 vs. 6)

Revision 62008/01/19 - Main.TerrenceMartin

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

WS Gram Testing

Line: 11 to 11
  The following is results from UCSD WS Gram testing with Pre-WS Gram for comparison.
Changed:
<
<

Test Setup

Gatekeeper

Dual Core Dual CPU Opteron 275 (64bit) 8GB RAM Centos 5 (RHEL) OSG 0.8.0 CE Condor 6.9.5

Submitters

Dual Processor Single Core 3.06Ghz Intel (32bit) 4GB RAM Centos 4 (RHEL) OSG 0.8.0 Client Condor 6.9.5

Dual Processor Dual Core 2.0Ghz Intel (64bit) 8GB RAM Centos 4 (RHEL) OSG 0.8.0 Client Condor 6.9.5

Test Scripts

Condor Submit Script WS GRAM

Universe=grid
Grid_Type = gt4
Jobmanager_Type = Condor
GlobusScheduler=https://osg-gw-5.t2.ucsd.edu:9443
executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh
globus_rsl=(condor_submit=('+SleepSlot' 'TRUE'))
transfer_executable=True
stream_output = False
stream_error  = False
WhenToTransferOutput = ON_EXIT
transfer_input_files = /data/tmp/myscript.sh
log    = /tmp/wsgram.log

arguments = 3600
output = output/wsgram.out.$(Process)
error  = output/wsgram.err.$(Process)
notification=Never
queue 2000

Condor Submit Script PreWS? (GT2) GRAM

universe=globus
GlobusScheduler=osg-gw-5.t2.ucsd.edu:/jobmanager-condor
executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh
globus_rsl=(condor_submit=('+SleepSlot' 'TRUE'))
transfer_executable=True
stream_output = False
stream_error  = False
WhenToTransferOutput = ON_EXIT
transfer_input_files = /data/tmp/myscript.sh
log    = /tmp/wsgram.log

arguments = 3600
output = output/wsgram.out.$(Process)
error  = output/wsgram.err.$(Process)
notification=Never
queue 2000

Test Program

#!/bin/sh

echo "starting WS Test...."
/usr/bin/whoami
/bin/hostname
chmod 755 myscript.sh
./myscript.sh
COUNT=$1
sleep $COUNT
echo "done WS Test.
>
>

Notable Findings

 
Changed:
<
<
Test Script

Downloaded to the node via GRAM and then called from the test program.

>
>
The following were some of the behaviors of GT4 found during testing
 
Changed:
<
<
#!/bin/sh
>
>

Most Important Issues

 
Changed:
<
<
echo "I am myscript!!!"
>
>
  • GT4 on loads of 1000 jobs or greater, or tests that involved two submitters with at least 1000 jobs each it was common to have up to 5% of submitted jobs go into a Hold state on the submitter(s). During larger tests the rate of held jobs increased as a percentage of total jobs. There were a variety of reasons given by the GRAM for these jobs to be held. When the jobs were released by condor_release command it was found that in most if not all cases the jobs did eventually finish successfully.
 
Changed:
<
<

Gridmanager Settings

>
>
  • One of the configuration changes to reduce the occurrence of certain GRAM errors involved increasing the maximum Java heap memory on the submitter to 1024MB of memory. This resulted in a much reduced frequency or the elimination of some of the errors that resulted in held jobs. While this change was successful in reducing or eliminating certain errors the increased resource consumption on the submitter is of concern. Particularly when considering multi-user submission nodes or nodes with multiple schedd.
 
Changed:
<
<
These settings were the ones used for most of the results below including the pre WS GT2 comparisons.
>
>
  • CPU load on the submitters was much higher with GT4 than it was with GT2
 
Changed:
<
<
MAX_GRIDMANAGER_LOG     = 10000000
GRIDMANAGER_DEBUG       = D_COMMAND D_FULLDEBUG
>
>
  • Removal of larger amounts of submitted jobs via Condor_rm (as low as 1000 jobs) is not possible without administrator intervention on the submitter and gatekeeper. This is a known WS Gram limitation that is apparently fixed in a more recent version.
 
Deleted:
<
<
# GT4 requires a gftp server to be running locally #GRIDFTP_URL_BASE = gsiftp://$(FULL_HOSTNAME) GRIDFTP_SERVER = $(LIBEXEC)/globus-gridftp-server GRIDFTP_SERVER_WRAPPER = $(LIBEXEC)/gridftp_wrapper.sh
 
Changed:
<
<
# Bump up the max jobs that can be submitted via condor-g GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=2000 GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=50
>
>

Remaining Findings

 
Changed:
<
<
JAVA_EXTRA_ARGUMENTS = -Xmx1024M GRIDMANAGER_GAHP_CALL_TIMEOUT = 600
>
>
  • GT4 was able to handle up to 1000 jobs that were submitted from 1 submit host. In the last test of 1000 submissions all jobs completed without any errors. This level of operation was achieved after a series of configuration modifications were performed to the standard OSG 0.8.0 WS GRAM installation and client configuration. A jar file was also provided that allowed for some additional configuration flexibility on the submitter.
 
Changed:
<
<

Submitter GFTP server configuration

>
>
  • GT4 was considerably slower to submit jobs to the gatekeeper than GT2 at higher job counts. At 2000 jobs submitted each from 2 submitters GT2 was able to schedule 4000 jobs in about 1 hour with GT2. GT4 took several hours and due to the jobs completing in 1 hour was never able to achieve more than slightly over 2K jobs in the queue. At lower job counts GT4 was more responsive.
 
Changed:
<
<
control_preauth_timeout 300
>
>
  • During testing it was found that once the submitter was working any additional job submissions by the test user, even if they were targeted to another gatekeeper, experienced a significant delay before the submitter sent them to the gatekeeper. The delay was approximately an hour.
 
Changed:
<
<

Gatekeeper GFTP server configuration

>
>
  • CPU load on the gatekeeper was slightly lower with GT4 than it was with GT2
 
Deleted:
<
<
control_preauth_timeout 300
 
Changed:
<
<

GT4 Server Side Settings

>
>
  • GUMS server load was not notably higher with GT4 than it was with GT2
 
Changed:
<
<
inetd 1
log_level ERROR,WARN,INFO
log_single /osglocal/osgce/globus/var/log/gridftp-auth.log
log_transfer /osglocal/osgce/globus/var/log/gridftp.log
control_preauth_timeout 300
>
>
  • The GT4 Gatekeeper service was stable during the testing. The only time the service was restarted was to perform a configuration change or in order to cleanly remove the persistent areas due to problems removing large numbers of jobs from the submission queue.
 
Changed:
<
<

GT4 Server Configuration

>
>
  • The GT4 gatekeeper never experienced excessive loads during the tests. The gatekeeper remained responsive.
 
Deleted:
<
<
See Attachments
 

Tests

Added:
>
>
The following Graphs show the loads of both the submitter(s) and the gatekeeper under different levels of total job submissions. All jobs were submitted as a single condor_submit command.
 

WS GRAM Test 1000 Jobs Submitted from 1 Submitter (1K total Jobs)

Changed:
<
<
This test is WS Gram All jobs finished but there were some held jobs. Probably in the 5% range.
>
>
During this test some of the jobs had to be released by condor_release as they became held. In a subsequent 1000 job test not shown here all jobs completed without any held jobs. This might indicated this is a threshold with the current configuration and code.
 

  • OSG GW 5 CE:
    gw5-condor.png
Added:
>
>
  • Ramp up time was fairly quick
 
  • OSG 5 Load 1K:
    gw5-load.png
Added:
>
>
  • Load spikes during submission and then trails off
 
  • OSG GW 5 Network 1K:
    gw5-network.png
Added:
>
>
  • As expected the network traffic was minimal with small spikes at job submission and then again at job completion
 
  • OSG GW 5 Memory 1K:
    gw5-mem.png
Added:
>
>
  • Memory consumption increased during the test and then returned to about pre-test levels.
 
  • UAF Load 1K:
    uaf-1-load.png
Added:
>
>
  • Submitter load spikes during initial submission
 
  • UAF Mem 1K:
    uaf-1-mem.png
Added:
>
>
  • The submission consumes some of the free ram
 

WS GRAM Test 2000 Jobs Submitted from 1 Submitters (2K total jobs)

Changed:
<
<
This test showed a higher count of held jobs. Other notable discoveries was that submission of addition trivial amounts of WS GRAM jobs, even targeting other CE were delayed by up to 1 hour.
>
>
This test showed a higher count of held jobs and an increase in resource consumption.
 
  • OSG 5 Condor Load 2K:
    osg-5-condor-load-2K.png
Line: 203 to 102
 

WS GRam 2K x 2 Submitters

Changed:
<
<
Significant amount of these jobs went into a hold state
>
>
Significant amount of these jobs went into a hold state going as high as around 7% held jobs before the test was stopped,
 

  • GUMS Load WS 2K x 2:
Line: 211 to 110
 
  • OSG 5 Condor Load WS 2K x 2:
    osg-5-condor-load-2kx2.png
Added:
>
>
  • The rate of jobs being queued on the gatekeeper was fairly low, taking 2 hours before the queue hit its peak and the peak was well under 4K jobs indicating a significant amount of jobs had already completed at that point.
 
  • OSG 5 Load WS 2K x 2:
    osg-5-load-2kx2.png

  • OSG 5 Mem WS 2K x 2:
    osg-5-mem-2kx2.png
Added:
>
>
  • Unused memory dropped very low in this test.
 
  • UAF 1 Load WS 2K x 2:
    uaf-1-load-2kx2.png
Line: 227 to 128
 

Followup WS GRAM 2000 x 2 submitters 5%+ Hold result

Changed:
<
<
This test resulted in a greater than 5% job hold rate and excessive gatekeeper load. Errors included problems with authentication.
>
>
This test was run with the same job counts as the previous test. Again a greater than 5% job hold rate and excessive gatekeeper load. Hold errors included problems with authentication.
 
  • Condor Load 2kx2 WS GRAM Test 2:
    osg-gw-5-CondorLoad-2kx2-2.png
Added:
>
>
  • As before the submission rate was fairly slow, about 2.5 hours to reach a peak of a little over 2000 jobs out of 4000 submitted total
 
  • OSG GW5 Load 2kx2 Test 2:
    osg-gw-5-load-2kx2-2.png
Line: 248 to 150
 
  • Comparison Condor Load WS vs Pre WS:
    comparisonload-2.png
Added:
>
>
  • This graph shows the difference in submission rate between GT2 and GT4 at this level
 
  • Comparison Condor Load WS vs Pre WS (long view):
    comparisonload.png
Added:
>
>
  • This is the same graph as above but at a lower level of detail.
 
  • Final Pre WS Condor Load:
    FinalLoad.png
Added:
>
>
  • This is the complete GT2 cycle
 
  • Final Pre WS System Load:
    finalsystemload.png
Added:
>
>
  • The initial load spike in this graph was at the tail end of the last GT4 test and likely caused by the removal of a few dozen lingering jobs. During the GT2 test the gatekeeper load was around 20-25 (approximately a load of 6/CPU).
 
  • UAF 1 System Load Pre WS:
    uaf-1loadprews.png
Added:
>
>
  • GT2 has a much lower CPU load on submitters than GT4
 
  • UAF 1 System Mem Pre WS:
    uaf-1memprews.png

  • UAF 2 System Load Pre WS:
    uaf-2loadprews.png
Added:
>
>
  • This graph is a bit spikey, possibly due to user activity on the submitter
 
  • UAF 2 System Mem Pre WS:
    uaf-2memprews.png
Deleted:
<
<

Notes

 
Changed:
<
<
  • With increasing jobs queued (within acceptable levels for pre-ws gram) about 5% of jobs become held for one reason or another. Configuration changes to timeouts have reduced the variety of timeouts.
>
>

Test Setup

The following details some of the hardware and software configurations used during testing. Included are the Condor submit scripts and scripts used to test the submission infrastructure. The focus of these tests was to exercise the submitter and gatekeeper abilities to handle increasing amounts of job request. To isolate the activity to submission the job itself was very light with minimal data transfer between submitter and gatekeeper.

 
Changed:
<
<
  • One of the more recent tests (2K x 1 submitter) resulted in 0 held jobs, however there are two jobs that never even started. A hold and release cycle got them started again.
>
>
The job submission counts were chosen based on experience running production OSG clusters. The job counts of 1000-2000 jobs per submitter, as well as the combination of multiple submitters are fairly typical in the Open Science Grid.
 
Changed:
<
<
  • In a followup 2k x 2 submitter test about 2 hours in 5% of the jobs (204/4000) had gone into various hold states.
>
>
All job submissions were submitted through the Condor-g infrastructure.

The test job run on the compute nodes consisted of a single shell script that went into a 3600 second sleep after running a small sub shell script. This sub shell script was downloaded via the GT infrastructure as an included file as part of the job.

The condor pool was configured to offer the test jobs ~1700 free job slots.

Gatekeeper

  • Dual Core Dual CPU Opteron 275 (64bit)
  • 8GB RAM
  • Centos 5 (RHEL)
  • OSG 0.8.0 CE (Configured with NFS Lite condor.pm)
  • Condor 6.9.5

Submitters

  • Dual Processor Single Core 3.06Ghz Intel (32bit)
  • 4GB RAM
  • Centos 4 (RHEL)
  • OSG 0.8.0 Client
  • Condor 6.9.5

  • Dual Processor Dual Core 2.0Ghz Intel (64bit)
  • 8GB RAM
  • Centos 4 (RHEL)
  • OSG 0.8.0 Client
  • Condor 6.9.5

Test Scripts

Condor Submit Script WS GRAM

 
Changed:
<
<
HoldReason? = "Globus error: GT4_GRAM_JOB_SUBMIT timed out" ... HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" ... HoldReason? = "Globus error: GT4_GRAM_JOB_SUBMIT timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_SUBMIT timed out" HoldReason? = "Globus error: GT4_GRAM_JOB_SUBMIT timed out" LastHoldReason? = "Spooling input data files" HoldReason? = "Globus error 155: the job manager could not stage out a file" HoldReason? = "Globus error 155: the job manager could not stage out a file" HoldReason? = "Globus error 155: the job manager could not stage out a file" HoldReason? = "Globus error 155: the job manager could not stage out a file" ... HoldReason? = "Globus error: org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException: \"/DC=org/DC=doegrids/OU=People/CN=Terr ence Martin 525658\" is not authorized to use operation: {http://www.globus.org/namespaces/2004/10/gram/job}createManagedJob on this service" HoldReason? = "Globus error: GT4_GRAM_JOB_DESTROY timed out" HoldReason? = "Globus error: org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException: \"/DC=org/DC=doegrids/OU=People/CN=Terr ence Martin 525658\" is not authorized to use operation: {http://www.globus.org/namespaces/2004/10/gram/job}createManagedJob on this service"
>
>
Universe=grid Grid_Type = gt4 Jobmanager_Type = Condor GlobusScheduler? =https://osg-gw-5.t2.ucsd.edu:9443 executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh globus_rsl=(condor_submit=('+SleepSlot' 'TRUE')) transfer_executable=True stream_output = False stream_error = False WhenToTransferOutput? = ON_EXIT transfer_input_files = /data/tmp/myscript.sh log = /tmp/wsgram.log

arguments = 3600 output = output/wsgram.out.$(Process) error = output/wsgram.err.$(Process) notification=Never queue 2000

 
Added:
>
>
Condor Submit Script PreWS? (GT2) GRAM
 
Changed:
<
<
  • Configuration changes to the submitter have resulted in Gridmanager java related process growing to greater than 1.1GB of used memory size. Each user that submits from a specific host will have their own java process. This is just the current largest Java process size. It may be necessary to allow Java to consume even more memory if > 2K jobs are submitted.
>
>
universe=globus
GlobusScheduler=osg-gw-5.t2.ucsd.edu:/jobmanager-condor
executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh
globus_rsl=(condor_submit=('+SleepSlot' 'TRUE'))
transfer_executable=True
stream_output = False
stream_error  = False
WhenToTransferOutput = ON_EXIT
transfer_input_files = /data/tmp/myscript.sh
log    = /tmp/wsgram.log
 
Changed:
<
<
  • CPU Load on the gatekeeper that is recieving WSGram jobs is similar to the load on the gatekeeper that is recieving Pre-WS Gram jobs
>
>
arguments = 3600 output = output/wsgram.out.$(Process) error = output/wsgram.err.$(Process) notification=Never queue 2000
 
Changed:
<
<
  • CPU Load on the submitter is much higher with WS Gram than Pre-WS Gram.
>
>
Test Script
 
Changed:
<
<
  • GUMS server load is slightly higher with WS GRAM than Pre WS Gram
>
>
#!/bin/sh
 
Changed:
<
<
  • Removal of larger amounts of submitted jobs via Condor_rm (as low as 1K) is not possible without manual intervention on the submitter and gatekeeper. WS Gram fails to properly remove the jobs from the gatekeeper which requires that the persisted directory on the gatekeeper be manually purged. This is a known WS Gram limitation that is apparently fixed in a more recent version.
>
>
echo "starting WS Test...." /usr/bin/whoami /bin/hostname chmod 755 myscript.sh ./myscript.sh COUNT=$1 sleep $COUNT echo "done WS Test.

Test Sub Script

Downloaded to the node via GRAM and then called from the test program.

#!/bin/sh

echo "I am myscript!!!"

Gridmanager Settings

These settings were the ones used for most of the results below including the pre WS GT2 comparisons.

MAX_GRIDMANAGER_LOG     = 10000000
GRIDMANAGER_DEBUG       = D_COMMAND D_FULLDEBUG

# GT4 requires a gftp server to be running locally
#GRIDFTP_URL_BASE = gsiftp://$(FULL_HOSTNAME)
GRIDFTP_SERVER = $(LIBEXEC)/globus-gridftp-server
GRIDFTP_SERVER_WRAPPER = $(LIBEXEC)/gridftp_wrapper.sh

# Bump up the max jobs that can be submitted via condor-g
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=2000
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=50
 
Changed:
<
<
  • Pretty slow ramp up time. 20 minutes to start 76 jobs of 2K in one test. ~15s/job
>
>
JAVA_EXTRA_ARGUMENTS = -Xmx1024M GRIDMANAGER_GAHP_CALL_TIMEOUT = 600

Submitter GFTP server configuration

control_preauth_timeout 300

Gatekeeper GFTP server configuration

control_preauth_timeout 300

GT4 Server Side Settings

inetd 1
log_level ERROR,WARN,INFO
log_single /osglocal/osgce/globus/var/log/gridftp-auth.log
log_transfer /osglocal/osgce/globus/var/log/gridftp.log
control_preauth_timeout 300

GT4 Server Configuration

 
Added:
>
>
Server Configuration
 

-- TerrenceMartin - 17 Jan 2008

Revision 52008/01/18 - Main.RamiVanguri

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

WS Gram Testing

Line: 9 to 9
 

Introduction

Changed:
<
<
The following is results from UCSD WS Gram testing with Pre-WS Gram for comparision.
>
>
The following is results from UCSD WS Gram testing with Pre-WS Gram for comparison.

Test Setup

Gatekeeper

Dual Core Dual CPU Opteron 275 (64bit) 8GB RAM Centos 5 (RHEL) OSG 0.8.0 CE Condor 6.9.5

Submitters

Dual Processor Single Core 3.06Ghz Intel (32bit) 4GB RAM Centos 4 (RHEL) OSG 0.8.0 Client Condor 6.9.5

Dual Processor Dual Core 2.0Ghz Intel (64bit) 8GB RAM Centos 4 (RHEL) OSG 0.8.0 Client Condor 6.9.5

Test Scripts

Condor Submit Script WS GRAM

Universe=grid
Grid_Type = gt4
Jobmanager_Type = Condor
GlobusScheduler=https://osg-gw-5.t2.ucsd.edu:9443
executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh
globus_rsl=(condor_submit=('+SleepSlot' 'TRUE'))
transfer_executable=True
stream_output = False
stream_error  = False
WhenToTransferOutput = ON_EXIT
transfer_input_files = /data/tmp/myscript.sh
log    = /tmp/wsgram.log

arguments = 3600
output = output/wsgram.out.$(Process)
error  = output/wsgram.err.$(Process)
notification=Never
queue 2000

Condor Submit Script PreWS? (GT2) GRAM

universe=globus
GlobusScheduler=osg-gw-5.t2.ucsd.edu:/jobmanager-condor
executable=/home/users/tmartin/Cluster_Tests/ENV_test/re.sh
globus_rsl=(condor_submit=('+SleepSlot' 'TRUE'))
transfer_executable=True
stream_output = False
stream_error  = False
WhenToTransferOutput = ON_EXIT
transfer_input_files = /data/tmp/myscript.sh
log    = /tmp/wsgram.log

arguments = 3600
output = output/wsgram.out.$(Process)
error  = output/wsgram.err.$(Process)
notification=Never
queue 2000

Test Program

#!/bin/sh

echo "starting WS Test...."
/usr/bin/whoami
/bin/hostname
chmod 755 myscript.sh
./myscript.sh
COUNT=$1
sleep $COUNT
echo "done WS Test.

Test Script

Downloaded to the node via GRAM and then called from the test program.

#!/bin/sh

echo "I am myscript!!!"

Gridmanager Settings

These settings were the ones used for most of the results below including the pre WS GT2 comparisons.

MAX_GRIDMANAGER_LOG     = 10000000
GRIDMANAGER_DEBUG       = D_COMMAND D_FULLDEBUG

# GT4 requires a gftp server to be running locally
#GRIDFTP_URL_BASE = gsiftp://$(FULL_HOSTNAME)
GRIDFTP_SERVER = $(LIBEXEC)/globus-gridftp-server
GRIDFTP_SERVER_WRAPPER = $(LIBEXEC)/gridftp_wrapper.sh

# Bump up the max jobs that can be submitted via condor-g
GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE=2000
GRIDMANAGER_MAX_PENDING_SUBMITS_PER_RESOURCE=50

JAVA_EXTRA_ARGUMENTS = -Xmx1024M
GRIDMANAGER_GAHP_CALL_TIMEOUT = 600

Submitter GFTP server configuration

control_preauth_timeout 300

Gatekeeper GFTP server configuration

control_preauth_timeout 300

GT4 Server Side Settings

inetd 1
log_level ERROR,WARN,INFO
log_single /osglocal/osgce/globus/var/log/gridftp-auth.log
log_transfer /osglocal/osgce/globus/var/log/gridftp.log
control_preauth_timeout 300

GT4 Server Configuration

See Attachments

 

Tests

Line: 201 to 343
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load 2K" date="1200614946" name="uaf-1-load-2k.png" path="uaf-1-load-2k.png" size="31368" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load 2K" date="1200614808" name="osg-5-condor-load-2K.png" path="osg-5-condor-load-2K.png" size="30651" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 1K" date="1200614220" name="gw5-load.png" path="gw5-load.png" size="27977" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="GT4 Server Configuration" date="1200696015" name="server-config.wsdd" path="server-config.wsdd" size="14671" user="Main.RamiVanguri" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Load 1K" date="1200614420" name="uaf-1-load.png" path="uaf-1-load.png" size="25876" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW5 Load 2kx2 Test 2" date="1200627926" name="osg-gw-5-load-2kx2-2.png" path="osg-gw-5-load-2kx2-2.png" size="37049" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Comparison Condor Load WS vs Pre WS (long view)" date="1200641467" name="comparisonload.png" path="comparisonload.png" size="34830" user="Main.TerrenceMartin" version="1"

Revision 42008/01/18 - Main.TerrenceMartin

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

WS Gram Testing

Line: 98 to 98
 

Pre-WS GRAM Comparision Test 2000 Jobs Submitted from 2 Submitters (4K Jobs)

Changed:
<
<
  • Of particular note is that rate of submission of Pre-WS GRAM jobs is approximated 1Hz. (1380/1356 jobs ~= 1.0Hz)
>
>
  • Rate of submission of Pre-WS GRAM jobs is approximated 1Hz. (1380/1356 jobs ~= 1.0Hz)
 
Added:
>
>
  • No jobs were held

  • Overall load was not problematic on the gatekeeper, and not significant on the submitter

  • Comparison Condor Load WS vs Pre WS:
    comparisonload-2.png

  • Comparison Condor Load WS vs Pre WS (long view):
    comparisonload.png

  • Final Pre WS Condor Load:
    FinalLoad.png

  • Final Pre WS System Load:
    finalsystemload.png

  • UAF 1 System Load Pre WS:
    uaf-1loadprews.png

  • UAF 1 System Mem Pre WS:
    uaf-1memprews.png

  • UAF 2 System Load Pre WS:
    uaf-2loadprews.png

  • UAF 2 System Mem Pre WS:
    uaf-2memprews.png
 

Notes

Line: 155 to 182
 
Deleted:
<
<
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem WS 2K x 2" date="1200622736" name="osg-5-mem-2kx2.png" path="osg-5-mem-2kx2.png" size="29864" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Network 1K" date="1200614396" name="gw5-network.png" path="gw5-network.png" size="25174" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Final Pre WS System Load" date="1200641519" name="finalsystemload.png" path="finalsystemload.png" size="33398" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load WS 2K x 2" date="1200622695" name="osg-5-condor-load-2kx2.png" path="osg-5-condor-load-2kx2.png" size="31226" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Mem 1K" date="1200614443" name="uaf-1-mem.png" path="uaf-1-mem.png" size="24084" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 2K" date="1200614817" name="osg-5-load-2K.png" path="osg-5-load-2K.png" size="32182" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load WS 2K x 2" date="1200622781" name="uaf-1-load-2kx2.png" path="uaf-1-load-2kx2.png" size="33090" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Memory 1K" date="1200614548" name="gw5-mem.png" path="gw5-mem.png" size="24426" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Comparison Condor Load WS vs Pre WS" date="1200641443" name="comparisonload-2.png" path="comparisonload-2.png" size="31656" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load WS 2K x 2" date="1200622716" name="osg-5-load-2kx2.png" path="osg-5-load-2kx2.png" size="33458" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load WS 2K x 2" date="1200622675" name="gums-load-2kx2.png" path="gums-load-2kx2.png" size="34877" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 System Load Pre WS" date="1200641545" name="uaf-1loadprews.png" path="uaf-1loadprews.png" size="31614" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Final Pre WS Condor Load" date="1200641492" name="FinalLoad.png" path="FinalLoad.png" size="33047" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Condor Load 2kx2 WS GRAM Test 2" date="1200627888" name="osg-gw-5-CondorLoad-2kx2-2.png" path="osg-gw-5-CondorLoad-2kx2-2.png" size="31404" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 2 System Mem Pre WS" date="1200641617" name="uaf-2memprews.png" path="uaf-2memprews.png" size="27133" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 System Mem Pre WS" date="1200641568" name="uaf-1memprews.png" path="uaf-1memprews.png" size="27563" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load 2K" date="1200614946" name="uaf-1-load-2k.png" path="uaf-1-load-2k.png" size="31368" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load 2K" date="1200614808" name="osg-5-condor-load-2K.png" path="osg-5-condor-load-2K.png" size="30651" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 1K" date="1200614220" name="gw5-load.png" path="gw5-load.png" size="27977" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Load 1K" date="1200614420" name="uaf-1-load.png" path="uaf-1-load.png" size="25876" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW5 Load 2kx2 Test 2" date="1200627926" name="osg-gw-5-load-2kx2-2.png" path="osg-gw-5-load-2kx2-2.png" size="37049" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Comparison Condor Load WS vs Pre WS (long view)" date="1200641467" name="comparisonload.png" path="comparisonload.png" size="34830" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 2 Load WS 2K x 2" date="1200622804" name="uaf-2-load-2kx2.png" path="uaf-2-load-2kx2.png" size="32033" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 2 System Load Pre WS" date="1200641591" name="uaf-2loadprews.png" path="uaf-2loadprews.png" size="36110" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Memory 2K" date="1200614956" name="uaf-1-mem-2k.png" path="uaf-1-mem-2k.png" size="27913" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem 2K" date="1200614827" name="osg-5-mem-2k.png" path="osg-5-mem-2k.png" size="29810" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load 2K" date="1200615000" name="Gums-Load-2K.png" path="Gums-Load-2K.png" size="35674" user="Main.TerrenceMartin" version="1"

Revision 32008/01/18 - Main.TerrenceMartin

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

WS Gram Testing

Line: 83 to 83
  uaf-2-load-2kx2.png
Added:
>
>

Followup WS GRAM 2000 x 2 submitters 5%+ Hold result

This test resulted in a greater than 5% job hold rate and excessive gatekeeper load. Errors included problems with authentication.

  • Condor Load 2kx2 WS GRAM Test 2:
    osg-gw-5-CondorLoad-2kx2-2.png

  • OSG GW5 Load 2kx2 Test 2:
    osg-gw-5-load-2kx2-2.png

  • OSG GW5 Mem 2kx2 Test 2:
    osg-gw-5-mem-2kx2-2.png
 

Pre-WS GRAM Comparision Test 2000 Jobs Submitted from 2 Submitters (4K Jobs)

Added:
>
>
  • Of particular note is that rate of submission of Pre-WS GRAM jobs is approximated 1Hz. (1380/1356 jobs ~= 1.0Hz)
 

Notes

  • With increasing jobs queued (within acceptable levels for pre-ws gram) about 5% of jobs become held for one reason or another. Configuration changes to timeouts have reduced the variety of timeouts.
Changed:
<
<
  • One of the more recent tests (2K x 1 submitter) resulted in 0 held jobs, however there are two jobs that never even started. A hold and release cycle got them started again. A 2k x 2 submitter test needs to be done to follow up.
>
>
  • One of the more recent tests (2K x 1 submitter) resulted in 0 held jobs, however there are two jobs that never even started. A hold and release cycle got them started again.

  • In a followup 2k x 2 submitter test about 2 hours in 5% of the jobs (204/4000) had gone into various hold states.

HoldReason = "Globus error: GT4_GRAM_JOB_SUBMIT timed out"
...
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
...
HoldReason = "Globus error: GT4_GRAM_JOB_SUBMIT timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_SUBMIT timed out"
HoldReason = "Globus error: GT4_GRAM_JOB_SUBMIT timed out"
LastHoldReason = "Spooling input data files"
HoldReason = "Globus error 155: the job manager could not stage out a file"
HoldReason = "Globus error 155: the job manager could not stage out a file"
HoldReason = "Globus error 155: the job manager could not stage out a file"
HoldReason = "Globus error 155: the job manager could not stage out a file"
...
HoldReason = "Globus error: org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException: \"/DC=org/DC=doegrids/OU=People/CN=Terr
ence Martin 525658\" is not authorized to use operation: {http://www.globus.org/namespaces/2004/10/gram/job}createManagedJob on this service"
HoldReason = "Globus error: GT4_GRAM_JOB_DESTROY timed out"
HoldReason = "Globus error: org.globus.wsrf.impl.security.authorization.exceptions.AuthorizationException: \"/DC=org/DC=doegrids/OU=People/CN=Terr
ence Martin 525658\" is not authorized to use operation: {http://www.globus.org/namespaces/2004/10/gram/job}createManagedJob on this service"
 
  • Configuration changes to the submitter have resulted in Gridmanager java related process growing to greater than 1.1GB of used memory size. Each user that submits from a specific host will have their own java process. This is just the current largest Java process size. It may be necessary to allow Java to consume even more memory if > 2K jobs are submitted.
Line: 122 to 165
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Memory 1K" date="1200614548" name="gw5-mem.png" path="gw5-mem.png" size="24426" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load WS 2K x 2" date="1200622716" name="osg-5-load-2kx2.png" path="osg-5-load-2kx2.png" size="33458" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load WS 2K x 2" date="1200622675" name="gums-load-2kx2.png" path="gums-load-2kx2.png" size="34877" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Condor Load 2kx2 WS GRAM Test 2" date="1200627888" name="osg-gw-5-CondorLoad-2kx2-2.png" path="osg-gw-5-CondorLoad-2kx2-2.png" size="31404" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load 2K" date="1200614946" name="uaf-1-load-2k.png" path="uaf-1-load-2k.png" size="31368" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load 2K" date="1200614808" name="osg-5-condor-load-2K.png" path="osg-5-condor-load-2K.png" size="30651" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 1K" date="1200614220" name="gw5-load.png" path="gw5-load.png" size="27977" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Load 1K" date="1200614420" name="uaf-1-load.png" path="uaf-1-load.png" size="25876" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW5 Load 2kx2 Test 2" date="1200627926" name="osg-gw-5-load-2kx2-2.png" path="osg-gw-5-load-2kx2-2.png" size="37049" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 2 Load WS 2K x 2" date="1200622804" name="uaf-2-load-2kx2.png" path="uaf-2-load-2kx2.png" size="32033" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Memory 2K" date="1200614956" name="uaf-1-mem-2k.png" path="uaf-1-mem-2k.png" size="27913" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem 2K" date="1200614827" name="osg-5-mem-2k.png" path="osg-5-mem-2k.png" size="29810" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load 2K" date="1200615000" name="Gums-Load-2K.png" path="Gums-Load-2K.png" size="35674" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW5 Mem 2kx2 Test 2" date="1200627955" name="osg-gw-5-mem-2kx2-2.png" path="osg-gw-5-mem-2kx2-2.png" size="30078" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 CE 1K" date="1200614144" name="gw5-condor.png" path="gw5-condor.png" size="27117" user="Main.TerrenceMartin" version="1"

Revision 22008/01/18 - Main.TerrenceMartin

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

WS Gram Testing

Line: 59 to 59
  Gums-Load-2K.png
Added:
>
>

WS GRam 2K x 2 Submitters

Significant amount of these jobs went into a hold state

  • GUMS Load WS 2K x 2:
    gums-load-2kx2.png

  • OSG 5 Condor Load WS 2K x 2:
    osg-5-condor-load-2kx2.png

  • OSG 5 Load WS 2K x 2:
    osg-5-load-2kx2.png

  • OSG 5 Mem WS 2K x 2:
    osg-5-mem-2kx2.png

  • UAF 1 Load WS 2K x 2:
    uaf-1-load-2kx2.png

  • UAF 2 Load WS 2K x 2:
    uaf-2-load-2kx2.png
 

Pre-WS GRAM Comparision Test 2000 Jobs Submitted from 2 Submitters (4K Jobs)

Line: 86 to 110
 -- TerrenceMartin - 17 Jan 2008
Added:
>
>

META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem WS 2K x 2" date="1200622736" name="osg-5-mem-2kx2.png" path="osg-5-mem-2kx2.png" size="29864" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Network 1K" date="1200614396" name="gw5-network.png" path="gw5-network.png" size="25174" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load WS 2K x 2" date="1200622695" name="osg-5-condor-load-2kx2.png" path="osg-5-condor-load-2kx2.png" size="31226" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Mem 1K" date="1200614443" name="uaf-1-mem.png" path="uaf-1-mem.png" size="24084" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 2K" date="1200614817" name="osg-5-load-2K.png" path="osg-5-load-2K.png" size="32182" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load WS 2K x 2" date="1200622781" name="uaf-1-load-2kx2.png" path="uaf-1-load-2kx2.png" size="33090" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Memory 1K" date="1200614548" name="gw5-mem.png" path="gw5-mem.png" size="24426" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load WS 2K x 2" date="1200622716" name="osg-5-load-2kx2.png" path="osg-5-load-2kx2.png" size="33458" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load WS 2K x 2" date="1200622675" name="gums-load-2kx2.png" path="gums-load-2kx2.png" size="34877" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load 2K" date="1200614946" name="uaf-1-load-2k.png" path="uaf-1-load-2k.png" size="31368" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load 2K" date="1200614808" name="osg-5-condor-load-2K.png" path="osg-5-condor-load-2K.png" size="30651" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 1K" date="1200614220" name="gw5-load.png" path="gw5-load.png" size="27977" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Load 1K" date="1200614420" name="uaf-1-load.png" path="uaf-1-load.png" size="25876" user="Main.TerrenceMartin" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 2 Load WS 2K x 2" date="1200622804" name="uaf-2-load-2kx2.png" path="uaf-2-load-2kx2.png" size="32033" user="Main.TerrenceMartin" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Memory 2K" date="1200614956" name="uaf-1-mem-2k.png" path="uaf-1-mem-2k.png" size="27913" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem 2K" date="1200614827" name="osg-5-mem-2k.png" path="osg-5-mem-2k.png" size="29810" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load 2K" date="1200615000" name="Gums-Load-2K.png" path="Gums-Load-2K.png" size="35674" user="Main.TerrenceMartin" version="1"

Revision 12008/01/18 - Main.TerrenceMartin

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

WS Gram Testing

Table of Contents

Introduction

The following is results from UCSD WS Gram testing with Pre-WS Gram for comparision.

Tests

WS GRAM Test 1000 Jobs Submitted from 1 Submitter (1K total Jobs)

This test is WS Gram All jobs finished but there were some held jobs. Probably in the 5% range.

  • OSG GW 5 CE:
    gw5-condor.png

  • OSG 5 Load 1K:
    gw5-load.png

  • OSG GW 5 Network 1K:
    gw5-network.png

  • OSG GW 5 Memory 1K:
    gw5-mem.png

  • UAF Load 1K:
    uaf-1-load.png

  • UAF Mem 1K:
    uaf-1-mem.png

WS GRAM Test 2000 Jobs Submitted from 1 Submitters (2K total jobs)

This test showed a higher count of held jobs. Other notable discoveries was that submission of addition trivial amounts of WS GRAM jobs, even targeting other CE were delayed by up to 1 hour.

  • OSG 5 Condor Load 2K:
    osg-5-condor-load-2K.png

  • OSG 5 Load 2K:
    osg-5-load-2K.png

  • OSG 5 Mem 2K:
    osg-5-mem-2k.png

  • UAF 1 Load 2K:
    uaf-1-load-2k.png

  • UAF 1 Memory 2K:
    uaf-1-mem-2k.png

  • GUMS Load 2K:
    Gums-Load-2K.png

Pre-WS GRAM Comparision Test 2000 Jobs Submitted from 2 Submitters (4K Jobs)

Notes

  • With increasing jobs queued (within acceptable levels for pre-ws gram) about 5% of jobs become held for one reason or another. Configuration changes to timeouts have reduced the variety of timeouts.

  • One of the more recent tests (2K x 1 submitter) resulted in 0 held jobs, however there are two jobs that never even started. A hold and release cycle got them started again. A 2k x 2 submitter test needs to be done to follow up.

  • Configuration changes to the submitter have resulted in Gridmanager java related process growing to greater than 1.1GB of used memory size. Each user that submits from a specific host will have their own java process. This is just the current largest Java process size. It may be necessary to allow Java to consume even more memory if > 2K jobs are submitted.

  • CPU Load on the gatekeeper that is recieving WSGram jobs is similar to the load on the gatekeeper that is recieving Pre-WS Gram jobs

  • CPU Load on the submitter is much higher with WS Gram than Pre-WS Gram.

  • GUMS server load is slightly higher with WS GRAM than Pre WS Gram

  • Removal of larger amounts of submitted jobs via Condor_rm (as low as 1K) is not possible without manual intervention on the submitter and gatekeeper. WS Gram fails to properly remove the jobs from the gatekeeper which requires that the persisted directory on the gatekeeper be manually purged. This is a known WS Gram limitation that is apparently fixed in a more recent version.

  • Pretty slow ramp up time. 20 minutes to start 76 jobs of 2K in one test. ~15s/job

-- TerrenceMartin - 17 Jan 2008

META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Network 1K" date="1200614396" name="gw5-network.png" path="gw5-network.png" size="25174" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Mem 1K" date="1200614443" name="uaf-1-mem.png" path="uaf-1-mem.png" size="24084" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 2K" date="1200614817" name="osg-5-load-2K.png" path="osg-5-load-2K.png" size="32182" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 Memory 1K" date="1200614548" name="gw5-mem.png" path="gw5-mem.png" size="24426" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Load 2K" date="1200614946" name="uaf-1-load-2k.png" path="uaf-1-load-2k.png" size="31368" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Condor Load 2K" date="1200614808" name="osg-5-condor-load-2K.png" path="osg-5-condor-load-2K.png" size="30651" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Load 1K" date="1200614220" name="gw5-load.png" path="gw5-load.png" size="27977" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF Load 1K" date="1200614420" name="uaf-1-load.png" path="uaf-1-load.png" size="25876" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="UAF 1 Memory 2K" date="1200614956" name="uaf-1-mem-2k.png" path="uaf-1-mem-2k.png" size="27913" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG 5 Mem 2K" date="1200614827" name="osg-5-mem-2k.png" path="osg-5-mem-2k.png" size="29810" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="GUMS Load 2K" date="1200615000" name="Gums-Load-2K.png" path="Gums-Load-2K.png" size="35674" user="Main.TerrenceMartin" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="OSG GW 5 CE 1K" date="1200614144" name="gw5-condor.png" path="gw5-condor.png" size="27117" user="Main.TerrenceMartin" version="1"
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback