WS Gram Testing

Table of Contents

Introduction

The following is results from UCSD WS Gram testing with Pre-WS Gram for comparision.

Tests

WS GRAM Test 1000 Jobs Submitted from 1 Submitter (1K total Jobs)

This test is WS Gram All jobs finished but there were some held jobs. Probably in the 5% range.

• OSG GW 5 CE:
  

• OSG 5 Load 1K:
  

• OSG GW 5 Network 1K:
  

• OSG GW 5 Memory 1K:
  

• UAF Load 1K:
  

• UAF Mem 1K:
  

WS GRAM Test 2000 Jobs Submitted from 1 Submitters (2K total jobs)

This test showed a higher count of held jobs. Other notable discoveries was that submission of addition trivial amounts of WS GRAM jobs, even targeting other CE were delayed by up to 1 hour.

• OSG 5 Condor Load 2K:
  

• OSG 5 Load 2K:
  

• OSG 5 Mem 2K:
  

• UAF 1 Load 2K:
  

• UAF 1 Memory 2K:
  

• GUMS Load 2K:
  

WS GRam 2K x 2 Submitters

Significant amount of these jobs went into a hold state

• GUMS Load WS 2K x 2:
  

• OSG 5 Condor Load WS 2K x 2:
  

• OSG 5 Load WS 2K x 2:
  

• OSG 5 Mem WS 2K x 2:
  

• UAF 1 Load WS 2K x 2:
  

• UAF 2 Load WS 2K x 2:
  

Pre-WS GRAM Comparision Test 2000 Jobs Submitted from 2 Submitters (4K Jobs)

Notes

• With increasing jobs queued (within acceptable levels for pre-ws gram) about 5% of jobs become held for one reason or another. Configuration changes to timeouts have reduced the variety of timeouts.

• One of the more recent tests (2K x 1 submitter) resulted in 0 held jobs, however there are two jobs that never even started. A hold and release cycle got them started again. A 2k x 2 submitter test needs to be done to follow up.

• Configuration changes to the submitter have resulted in Gridmanager java related process growing to greater than 1.1GB of used memory size. Each user that submits from a specific host will have their own java process. This is just the current largest Java process size. It may be necessary to allow Java to consume even more memory if > 2K jobs are submitted.

• CPU Load on the gatekeeper that is recieving WSGram jobs is similar to the load on the gatekeeper that is recieving Pre-WS Gram jobs

• CPU Load on the submitter is much higher with WS Gram than Pre-WS Gram.

• GUMS server load is slightly higher with WS GRAM than Pre WS Gram

• Removal of larger amounts of submitted jobs via Condor_rm (as low as 1K) is not possible without manual intervention on the submitter and gatekeeper. WS Gram fails to properly remove the jobs from the gatekeeper which requires that the persisted directory on the gatekeeper be manually purged. This is a known WS Gram limitation that is apparently fixed in a more recent version.

• Pretty slow ramp up time. 20 minutes to start 76 jobs of 2K in one test. ~15s/job

-- TerrenceMartin - 17 Jan 2008