Apr 2011

Condor 7.6.0, glideinWMS v2_5_1 (+minor patches), loadtest_condor 1.1 (+minor patches)

This time the test was about Negotiator scalability.

The test consisted in running jobs with 10k-15k glideins, once with a simple requirement, and once with a complex one that created one autocluster per job:

• simple:
  Requirements=True
• complex:
  Requirements = ( ( stringListMember(GLIDEIN_Site,string(ClusterId? )) || stringListMember(GLIDEIN_Gatekeeper,string(ClusterId? )) || (GLIDEIN_Fake=?=UNDEFINED)) && Arch=!Dummy) ) && ( ( Memory > 1 ) && ( Disk >= 1 )

Simple glidein start expression

Using a single sleeper pool, with a accept all Start condition, and with the negotiator limited to 20s per cycle
NEGOTIATOR_MAX_TIME_PER_SUBMITTER=40
NEGOTIATOR_MAX_TIME_PER_PIESPIN=20
the system behaved pretty much the same way with either the simpler or the complex job requirements.

However, looking closer to the Negotiator behavior, it is clear that most jobs don't get considered for matching;the NegotiatorLog? has a
Reached max time per spin: 20 ... stopping
line at the end of each cycle, and the Negotiator ClassAd? reports that only ~250 job, out of a total of ~1.5k have been considered for matchmaking:
LastNegotiationCycleNumJobsConsidered0 = 254
LastNegotiationCycleRejections0 = 234
LastNegotiationCycleNumIdleJobs0 = 1620
LastNegotiationCycleTotalSlots0 = 15081
LastNegotiationCycleCandidateSlots0 = 2002

Selective glidein start expression

To test how the above described behaviour affects the system (when there are many autoclusters), new glideins were configured to only access a subset of the jobs:
GLIDEIN_Entry_Start="(round(ClusterId? /10)*10==ClusterId)"

As expected, only the first ~20 idle jobs that matched (out of ~200) started running, even as there were plenty (>200) unclaimed glideins in the system.
The net result was both delayed job execution and wasted cpu cycles.

So this configuration is not really functional.

To correct for the above, the negotiator limits were commented out:
# NEGOTIATOR_MAX_TIME_PER_SUBMITTER=40
#NEGOTIATOR_MAX_TIME_PER_PIESPIN=20

All the deserving jobs thus started to run, but at the expense of the negotiator cycle time, which now increased to ~3 minutes (with 1.2k idle jobs in the queue) compared to ~50s it took before.