• For detailed results, see bench_30min_10k.ods.
  
  

OSG use of GRAM5

OSG is planning in officially suporting GRAM5 in the near term.

The first step in the process is represented by this planning document.

• Ran 10k 30 minute jobs against both GT2 and GRAM5 on osg-gw-5, using Condor-G v7.4.1 on uaf-1
• GRAM5 finished in about 6 hours, GT2 took about 9 hours.
• Condor-G numbers were much closer to reality under GRAM5 than under GT2.
  • Under GRAM5 Condor-G noticed the termination of the last file close to the 6 hour mark, although it was not always that close during the run.
  • Under GT2, Condor-G thought jobs were running almost 14 hours.
• For detailed results, see bench_30min_10k.ods.
  
  

GRAM 5 alpha benchmarking

• patch lib/perl/Globus/GRAM/JobManager/condor.pm, to enable file transfer
  298,299c298,299
  < #$requirements = "OpSys = \"" . $description->condor_os() . "\" ";
< #$requirements . " && Arch = \"" . $description->condor_arch() . "\" ";
---
> $requirements = "OpSys \"" . $description->condor_os() . "\" ";
> $requirements . " && Arch == \"" . $description->condor_arch() . "\" ";
  318,347d317
  <
  < ####################
  < # add input files
  < my @flist;
  <
  < my $sdir;
  < opendir($sdir,$description->directory());
  < my @sfiles = grep { !/^\./} readdir($sdir);
  < close $sdir;
  <
  < foreach $f ( @sfiles ) {
  < my $fpath = $description->directory() . "/" . $f;
  < if (-d $fpath) {
  < # do nothing
  < } else {
  < my $age = -M $fpath;
  < { #if ( $age < $0.01 ) {
  < # protection for when the dir is shared with other jobs
  < # get only recent files
  < $fpath =~ s{\/\/}{\/}g;
  < push (@flist,"$fpath");
  < }
  < }
  < }
  <
  < print SCRIPT_FILE "transfer_input_files = " . join(",",@flist) . "\n";
  < print SCRIPT_FILE "should_transfer_files = YES\n";
  < print SCRIPT_FILE "when_to_transfer_output = ON_EXIT\n";
  < ###################
  <
  

GRAM 5 RC benchmarking

GRAM5 rc was installed on osg-gw-5.t2.ucsd.edu.

• No major problems found anymore... Condor-G timed out on a few jobs putting them on hold... likely a Condor-G problem
• The benchmarking seemed limited by the submit machine CPU (uaf-1); the jobs were submittied to GRAM5 as fast as jobs were being submitted to the client's schedd
• Rates of up to ~2Hz (100 jobs/min) were observed
• Detailed numbers can be found at rates_gram5rc_ucsd.pdf

Backward compatibility, using Condor-G's gt2 mode:

• everything worked, but only 10 jobs at a time were in the CE queue
• enough to allow casual use ofa GRAM5 CE, but condor-G 7.4+with its gt5 mode is NEEDED to useon large scale

• patch lib/perl/Globus/GRAM/JobManager/condor.pm, to enable file transfer
  298,299c298,299
  < #$requirements = "OpSys = \"" . $description->condor_os() . "\" ";
< #$requirements . " && Arch = \"" . $description->condor_arch() . "\" ";
  ---
  > $requirements = "OpSys \"" . $description->condor_os() . "\" ";
> $requirements . " && Arch == \"" . $description->condor_arch() . "\" ";
  318,347d317
  <
  < ####################
  < # add input files
  < my @flist;
  <
  < my $sdir;
  < opendir($sdir,$description->directory());
  < my @sfiles = grep { !/^\./} readdir($sdir);
  < close $sdir;
  <
  < foreach $f ( @sfiles ) {
  < my $fpath = $description->directory() . "/" . $f;
  < if (-d $fpath) {
  < # do nothing
  < } else {
  < my $age = -M $fpath;
  < { #if ( $age < $0.01 ) {
  < # protection for when the dir is shared with other jobs
  < # get only recent files
  < $fpath =~ s{\/\/}{\/}g;
  < push (@flist,"$fpath");
  < }
  < }
  < }
  <
  < print SCRIPT_FILE "transfer_input_files = " . join(",",@flist) . "\n";
  < print SCRIPT_FILE "should_transfer_files = YES\n";
  < print SCRIPT_FILE "when_to_transfer_output = ON_EXIT\n";
  < ###################
  <
  
• add -seg-module condor to etc/grid-services/jobmanager-condor
• Start
  

  $GLOBUS_LOCATION/sbin/globus-job-manager-event-generator \
   -scheduler $lrm \
   -background \
   -pidfile $GLOBUS_LOCATION/var/seg-$lrm.pid
  

GRAM5 beta2

Just installed GRAM2 beta2 on Nov 25th.

Steps:

• copy over etc/gridftp.conf
• create var/log
• copy over the patched lib/perl/Globus/GRAM/JobManager/condor.pm

• copied VDT $GLOBUS_LOCATION/etc/gridftp.cong
• created $GLOBUS_LOCATION/var/log (used by gridFTP)

OSG jobmanager-condor

OSG uses a patched/augmented version jobmanager-condor (condor.in->condor.pm). The jobmanager-condor changed slightly between GT2 and GRAM5, so we had to merge the two.

The work was performed by Christopher Theissen, and is documented at here.

• The initial install used the VDT perl, and this created some problems. Make sure you use the system perl.
  Configured in globus.v5b1/libexec/globus-sh-tools-vars.sh

GRAM 5 beta 1

Installation

While up to alpha3, GRAM5 played well on top of a previous installation of a 1.10.1 VDT, beta1 needs some changes:

• gridfrtp needs and additional parameter (-i).
  Need to change $VDT/vdt/services/vdt-run-gsiftp.sh and add -i
  #!/bin/sh

. /osglocal/osgce/setup.sh
exec /osglocal/osgce/globus/sbin/globus-gridftp-server -i -c /osglocal/osgce/globus/etc/gridftp.conf
• The gass cache needs an explicit definition of PATH, or it will not work:
  [0939] root@osg-gw-5 /osglocal/osgce/globus/bin# cat globus-gass-cache
#!/bin/bash

export PATH=/usr/bin

exec /osglocal/osgce/globus/bin/globus-gass-cache.org "$@"
• jobmanager-fork is not the default anymore; one needs to change the symlink in globus/etc/grid-services

• Submitting from a machine on the other side of the world, the job turnaround is about 17 jobs/min. Peaks of 30 jobs/min has also been observed.
  If monitoring is not needed, 200 jobs/min (3Hz) sustained rate should be acchievable.
  For more details: rates_gt5a2_ucsd_r2.pdf
  Note: No clear conclusions, yet... I have found that GRAM5 gets slower in time, so the results may be affected by that.

• Submitting from a machine on a local LAN, 1Hz (50 jobs/min) job turnaround is easy to acchieve. Peaks of 3Hz (170 jobs/min) has also been observed.
  If monitoring is not needed, 3Hz (200 jobs/min) sustained rate should be acchievable.
  For more details: rates_gt5a2_ucsd.pdf
• Submitting from a machine on the other side of the world, the job turnaround is about 17 jobs/min. Peaks of 30 jobs/min has also been observed.
  If monitoring is not needed, 200 jobs/min (3Hz) sustained rate should be acchievable.
  Network latency is obviously a major scalability limit.
  For more details: rates_gt5a2_ucsd_r2.pdf

GRAM 5 benchmarking

GRAM5 alpha2 was insatlled on osg-gw-5.t2.ucsd.edu.

• File transfer is not working properly. All further tests are without input or output files.
• Submitting from a machine on a local LAN, 1Hz (50 jobs/min) job turnaround is easy to acchieve. Peaks of 3Hz (170 jobs/min) has also been observed.
  If monitoring is not needed, 7Hz (400 jobs/min) is acchievable.
  For more details: rates_gt5a2_ucsd.pdf

GRAM5 tests

This page is meant to contain the information we gather while testing GRAM5.

GRAM5 validation tests

Jeff and Igor did a first round of tests to check if GRAM5 (alpha2) indeed worked and how it compared to GT2 on a small scale.

• gram5basetest.pdf: GRAM5 validation test
• gram5basetest.odt: GRAM5 validation test - source

-- IgorSfiligoi - 2009/08/12