Difference: GRAM5Test1 (11 vs. 12)

Revision 122009/12/21 - Main.IgorSfiligoi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

GRAM5 tests

Line: 9 to 9
 Jeff and Igor did a first round of tests to check if GRAM5 (alpha2) indeed worked and how it compared to GT2 on a small scale.

Changed:
<
<

GRAM 5 benchmarking

>
>

GRAM 5 alpha benchmarking

  GRAM5 alpha2 was installed on osg-gw-5.t2.ucsd.edu.
  • File transfer is not working properly. All further tests are without input or output files.
Line: 38 to 38
 Steps:
  • copy over etc/gridftp.conf
  • create var/log
Changed:
<
<
  • patch lib/perl/Globus/GRAM/JobManager/condor.pm, to enable file transfer
    298,299c298,299
    < #$requirements = "OpSys = \"" . $description->condor_os() . "\" ";
    < #$requirements .
    " && Arch = \"" . $description->condor_arch() . "\" ";
    ---
    > $requirements = "OpSys
    \"" . $description->condor_os() . "\" ";
    > $requirements .
    " && Arch == \"" . $description->condor_arch() . "\" ";
    318,347d317
    <
    < ####################
    < # add input files
    < my @flist;
    <
    < my $sdir;
    < opendir($sdir,$description->directory());
    < my @sfiles = grep { !/^\./} readdir($sdir);
    < close $sdir;
    <
    < foreach $f ( @sfiles ) {
    < my $fpath = $description->directory() . "/" . $f;
    < if (-d $fpath) {
    < # do nothing
    < } else {
    < my $age = -M $fpath;
    < { #if ( $age < $0.01 ) {
    < # protection for when the dir is shared with other jobs
    < # get only recent files
    < $fpath =~ s{\/\/}{\/}g;
    < push (@flist,"$fpath");
    < }
    < }
    < }
    <
    < print SCRIPT_FILE "transfer_input_files = " . join(",",@flist) . "\n";
    < print SCRIPT_FILE "should_transfer_files = YES\n";
    < print SCRIPT_FILE "when_to_transfer_output = ON_EXIT\n";
    < ###################
    <
>
>
  • patch lib/perl/Globus/GRAM/JobManager/condor.pm, to enable file transfer
    298,299c298,299
    < #$requirements = "OpSys = \"" . $description->condor_os() . "\" ";
    < #$requirements .
    " && Arch = \"" . $description->condor_arch() . "\" ";
    ---
    > $requirements = "OpSys \"" . $description->condor_os() . "\" ";
    > $requirements .
    " && Arch == \"" . $description->condor_arch() . "\" ";
    318,347d317
    <
    < ####################
    < # add input files
    < my @flist;
    <
    < my $sdir;
    < opendir($sdir,$description->directory());
    < my @sfiles = grep { !/^\./} readdir($sdir);
    < close $sdir;
    <
    < foreach $f ( @sfiles ) {
    < my $fpath = $description->directory() . "/" . $f;
    < if (-d $fpath) {
    < # do nothing
    < } else {
    < my $age = -M $fpath;
    < { #if ( $age < $0.01 ) {
    < # protection for when the dir is shared with other jobs
    < # get only recent files
    < $fpath =~ s{\/\/}{\/}g;
    < push (@flist,"$fpath");
    < }
    < }
    < }
    <
    < print SCRIPT_FILE "transfer_input_files = " . join(",",@flist) . "\n";
    < print SCRIPT_FILE "should_transfer_files = YES\n";
    < print SCRIPT_FILE "when_to_transfer_output = ON_EXIT\n";
    < ###################
    <
 
  • add -seg-module condor to etc/grid-services/jobmanager-condor
  • Start
    $GLOBUS_LOCATION/sbin/globus-job-manager-event-generator \
    -scheduler $lrm \
    -background \
    -pidfile $GLOBUS_LOCATION/var/seg-$lrm.pid
Added:
>
>

GRAM 5 RC benchmarking

GRAM5 rc was installed on osg-gw-5.t2.ucsd.edu.

  • No major problems found anymore... Condor-G timed out on a few jobs putting them on hold... likely a Condor-G problem
  • The benchmarking seemed limited by the submit machine CPU (uaf-1); the jobs were submittied to GRAM5 as fast as jobs were being submitted to the client's schedd
  • Rates of up to ~2Hz (100 jobs/min) were observed
  • Detailed numbers can be found at rates_gram5rc_ucsd.pdf

Backward compatibility, using Condor-G's gt2 mode:

  • everything worked, but only 10 jobs at a time were in the CE queue
  • enough to allow casual use ofa GRAM5 CE, but condor-G 7.4+with its gt5 mode is NEEDED to useon large scale
  -- IgorSfiligoi - 2009/08/12
Line: 50 to 61
 
META FILEATTACHMENT attachment="rates_gt5a2_ucsd.odt" attr="" comment="GRAM5 alpha2 benchmarks - UCSD LAN" date="1251142757" name="rates_gt5a2_ucsd.odt" path="rates_gt5a2_ucsd.odt" size="84723" stream="rates_gt5a2_ucsd.odt" tmpFilename="/tmp/F7pHLKSE7v" user="IgorSfiligoi" version="1"
META FILEATTACHMENT attachment="rates_gt5a2_ucsd_r2.pdf" attr="" comment="GRAM5 alpha2 benchmarks - Italy to UCSD" date="1251236990" name="rates_gt5a2_ucsd_r2.pdf" path="rates_gt5a2_ucsd_r2.pdf" size="542342" stream="rates_gt5a2_ucsd_r2.pdf" tmpFilename="/tmp/iLC4JpJHtM" user="IgorSfiligoi" version="1"
META FILEATTACHMENT attachment="rates_gt5a2_ucsd_r2.odt" attr="" comment="GRAM5 alpha2 benchmarks - Italy to UCSD" date="1251237000" name="rates_gt5a2_ucsd_r2.odt" path="rates_gt5a2_ucsd_r2.odt" size="85702" stream="rates_gt5a2_ucsd_r2.odt" tmpFilename="/tmp/fQc4gmN6cA" user="IgorSfiligoi" version="1"
Added:
>
>
META FILEATTACHMENT attachment="rates_gram5rc_ucsd.odt" attr="" comment="GRAM5 rc benchmarks - UCSD LAN" date="1261417592" name="rates_gram5rc_ucsd.odt" path="rates_gram5rc_ucsd.odt" size="52127" stream="rates_gram5rc_ucsd.odt" tmpFilename="/tmp/IyVYPNf9T8" user="IgorSfiligoi" version="1"
META FILEATTACHMENT attachment="rates_gram5rc_ucsd.pdf" attr="" comment="GRAM5 rc benchmarks - UCSD LAN" date="1261417660" name="rates_gram5rc_ucsd.pdf" path="rates_gram5rc_ucsd.pdf" size="504288" stream="rates_gram5rc_ucsd.pdf" tmpFilename="/tmp/ENXExzpirE" user="IgorSfiligoi" version="1"
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback