Code Documentation Guidelines

• Try to document your code so we can easily generate API documentation using epydoc
• Refer to http://epydoc.sourceforge.net/epytext.html for EpyText? documentation
• You can refer to a simple example below. Refer to epydoc example for details

 """
 Return the x intercept of the line M{y=m*x+b}. The X{x intercept}
 of a line is the point at which it crosses the x axis (M{y=0}).

 This function can be used in conjuction with L{z_transform} to
 find an arbitrary function's zeros.

 @type m: number
 @param m: The slope of the line.
 @type b: number
 @param b: The y intercept of the line. The X{y intercept} of a
 line is the point at which it crosses the y axis (M{x=0}).
 @rtype: number
 @return: the x intercept of the line M{y=m*x+b}.
 """
 return -b/m

• Add following to your ~/.ssh/config

   Host cdcvs.fnal.gov
   ForwardX11 = no
   GSSAPIAuthentication yes
   GSSAPIDelegateCredentials yes
   

• Checkout the release manager code

  svn checkout svn+ssh://p-gwms-release-manager@cdcvs.fnal.gov/cvs/projects/gwms-release-manager

• Create the checksum files and release tarballs for your release. Run the command below giving the release version, --source-dir=<path to glideinwms code> and --release-dir=<path to create release related files>. Release tarballs will be created in <path to create release related files>/v2_5_1/

   # For example for release v2_5_1 run the command below
   gwms-release-manager/trunk/bin/make_release.py --release-version=v2_5_1 --source-dir=/tmp/release/glideinWMS --release-dir=/tmp/release/distro
   

svn checkout svn+ssh://p-gwms-release-manager@cdcvs.fnal.gov/cvs/projects/gwms-release-manager

Instructions for release v2_5 and later

Starting glideinWMS v2_5, glideinWMS releases are managed by gwms_release_manager package. Release manager package is in Fermi Redmine

gwms_release_manager automates following tasks for you

• Creates the required checksum files in glideinWMS/etc directory
• Strips off the non-essential files and creates 3 release tarballs (factory, frontend and full glideinWMS release).
• Anything you want to be automated during the release process should go in the release manager

To get access to the release manager:

• Register with Fermi Redmine
• Check Parag or Burt to get access to the release manager code repository
• Add following to your ~/.ssh/config

       Host cdcvs.fnal.gov
           ForwardX11 = no
           GSSAPIAuthentication yes
           GSSAPIDelegateCredentials yes
       

• Get a Kerberos principle (kinit)
• Checkout the release manager code

  svn checkout svn+ssh://p-gwms-release-manager@cdcvs.fnal.gov/cvs/projects/gwms-release-manager/branches

To make a release:

• Make the changes and test them.
• Update the tags file with the release notes and release version (glideinWMS/doc/tags.txt)
• Update the documentation pages glideinWMS/doc/download.html and glideinWMS/doc/history.html including possible links to be release tarballs
• Create the checksum files and release tarballs for your release. Run the command below giving the release version, --source-dir=<path to glideinwms code> and --release-dir=<path to create release related files>. Release tarballs will be created in <path to create release related files>/v2_5_1/

       # For example for release v2_5_1 run the command below
       gwms-release-manager/trunk/bin/make_release.py --release-version=v2_5_1 --source-dir=/tmp/release/glideinWMS --release-dir=/tmp/release/distro
       

• Commit the changes to CVS.
• Tag the release in CVS
• Copy the release tarballs to the web area
• Check out the doc subdirectory in the web area.
• Create the required symlinks to latest docs directory.
• Announce the release to following mailing lists: osg-wms@opensciencegrid.org, cms-dct-wms@fnal.gov, glideinwms-support@fnal.gov, corralwms@fnal.gov
• If you made any changes to the release manager to accommodate the release, commit those changes.
• Tag the gwms_release_manager with same tag name as the glideinWMS release.

Instructions for release v2_4_x and earlier

cvs co -r v2_5_1 glideinWMS

• Update the download page and the history page with the required release information.

• Version v2+: Branch branch_v2plus. This is for now v2.5+and should be default for committing changes for v2.5 and above. To facilitate merging features for v2.5.1 until v2.5 is released you can either use a personal branch or use branch_v2_5plus_temp. Please don't use branch_v2_5plus_temp without asking Parag as this branch may go away soon once v2.5 is released.

Testing

Testing Framework

Testing Guidelines

Nightly NMI builds

Code Branching Guidelines

Code Formatting Guidelines

Testing Guidelines

• Follow Code TestingGuideLines?

Documentation Guidelines (Needs more work)

• Version v2+: Branch branch_v2plus. This is for now v2.5+and should be default for commiting changes for v2.4.3 and above.
• Version v2.4+: Branch branch_v2_4plus. Should be used in case we need to release v2.4.3+ before we release v2.5.

• Create and put the glideinWMS_<version>.tar.gz file in the web area

• Version v2.4+: Branch branch_v2_4plus. Should be used in case we need to release v2.4.2+ before we release v2.5.

• Set following environment variables before you checkout the code from CVS.

• Version v2+: Branch branch_v2plus. This is for now v2.5+and should be default for commiting changes for v2.4.1 and above.

• Version v2+: Branch branch_v2plus. This is for now v2.5+

• Version v2.4+: Branch branch_v2_4plus. Anything related to v2.4 should go here.

Production Branches

• Version v2.3+: Branch branch_v2_3plus. This was created since we wanted to release changes to v2.3 but we were not yet ready for v2.4

Non-Production Branches

• Branches not in the list above are development/testing branches owned by individual developers.

• Follow Code BranchingGuideLines
• Follow Code FormattingGuideLines

glideinWMS Project Developer's Home Page

Task List

Getting Started

Gaining access to the glideinWMS code

• One needs a valid Fermilab Kerberos principle and a krb5 ticket
• You need write access to the cdcvs CVS repository. If you don't, open a Servicedesk Ticket at http://servicedesk.fnal.gov requesting access to the cdcvs.
• You need write access to the glideinWMS CVS module. Check with one of the existing glideinWMS team members or email glideinwms-support@fnal.gov
• Set following environment variables before you checkout the code from CVS.

        export CVSROOT=cvsuser@cdcvs.fnal.gov:/cvs/cd
        export CVSRSH=ssh
     

Current CVS structure

• Version v2+: Branch branch_v2plus
• Version v3+: CVS Head

Making Changes to glideinWMS Code

• Follow BranchingGuideLines
• Follow FormattingGuideLines
• Document your changes. This is very important!
• Update the tags.txt file with notes on your recent changes. The information in this file act as release notes, so updating this file is very important.

Making a glideinWMS release

• Make the changes and test them.
• Update the tags file with the release notes and release version (glideinWMS/doc/tags.txt)
• Commit the changes to CVS.
• Tag the release in CVS
• Create and put the glideinWMS_.tar.gz file in the web area
• Check out the doc subdirectory in the web area.
• Create the required symlinks to latest docs directory.
• Update the main page, the download page and the history page with the required information.
• Announce the release to following mailing lists: osg-wms@opensciencegrid.org, cms-dct-wms@fnal.gov, glideinwms-support@fnal.gov

Recent code changes

Add SEC_TOOL_SESSION_DURATION = 60 to prevent daemon bloat.

Add better protections against failures during condor_q.

Fixed a small logical error as well.

Tagged it as snapshot_091214.

- IgorSfiligoi - 2009/12/07- head/frontend

Restructured glideinFrontendInterface, to make it a proper library module.

Removed any dependence of glideinFrontendLib in the process.

Restructured glideinFrontendInterface, to make it a proper library module.
Removed any dependence of glideinFrontendLib in the process.

Added a MultiAdvertizeWork class to glideinFrontendInterface so one can make optimizations in there without changing the interface (too much).

Several changes needed to glideinFrontendElement to use the changed interface.

Tagged it as snapshot_091207.

Change the default of

GLEXEC_JOB

from False to True. All new versions of condor require this to be true.

Frontend will now only communicate with trusted factories. This is acchieved by using AuthenticatedIdentity? attribute of the classads.

The frontend config needed to be changed to acchieve this. But the previous format was not very apt for this, as all the factory collectors were listed as a single string. So the new frontend config now has a proper XML tree for the factories:

<factory collectors="node1,node2"/>

<factory>
  <collectors>
    <collector node="node1" classad_identity="id1@node1"/>
    <collector node="node2" classad_identity="id2@domain2"/>
  </collectors>
</factory>

To keep the consistency between the factory collectors and job schedds, I also changed the schedds from a string to a XML tree:

<job schedds="node1,sched2@node2"/>

<job>
  <schedds>
    <schedd fullname="node1"/>
    <schedd fullname="schedd2@node2"/>
  </schedds>
</job>

The installer has been updated to properly create the new config.

Changesd protocol between frontend and factory.

Frontend classad added

 ReqEncIdentity

The factory check this and after decoding, compares it to AuthenticatedIdentity? . This prevents replay attacks

The frontend config changed to add this info:

 <security classad_identity="identity" classad_proxy="fname">

The factory, the frontend and the installer were modified.

The documentation was updated to reflect the change.

-- IgorSfiligoi - 2009/10/26 - head/install&creation/web_base

Found and fixed the GLEXEC problem.

I had enabled publish_glidein for GLEXEC_BIN. Since it is not quoted, it cannot be exported, and this created a lot of problems for Condor.

Task List

TaskList

Improved the proxy_url="OSG" handling in glidein_startup.sh.

It was way too simplistic and would fail on some OSG sites.

-- IgorSfiligoi - 2009/10/15 - head/creation

Add debug option to both create_glidein and create_frontend. Will print out the full traceback in case of problems.

Implemented support for multiple condor binaries.

The config changes from

<condor base_dir="/home/igorfactory/glidecondor"/>

to

 <condor_tarballs>
   <condor_tarball arch="x86" os="linux-rhel3" base_dir="/home/igorfactory/glidecondor_rhel3"/>
   <condor_tarball arch="x86" os="linux-rhel5" base_dir="/home/igorfactory/glidecondor_rhel5"/>
</condor_tarballs>

The factory must also define the gollowing attributes

<attr name="CONDOR_ARCH" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="x86"/>
<attr name="CONDOR_OS" const="True" glidein_publish="False" job_publish="False" parameter="True" publish="False" type="string" value="linux-rhel3"/>

If not defined, it defaults to "default". A value of "auto" for either or both of the above will enable auto-detection of that component.

The implementation creates one tarball per os/arch pair. The "condor_platform_select.sh" script will enable the downloading of the appropriate tarball at runtime.

The plan is to change also the installer and update the documentation before merging back into the head.

Add debug option to both create_glidein and create_frontend. Will print out the full traceback in case of problems.

Factory now only checks X509_USER_PROXY if it needs it.

The installer now asks for the collector port number. This allows several users to coexist on the same node.

Recent code changes

Made VDT optional. openssl used anywhere where grid-proxy-info was used before.
User can now provide its ownTRUSTED_CA dir and its own voms-proxy-info.

Addedd support for VDT 2.0, and made it the default.
Also added Globus-Client and Myproxy-Client in the minimal VDT install.

Developer access to the code

To access the code in read-write mode, one first needs a valid krb5 ticket and be explicitly authorized.
Once that is acchieved,
set CVSROOT to cvsuser@cdcvs.fnal.gov:/cvs/cd
and CVSRSH to ssh.

glideinWMS Project TWiki Home Page

This page should be used by the glideinWMS development team for the project related information.

-- IgorSfiligoi - 2009/09/29