Preamble support in the LRMS adapter

Contents

• Problem Statement and the Solution
• How the Solution is implemented
• Modifying the SGE adapter
• Modifying the PBS adapter
• Improving the idea
• Questions?

Problem Statement and the Solution

One often wants to execute some setup work before running an application. A typical example is setting the environment by way of the module command before running an MPI application.

The straightforward, but not elegant, way of doing that is to create a script, say mpijob.sh, which contains

    module load mpi
    mpirun -np 16 my_mpi_app

This has several disadvantages: (1) it requires to submit a script (myjob.sh) to the execution site id addition to the job description; (2) it requires hard-coding in the script the number of processes; (3) it forces the user to specfiy low-level and site-dependent information such as the MPI-launcher program, e.g., mpirun or mpiexec.

A better way is to include the setup work as part of the job description submitted to Globus, and to leave the MPI-launching mechanism to be handled by the execution site. That is, we would like something like

<job>
  <executable>my_mpi_app</executable>
  <directory>${GLOBUS_USER_HOME}</directory>
  ...
  <count>16</count>
  ...
  <jobType>mpi</jobType>
  <extensions>
    <preamble>
    module load mpi
    </preamble>
  </extensions>
</job>

How the Solution is implemented

To implement the solution, one has to insert in the LRMS adapter code that:

• checks if the job description contains the <preamble> element;
• if the job description contains the <preamble> element, then gets the text contents of this element from the job description and inserts it in the LRMS job script just before inserting the text related to the application (the <executable> and <directory> elements)

Next we show what changes need to be made to the SGE and PBS adapters.

Modifying the SGE adapter

To support the preamble for SGE, insert in the file $GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/sge.pm just before the lines

    $sge_job_script->print("# Change to directory requested by user\n");
    $sge_job_script->print('cd ' . $description->directory() . "\n");

these lines

    #
    # Insert the application preamble
    #
    if( $description->preamble() ) {
         $self->log("INFO: Preamble:\n");
         $self->log( $description->preamble() );
         $sge_job_script->print("\n# Preamble\n");
         $sge_job_script->print( $description->preamble() .  "\n\n");
    }

Modifying the PBS adapter

To support the preamble for PBS, insert in the file $GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/pbs.pm just before the lines

    $rsh_env = '';
    $library_vars{LD_LIBRARY_PATH} = 0;

these lines

    #
    # Insert the application preamble
    #
    if(  $description->preamble() ) {
        $self->log("INFO: Preamble:\n");
        $self->log( $description->preamble() );
        print JOB "\n# Preamble\n";
        print JOB  $description->preamble() .  "\n\n";
    }

Improving the idea

The idea can be used to implement a postamble, where one puts commands to be executed after the application runs, e.g., file cleanup or file copying.

Questions?

If you have questions, contact Gabriel Mateescu, mateescu@acm.org