middleware:Globus/421

Introduction

The Globus Toolkit, currently at version 4, is an open source toolkit for building computing grids developed and provided by the Globus Alliance.
Simplified Globus Toolkit 4 architecture The Globus Toolkit system consists of several components which interact over the network A provider site. The user uses the generic graphical User Interface (UI) for resources access, authentication, authorization to use the resources. It provides CLI tools and API to perform some basic operations and development. From UI user can access the functionalities offered by the Information, Workload and Data management systems. All connections need to pass the Gatekeeper, which acts as a generic interface to the cluster, managed by the Local Resource Management System (LRMS). A Storage Element (SE) provides uniform access to data storage resources. It may control simple disk servers, large disk arrays or tape-based Mass Storage Systems (MSS). Most storage resources are managed by a Storage Resource Manager (SRM) - a middleware service providing capabilities like transparent file migration from disk to tape, file pinning, space reservation, etc. The Information Service (IS) provides information about the Grid resources and their status. The resources are discovered via IS. The published information is also used for monitoring and accounting purposes. Globus Monitoring and Discovery Service (MDS) used for resource discovery and to publish the resource status.	Package:    Globus Toolkit 4.0.8  os:             Scientific Linux version 4.8 64 bit  server:        dgiref-globus40.fzk.de  manuals:   server v. 4.0.8 / services  monitoring: monitoring page Package:    Globus Toolkit 5.0.4  os:             Scientific Linux version 5.6 64 bit  server:        dgiref-globus.fzk.de  manuals:   server v. 5.0.4  monitoring: monitoring page Archive links globus toolkit server v. 4.2.1 globus toolkit server v. 4.0.7 Information links Operating systems The globus community Globus updates and patches Globus usage statistics Download links codebase binary repo Guidelines links guide to dgridmap Files links Example of dgridmap pbs.pm

Globus Toolkit server v.4.2.1

Prepare

Operating system

• Scientific Linux v.5.4 64 bit

Configure the following settings for the server:

• proxy
• ntp
• script:/etc/resolv.conf
• configure the postgres update

Additional Software

• torque & maui client
• java SDK >= 1.6.0 (from Sun, IBM, HP, or BEA (do not use GCJ))
• PostgreSQL 8.1.9 (postgresql, postgresql-libs, postgresql-server)
• Perl 5.8.8 (with XML::PARSER)
• sudo
• ant version >= 1.6.5

To make all libraries from ant available do the following as root:

 $ cd /usr/share/ant/lib
 $ ln -s /usr/share/java/ant.jar ant.jar
 $ ln -s /usr/share/java/ant-launcher.jar ant-launcher.jar

Each grid user needs a certificates directory . As grid user create this directory by: mkdir -p $home/.$user_globus chown $user_grid /localhome/.$user_globus

Firewall configuration

The GT4 frontend runs GRAM + MDS + GridFTP + RFT services (how to open port in firewall).

Install

The installation procedure includes the following steps:

• Download the GT binary
• Install with gpt-install
• Post-installation with gpt-postinstall

Configure

To configure the gsissh, please use the Globus services page. Please check the grid certificates allocation before to proceed.

• Configure Globus authorization
• Grid Security Infrastructure

Globus user must be authenticated to use the services. There are two ways for user authentication in Globus - manually or automatically.

• automatically

There is the dgridmap script for automatic generation of the grid-mapfile. It is mandatory to execute the dgridmap script regularly (e.g. cron minimum once per 24 hours) for update of the Globus grid-mapfile (see section join D-Grid).

Grid-mapfile: The grid-mapfile located in the directory /etc/grid-security/ is necessary for the successful launch of the Globus container. If you have none of it, an empty file with the name grid-mapfile should be created.

• manually

as grid-user extract the Distinguished name (DN) as the following:

       su $USER
       grid-cert-info -subject /C=DE/O=GridGermany/OU=Leibniz-Rechenzentrum/CN=griduser1

Now, the grid-user should be included to the grid-mapfile. For this, execute as root:

       su root
       grid-mapfile-add-entry -dn "<DN>" -ln <locale user name for griduser1>

• Configure /etc/services

To make the Globus services available, the following ports must be inserted as root in the file /etc/services:

       gsiftp             2811/tcp 
       gsigatekeeper      2119/tcp

• Configure /etc/hosts

In /etc/hosts the IP address of the server must be mapped to FQDN as follows: <IP ADDRESS> <FQDN> <hostname>

• Configure /etc/hosts.allow

Insert as 'root' the following entries in the file /etc/hosts.allow:

       globus-gatekeeper:ALL:ALLOW
       globus-gridftp-server:ALL:ALLOW

The required configuration for every Globus service is documented in a separate section using the links from service section

Proceed

• Globus Usage statistics

• The Globus Toolkit provides functions for submitting personal user information to the Globus Alliance. Additional information about usage statistics is presented here.
• These functions should be deactivated in the the D-Grid reference installation. This can be done as follows:

• for Java components

• in file $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd the attribute <globalConfiguration> comment the parameter usageStatisticsTargets with the:

           <!--
           <parameter name="usageStatisticsTargets" value="usage-stats.globus.org:4810"/> 
           -->

• for C components:

• Setup the environment variable GLOBUS_USAGE_OPTOUT with value 1.

• Globus start script

• create the /etc/init.d/globus-container file, which serves as start/stop script for the container with the following content:

#!/bin/sh -e
# use RedHat feature:
# chkconfig: 345 80 50
#
# A script to start and stop the Globus Container
#
### BEGIN INIT INFO
# Provides: globus-container
# Required-Start: $network $remote_fs
# Required-Stop: $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
## Description: Start/stop the globus-container service.
### END INIT INFO
case "$1" in
  start)
   su - globus -c "/usr/local/globus/sbin/globus-start-container-detached -p 8443"
     ;;
   stop)
  su - globus -c "/usr/local/globus/sbin/globus-stop-container-detached"
     ;;
  restart)
   $0 stop
   sleep 1
   $0 start
   ;;
  *)
  printf "Usage: $0 {start|stop|restart}\n" >&2
  exit 1
  ;;
esac

• the /etc/init.d/globus-container should be executed by root:

Do not run the /etc/init.d/globus-container before install the Globus Toolkit services.

Initial test

• Using a JDL file (job description language)
  • The following job is described in XML format using the job description language (JDL). It is stated as a parameter during the job submission (see below).
  • First, create as grid user the file torqueJob.xml and fill it with the following content (see script section #Using a JDL file)
• Using the voms-proxy-init (e.g. voms-proxy-init -voms dgtest:/dgtest/admin:softwareadmin)

Update

• Re-install Globus

Execute commands in installation directory as user 'globus'

• Deinstallation

This procedure run as 'root' will delete the whole installation and configuration of Globus from the machine:

• Remove installation directories and files
• Remove the Globus entries, if they are available.
• remove for GSI
• remove for GRAM
• remove for RFT
• delete postgressql packages with YaST or YUM

• Update

The updates for the 64-bit architecture are available. For the 32-bit machines were the same packages with gcc32 packages installed. More current fixes can be found here.

All the updates can be downloaded from here and must be installed by user globus:

Globus Toolkit services

Introduction

The Globus Toolkit service components can be separated into the following groups:

1. Web-Service based components (WS-GRAM, RFT, MDS4)
2. Non-Web-Service based components (GridFTP , GSISSH)

Web-Service components

WS-GRAM

Configure WS-GRAM

After successful installation of the recommended D-Grid package Globus is aready configured to use PBS (more precisely TORQUE) as the Local Resource Management System (LRMS). The interface for submitting jobs to the LRMS is provided by a component called Scheduler Adapter, which is basically a Perl module called <$GLOBUS_LOCATION>/lib/perl/Globus/GRAM/JobManager/pbs.pm. It should be patched as discribed below.

Patching the Scheduler Adapter

Line 387-388:

    elsif($description->jobtype() eq 'mpi' ||
          $description->jobtype() eq 'multiple')

should be replaced by:

    elsif(
          $description->jobtype() eq 'mpi' ||
           ($description->jobtype() eq 'multiple' and
            ($description->host_count() > 1 or $description->count() > 1) 
           )
         )

At line 408:

            print CMD "#!/bin/sh\n";

should be added:

            print CMD "#!/bin/sh\n";
            print CMD ". /etc/profile";

A more rigorous patch is possible, if the Mpiexec implementation by Pete Wyckoff is installed on the cluster. Note that this version of Mpiexec is not fully compatible with those provided with MPICH and other MPI iomplementations. Most notable, the parameter '-machinefile' does not exist.

Configuring Sudo

In order to submit jobs on behalf of a user Globus needs to be authorized to invoke specific commands via sudo (Super User Do). To this end edit the file /etc/sudoers.

Add the following lines to /etc/sudoers:

#
# Disable "ssh hostname sudo <cmd>", because it will show the password in clear.
#         You have to run "ssh -t hostname sudo <cmd>".
#
# Defaults    requiretty
 
# Globus GRAM entries
 
globus  ALL=(ALL) NOPASSWD: \
    /usr/local/globus/libexec/globus-gridmap-and-execute \
      -g /etc/grid-security/grid-mapfile \
      /usr/local/globus/libexec/globus-job-manager-script.pl *
 
globus  ALL=(ALL) NOPASSWD: \
    /usr/local/globus/libexec/globus-gridmap-and-execute \
      -g /etc/grid-security/grid-mapfile \
      /usr/local/globus/libexec/globus-gram-local-proxy-tool *

References

• WS-GRAM Scheduler Interface Tutorial
  

• WS-GRAM Job Description RSL parameter
  

• Mpiexec by Pete Wyckoff

Pre-WS GRAM configuration

• As root user create the /etc/xinetd.d/gsigatekeeper file and insert the following data:

service gsigatekeeper 
{ 
socket_type = stream 
protocol  = tcp 
wait  = no 
user = root 
env  += LD_LIBRARY_PATH=<$GLOBUS_LOCATION>/lib 
env += GLOBUS_TCP_PORT_RANGE=20000,25000 
server = <$GLOBUS_LOCATION>/sbin/globus-gatekeeper 
server_args = -conf <$GLOBUS_LOCATION>/etc/globus-gatekeeper.conf 
disable = no 
}

• Restart as root user the xinetd daemon:

$ /etc/init.d/xinetd restart

GRAM tests

To be sure that GRAM accepts jobs, execute as grid-user:

For WS-GRAM

> globusrun-ws -submit -F <FQDN of the Globus Frontend> -s -c /bin/hostname
 Delegating user credentials...Done.
 Submitting job...Done.
 Job ID: uuid:66720d6a-6aac-11dd-82c4-af7ae8031d29
 Termination time: 08/16/2008 09:27 GMT
 Current job state: Pending
 Current job state: Active
 Current job state: CleanUp-Hold
 dgiref-globus.fzk.de
 Current job state: CleanUp
 Current job state: Done
 Destroying job...Done.
 Cleaning up any delegated credentials...Done.

For Pre-WS-GRAM

> globus-job-run localhost:2119/jobmanager-fork /bin/date
Fri Dec 21 10:59:52 CEST 2007

Turn off fork scheduler

To turn off the fork scheduler, rename the following configuration files, and restart the container as root user:

$ cd $GLOBUS_LOCATION/etc/gram-service-Fork 
$ mv  jndi-config.xml jndi-config.xml_save
$ cd $GLOBUS_LOCATION/etc/grid-services
$ mv jobmanager-fork jobmanager-fork.save
$ /etc/init.d/globus-container restart

Disabling the fork scheduler can be tested by a grid user as following:

For WS-GRAM

> globusrun-ws -submit  -c /bin/hostname
Submitting job...Failed.
globusrun-ws: Error submitting job
globus_soap_message_module: SOAP Fault
Fault code: soapenv:Server.userException
Fault string: java.rmi.RemoteException: Job creation failed.; nested exception is:
java.rmi.RemoteException: The Managed Job Factory Service at 
https://10.156.10.69:8443/wsrf/services/ManagedJobFactoryService does not have a resource with key "Fork".
 
> globusrun-ws -submit -Ft Fork -c /bin/hostname
Submitting job...Failed.
globusrun-ws: Error submitting job
globus_soap_message_module: SOAP Fault
Fault code: soapenv:Server.userException
Fault string: java.rmi.RemoteException: Job creation failed.; nested exception is:
java.rmi.RemoteException: The Managed Job Factory Service at 
https://10.156.10.69:8443/wsrf/services/ManagedJobFactoryService does not have a resource with key "Fork".

For Pre-WS-GRAM

> globus-job-run localhost:2119/jobmanager-fork /bin/date
GRAM job submission failed because the gatekeeper failed to find the requested service (error code 93)

RFT

RFT configuration

Configure the PostgreSQL database:

As root user edit the /etc/sysconfig/postgresql file, to ensure that TCP/IP connections (option -i) are allowed:

$ vi /etc/sysconfig/postgresql
# Add: 
POSTGRES_OPTIONS="-i"

As the postgres user initialize the new database:

> initdb -D /var/lib/pgsql/data

As the postgres user allow access to the globus user as follows:

> vi /var/lib/pgsql/data/pg_hba.conf

Add to the end of file:

host[TAB]rftDatabase[TAB]globus[TAB]<IP-addresses of GT4 frontends>[TAB]255.255.255.255[TAB][md5/trust]

[TAB] stands for Tab input. For security reasons, it is recommended that you have a password protection. If the password protection is desired, md5 is used, otherwise trust will be used. If a password protection is set, you must give a password to every database user.

Restart as root user the PostgreSQL database server:

$ /etc/init.d/postgresql restart

As postgres user create a database ID for the user globus.

> createuser globus

Answer to the next question with "y". The password for the database ID globus will be asked.

Execute The shell script /usr/local/bin/globus-env-setup.sh as postgres user:

> . /usr/local/bin/globus-env-setup.sh

Now create the RFT database as postgres user:

> createdb rftDatabase 
> psql -d rftDatabase -f  $GLOBUS_LOCATION/share/globus_wsrf_rft/rft_schema.sql

As globus user configure the using of the RFT database.

% vi $GLOBUS_LOCATION/etc/globus_wsrf_rft/jndi-config.xml
#If the password protection is active, setup the chosen password 'foo'. Otherwise leave an empty string.
<resource name="dbConfiguration"
            type="org.globus.transfer.reliable.service.database.RFTDatabaseOptions">
            <resourceParams>
            <parameter>
                <name>
                driverName
                </name>
                <value>
                org.postgresql.Driver
                </value>
            </parameter>
            <parameter>
                <name>
                connectionString
                </name>
                <value>
                jdbc:postgresql://dgiref-globus.fzk.de/rftDatabase
                </value>
            </parameter>
            <parameter>
                <name>
                userName
                </name>
                <value>
                globus
                </value>
            </parameter>
            <parameter>
                <name>
                password
                </name>
                <value>
                </value>
            </parameter>
            </resourceParams>
        </resource>

Restart as root user the Globus Container:

$ /etc/init.d/globus-container restart

RFT test

If the RFT service is properly configured, it should be possible to copy a test file as follows:

As globus user copy the file transfer.xfr to a temporary directory like /tmp.

In the file /tmp/transfer.xfr replace the entry localhost by the Globus frontend FQDN and create an empty test file:

% cp $GLOBUS_LOCATION/share/globus_wsrf_rft_test/transfer.xfr /tmp
% touch /tmp/rftTest.tmp

Start the test as grid user:

> rft -h <FQDN> -f /tmp/transfer.xfr

MDS

MDS4 configuration

To show the FQDN instead of the IP address of the Globus frontend in the WebMDS, insert as the globus user the following entries in $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd to the <globalConfiguration> tag:

<globalConfiguration> 
 <parameter name="logicalHost" value="<FQDN>"/> 
 <parameter name="publishHostName" value="true"/> 
 ... 
</globalConfiguration>

To register your site to the D-Grid wide MDS Index located at the LRZ in Munich and the redundant MDS Index located at KIT in Karlsruhe, insert as the globus user the following index URLs in $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/hierarchy.xml to the <upstream> tag:

    <config>                                            <config> 
           <upstream>$URL1</upstream>      OR                    <upstream>$URL</upstream>
           <upstream>$URL2</upstream>                   </config>
    </config>

According to the Site Hierarchy and the belonging to one VO the values for URL, URL1 and URL2 must be the following:

For Kerndgrid Sites

NOT new Globus installation in the Site

If a MDS Site-Index with the name "site-index.mysite.de" listen on port 8443 is already available (e.g. more than one Globus Installations in the site):

URL = https://site-index.mysite.de:8443/wsrf/services/DefaultIndexService

NEW Globus installation in the Site

If this is the first Globus installation in the Site, this assumes that the Site Index itself should be registered to the Kerndgrid MDS index hosted at the LRZ and KIT:

URL1 = https://mds-dgi.lrz.de:8445/wsrf/services/DefaultIndexService

URL2 = https://dgrid-mds.scc.kit.edu:8443/wsrf/services/DefaultIndexService

For Globus 4.2 MDS please use:

URL = https://mds2-dgi.lrz.de:8445/wsrf/services/DefaultIndexService

In this case, the geomaint sensor must be also installed and configured. Please specify the appropriate name for your site as registered in the D-Grid Resources Registration Service GRRS in the configuration of the sensor!

For Community Sites

NOT new Globus installation in the Site

If a Site-Index with the name "site-index.mysite.de" listen on port 8443 is already available (e.g. many Globus Installations in the site):

URL = https://site-index.mysite.de:8443/wsrf/services/DefaultIndexService

NEW Globus installation in the Site

If this is the first Globus installation in the Site, that assumes that the Site Index should be registered itself to the Community-Index:

URL = https://index.mycommunity.de:8443/wsrf/services/DefaultIndexService

Please ask the community leader about the correspondent Community-Index address.

NEW Community-Index installation in the community

If this is a first Globus installation in the whole Community, an additional Community-Index should be setup (on this or another computer). This Community-Index register itself direct to the central D-Grid MDS4 Indexes at the LRZ and KIT:

URL1 = https://mds-dgrid.lrz.de:8443/wsrf/services/DefaultIndexService

URL2 = https://dgrid-mds.scc.kit.edu:8443/wsrf/services/TopIndexService

For Globus 4.2 MDS please use:

URL = https://mds2-dgrid.lrz.de:8443/wsrf/services/DefaultIndexService

In this case, the geomaint sensor must be also installed and configured. Please specify the appropriate name for your site as registered in the D-Grid Resources Registration Service GRRS in the configuration of the sensor!

Please send your index address to mab<nospam>d-grid.de so that your site index is visible from the central MDS sites LRZ WebMDS and the redundant KIT WebMDS

Additional information e.g. the list of available Community-addresses can be found here.

MDS4 test

Restart the container as root after previous configuration steps.

$ /etc/init.d/globus-container restart

After around 10 minutes the information about your installation will appear under the LRZ WebMDS and the redundant KIT WebMDS. For Globus 4.2 the information will appear under LRZ WebMDS 4.2

To see the provided monitoring data from a grid client do the following as user:

> wsrf-query -s https://<FQDN>:8443/wsrf/services/DefaultIndexService

Geomaint sensor

The GeoMaint sensor as part of the Globus MonMan incubator project must be installed on every globus Site index server to forward monitoring data to the central MDS Index Service. The site specific monitoring data gathered by the sensor contains the geographical coordinates as well as informations about maintenance and contact persons. These informations are used by LRZ WebMDS, LRZ WebMDS 4.2 and the redundant KIT WebMDS to clarify the current site status in the topology map.

Geomaint can be downloaded from MonMan repository. A guide to the installation can be found here or in the Readme File.

Current version for Geomaint 1.2.3 sensor is compatible with Globus Toolkit 4.0.x and Globus Toolkit 4.2.x .

vi $GLOBUS_LOCATION/libexec/infoprovider/conf/site.conf
 
# Geolocation
 
site.location=Garching, Deutschland
 
site.latitude=48.26166   #at least 5 decimal symbols in the position
 
site.longitude=11.66638  #at least 5 decimal symbols in the position
 
site.web=http://mabtest.lrz-muenchen.de
 
site.sponsor=BMBF
 
...
 
# configuration of the currently ongoing maintenance work
# 1 stands for a scheduled maintenance work and 
# 2 maintenance now
# 0 represents no current or planned maintenance(running)
 
#site.maintenance=3unconfigured site sensor
 
#site.maintenance=2cluster is down for maintenance until 7pm
 
#site.maintenance=1maintenance today from 5pm to 7pm
 
#site.maintenance=0Running

Ganglia: resource monitoring

Ganglia cluster monitoring is used to monitor individual needed Compute Nodes. It provides parameters such as Operating system RAM size or processor architecture. It is recommended to install Ganglia to the computing nodes to provide cluster information in the MDS4. This enables the grid users and resource brokers to match their requirements of computing resources.

The information about installing Ganglia and the software packages can be found here. A guide to link the cluster Ganglia Monitoring toolkits to the Globus Toolkit can be found here.

To enable the Ganglia information provider for MDS4 execute the following command:

$ mds-gluerp-configure pbs ganglia $GLOBUS_LOCATION/etc/gram-service-PBS/gluerp-config.xml

If a Ganglia monitoring daemon is not installed on the Globus Frontend you must enter the hostname and port where Ganglia is running by editing the file

$ $GLOBUS_LOCATION/etc/gram-service-PBS/gluerp-config.xml

After that you have to restart the Globus Container.

$ /etc/init.d/globus-container restart

Non-Web-Service components

GridFTP

configuration

As root user insert the following data to the /etc/xinetd.d/gsiftp file:

service gsiftp 
{ 
instances  = 100 
socket_type  = stream 
wait  = no 
user  = root
env  += LD_LIBRARY_PATH=<$GLOBUS_LOCATION>/lib
env  += GLOBUS_TCP_PORT_RANGE=20000,25000
server  = <$GLOBUS_LOCATION>/sbin/globus-gridftp-server
server_args  = -i
nice = 10
disable = no
}

Restart the xinetd daemon as the root user:

$ /etc/init.d/xinetd restart

GridFTP test

To be sure that GridFTP can successfully transfer data, execute as grid-user:

> grid-proxy-init
> globus-url-copy gsiftp://localhost/etc/hosts file:///tmp/hosts_copy 
> ls /tmp/hosts_copy

GSISSH

Firewall configuration Access via GSISSH (TCP port 2222) to the interactive node should be allowed (how to open port in firewall).

configure

To configure the 'gsissh' start script do the following as 'root':

su root
cp $GLOBUS_LOCATION/sbin/SXXsshd /etc/init.d/gsisshd
chkconfig --add gsisshd

As 'globus' user change the port number from 22 to 2222 in the following files:

su globus
cat $GLOBUS_LOCATION/etc/ssh/ssh_config
cat $GLOBUS_LOCATION/etc/ssh/sshd_config

As 'root' start the 'gsissh' daemon with the following command:

su root
 /etc/init.d/gsisshd start

setting up as service

• In order to work with the GSI-SSH Service, the following line must be inserted as 'root' in the file /etc/services:

gsissh             2222/tcp

• In /etc/hosts the IP address of the server must be mapped to the FQDN as follows:

<IP ADDRESS>    <FQDN>    <hostname>

• Additionaly, insert as 'root' the following entries in the file /etc/hosts.allow:

echo "gsisshd:ALL:ALLOW" >> /etc/hosts.allow

proceed

service gsisshd [start|stop]

Attribute-based authorization

Install

The original description can be downloaded from d-grid site.

To apply the Attribute-based authorization to the Globus Toolkit, the following components should be additionally installed:

• Virtual Organization Membership Server (VOMS) Authorization Interceptors for the Globus Web-Services (GRAM and RFT. The service security should be configured accordingly).
• Authorization callout C-API for the Globus GridFTP should be used for attributes mapping.
• The aba-glite.tar.gz file regarding the attribute based authorization is supposed to be installed in /opt/glite

Configure

1. setup the voms server certificates
2. Configure the sudo for WSGRAM
3. Mappings for VOMS FQAN
4. ln -s /etc/grid-security/voms-attr-authz /etc/grid-security/voms-lcmaps-mapfile
5. configure the /opt/glite/etc/vomses file (see the file /opt/glite/etc/vomses.template for instructions, for accessing the voms server)

To avoid the error message: "<username> is not in the grid mapfile", all local accounts, which attributes should be mapped, the similar empty record in the grid-mapfile should be assigned: vi /etc/grid‐security/grid‐mapfile "" uhdtsgm

JavaGAT adaptor

To use the GAT Globus Tookit adaptors, do the following:

1. locate the personnel certificates files userkey.pem and usercert.pem in the directory $HOME/.globus
2. locate the host certificates of the Grid hosts you like to access in the directory $HOME/.globus/certificates.
3. create a proxy certificate with $GAT LOCATION/bin/grid-proxy-init.
4. The dataset $HOME/.globus/cog.properties should exists and to be like:

cat $HOME/.globus/cog.properties
 
#Java CoG Kit Configuration File
#usercert: The path to the file containing your dgrid certificate.
usercert=/home/dgdt0000/.globus/usercert.pem
# userkey: The path to the file containing your Grid key.
userkey=/home/dgdt0000/.globus/userkey.pem
# proxy: The name under which your proxy certificate which you create with grid-proxy-init is stored.
proxy=/tmp/x509up_u1000
#cacert: The path of the directory, which contains the host certificates.
#cacert=/etc/grid-security/certificates
cacert=/home/dgdt0000/.globus/cog-certificates

MPI extension for Globus 4.2

Globus Toolkit package provided by the D-Grid reference installation

From a user's perspective the change allows to use extensions to load the mpi module before the job starts as shown below:

  <extensions>
     <preamble> module load mpi </preamble>
  </extensions>
</job>

In the latest globus 4.2 reference package the changes provided below are already made.

Globus Toolkit package do not provided by the D-Grid reference installation

Administrative changes

It is necessary to change the file $GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager/pbs.pm. The output of diff pbs.pm.orig pbs.pm is shown below containing all the necessary changes including the ones proposed by Benjamin Henne (the changes below the diff entry 280c311 are from Benjamin and not related to this extension).

patch pbs.pm -

The patch program will wait for your input. Paste the diff output below and when finished press enter and then ctrl-D (strg-D auf Deutsch) TWICE (ZWEIMAL).

30c30
<     $mpirun         = 'no';
---
>     $mpirun         = '/home/software/mpi/mpiexec_wrapper';
37,39c37,39
<     $softenv_dir    = '';
<     $soft_msc       = "$softenv_dir/bin/soft-msc";
<     $softenv_load   = "$softenv_dir/etc/softenv-load.sh";
---
>     #$softenv_dir    = '';
>     #$soft_msc       = "$softenv_dir/bin/soft-msc";
>     #$softenv_load   = "$softenv_dir/etc/softenv-load.sh";
41a42,62
> sub new
> {
> 
>     my $cln   = shift;
>     my $class = ref($cln) || $cln;
> 
>     my $self  = $class->SUPER::new(@_);
>     bless $self, $class;
> 
>     $self->log("INFO: exec new from sge::new\n");
> 
>     my $description = $self->{JobDescription};
>     use Globus::GRAM::ExtensionsHandler;
>     new Globus::GRAM::ExtensionsHandler($class, $self, $description);
> 
> #        use Data::Dumper;
> #        $self->log("INFO: job description\n");
> #        $self->log(Dumper($description));
> #       if (defined($description->march)){$self->log("INFO: march available\n");}
>     return $self;
> }
268,275c289,296
<     if ($softenv_dir ne '')
<     {
<         $self->setup_softenv(
<             $self->job_dir() . '/pbs_softenv_job_script',
<             $soft_msc,
<             $softenv_load,
<             *JOB);
<     }
---
>     #if ($softenv_dir ne '')
>     #{
>     #    $self->setup_softenv(
>     #        $self->job_dir() . '/pbs_softenv_job_script',
>     #        $soft_msc,
>     #        $softenv_load,
>     #        *JOB);
>     #}
276a298,307
>     
>     #
>     # If preamble defined, insert it 
>     #
>     if(  $description->preamble() ) {
>         $self->log("INFO: Preamble:\n");
>         $self->log( $description->preamble() );
>         print JOB "\n# Preamble\n";
>         print JOB  $description->preamble() .  "\n\n";
>     }
280c311
<     $library_vars{LD_LIBRARY_PATH} = 0;
---
>     #$library_vars{LD_LIBRARY_PATH} = 0;
444c475
<                     $machinefilearg = ' -machinefile $PBS_NODEFILE';
---
>                 #    $machinefilearg = ' -machinefile $PBS_NODEFILE';
469c500
<                     print JOB ' -machinefile $PBS_NODEFILE';
---
>                 #    print JOB ' -machinefile $PBS_NODEFILE';
481,490c512,525
< hosts=\`cat \$PBS_NODEFILE\`;
< counter=0
< while test \$counter -lt $count; do
<     for host in \$hosts; do
<         if test \$counter -lt $count; then
<             $remote_shell \$host "/bin/sh $cmd_script_name; echo \\\$? > $exit_prefix.\$counter" < $stdin &
<             counter=\`expr \$counter + 1\`
<         else
<             break
<         fi
---
> if [ $count -eq 1 ]; then
>     /bin/sh $cmd_script_name; echo \$? > $exit_prefix.0 < $stdin
> else
>     hosts=\`cat \$PBS_NODEFILE\`;
>     counter=0
>     while test \$counter -lt $count; do
>         for host in \$hosts; do
>             if test \$counter -lt $count; then
>                 $remote_shell \$host "/bin/sh $cmd_script_name; echo \\\$? > $exit_prefix.\$counter" < $stdin &
>                 counter=\`expr \$counter + 1\`
>             else
>                 break
>            fi
>         done
492,493c527,528
< done
< wait
---
>     wait
> fi

The file $GLOBUS_LOCATION/lib/perl/Globus/GRAM/ExtensionsHandler.pm should also be changed! The diff is provided below.

15a16,17
>     # LRZ: added $jm arg to get access to the JM log file
>     my $jm      = shift;
26a29,30
>     # LRZ: set log file handle
>     $self->{log} = $jm->{log} if ( $jm->{log} );

Finally the file <code>$GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager.pm</code> should be changed as follows.
 
93c93,97
<     if ($description->xml_extensions())
---
>     #LRZ# if ($description->xml_extensions())
>     #LRZ#{
>     #LRZ#    new Globus::GRAM::ExtensionsHandler($class, $description);
>     #LRZ#}
>     if( ! defined($description->logfile()) )
95c99,103
<         new Globus::GRAM::ExtensionsHandler($class, $description);
---
>         my $WHOAMI = `whoami`;
>         chomp $WHOAMI;
> 
>         my $logfile = "/tmp/$WHOAMI-jobmanager.log";
>         $description->add('logfile', $logfile);
413a422
> # chdir("/"); is missing from
430c439
<     chdir("/");
---
>     #chdir("/");