middleware:Globus/420

From Dgiref

Introduction

The Globus Toolkit, currently at version 4, is an open source toolkit for building computing grids developed and provided by the Globus Alliance.

Simplified Globus Toolkit 4 architecture

The Globus Toolkit system consists of several components which interact over the network

A provider site.
The user uses the generic graphical User Interface (UI) for resources access, authentication, authorization to use the resources. It provides CLI tools and API to perform some basic operations and development.
From UI user can access the functionalities offered by the Information, Workload and Data management systems.
All connections need to pass the Gatekeeper, which acts as a generic interface to the cluster, managed by the Local Resource Management System (LRMS).
A Storage Element (SE) provides uniform access to data storage resources. It may control simple disk servers, large disk arrays or tape-based Mass Storage Systems (MSS). Most storage resources are managed by a Storage Resource Manager (SRM) - a middleware service providing capabilities like transparent file migration from disk to tape, file pinning, space reservation, etc.
The Information Service (IS) provides information about the Grid resources and their status. The resources are discovered via IS. The published information is also used for monitoring and accounting purposes. Globus Monitoring and Discovery Service (MDS) used for resource discovery and to publish the resource status.

Package:    Globus Toolkit 4.0.8
os:             Scientific Linux version 4.8 64 bit
server:        dgiref-globus40.fzk.de
manuals: server v. 4.0.8 / services
monitoring: monitoring page

Package:    Globus Toolkit 5.0.4
os:             Scientific Linux version 5.6 64 bit
server:        dgiref-globus.fzk.de
manuals: server v. 5.0.4
monitoring: monitoring page

Archive links

Information links

Download links

Guidelines links

guide to dgridmap

Files links

Example of dgridmap
pbs.pm

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Globus Toolkit server v.4.2

Prepare

Operating system

Scientific Linux v.4.8 64 bit

Optimizing the configuration: 

Use minimal operating system installation without firewall.
To verify installed packages use the command 
rpm -qa | grep package_name
Install the following additional packages:
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command: 
chkconfig <SERVICE> off

Configure the following settings for the server:

Additional Software

Torque and maui client
java SDK > 1.5.0 (from Sun, IBM, HP, or BEA (do not use GCJ))
ant 1.6.5

To make all libraries from ant available do the following as root:

 $ cd /usr/share/ant/lib
 $ ln -s /usr/share/java/ant.jar ant.jar
 $ ln -s /usr/share/java/ant-launcher.jar ant-launcher.jar

PostgreSQL 8.1.9 (postgresql, postgresql-libs, postgresql-server)
Perl 5.8.8 (with XML::PARSER)
sudo

administrator's script: prepare.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# prepare the installation of Globus 4.2.1

# Declare the variables section ------------
# user_globus=globus local user
# user_grid=non root and non globus grid user
# path_globus=globus localtion
# path_certificates=certificates location
# path_gpt=gpt-3.2 location
# JAVA_HOME=java home location
# ANT_HOME=ant home location
# Please insert your actual configuration
# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
user_globus=globus
user_grid=dgdt0001
path_globus=/usr/local/globus
path_certificates=/etc/grid-security/certificates
path_gpt=/usr/local/gpt-3.2
JAVA_HOME=/usr/java/jdk1.6.0_13/
ANT_HOME=/usr/share/ant
# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

wget -O /etc/yum.repos.d/sl-dgiref.repo http://mirror.scc.kit.edu/downloads/yum.repo/sl-dgiref.repo

yum -y install perl sudo postgresql postgresql-libs postgresql-server perl-XML-Parser
yum -y install jdk-1.6.0_13-fcs ant gcc java-1.6.0-sun-compat

cd /usr/share/ant/lib
ln -s /usr/share/java/ant.jar ant.jar
ln -s /usr/share/java/ant-launcher.jar ant-launcher.jar

# Create the user globus as the Globus administrator and the group globus:
# su root
groupadd $user_globus
useradd -m -g $user_globus -d /localhome/$user_globus $user_globus

umask 022

# Create the installation directory for Globus /usr/local/globus and set the globus user as owner:
mkdir -p $path_globus
chown -R $user_globus.$user_globus $path_globus
chmod 755 $path_globus
chmod 755 $path_gpt

# Create the directory for authorization and authentication functions (GSI):
mkdir -p $path_certificates

# Each grid user needs a certificates directory . As grid user create this directory by:
mkdir -p /localhome/.$user_globus
chown $user_grid /localhome/.$user_globus

### Environment Variables
# As '''root''' create a new setup file called ''/usr/local/bin/globus-env-setup.sh'':
if [ ! -f /usr/local/bin/globus-env-setup.sh ];
then
echo "\
export GPT_LOCATION=${path_gpt}
export GLOBUS_LOCATION=${path_globus}
export GLOBUS_USAGE_OPTOUT=1
export GLOBUS_TCP_PORT_RANGE=20000,25000
export JAVA_HOME=${JAVA_HOME}
export ANT_HOME=${ANT_HOME}
export CLASSPATH=\$CLASSPATH:\$JAVA_HOME/lib
export PATH=\$ANT_HOME/bin:\$JAVA_HOME/bin:\$PATH
export PATH=\$GLOBUS_LOCATION/bin:\$GLOBUS_LOCATION/sbin:\$PATH
export PATH=\$GLOBUS_LOCATION/etc:\$GPT_LOCATION/sbin:\$PATH
if [ -r \$GLOBUS_LOCATION/etc/globus-user-env.sh ]; then
. \$GLOBUS_LOCATION/etc/globus-user-env.sh
else
echo "error: globus may not be installed yet"
fi
" > /usr/local/bin/globus-env-setup.sh
fi

# The file should be marked as executable:
chmod +x /usr/local/bin/globus-env-setup.sh

#In order to execute this script for every user automatically, link in the directory /etc/profile.d as follows:
ln -s /usr/local/bin/globus-env-setup.sh /etc/profile.d

# GPT
# # As 'root' create the directory /usr/local/gpt-3.2 for the GPT installation and for
# the user 'globus' must have the necessary privileges:
# su root
mkdir -p $path_gpt
chown -R $user_globus.$user_globus $path_gpt

# create directory for the grid host certificates
mkdir /etc/grid-security/
# after copy the host certificate and host key into /etc/grid-security/

</td> </tr></table>

Install

The installation procedure includes the following steps:

GPT - Grid Packaging Toolkit will be needed to install the Globus toolkit.
Globus Toolkit installation

administrator's script: install.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# installation script globus toolkit 4.2.1

# Declare the variables section ------------
# BASE_URL="http://mirror.scc.kit.edu/downloads/src/globus"
# PACKAGE_GPT_32="gpt-3.2-src"
# PACKAGE_GLOBUS="Lrz_SL45-gt4.2_binary-x86_64-unknown-linux-gnu-bin"
# workdir=/tmp
# Please insert your actual configuration
# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BASE_URL="http://mirror.scc.kit.edu/downloads/src/globus/2009.2"
PACKAGE_GPT_32="gpt-3.2-src"
PACKAGE_GLOBUS="Lrz_SL45-gt4.2_64_binary-x86_64-unknown-linux-gnu-bin.tar.gz"
workdir=/tmp
# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# Download the GPT-3.2 and Globus Toolkit packages
#su globus
cd $workdir

if [ ! -f ${PACKAGE_GPT_32}.tar.gz ];
then
wget ${BASE_URL}/${PACKAGE_GPT_32}.tar.gz
fi
if [ ! -f ${PACKAGE_GLOBUS}.tar.gz ];
then
wget ${BASE_URL}/${PACKAGE_GLOBUS}.tar.gz
fi

# Install GPT as the user '''globus''':
su - $user_globus

. /usr/local/bin/globus-env-setup.sh

if [ ! -f /usr/local/gpt-3.2/sbin/gpt-build ];
then
tar zxvf ${PACKAGE_GPT_32}.tar.gz
cd gpt-3.2
./build_gpt
cd ..
rm -rf gpt-3.2/
rm gpt-3.2-src.tar.gz
fi

# Install packages:
# install (configuration option was ''./configure --prefix=$GLOBUS_LOCATION --enable-wsgram-pbs'')
$GPT_LOCATION/sbin/gpt-install $workdir/${PACKAGE_GLOBUS}.tar.gz
$GPT_LOCATION/sbin/gpt-postinstall

</td> </tr></table>

Configure

To configure the gsissh, please use the Globus services page

Please check the grid certificates allocation before to proceed.

Grid Security Infrastructure: In order to use the Globus services Globus user must be authenticated. There are two ways for user authentication in Globus:

manually

as grid-user extract the Distinguished name (DN) as the following:

       su $USER
       grid-cert-info -subject /C=DE/O=GridGermany/OU=Leibniz-Rechenzentrum/CN=griduser1

Now, the grid-user should be included to the grid-mapfile. For this, execute as root:

       su root
       grid-mapfile-add-entry -dn “<DN>” -ln <locale user name for griduser1>

automatically

There is the dgridmap script for automatic generation of the grid-mapfile. It is mandatory to execute the dgridmap script regularly (e.g. cron minimum once per 24 hours) for update of the Globus grid-mapfile (see section join D-Grid).

Grid-mapfile:

The grid-mapfile located in the directory /etc/grid-security/ is necessary for the successful launch of the Globus container. If you have none of it, an empty file with the name grid-mapfile should be created.

In order to work with the Globus services, the following ports must be inserted as root in the file /etc/services:

gsiftp             2811/tcp 
gsigatekeeper      2119/tcp

In /etc/hosts the IP address of the server must be mapped to FQDN as follows: <IP ADDRESS> <FQDN> <hostname>

Furthermore, insert as 'root' the following entries in the file /etc/hosts.allow:

globus-gatekeeper:ALL:ALLOW
globus-gridftp-server:ALL:ALLOW

The extensions in order to enable Globus job submission to Torque is documented here.

The required configuration for every Globus service is documented in a separate section using the links from service section

administrator's script: configure.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# configure script for globus 4.2.1
# configure GSI Security

# cp yourhostkey.pem hostkey.pem
# cp yourhostcert.pem hostcert.pem
# cp hostkey.pem containerkey.pem
# cp hostcert.pem containercert.pem
# chown globus.globus container*.pem:
# set certificate privileges
chmod 400 /etc/grid-security/*key.pem
chmod 644 /etc/grid-security/*cert.pem
#create an empty gridmap-file
touch /etc/grid-security/gridmap-file

</td> </tr></table>

Proceed

Globus Usage statistics

The Globus Toolkit provides functions for submitting personal user information to the Globus Alliance. These functions should be deactivated in the the D-Grid reference installation. Additional information about usage statistics is presented here.

This can be done as follows:

for Java components: in file $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd the attribute <globalConfiguration> comment the parameter usageStatisticsTargets with the:

     <!--
     <parameter name="usageStatisticsTargets" 
     value="usage-stats.globus.org:4810"/> 
     -->

for C components: Setup the environment variable GLOBUS_USAGE_OPTOUT with value 1.

Globus start script

The file /etc/init.d/globus-container serves as start/stop script for the container.

The contents of the file is as follows:

#!/bin/sh -e
# use RedHat feature:
# chkconfig: 345 80 50
#
# A script to start and stop the Globus Container
#
### BEGIN INIT INFO
# Provides: globus-container
# Required-Start: $network $remote_fs
# Required-Stop: $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
## Description: Start/stop the globus-container service.
### END INIT INFO
case "$1" in
  start)
   su - globus -c "/usr/local/globus/sbin/globus-start-container-detached -p 8443"
     ;;
   stop)
  su - globus -c "/usr/local/globus/sbin/globus-stop-container-detached"
     ;;
  restart)
   $0 stop
   sleep 1
   $0 start
   ;;
  *)
  printf "Usage: $0 {start&#124;stop&#124;restart}\n" >&2
  exit 1
  ;;
esac

The /etc/init.d/globus-container should be executed by root:

administrator's script: proceed.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# proceed script for globus 4.2.1

su root
touch /etc/init.d/globus-container
chmod a+x /etc/init.d/globus-container
# edit and save the globus-contaner init script
vi /etc/init.d/globus-container
# To ensure that the globus container starts automaticly after every reboot of the server:
chkconfig -add globus-container

</td> </tr></table>

Initial test

Using a JDL file
- The following job is described in XML format using the job description language (JDL). It is stated as a parameter during the job submission (see below).
- First, create as grid user the file torqueJob.xml and fill it with the following content (see script section #Using a JDL file)

A detailed description of the job description language can be found here.

Generic Test
Torque Test

administrator's script: test.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# test for globus toolkit 4.2.1
griduser='griduser'
gridclient="ui.d-grid.de"

# Generic Test
# Log on into a grid client (eg. with ssh)
ssh $griduser@$gridclient

#GSI Authentication

grid-proxy-init

# Your identity: /C=DE/O=GermanGrid/OU=FZK/CN=Grid User
# Enter GRID pass phrase for this identity:
# Creating proxy .............. Done
# Your proxy is valid until: Fri Aug 15 23:27:06 2008

#Test Counter Service
# as griduser:
counter-client -s https://<FQDN>:8443/wsrf/services/CounterService

#Got notification with value: 3
#Counter has value: 3
#Got notification with value: 13

# Torque Test --------------------------------------------
# The successful integration of torque with Globus can be tested as grid user like here:
globusrun-ws -submit -F <FQDN> -Ft PBS -c /bin/sleep 120

#Using a JDL file
echo " <job>
<executable>/bin/echo</executable>
<directory>/tmp</directory>
<argument>***Hello D-Grid user***</argument>
<stdout>/tmp/stdout_torque</stdout>
<stderr>/tmp/stderr_torque</stderr>
<queue>dgiseq</queue>
</job>" > torqueJob.xml

#Then submit the grid job as griduser
globusrun-ws -submit -F <FQDN> -Ft PBS -f torqueJob.xml

</td> </tr></table>

Update

Re-install Globus

Execute commands in installation directory as user 'globus'

Deinstallation

This procedure run as 'root' will delete the whole installation and configuration of Globus from the machine:

Remove installation directories and files
Remove the Globus entries, if they are available.
remove for GSI
remove for GRAM
remove for RFT
delete postgressql packages with YaST or YUM

Update

The updates for the 64-bit architecture are available. For the 32-bit machines were the same packages with gcc32 packages installed. More current fixes can be found here.

WARNING:  Globus 4.0.7 fixes:
It is highly recommended to install the following updates after the globus installation.

All the updates can be downloaded from here and must be installed by user globus:

administrator's script: update.sh

<tr style="align:center; valign:top; background-color:#FFF; border:dotted 1px #A8A8A8; margin:0.1em 0em; font-size:95%; color:#808000; width:95%; height:1%; font-family:'Verdana','Georgia'; font-size:11.5px; padding:0 7px;" >

<td>

# updates for the globus 4.2.1
griduser='griduser'

# REINSTALLATION
su globus
make distclean
rm -rf $GLOBUS_LOCATION/*
cd /tmp
$GPT_LOCATION/sbin/gpt-install <GT4_binary.tar.gz>
$GPT_LOCATION/sbin/gpt-postinstall

# DEINSTALL
## 1.1. Remove installation directories and files
su root
rm /etc/profile.d/globus-env-setup.sh
rm /usr/local/bin/globus-env-setup.sh
rm -rf /usr/local/gpt-3.2
rm -rf /usr/local/globus

## 1.2. Remove the Globus entries, if they are available.
vi /etc/hosts.allow

## 1.3. Delete user 'globus'
userdel globus
rm -rf /home/globus
groupdel globus

## 1.4. for GSI:
rm -rf /etc/grid-security

su $griduser
rm -rf ~/.globus (for the grid user)

## 1.5. for GRAM:
su root
rm /etc/xinetd.d/gsi*
vi /etc/sudoers

## 1.6. for RFT:
su root
rm -rf /var/lib/pgsql/data/*

## 1.7. Delete the following packages with YUM:
yum remove postgressql
yum remove postgressql-libs
yum remove postgressql-server

# UPDATE
# as 'globus' user install the new updates after downlowding them
# from http://www.globus.org/toolkit/advisories.html:
su globus
gpt-build -update globus_java_ws_core_common-1.15.tar.gz
gpt-build -update globus_java_ws_core-3.18.tar.gz
gpt-build -update globus_gass_cache-5.4.tar.gz gcc64dbg
gpt-build -update globus_gsi_cert_utils-5.5.tar.gz gcc64dbg

</td> </tr></table>

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Globus Toolkit services

middleware:Globus/420

Contents

Introduction

Globus Toolkit server v.4.2

Prepare

Install

Configure

Proceed

Initial test

Update

Globus Toolkit services

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox