middleware:Globus/50/server

From Dgiref

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Globus Toolkit server v.5.0.1

Prepare

Operating system

Scientific Linux v.5.4 64 bit

Optimizing the configuration: 

Use minimal operating system installation without firewall.
To verify installed packages use the command 
rpm -qa | grep package_name
Install the following additional packages:
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command: 
chkconfig <SERVICE> off

Configure the following settings for the server:

Additional Software

torque & maui client
java SDK >= 1.6.0 (from Sun, IBM, HP, or BEA (do not use GCJ))
Grid Packaging Toolkit (GPT)
Perl 5.8.8 (with XML::PARSER)
sudo
ant version >= 1.6.5

To make all libraries from ant available do the following as root:

 $ cd /usr/share/ant/lib
 $ ln -s /usr/share/java/ant.jar ant.jar
 $ ln -s /usr/share/java/ant-launcher.jar ant-launcher.jar

Each grid user needs a certificates directory . As grid user create this directory by:

mkdir -p $home/.$user_globus
chown $user_grid /localhome/.$user_globus

Firewall configuration

The GT5 frontend runs Grid Security Infrastructure (GSI) FTP and GSI GATEKEEPER services (how to open port in firewall).

Service	Incoming ports (TCP)	Change to default default
gsigatekeeper	2119	No
gsiftp	2811	No

administrator's script: prepare.sh

```
 
```
```
#!/bin/sh
```
```
 
```
```
su root
```

# prepare the installation of Globus 4.0.8

# Declare the variables section ------------

```
# user_globus=globus local user
```

# user_grid=non root and non globus grid user

```
# path_globus=globus localtion
```

# path_certificates=certificates location

```
# path_gpt=gpt-3.2 location
```
```
# JAVA_HOME=java home location
```
```
# ANT_HOME=ant home location
```

# Please insert your actual configuration

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
user_globus=globus
```
```
 
```
```
path_security=/etc/grid-security
```

path_certificates=${path_security}/certificates

```
path_workdir=/usr/local
```

path_globus=${path_workdir}/globus-5.0.1

```
 
```
```
path_java=${path_workdir}/jdk1.6.0_16
```
```
JAVA_HOME=/etc/alternatives/jdk
```
```
 
```

BASE_URL="http://mirror.scc.kit.edu/downloads/src"

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
 
```
```
yum -y install sudo xinetd java
```

yum -y install gcc perl.x86_64 perl-XML-Parser

```
yum -y install mysql-server.x86_64
```
```
 
```
```
service mysqld restart
```
```
chkconfig mysqld on
```

#------------------------------------------------------ install SUN java

wget -O ${path_workdir}/jdk-6u16-linux-x64.bin ${BASE_URL}/misc/jdk-6u16-linux-x64.bin

chmod +x ${path_workdir}/jdk-6u16-linux-x64.bin

```
cd ${path_workdir}
```
```
./jdk-6u16-linux-x64.bin
```

rm ${path_workdir}/jdk-6u16-linux-x64.bin

```
 
```
```
# link to the /opt/java
```
```
rm /usr/java/latest
```
```
mkdir /usr/java/
```
```
ln -s ${path_java} /usr/java/latest 
```
```
rm -f /etc/alternatives/java
```

ln -s ${path_java}/bin/java /etc/alternatives/java

ln -s ${path_java}/ /etc/alternatives/jdk

```
 
```
```
# test after restart bash session
```
```
java -version
```
```
 
```
```
# Create user and group globus:
```
```
groupadd $user_globus 
```

useradd -m -g $user_globus -d /localhome/$user_globus $user_globus

```
umask 022 
```
```
 
```

##------------------------------------------------------ (GSI):

```
mkdir -p $path_certificates
```

# after copy the host certificate and host key into /etc/grid-security/, configure GSI Security

# cp yourhostkey.pem ${path_security}/hostkey.pem

# cp yourhostcert.pem ${path_security}/hostcert.pem

cp ${path_security}/hostkey.pem ${path_security}/containerkey.pem

cp ${path_security}/hostcert.pem ${path_security}/containercert.pem

chown $user_globus.$user_globus ${path_security}/container*.pem

```
# set certificate privileges
```
```
chmod 400 ${path_security}/*key.pem
```
```
chmod 644 ${path_security}/*cert.pem
```
```
#create an empty gridmap-file
```
```
touch ${path_security}/grid-mapfile
```

Install

Relocatable packages. This means that they can be installed in other folders apart from the default one. You can do this using the following command: rpm -Uvh --relocate /opt/globus-5.0.1=/usr/local/globus-5.0.1

Download rpms: globus-toolkit-5.0.1-0.1.x86_64.rpm, globus-5.0.1-AbA-extentions-0.1-0.1.x86_64.rpm.

What the globus-toolkit-5.0.1-0.1.x86_64.rpm is doing:
1. Creates the user globus if it doesn't exist in the system and sets ownership of the globus folder to globus.
2. Patches the pbs.pm file in the /opt/globus-5.0.1/lib/perl/Globus/GRAM/JobManager.pbs.pm
3. The package is not using the globus openssl libraries but those of the system.
4. Creates the folder /opt/globus-5.0.1/etc/DGridSetupFiles/ containing the following:

globus.sh: Globus environment setup. Can be copied to /etc/profile.d
adds gsigatekeeper and gsiftp in /etc/services if they do not exist
xinetd/gsiftp and xinetd/gsigatekeeper: Should be used with xinetd to start the gatekeeper and gridftp (just copy them to /etc/xinetd.d/ and do /etc/init.d/xinetd restart)
changes the /opt/globus-5.0.1/etc/globus-gatekeeper.conf and globus-job-manager.conf to reasonable, d-grid default values.
Sets default port for gsissh to 2222 and sets up a link called /opt/globus-5.0.1/sbin/gsisshd pointing to /opt/globus-5.0.1/sbin/sshd (that is so that the daemon reports in syslog as 'gsisshd')
Creates a preliminary script to set the environment variables: VO_VONAME_SW_DIR have the value $DGRID_VO_DIRECTORY/VONAME. The script needs to be modified so that the DGRID_VO_DIRECTORY gets the correct value.

Important note: if there is no host certificate in /etc/grid-security you should edit the /opt/globus-5.0.1/etc/globus-job-manager.conf to specify your globus-gatekeeper-subject. Finally it needs access to the grrs mysql server in Julich to receive the list of VO's.

What the globus-5.0.1-AbA-extentions-0.1-0.1.x86_64.rpm is doing:
1. The glite AbA related sources have been compiled so that they use system openssl and not globus openssl
2. Creates the files: /opt/glite/etc/lcmaps/lcmaps.db /opt/glite/etc/lcas/lcas.db with appropriate content
3. Outputs as a note during installation a command to receive the certificate for dgrid-voms.fzk.de in /etc/grid-security/vomsdir/dgrid-voms.fzk.de.7171.pem (this is not done automatically due to internet connectivity issues)
4. Creates the appropriate files and links under /etc/grid-security for the voms-attr-mappings
5. Due to a bug in gsisshd or in the lcas gt4 interface, the /etc/gsisshd script has to be changed. The script changes it only if it finds it in /etc/init.d/gsisshd otherwise it prints a message
6. Creates the /opt/glite/etc/vomses file with the necessary voms contact strings for each VO
7. Changes the /etc/xinetd.d/gsiftp and gsigatekeeper files if it finds them to include the new glite libraries

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```

BASE_URL=http://mirror.scc.kit.edu/downloads/rpms/globus/2010.2/

```
 
```

wget ${BASE_URL}/globus-toolkit-5.0.1-0.1.x86_64.rpm

wget ${BASE_URL}/globus-5.0.1-AbA-extentions-0.1-0.1.x86_64.rpm

```
# globus package
```

rpm -ihv globus-toolkit-5.0.1-0.1.x86_64.rpm

```
# ABA
```

rpm -ihv globus-5.0.1-AbA-extentions-0.1-0.1.x86_64.rpm

```
 
```
```
# 
```

openssl s_client -connect dgrid-voms.fzk.de:15001 -ssl3 2> /dev/null | sed -n '/CERTIFICATE/,/CERTIFICATE/p' > /etc/grid-security/vomsdir/dgrid-voms.fzk.de.7171.pem

Configure

To configure the globus 5.0.1 do:
- Have the hostcerts in /etc/grid-security, torque should also be installed.
- copy /opt/globus-5.0.1/etc/DGridSetupFiles/xinetd/gsiftp and xinetd/gsigatekeeper to /etc/xinetd.d/
- copy /opt/globus-5.0.1/sbin/SXXsshd to /etc/init.d/gsisshd
- set up the grid-mapfile and the /etc/grid-security/certificates folder
- check the /opt/glite-5.0.1/lib/perl/Globus/GRAM/JobManager/pbs.pm script if it contains correct values for the pbs commands.
To configure the ABA extensions do:
- run the openssl command that is produced as a note in the package: "openssl s_client -connect dgrid-voms.fzk.de:15001 -ssl3 2> /dev/null | sed -n '/CERTIFICATE/,/CERTIFICATE/p' > /etc/grid-security/vomsdir/dgrid-voms.fzk.de.7171.pem"
- edit the file /etc/grid-security/voms-attr-mappings along with the relevant users which reflect the roles per VO

administrator's script: configure.sh

```
 
```
```
#!/bin/sh
```
```
su - root
```
```
 
```
```
echo "\
```
```
globus-gatekeeper:ALL:ALLOW
```
```
globus-gridftp-server:ALL:ALLOW
```
```
" > /etc/hosts.allow
```
```
 
```
```
grep -r gsiftp /etc/services
```

# gsiftp          2811/tcp                        # GSI FTP

# gsiftp          2811/udp                        # GSI FTP

```
 
```
```
grep -r gsigatekeeper /etc/services
```

# gsigatekeeper   2119/tcp                        # GSIGATEKEEPER

# gsigatekeeper   2119/udp                        # GSIGATEKEEPER

```
 
```
```
cat /opt/glite/etc/lcas/lcas.db
```

pluginname=/opt/glite/lib64/modules/lcas_userban.mod

pluginargs="/etc/grid-security/grid-mapfile.deny"

```
 
```
```
cat /etc/xinetd.d/gsigatekeeper
```
```
service gsigatekeeper
```
```
{
```
```
socket_type             = stream
```
```
protocol                = tcp
```
```
wait                    = no
```
```
user                    = root
```

env                     = GLOBUS_LOCATION=/opt/globus-5.0.1

env                     += LD_LIBRARY_PATH=/opt/globus-5.0.1/lib:/opt/glite/lib64

env                     += LCMAPS_LOG_LEVEL=5

env                     += LCAS_LOG_LEVEL=5

env                     += LCMAPS_DEBUG_LEVEL=5

env                     += LCAS_DEBUG_LEVEL=5

server                  = /opt/globus-5.0.1/sbin/globus-gatekeeper

server_args             = -conf /opt/globus-5.0.1/etc/globus-gatekeeper.conf

```
disable                 = no
```
```
}
```
```
 
```
```
cat /etc/xinetd.d/gsiftp
```
```
service gsiftp
```
```
{
```
```
instances               = 100
```
```
socket_type             = stream
```
```
wait                    = no
```
```
user                    = root
```

env                     += GLOBUS_LOCATION=/opt/globus-5.0.1

env                     += LD_LIBRARY_PATH=/opt/globus-5.0.1/lib:/opt/glite/lib64

env                     += GLOBUS_TCP_PORT_RANGE=20000,25000

env                     += GLOBUS_HOSTNAME=dgiref-globus50.fzk.de

server                  = /opt/globus-5.0.1/sbin/globus-gridftp-server

```
server_args             = -i
```
```
log_on_success          += DURATION
```
```
disable                 = no
```
```
}
```
```
 
```
```
 
```

# Creates a preliminary script to set the environment variables

cat /opt/globus-5.0.1/etc/DGridSetupFiles/extras/Set_VO_SW_DIR.sh

```
#!/bin/sh
```
```
#Configuration:
```
```
MYSQLCMD=/usr/bin/mysql
```
```
UNAME="GRRS_UNAME"
```
```
PW="GRRS_PASS"
```
```
GRRS="zam275-v1.zam.kfa-juelich.de"
```
```
DGRID_VO_DIRECTORY="/opt/globus-5.0.1/"
```
```
 
```

VOOUT=$($MYSQLCMD -u $UNAME --password=$PW -h $GRRS -P 3306 -e "Select vo_long from dgrid_vo_list" dgrid)

```
echo "#!/bin/sh"
```

echo "export DGRID_VO_DIRECTORY=$DGRID_VO_DIRECTORY"

for i in $(echo "$VOOUT" | grep -v vo_long  | sed "s/[|+ ]//g") ; do

        echo "export VO_${i}_SW_DIR=\"\$DGRID_VO_DIRECTORY\"/${i}" ;

```
done
```
```
 
```

# check the /opt/glite-5.0.1/lib/perl/Globus/GRAM/JobManager/pbs.pm script if it contains correct values for the pbs commands.

cat /opt/globus-5.0.1/lib/perl/Globus/GRAM/JobManager/pbs.pm

```
# ...
```
```
# BEGIN
```
```
#{
```
```
#    $mpiexec        = 'no';
```
```
#    $mpirun         = 'no';
```
```
#    $qsub           = '/usr/bin/qsub';
```

#    $qstat          = '/usr/bin/qstat';

```
#    $qdel           = '/usr/bin/qdel';
```
```
#    $cluster        = 1;
```
```
#    $cpu_per_node   = 1;
```
```
#    $remote_shell   = '/usr/bin/ssh';
```
```
#    $softenv_dir    = '';
```

#    $soft_msc       = "$softenv_dir/bin/soft-msc";

#    $softenv_load   = "$softenv_dir/etc/softenv-load.sh";

```
#}
```
```
# ...
```
```
 
```
```
# make some links
```

ln -s /opt/globus-5.0.1/sbin/SXXsshd  /etc/init.d/gsisshd

ln -s /opt/globus-5.0.1/etc/DGridSetupFiles/globus.sh  /etc/profile.d/globus.sh

ln -s /opt/globus-5.0.1/etc/DGridSetupFiles/xinetd/gsiftp  /etc/xinetd.d/gsiftp

ln -s /opt/globus-5.0.1/etc/DGridSetupFiles/xinetd/gsigatekeeper  /etc/xinetd.d/gsigatekeeper

ln -s /opt/globus-5.0.1/etc/DGridSetupFiles/extras/Set_VO_SW_DIR.sh /etc/profile.d/Set_VO_SW_DIR.sh

```
 
```
```
# ABA
```

#edit the file /etc/grid-security/voms-attr-mappings along with the relevant users which reflect the roles per VO

```
echo "
```

\"/dgtest/admin/Role=softwareadmin/Capability=NULL\" dgdtsgm

" > /etc/grid-security/voms-attr-mappings

Proceed

(To startup the globus-services on boot you need to use the chkconfig command as usual)

/etc/init.d/gsisshd start
/etc/init.d/xinetd start

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
# start containers
```
```
/etc/init.d/gsisshd restart
```
```
/etc/init.d/xinetd restart 
```
```
 
```
```
chkconfig gsisshd on
```
```
chkconfig xinetd on
```

Initial test

Using a JDL file (job description language)
- The following job is described in XML format using the job description language (JDL). It is stated as a parameter during the job submission (see below).
- First, create as grid user the file torqueJob.xml and fill it with the following content (see script section #Using a JDL file)
Using the voms-proxy-init (e.g. voms-proxy-init -voms dgtest:/dgtest/admin:softwareadmin)

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
 
```
```
# test for globus toolkit 4.0.8
```
```
griduser='griduser'
```
```
gridclient="ui.d-grid.de"
```
```
 
```
```
# Generic Test
```

# Log on into a grid client (eg. with ssh)

```
ssh $griduser@$gridclient
```
```
 
```
```
#GSI Authentication
```
```
 
```
```
grid-proxy-init 
```
```
 
```

# Your identity: /C=DE/O=GermanGrid/OU=FZK/CN=Grid User

# Enter GRID pass phrase for this identity:

```
# Creating proxy .............. Done
```

# Your proxy is valid until: Fri Aug 15 23:27:06 2008

```
 
```
```
#Test Counter Service
```
```
# as griduser:
```

counter-client -s https://<FQDN>:8443/wsrf/services/CounterService

```
 
```
```
#Got notification with value: 3
```
```
#Counter has value: 3
```
```
#Got notification with value: 13
```
```
 
```

# Torque Test --------------------------------------------

# The successful integration of torque with Globus can be tested as grid user like here:

globusrun-ws -submit -F <FQDN> -Ft PBS  -c /bin/sleep 120

```
 
```
```
#Using a JDL file
```
```
echo " <job>
```
```
  <executable>/bin/echo</executable>
```
```
  <directory>/tmp</directory>
```

  <argument>***Hello D-Grid user***</argument>

```
  <stdout>/tmp/stdout_torque</stdout>
```
```
  <stderr>/tmp/stderr_torque</stderr>
```
```
  <queue>dgiseq</queue>
```
```
 </job>" > torqueJob.xml
```
```
 
```
```
#Then submit the grid job as griduser
```

globusrun-ws -submit -F <FQDN> -Ft PBS  -f torqueJob.xml

middleware:Globus/50/server

Contents

Globus Toolkit server v.5.0.1

Prepare

Install

Configure

Proceed

Initial test

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox