cluster:Interactive/2
Contents |
Introduction
|
The current D-Grid reference installation is combined together the login server with the interactive node on one server.
That allow the connection and access from the User Interface into the D-Grid reference installation site and to work with the grid applications and also to compile and test jobs using local libraries. |
|
Login server
Prepare
- Operating system
- Scientific Linux version 5.4 64 bit
Optimizing the configuration:
Use minimal operating system installation without firewall. To verify installed packages use the command
-
rpm -qa | grep package_name
Install the following additional packages:
-
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command:
-
chkconfig <SERVICE> off
Configure the following settings for the server:
- Additional software
- Perl version >= 5.8.8 (with XML::PARSER)
- Firewall configuration
- Access via GSISSH (TCP port 2222) to the gsissh server should be allowed (how to open port in firewall).
administrator's script: prepare.sh
#!/bin/bash# prepare the interactive node installation# Declare the variables section ------------# Please insert your actual configuration# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~echo `su -`
path_gpt=/usr/local/gpt-3.2
path_globus=/usr/local/globus
# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#-> start routineyum -y install gcc
echo "\
export GPT_LOCATION=${path_gpt}export GPT_INSTALL_LOCATION=${path_gpt}export GLOBUS_LOCATION=${path_globus}export PATH=\$GPT_LOCATION/sbin:\$PATH" > /usr/local/bin/gpt-env-setup.sh
# The file should be marked as executable:chmod +x /usr/local/bin/gpt-env-setup.sh
source /usr/local/bin/gpt-env-setup.sh
#In order to execute this script for every user automatically, link in the directory /etc/profile.d as follows:ln -s /usr/local/bin/gpt-env-setup.sh /etc/profile.d
# create user Globusgroupadd globus
useradd -m -g globus -d /localhome/globus globus
umask 022mkdir -p $path_globus $path_gpt
chown -R globus.globus $path_globus
chown -R globus.globus $path_gpt
chmod 755 $path_globus
chmod 755 $path_gpt
#<- end routine
Install
- downloads and install the Grid Packaging Toolkit (GPT)
- download gsissh package
- install gsissh by using the gpt toolkit
- execute gpt-postinstall
administrator's script: install.sh
#!/bin/bash# As user 'globus' download the binary package for gsisshsu - globusworkdir=/tmp
PACKAGE=Lrz_SL5.3_gsissh-gsissh4.4-jdk-1.5-x86_64-unknown-linux-gnu-bin.tar.gz
# GSI-SSH installation# As user 'globus' download the binary package for gsissh# download the gsissh packagewget -O $workdir/gpt-3.2-src.tar.gz http://mirror.scc.kit.edu/downloads/src/globus/2009.2/gpt-3.2-src.tar.gz
cd $workdir
tar zxvf $workdir/gpt-3.2-src.tar.gz
cp -rf $workdir/gpt-3.2/* /usr/local/gpt-3.2/
cd /usr/local/gpt-3.2/
./build_gptrm -rf $workdir/gpt-3.2*
wget -O $workdir/$PACKAGE http://mirror.scc.kit.edu/downloads/src/interactive/2010.1/$PACKAGE
$GPT_LOCATION/sbin/gpt-install $workdir/$PACKAGE 2>&1 | tee gsissh-install.log
# Bundle Lrz_SL5.3_gsissh successfully installed.# Cleaning up temp locations$GPT_LOCATION/sbin/gpt-postinstall 2>&1 | tee gsissh-postinstall.log
Configure
Please check the grid certificates allocation before to proceed.
WARNING: Grid-mapfile:
The grid-mapfile located in the directory /etc/grid-security/ is necessary for the successful launch of the gsissh.
If you have none of it, an empty file with the name grid-mapfile should be created.
To use the gsissh, user globus must be authenticated. There are two ways for user authentication in Globus:
1 - manually
- as
grid-userextract the Distinguished name (DN) as the following:
su $USER grid-cert-info -subject /C=DE/O=GridGermany/OU=Leibniz-Rechenzentrum/CN=griduser1
- Now, the grid-user should be included to the grid-mapfile. For this, execute as
root:
su root grid-mapfile-add-entry -dn “<DN>” -ln <locale user name for griduser1>
2 - automaticly
- There is the dgridmap script for automatic generation of the grid-mapfile. It is mandatory to execute the dgridmap script regularly (e.g. cron minimum once per 24 hours) for update of the Globus grid-mapfile (see section join D-Grid).
administrator's script: configure.sh
#!/bin/bashGSISSHD_EXEC_NAME=gsisshd# GSISSHD Default Configuration for D-GRID#############################################if [ ! -d $GLOBUS_LOCATION/etc/ssh ] ; then
mkdir -p $GLOBUS_LOCATION/etc/ssh
fiif [ ! -f $GLOBUS_LOCATION/sbin/gsisshd ] ; then
ln -s $GLOBUS_LOCATION/sbin/sshd $GLOBUS_LOCATION/sbin/gsisshd
ficat > $GLOBUS_LOCATION/etc/ssh/sshd_config << EOF
# \$OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp \$# This is the gsisshd server system-wide configuration file. See# sshd_config(5) for more information.# This gsisshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin# The strategy used for options in the default sshd_config shipped with# OpenSSH is to specify options with their default value where# possible, but leave them commented. Uncommented options change a# default value.Port 2222#AddressFamily any#ListenAddress 0.0.0.0#ListenAddress ::# Disable legacy (protocol version 1) support in the server for new# installations. In future the default will change to require explicit# activation of protocol 1Protocol 2# HostKey for protocol version 1#HostKey /etc/ssh/ssh_host_key# HostKeys for protocol version 2#HostKey /etc/ssh/ssh_host_rsa_key#HostKey /etc/ssh/ssh_host_dsa_key# Lifetime and size of ephemeral version 1 server key#KeyRegenerationInterval 1h#ServerKeyBits 1024# Logging# obsoletes QuietMode and FascistLogging#SyslogFacility AUTH#LogLevel INFO# Authentication:#LoginGraceTime 2m#PermitRootLogin yes#StrictModes yes#MaxAuthTries 6#MaxSessions 10#RSAAuthentication yes#PubkeyAuthentication yes#AuthorizedKeysFile .ssh/authorized_keys# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts#RhostsRSAAuthentication no# similar for protocol version 2#HostbasedAuthentication no# Change to yes if you don't trust ~/.ssh/known_hosts for# RhostsRSAAuthentication and HostbasedAuthentication#IgnoreUserKnownHosts no# Don't read the user's ~/.rhosts and ~/.shosts files#IgnoreRhosts yes# To disable tunneled clear text passwords, change to no here!#PasswordAuthentication yes#PermitEmptyPasswords no# Change to no to disable s/key passwords#ChallengeResponseAuthentication yes# Kerberos options#KerberosAuthentication no#KerberosOrLocalPasswd yes#KerberosTicketCleanup yes#KerberosGetAFSToken no# Session hooks: if allowed, specify the commands to execute#AllowSessionHooks yes#SessionHookStartupCmd /bin/true#SessionHookShutdownCmd /bin/true# GSSAPI options#GSSAPIAuthentication yes#GSSAPIDelegateCredentials yes#GSSAPICleanupCredentials yes#GSSAPIStrictAcceptorCheck yes#GSSAPIKeyExchange yes# Set this to 'yes' to enable PAM authentication, account processing,# and session processing. If this is enabled, PAM authentication will# be allowed through the ChallengeResponseAuthentication and# PasswordAuthentication. Depending on your PAM configuration,# PAM authentication via ChallengeResponseAuthentication may bypass# the setting of "PermitRootLogin without-password".# If you just want the PAM account and session checks to run without# PAM authentication, then enable this but set PasswordAuthentication# and ChallengeResponseAuthentication to 'no'.#UsePAM yes#AllowAgentForwarding yes#AllowTcpForwarding yes#GatewayPorts noX11Forwarding yes#X11DisplayOffset 10#X11UseLocalhost yes#PrintMotd yes#PrintLastLog yes#TCPKeepAlive yes#UseLogin no#UsePrivilegeSeparation yes#PermitUserEnvironment no#Compression delayed#ClientAliveInterval 0#ClientAliveCountMax 3#UseDNS yesPidFile $GLOBUS_LOCATION/var/gsisshd.pid#MaxStartups 10#PermitTunnel no#ChrootDirectory none# no default banner path#Banner none# override default of no subsystemsSubsystem sftp $GLOBUS_LOCATION/libexec/sftp-server# the following are HPN related configuration options# tcp receive buffer polling. disable in non autotuning kernels#TcpRcvBufPoll yes# allow the use of the none cipher#NoneEnabled no# disable hpn performance boosts.#HPNDisabled no# buffer size for hpn to non-hpn connections#HPNBufferSize 2048# Example of overriding settings on a per-user basis#Match User anoncvs# X11Forwarding no# AllowTcpForwarding no# ForceCommand cvs serverEOF#GLOBUS_SETUP_SCRIPTcat > $GLOBUS_LOCATION/libexec/globus-script-initializer << EOF
exec_prefix=\${GLOBUS_LOCATION}prefix=\$GLOBUS_LOCATIONsbindir=\${exec_prefix}/sbinbindir=\${exec_prefix}/binlibdir=\${exec_prefix}/liblibexecdir=\${exec_prefix}/libexecincludedir=\${exec_prefix}/includedatadir=\${prefix}/sharesysconfdir=\${prefix}/etcsharedstatedir=\${prefix}/comlocalstatedir=\${prefix}/vartmpdir=/tmplocal_tmpdir=/tmpsecure_tmpdir=/tmpDELIM=if [ -n "\${LD_LIBRARY_PATH}" ]; thenDELIM=:fiLD_LIBRARY_PATH="\${GLOBUS_LOCATION}/lib\${DELIM}\${LD_LIBRARY_PATH}"DELIM=if [ -n "\${DYLD_LIBRARY_PATH}" ]; thenDELIM=:fiDYLD_LIBRARY_PATH="\${GLOBUS_LOCATION}/lib\${DELIM}\${DYLD_LIBRARY_PATH}"if [ -z "\${LIBPATH}" ]; thenLIBPATH="/usr/lib:/lib"fiLIBPATH="\${GLOBUS_LOCATION}/lib:\${LIBPATH}"DELIM=if [ -n "\${SHLIB_PATH}" ]; thenDELIM=:fiSHLIB_PATH="\${GLOBUS_LOCATION}/lib\${DELIM}\${SHLIB_PATH}"if [ -d \$GLOBUS_LOCATION/lib/sasl ]; thenDELIM=if [ -n "\${SASL_PATH}" ]; thenDELIM=:fiSASL_PATH="\${GLOBUS_LOCATION}/lib/sasl\${DELIM}\${SASL_PATH}"fiexport LD_LIBRARY_PATH DYLD_LIBRARY_PATH LIBPATH SHLIB_PATH SASL_PATHif [ -n "\${LD_LIBRARYN32_PATH}" ]; thenDELIM=""if [ "X\${LD_LIBRARYN32_PATH}" != "X" ]; thenDELIM=:fiLD_LIBRARYN32_PATH="\${GLOBUS_LOCATION}/lib\${DELIM}\${LD_LIBRARYN32_PATH}"export LD_LIBRARYN32_PATHfiif [ -n "\${LD_LIBRARY64_PATH}" ]; thenDELIM=""if [ "X\${LD_LIBRARY64_PATH}" != "X" ]; thenDELIM=:fiLD_LIBRARY64_PATH="\${GLOBUS_LOCATION}/lib\${DELIM}\${LD_LIBRARY64_PATH}"export LD_LIBRARY64_PATHfiglobus_source () {# Check if file exists and source itif [ ! -f "\$1" ] ; then\${GLOBUS_SH_PRINTF-printf} "\$1 not found.\n" >&2exit 1fi. "\$1"}EOF#INIT SCRIPT FOR GSISSHDcat > /etc/init.d/gsissh << EOF
#!/bin/sh## Init file for GSI-OpenSSH server daemon## chkconfig: 2345 55 25# description: GSI-OpenSSH server daemon#### BEGIN INIT INFO# Provides: gsisshd# Required-Start: \$network# Required-Stop: \$network# Default-Start: 2 3 4 5# Default-Stop: 0 1 6# Description: Start the gsisshd daemon### END INIT INFOGLOBUS_LOCATION="/usr/local/globus"
export GLOBUS_LOCATION. \${GLOBUS_LOCATION}/libexec/globus-script-initializer
PID_FILE=\${localstatedir}/gsisshd.pid
## SSHD arguments can be added here within the following# set of double quotes.#SSHD_ARGS="-f $GLOBUS_LOCATION/etc/ssh/sshd_config"
do_start()
{if [ ! -d \$localstatedir ]; then
mkdir -p \$localstatedir
fiecho -n "Starting up GSI-OpenSSH gsisshd server... "
\${sbindir}/gsisshd \$SSHD_ARGS > /dev/null 2>&1 &
if [ \$? -eq 0 ]; then
echo "done."
elseecho "failed to start GSI-OpenSSH gsisshd server!"
fi}do_stop()
{echo -n "Stopping the GSI-OpenSSH gsisshd server... "
pid=\`cat \$PID_FILE\`
kill -TERM \$pid
sleep 2
kill -TERM \$pid 2> /dev/null
rm -f \$PID_FILE
echo "done."
}case "\$1" in
start)if [ ! -f \$PID_FILE ]; then
do_start
elsepid=\`cat \$PID_FILE\`
psout=\`ps -A | grep \$pid | grep -v grep | awk "{if (\\\\\\\$1 == \$pid) print}"\`
if [ "x\$psout" = "x" ]; then
echo "Found stale gsisshd pid file... removing it."
rm -f \$PID_FILE
do_start
elseecho "GSI-OpenSSH gsisshd server is already running!"
fifi;;stop)if [ -f \$PID_FILE ] ; then
do_stop
elseecho "The server's pid file does not exist! Are you sure the server is running?"
fi;;restart)\$0 stop\$0 start;;*)
echo "Usage: \$0 (start|stop|restart)"
exit 1
esacexit 0
EOF
#Exec permissions for startup scriptchmod +x /etc/init.d/gsissh
#Add to bootup sequencechkconfig --add gsisshif grep -q gsissh /etc/services ; then
echo "gsissh 2222/tcp" >> /etc/services ;
fiif grep -q gsisshd /etc/hosts.allow ; then
echo "gsisshd:ALL:ALLOW" >> /etc/hosts.allow
fi
Proceed
To start/stop services use the commands:
administrator's script: proceed.sh
#!/bin/bashservice gsissh [start|stop|restart]
Initial test
To test the following commands can be used:
administrator's script: test.sh
#!/bin/bash# initial test for interactive node# Log on into a grid client (eg. with ssh)ssh griduser@gridclient
# logged in griduser@gridclient# GSI Authenticationgrid-proxy-init
# Your identity: /C=DE/O=GermanGrid/OU=FZK/CN=Grid User# Enter GRID pass phrase for this identity:# Creating proxy ...................................................... Done# Your proxy is valid until: Fri Aug 15 23:27:06 2008# GSI-SSH Access to login nodegsissh -p 2222 dgiref-login.fzk.de
#********************************************************## Welcome to the D-Grid-Referenzinstallation:## - Interactive node Installation:## * GSI-SSH (Globus 4.0.7 64 bit)# * gLite WN_TAR#**********************************************************# dgdt0001@dgiref-login.fzk.de:~>
Update
- deinstall software by the user
root
administrator's script: update.sh
#!/bin/bash# Deinstallation# This procedure run as 'root' will delete the whole installation and configuration of Globus from the machine:# Remove installation directories and filessu rootrm /etc/profile.d/globus-env-setup.sh
rm /usr/local/bin/globus-env-setup.sh
rm -rf /usr/local/globus
# Remove the Globus entries, if they are available.vi /etc/hosts.allow
# Delete user 'globus'userdel globus
rm -rf /home/globus
groupdel globus
# for GSI:rm -rf /etc/grid-security
su 'grid-user'
rm -rf ~/.globus (for the grid user)
# Remove the GSI-SSH start script:su rootchkconfig --del gsisshdrm -f /etc/init.d/gsisshd
Worker node
Prepare
- Operating system
- Scientific Linux version 5.6 64 bit
Optimizing the configuration:
Use minimal operating system installation without firewall. To verify installed packages use the command
-
rpm -qa | grep package_name
Install the following additional packages:
-
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command:
-
chkconfig <SERVICE> off
Configure the following settings for the server:
- proxy
- ntp
- script:/etc/resolv.conf
- Torque for WN
- UMD Repo
- Firewall configuration
Allowing incoming connections directed to the WNs is optional and Resource Providers can freely decide whether to permit them on a voluntary base. However, when such inbound connections are blocked, data transfers using GridFTP will be forced to work in "single-stream" mode and their performance might be accordingly degraded (how to open port in firewall).
| Service | Incoming ports (TCP) | Change to default configuration |
| GridFTP | 20000-25000 | Yes |
| WN should have an access to external network |
administrator's script: prepare.sh
#!/bin/bash# install umd#clean oldsudo - su
rm /etc/yum.repos.d/UMD* /etc/yum.repos.d/epel*
wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
wget http://repository.egi.eu/sw/production/umd/1/sl5/x86_64/updates/umd-release-1.0.2-1.el5.noarch.rpm
rm -f epel-release-5-4.noarch.rpm umd-release-1.0.2-1.el5.noarch.rpm
yum install epel-release-5-4.noarch.rpm
yum install yum-prioritiesyum install umd-release-1.0.2-1.el5.noarch.rpm
sed -i -e "s/priority=.*/priority=5/g" /etc/yum.repos.d/UMD-1-base.repo
sed -i -e "s/priority=.*/priority=4/g" /etc/yum.repos.d/UMD-1-updates.repo
# Missing packages installationyum -y install gcc gcc-c++ openssl
#<- end routine
Install
There are some types of packages to install into the Cluster Node that it will provide the Worker Node functionality
- glite-WN packages to operate with Grid middleware
administrator's script: install.sh
#!/bin/bash# install worker node packages# install glite-WN-3.2yum -y install glite-yaim-torque-utils.noarch yaim-voms.x86_64 glite-yaim-clients.noarch glite-yaim-core.noarch glite-yaim-mpi.noarch glite-yaim-torque-client.noarch lcgutil-yaim.noarch lfc-yaim.noarch yaim-glexec-wn.noarch
Configure
- Mount File system
- Configure users
- Prepare WNs for gLite
- The packages for the gLite middleware and OGSA-DAI will be provided by NFS server.
- The middleware configuration for all Worker nodes is unique to any WN.
- This require the corresponding WN performs the write rights for the configuration scripts to the directory /opt/glite-MW.
- This directory mounts with appropriate write rights.
- Rights configuration can be changed later, after general configuration.
- The specific configuration can be implemented using the prepared templates on the: http://www.d-grid.de/index.php?id=132
- The info.def, groups.conf and users.conf files are required for the WN configuration.
| Note: The site-info.def have the JAVA_LOCATION which should be configured!
|
WARNING:
The dgrid_env.sh script should be edited and the variables VOS, INSTALL_ROOT and DGRID_VO_DIRECTORY adjusted. The script ensures that only the D-Grid VOs users used the middleware environment variables.
| The dgrid_env.sh is calling another script - grid_env.sh
|
- Optional adjustment: In order to accelerate the WN configuration, the Certificates and CRLs configurations can be denied (they will be executed on the gLite-CE). This require removing the following functions from the
$GLITE_DIR/glite/yaim/scripts/node-info.def:- install_certs_userland
- config_fix_edg-fetch-crl-cron
- config_crl
by using the function TAR_WN_FUNCTIONS.
| The following error message is NOT important: [ERROR] Failed to add group
|
administrator's script: configure.sh
Update
administrator's script: update.sh