cluster:Torque/216/server

From Dgiref

See also troubleshooting for this page.

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Torque/216/server

Prepare

Operating system: Scientific Linux version 4.5 64 bit

Optimizing the configuration: 

Use minimal operating system installation without firewall.
To verify installed packages use the command 
rpm -qa | grep package_name
Install the following additional packages:
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command: 
chkconfig <SERVICE> off

Configure the following settings for the server:

Firewall configuration

Be sure that if you have firewalls running on the server or node machines that you allow connections on the appropriate ports for each machine. TORQUE pbs_mom daemons use UDP port 1023 and the pbs_server/pbs_mom daemons use ports 15001-15004 by default (how to open port in firewall).

Firewall based issues are often associated with server to mom communication failures and messages such as 'premature end of message' in the log files.

Also, the tcpdump program can be used to verify the correct network packets are being sent.

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare
```

REPO_URL="http://dgiref.d-grid.de/svn/dgiref/PROD/cf3/repl/repos/external/"

wget -O /etc/yum.repos.d/sl-dgiref.repo ${REPO_URL}/sl-dgiref.repo

Install

The D-Grid reference installation has the Torque batch system for jobs management. To install the Torque batch system:

Torque server installation (on which the pbs_server and maui scheduler are running)
Torque clients installation (gLite-CE, Globus GRAM und UNICORE NJS) which sent a Batch-jobs
Worker Nodes (cluster, on which the pbs_mom daemon is running)

WARNING: 
The three middleware frontends for Job Submission (gLite-CE, Globus GRAM and Unicore NJS) must be configured as torque clients with MAUI. The following packages are required in each of the three front ends (unless specifically stated, the steps as root):

torque
torque-client
maui
maui-client

Unless specifically stated, the installation and configuration steps as root user.

Selecting the cluster scheduler is an important decision and significantly affects cluster utilization, responsiveness, availability, and intelligence. The default TORQUE scheduler, pbs_sched, is very basic and will provide poor utilization of your cluster’s resources. Other options, such as Maui Scheduler is highly recommended. If using pbs_sched, start this daemon now. If using Maui, refer to the next installation steps.

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install torque server
```
```
# Choose the OS architecture: 
```
```
OS_arch="x86_64"   # x86_64 for 64 bit
```
```
# OS_arch="i686"   # i686 for 32 bit
```
```
 
```
```
# Install packages
```

yum -y install torque-2.1.6-1cri_2dgrid_sl4.${OS_arch} torque-server-2.1.6-1cri_2dgrid_sl4.${OS_arch} torque-client-2.1.6-1cri_2dgrid_sl4.${OS_arch}

yum -y install maui-3.2.6p19-snap1171482917_1dgrid_sl4.${OS_arch} maui-server-3.2.6p19-snap1171482917_1dgrid_sl4.${OS_arch}

Configure

Configure torque server
- Setup WNs addresses. Configuration file /var/spool/pbs/server_priv/nodes on the torque server must hold all host names of the WNs information.
- Fill out the configuration file /etc/hosts.equiv with the list of the FQDNs for all Torque Clients, e.g. gLite-CE, Globus GRAM and Unicore NJS
Configure maui
Create users for gLite: edginfo und rgma with the same UIDs like in gLite-CE. These users information are required to gLite with. The gLite-CE asks the MAUI server about this.
Create queues

The D-Grid operational concept provides for the two queues: dgiseq for sequential and dgipar for parallel jobs (the currently assigned queue limits are based on the values, as they are used in GridKa)

Make NFS export /var/spool/pbs/server_logs

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# configure Torque server and maui
```
```
# Torque Server configuration
```
```
HOST=`hostname -f`
```
```
echo $HOST > /var/spool/pbs/server_name
```
```
 
```
```
# Setup WNs addresses 
```
```
WN=<FQDN for WNs> 
```
```
NP=<Number of processors in WN>
```

echo "$WN np=$NP" >> /var/spool/pbs/server_priv/nodes

```
 
```
```
 
```
```
#Fill out the configuration
```
```
echo "$GLITE_HOST" >> /etc/hosts.equiv
```
```
echo "$GLOBUS_HOST" >> /etc/hosts.equiv
```

echo "$UNICORE_HOST" >> /etc/hosts.equiv

```
 
```
```
 
```
```
# Maui configuration
```
```
HOST=`hostname -f`  
```

# In some cases the alias host name command HOST=`hostname -a` should be run. But the previous should work.

sed -e "s/localhost/$HOST/" -i /var/spool/maui/maui.cfg

```
 
```
```
cat << EOF >> /var/spool/maui/maui.cfg
```

# The users edginfo and rgma query Maui from the glite-CE to provide data to the InfoSystem

```
ADMIN3            edginfo rgma 
```

# Set the delay to 1 minute before Maui tries to run a job again,

# in case it failed to run the first time.

```
# The default value is 1 hour. 
```
```
DEFERTIME       00:01:00 
```
```
# Necessary for MPI grid jobs 
```
```
ENABLEMULTIREQJOBS TRUE 
```
```
EOF 
```
```
 
```
```
# Create users for gLite 
```
```
useradd -u $EDGINFO_UID edginfo 
```
```
useradd -u $RGMA_UID rgma
```
```
 
```
```
# Create queues
```
```
 
```
```
qmgr <<+ 
```
```
create queue dgiseq 
```

set queue dgiseq queue_type = Execution

```
set queue dgiseq Priority = 20 
```

set queue dgiseq resources_max.cput = 48:00:00

set queue dgiseq resources_max.pcput = 48:00:00

set queue dgiseq resources_max.walltime = 96:00:00

set queue dgiseq resources_max.ncpus = 1

set queue dgiseq resources_max.nodect = 1

set queue dgiseq resources_default.cput = 48:00:00

set queue dgiseq resources_default.pcput = 48:00:00

set queue dgiseq resources_default.walltime = 96:00:00

set queue dgiseq resources_default.nodes = nodes=1:ppn=1

```
set queue dgiseq enabled = True
```
```
set queue dgiseq started = True
```
```
create queue dgipar
```
```
set queue dgipar queue_type = Execution
```
```
set queue dgipar Priority = 10
```

set queue dgipar resources_max.cput = 960:00:00

set queue dgipar resources_max.pcput = 48:00:00

set queue dgipar resources_max.ncpus = 40

set queue dgipar resources_max.nodect = 10

set queue dgipar resources_max.walltime = 96:00:00

set queue dgipar resources_default.ncpus = 2

set queue dgipar resources_default.cput = 96:00:00

set queue dgipar resources_default.pcput = 48:00:00

set queue dgipar resources_default.walltime = 96:00:00

```
set queue dgipar enabled = True
```
```
set queue dgipar started = True
```
```
+
```
```
 
```

# Make NFS export /var/spool/pbs/server_logs

```
GLOBUS_HOST=<IP of GLOBUS-GRAM> 
```

echo "/var/spool/pbs/server_logs   ${GLOBUS_HOST}(ro,async,no_root_squash)" >> /etc/exports

```
exportfs -r
```
```
 
```
```
exit 0
```

Proceed

To start / stop use the commands:

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
# proceed
```
```
# start server
```
```
chkconfig pbs_server on 
```
```
/etc/init.d/pbs_server start
```
```
 
```
```
## start maui
```
```
chkconfig maui on 
```
```
/etc/init.d/maui start
```
```
 
```
```
exit 0
```

Initial test

The pbs_server daemon was started on the TORQUE server when the torque.setup file was executed or when it was manually configured. It must now be restarted so it can reload the updated configuration changes.

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
# shutdown server
```
```
qterm -t quick
```
```
# start server
```
```
chkconfig pbs_server on 
```
```
/etc/init.d/pbs_server start
```

# verify all queues are properly configured

```
qstat -q
```
```
# view additional server configuration
```
```
qmgr -c 'p s'
```

# verify all nodes are correctly reporting

# If everything Ok, you can see the all WNs details with the WN status "free".

```
pbsnodes -a 
```
```
# submit a basic job
```
```
echo "sleep 30" | qsub
```
```
# verify jobs display
```
```
qstat
```
```
# check queues:
```
```
qstat -Q
```

#Queue   |   Max  | Tot |  Ena |  Str  | Que  | Run  | Hld  | Wat  | Trn  | Ext T

#dgiseq  |     0  |   0 |  yes |  yes  |   0  |   0  |   0  |   0  |   0  |   0 E

#dgipar  |     0  |   0 |  yes |  yes  |   0  |   0  |   0  |   0  |   0  |   0 E

```
 
```
```
exit 0
```

Update

Upgrade: Upgrading TORQUE can generally be done without shutting down the whole cluster and disrupting running jobs. Simply build and install the new version and restart the daemons. Here is the safest procedure for upgrading TORQUE:

Kill the scheduler.
Wait a few minutes for all new jobs to complete startup.
1. All running jobs in qstat -a have some elapsed walltime.
Restart pbs_server.
Verify the new pbs_server is working correctly.
1. nodes should come up (not down or state-unknown)
2. job walltimes should increase
If upgrading from an earlier 2.1 build, MOMs can automatically restart themselves with:
1. momctl -q enablemomrestart=1 -h :ALL
2. Start the scheduler.
If upgrading from 2.0 or earlier,
1. Restart MOMs on all idle nodes.
2. Wait a minute, make sure node and job states are updating correctly.
3. Delete the previous static archive library files: (libattr.a, libcmds.a, liblog.a, libnet.a, libpbs.a, libsite.a
4. Mark busy nodes offline.
5. Start the scheduler.
6. Restart MOMs on offline nodes after their jobs exit.

All external software like maui, perl-PBS, or pbs_python built with the 2.0.x static archives will need to be rebuilt with the newer 2.1.x shared libraries.

administrator's script: update.sh

```
 
```
```
#!/bin/bash 
```
```
# Reinstall packages:
```

rpm -Uvh --force package-to-reinstall.rpm

```
 
```
```
# delete installation:
```
```
rpm -Uvh --force package-to-delete.rpm
```
```
exit 0
```

cluster:Torque/216/server

Contents

Torque/216/server

Prepare

Install

Configure

Proceed

Initial test

Update

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox