cluster:Torque/236

From Dgiref

Introduction

TORQUE (Tera-scale Open-source Resource and QUEue manager) is a resource manager providing control over batch jobs and distributed compute nodes. Maui is an advanced cluster scheduler capable of optimizing scheduling and node allocation decisions. It allows site administrators extensive control over which jobs are considered eligible for for scheduling, how the jobs are prioritized, and where these jobs are run. Maui supports advance reservations, QOS levels, backfill, and allocation management. Each of these features, if enabled, may require some adjustment on the part of the user to optimize system performance.

TORQUE and Maui are the parts of the batch system to submit jobs into the cluster. Batch systems are comprised of four different component types:

a batch system will have a master node where pbs_server is running. Depending on the needs of the systems, a master node may be dedicated to this task or may fulfill the roles of other components as well.
Submit/Interactive Nodes provide an entry point to the system for users to be able to manage their workload. For these nodes, users are able to submit and track their jobs. Additionally, some sites have one or more nodes reserved for interactive use, such as testing and troubleshooting environment problems. These nodes will have client commands (e.g., qsub, qhold, etc) available.
Compute Nodes are the work horses of the system. Their role is to execute submitted jobs. On each compute node, pbs_mom will be running to start, kill and manage submitted jobs. It communicates with pbs_server on the master node.
Some systems are organized for the express purpose of managing a collection of resources beyond compute nodes. Resources can include high-speed networks, storage systems, license managers, etc. Availability of these resources is limited and need to be managed intelligently to promote fairness and increased utilization.

The life cycle of a job can be divided into four stages (basic job flow)

creation. Typically, a submit script is written to hold all of the job parameters. These parameters could include how long a job should run (i.e., wall time), what resources are necessary to run and what to execute.
submission. A job is submitted with the command qsub. Once submitted, the policies set by the administration and technical staff of the site will dictate the priority of the job and therefore, when it will start executing.
execution. Jobs often spend most of their life cycle executing. While a job is running, its status can be queried with qstat.
finalization. When a job has completed, by default, the stdout and stderr files will be copied to the directory where the job was submitted.

Package:    torque v. 2.5.7 (MAUI v.326p21)
os:             Scientific Linux version 5.6 64 bit
server:        dgiref-batch.fzk.de
manuals: server / client / wn client

Archive links

torque server / client v. 2.3.6
maui server / client v. 3.2.6p19

Information links

Download links

Guidelines links

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

TORQUE server v.2.3.6

Prepare

Operating system: Scientific Linux version 5.4 64 bit

If you want to use the maui scheduler instead of the torque scheduler, do not start the TORQUE's pbs_sched daemon after the torque installation.

Firewall configuration

Be sure that if you have firewalls running on the server or node machines that you allow connections on the appropriate ports for each machine. TORQUE pbs_mom daemons use UDP port 1023 and the pbs_server/pbs_mom daemons use ports 15001-15004 by default (how to open port in firewall).

Firewall based issues are often associated with server to mom communication failures and messages such as 'premature end of message' in the log files.

Also, the tcpdump program can be used to verify the correct network packets are being sent.

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare torque server
```
```
 
```

# Declare the variables section ------------

# Please insert your actual configuration

```
# WNs=(WNs addresses)
```
```
# np=number of processors pro WN
```
```
# declare middleware servers
```

# middleware=(declare middleware servers)

```
# HOST=current host name
```

# EDGINFO_UID=User id for edginfo as on GLITE server

# RGMA_UID=User id for rgma as on GLITE server

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
WNs=(dgiref-iwn01.fzk.de)
```
```
np=2
```
```
middleware=(sn06 sn04 sn03 sn02)
```
```
HOST=`hostname -f`
```
```
# check UIDs on glite machine for users
```
```
# edginfo, rgma 
```
```
EDGINFO_UID=101
```
```
RGMA_UID=102
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
 
```
```
#-> start routine
```

REPO_URL="http://dgiref.d-grid.de/svn/dgiref/PROD/cf3/repl/repos/external/"

wget -O /etc/yum.repos.d/sl-dgiref.repo ${REPO_URL}/sl-dgiref.repo

```
 
```
```
# Create users for gLite 
```

echo `useradd -u $EDGINFO_UID edginfo -d /localhome/edginfo`

echo `useradd -u $RGMA_UID rgma -d /localhome/rgma`

```
#<- end routine
```

Install

Torque client and server should be installed on the torque server host (client includes qmgr).

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install torque server
```
```
# load parameters from prepare section
```
```
cd `dirname $0`
```
```
source prepare.sh
```
```
#-> start routine
```

yum -y install torque-2.3.6-1cri.sl5 torque-server-2.3.6-1cri.sl5

yum -y install torque-client-2.3.6-1cri.sl5

```
#<-end routine
```

Configure

Configure torque server
- Setup WNs addresses. Configuration file /var/spool/pbs/server_priv/nodes on the torque server must hold all host names of the WNs information.
- Fill out the configuration file /etc/hosts.equiv with the list of the FQDNs for all Torque Clients, e.g. gLite-CE, Globus GRAM and Unicore NJS
Configure maui
Create users for gLite: edginfo und rgma with the same UIDs like in gLite-CE. These users information are required to gLite with. The gLite-CE asks the MAUI server about this.
Create queues

The D-Grid operational concept provides for the two queues: dgiseq for sequential and dgipar for parallel jobs (the currently assigned queue limits are based on the values, as they are used in GridKa)

Make NFS export /var/spool/pbs/server_logs

do not forget to update the queues for every VO with
qmgr -c "set queue dgiseq acl_groups += ${vo_names}"
qmgr -c "set queue dgipar acl_groups += ${vo_names}"

More: http://www.clusterresources.com/torquedocs21/4.1queueconfig.shtml

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# configure torque server 
```
```
# load parameters from prepare section
```
```
cd `dirname $0`
```
```
source prepare.sh
```
```
 
```
```
#-> begin routine
```
```
function sendTo {
```
```
shift $(($OPTIND - 1)) 
```
```
file_name=${!#}
```
```
 
```

mv -f ${file_name} ${file_name}-`date +"%Y%m%d"`

```
touch ${file_name}
```
```
 
```
```
for host in $@
```
```
do
```

        if [ "$host" != "${file_name}" ]

```
        then
```

                if [ "${file_name}" == "/var/spool/pbs/server_priv/nodes" ]

```
                then
```

                        echo "${host} np=${np}" >> ${file_name}

```
                else
```

                        echo "${host}" >> ${file_name}

```
                fi
```
```
        fi
```
```
done
```
```
}
```
```
# setup configuration files
```

sendTo "${HOST}" "/var/spool/pbs/server_name"

```
# generate list of WNs
```

sendTo "${WNs[@]}" "/var/spool/pbs/server_priv/nodes"

# store list of middleware servers into /etc/hosts.equiv

sendTo "${middleware[@]}" "/etc/hosts.equiv"

```
 
```
```
# Create queues
```
```
echo `qmgr <<+ 
```
```
create queue dgiseq 
```

set queue dgiseq queue_type = Execution

```
set queue dgiseq Priority = 20 
```

set queue dgiseq resources_max.cput = 48:00:00

set queue dgiseq resources_max.pcput = 48:00:00

set queue dgiseq resources_max.walltime = 96:00:00

set queue dgiseq resources_max.ncpus = 1

set queue dgiseq resources_max.nodect = 1

set queue dgiseq resources_default.cput = 48:00:00

set queue dgiseq resources_default.pcput = 48:00:00

set queue dgiseq resources_default.walltime = 96:00:00

set queue dgiseq resources_default.nodes = nodes=1:ppn=1

```
set queue dgiseq enabled = True
```
```
set queue dgiseq started = True
```
```
create queue dgipar
```
```
set queue dgipar queue_type = Execution
```
```
set queue dgipar Priority = 10
```

set queue dgipar resources_max.cput = 960:00:00

set queue dgipar resources_max.pcput = 48:00:00

set queue dgipar resources_max.ncpus = 40

set queue dgipar resources_max.nodect = 10

set queue dgipar resources_max.walltime = 96:00:00

set queue dgipar resources_default.ncpus = 1

set queue dgipar resources_default.cput = 96:00:00

set queue dgipar resources_default.pcput = 48:00:00

set queue dgipar resources_default.walltime = 96:00:00

```
set queue dgipar enabled = True
```
```
set queue dgipar started = True
```
```
+`
```
```
 
```
```
#<- end routine
```

Proceed

To start / stop use the commands:

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
# proceed
```
```
# start server
```
```
echo `chkconfig pbs_server on` 
```
```
echo `/etc/init.d/pbs_server start`
```

Initial test

The result of the following command qstat -Q should be something like:

# Queue   |   Max  | Tot |  Ena |  Str  | Que  | Run  | Hld  | Wat  | Trn  | Ext T
# dgiseq  |     0  |   0 |  yes |  yes  |   0  |   0  |   0  |   0  |   0  |   0 E
# dgipar  |     0  |   0 |  yes |  yes  |   0  |   0  |   0  |   0  |   0  |   0 E

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```

# verify all queues are properly configured

```
qstat -q
```
```
# view additional server configuration
```
```
qmgr -c 'p s'
```

# verify all nodes are correctly reporting

# If everything Ok, you can see the all WNs details with the WN status "free".

```
pbsnodes -a 
```
```
# submit a basic job
```
```
echo "sleep 30" | qsub
```
```
# verify jobs display
```
```
qstat
```
```
# check queues:
```
```
qstat -Q
```
```
 
```
```
# shutdown server
```
```
qterm -t quick
```
```
#<- end routine
```

Update

Upgrading TORQUE can generally be done without shutting down the whole cluster and disrupting running jobs. Simply build and install the new version and restart the daemons. Here is the safest procedure for upgrading TORQUE:

Kill the scheduler.
Wait a few minutes for all new jobs to complete startup.
1. All running jobs in qstat -a have some elapsed walltime.
Restart pbs_server.
Verify the new pbs_server is working correctly.
1. nodes should come up (not down or state-unknown)
2. job walltimes should increase
If upgrading from an earlier 2.1 build, MOMs can automatically restart themselves with:
1. momctl -q enablemomrestart=1 -h :ALL
2. Start the scheduler.
If upgrading from 2.0 or earlier,
1. Restart MOMs on all idle nodes.
2. Wait a minute, make sure node and job states are updating correctly.
3. Delete the previous static archive library files: (libattr.a, libcmds.a, liblog.a, libnet.a, libpbs.a, libsite.a
4. Mark busy nodes offline.
5. Start the scheduler.
6. Restart MOMs on offline nodes after their jobs exit.

All external software like maui, perl-PBS, or pbs_python built with the 2.0.x static archives will need to be rebuilt with the newer 2.1.x shared libraries.

administrator's script: update.sh

```
 
```
```
#!/bin/bash
```
```
# remove torque server
```
```
yum remove torque
```
```
# update torque server
```
```
yum update torque
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Maui server v.3.2.6p21

Prepare

It is required to install the Torque server (version 2.1.6 or version 2.3.6) before the maui scheduler packages.

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare maui
```
```
#-> start routine
```

REPO_URL="http://dgiref.d-grid.de/svn/dgiref/PROD/cf3/repl/repos/external/"

wget -O /etc/yum.repos.d/sl-dgiref.repo ${REPO_URL}/sl-dgiref.repo

```
#<- end routine
```

Install

To install Maui, the following RPMs are needed :

maui
maui-server

Install rpms from dgiref binary repository.

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install maui server
```
```
#-> start routine
```
```
# SL 5 installation:
```

yum -y install maui-3.2.6p21-snap.1234905291.5.el5.x86_64 maui-server-3.2.6p21-snap.1234905291.5.el5.x86_64

```
#<- end routine
```

Configure

By default, batch systems like OpenPBS or Torque have their own job scheduler. Normally, this is much more simple-minded than Maui; for example, in the case of OpenPBS and Torque, the default is a simple fifo scheduler. Therefore, before Maui can be used, the batch system must be told to use it.

With OpenPBS or Torque, this is accomplished setting the scheduling state of the server to False:

qmgr -c "set server scheduling = False"
qmgr -c "list server"
  ...
  server_state = Idle
  scheduling = False
  ...

The scheduling policies are defined in Maui’s configuration file : /var/spool/maui/maui.cfg

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# configure maui server
```
```
#-> start routine
```
```
HOST=`hostname -f`  
```

# In some cases the alias host name command HOST=`hostname -a` should be run. But the previous should work.

echo `sed -i "s/localhost/$HOST/" /var/spool/maui/maui.cfg`

```
 
```
```
cat >> /var/spool/maui/maui.cfg << EOF
```

# The users edginfo and rgma query Maui from the glite-CE to provide data to the InfoSystem

```
ADMIN3            edginfo edguser rgma 
```

# Set the delay to 1 minute before Maui tries to run a job again,

# in case it failed to run the first time.

```
# The default value is 1 hour. 
```
```
DEFERTIME       00:01:00 
```
```
# Necessary for MPI grid jobs 
```
```
ENABLEMULTIREQJOBS TRUE
```
```
 
```
```
EOF
```
```
 
```
```
qmgr -c "set server scheduling = False"
```
```
qmgr -c "list server"
```
```
 
```
```
#<- end routine
```

Proceed

Start service maui

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
# proceed
```
```
#-> start routine
```
```
## start maui
```
```
chkconfig maui on 
```
```
/etc/init.d/maui start
```
```
 
```
```
# MAUI can be tested with
```
```
showq
```
```
diagnose -g
```
```
#<- end routine
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

TORQUE client v.2.3.6

Prepare

Operating system: Scientific Linux version 5.4 64 bit

If you want to use the maui scheduler instead of the torque scheduler, do not start the TORQUE's pbs_sched daemon after the torque installation.

Firewall configuration

Be sure that if you have firewalls running on the server or node machines that you allow connections on the appropriate ports for each machine. TORQUE pbs_mom daemons use UDP port 1023 and the pbs_server/pbs_mom daemons use ports 15001-15004 by default (how to open port in firewall).

Firewall based issues are often associated with server to mom communication failures and messages such as 'premature end of message' in the log files.

Also, the tcpdump program can be used to verify the correct network packets are being sent.

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare torque client
```
```
#-> start routine
```

REPO_URL="http://dgiref.d-grid.de/svn/dgiref/PROD/cf3/repl/repos/external/"

wget -O /etc/yum.repos.d/sl-dgiref.repo ${REPO_URL}/sl-dgiref.repo

```
#<- end routine
```

Install

MAUI use encrypted connections between the client and server. Symmetric encryption keys embedded in the binaries. Therefore absolutely necessary to install, RPM packages for clients and servers from the same source, i.e. with the same keys in the binaries!

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install torque client
```
```
# load parameters from prepare section
```
```
#-> start routine
```
```
# Choose the OS architecture: 
```

# OS_arch="x86_64"   # x86_64 for 64 bit

```
OS_arch="i386"   # i386 for 32 bit
```
```
# install for SL4
```

yum -y install torque-2.3.6-1cri.slc4.${OS_arch}

yum -y install torque-client-2.3.6-1cri.slc4.${OS_arch}

```
# install for SL5
```

yum -y install torque-2.3.6-1cri.sl5 torque-client-2.3.6-1cri.sl5

```
 
```

# only for WN -----------------------------------------------------

yum -y install torque-mom-2.3.6-1cri.slc4.${OS_arch}

```
# or
```

yum -y install torque-mom-2.3.6-1cri.sl5

```
#<- end routine
```

Configure

For each compute host, the MOM server must be configured to trust the pbs_server daemon. In TORQUE 2.0.0p5 and later, this can also be done by creating the $(TORQUECFG)/server_name file and placing the server hostname inside.
Additional config parameters may be added to $(TORQUECFG)/mom_priv/config (see the MOM config page for details.)

Data management allows jobs’ data to be staged in/out or to and from the server and compute nodes.

For shared filesystems (i.e., NFS, DFS, AFS, etc.) use the $usecp parameter in the mom_priv/config files to specify how to map a user’s home directory. (Example: $usecp gridmaster.tmx.com:/home /home)
For local, non-shared filesystems, rcp or scp must be configured to allow direct copy without prompting for passwords (key authentication, etc.)
To use the PBS StageIN /out mechanism to copy the input/output files between CEs and Worker Nodes, add the configuration into /var/spool/pbs/mom_priv/config like this:

$usecp *:/home /home
$usecp *:/srv/nfs/home /srv/nfs/home

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# configure
```

# Declare the variables section ------------

# Please insert your actual configuration

```
# TORQUE_SERVER="torque server name"
```

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
TORQUE_SERVER="sn05"
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
#-> start routine
```
```
# server_name configuration
```

echo "${TORQUE_SERVER}" > /var/spool/pbs/server_name

```
 
```
```
echo "\
```
```
\$pbsserver ${TORQUE_SERVER}
```
```
\$logevent 255
```
```
\$ideal_load 4
```
```
\$max_load 10
```
```
\$usecp *:/home /home
```
```
\$usecp *:/srv/nfs/home /srv/nfs/home
```
```
"  > /var/spool/pbs/mom_priv/config
```
```
#<- end routine
```

Initial test

From a user account, it should be possible to use a 'Hello World' job submitting, as well as an interactive shell on a WN
The job results are as files STDIN.o<JOBID> (std-output) and STDIN.e<JOBID> (std-error).
test MAUI
The test on the gLite-CE should work as edginfo user configuration of gLite-packages.
To check the status of the job query, the following command is used within the lifetime of submitted jobs:

qstat 
# The result should be something like the following: 
# Job id  |   Name    |   User  | Time Use  | S | Queue
# 6.node1 |   STDIN   |  user1  | 0         | R |  batch

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
# test
```

# Declare the variables section ------------

# Please insert your actual configuration

```
# USER='grid user'
```
```
# QUEUE=queue name
```

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

USER='dgdt0001' 			                # grid user, e.g. dgdt0001

```
QUEUE=dgiseq
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
 
```
```
#-> start routine 
```

su ${USER}	                                # as the grid user

echo "sleep 60; hostname" | /usr/bin/qsub

```
# Then you get e.g. the following:
```
```
# 6.dgiref-isn02.fzk.de
```

echo "/bin/hostname" | qsub -q ${QUEUE}

```
qsub -I -q ${QUEUE}
```
```
qstat 
```

# The output files should be stored by the end of the jobs execution in the grid-user home directory:

```
ls ~
```
```
# STDIN.e6  STDIN.o6
```
```
#<- end routine
```

Update

To update use the standart rpm command syntax.

administrator's script: update.sh

```
 
```
```
#!/bin/bash
```
```
# remove torque client
```
```
yum remove torque
```
```
# update torque client
```
```
yum update torque
```
```
exit 0
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

Maui client v.3.2.6p21

Prepare

It is required to install the Torque client before the maui scheduler packages. It is required to install the Torque client (version 2.1.6 or version 2.3.6 depending on the Torque server version) before the maui scheduler packages.

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare maui
```
```
#-> start routine
```

REPO_URL="http://dgiref.d-grid.de/svn/dgiref/PROD/cf3/repl/repos/external/"

wget -O /etc/yum.repos.d/sl-dgiref.repo ${REPO_URL}/sl-dgiref.repo

```
#<- end routine
```

Install

To install Maui, the following RPMs are needed :

maui
maui-client

Install rpms from dgiref binary repository.

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install maui client
```
```
#-> start routine
```
```
# Choose the OS architecture: 
```

# OS_arch="x86_64"   # x86_64 for 64 bit

```
OS_arch="i686"   # i686 for 32 bit
```
```
# SL 4 installation:
```

yum -y install maui-3.2.6p21-dgrid_snap.1224706197.2.${OS_arch} maui-client-3.2.6p21-dgrid_snap.1224706197.2.${OS_arch}

```
# SL 5 installation:
```

yum -y install maui-3.2.6p21-snap.1234905291.5.el5.x86_64 maui-client-3.2.6p21-snap.1234905291.5.el5.x86_64

```
#<- end routine
```

Configure

Assign the torque server into /var/spool/maui/maui.cfg

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# configure maui client
```
```
TORQUE_SERVER="dgiref-batch"
```
```
#-> start routine
```

echo `sed -i "s/localhost/${TORQUE_SERVER}/g" /var/spool/maui/maui.cfg`

```
#<- end routine
```

Initial test

test with diagnose command

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
# MAUI can be tested with
```
```
showq
```
```
showstats -u
```
```
diagnose -g
```
```
#<- end routine
```

cluster:Torque/236

Contents

Introduction

TORQUE server v.2.3.6

Prepare

Install

Configure

Proceed

Initial test

Update

Maui server v.3.2.6p21

Prepare

Install

Configure

Proceed

TORQUE client v.2.3.6

Prepare

Install

Configure

Initial test

Update

Maui client v.3.2.6p21

Prepare

Install

Configure

Initial test

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox