cluster:Torque/257

From Dgiref

Introduction

TORQUE (Tera-scale Open-source Resource and QUEue manager) is a resource manager providing control over batch jobs and distributed compute nodes. Maui is an advanced cluster scheduler capable of optimizing scheduling and node allocation decisions. It allows site administrators extensive control over which jobs are considered eligible for for scheduling, how the jobs are prioritized, and where these jobs are run. Maui supports advance reservations, QOS levels, backfill, and allocation management. Each of these features, if enabled, may require some adjustment on the part of the user to optimize system performance.

TORQUE and Maui are the parts of the batch system to submit jobs into the cluster. Batch systems are comprised of four different component types:

a batch system will have a master node where pbs_server is running. Depending on the needs of the systems, a master node may be dedicated to this task or may fulfill the roles of other components as well.
Submit/Interactive Nodes provide an entry point to the system for users to be able to manage their workload. For these nodes, users are able to submit and track their jobs. Additionally, some sites have one or more nodes reserved for interactive use, such as testing and troubleshooting environment problems. These nodes will have client commands (e.g., qsub, qhold, etc) available.
Compute Nodes are the work horses of the system. Their role is to execute submitted jobs. On each compute node, pbs_mom will be running to start, kill and manage submitted jobs. It communicates with pbs_server on the master node.
Some systems are organized for the express purpose of managing a collection of resources beyond compute nodes. Resources can include high-speed networks, storage systems, license managers, etc. Availability of these resources is limited and need to be managed intelligently to promote fairness and increased utilization.

The life cycle of a job can be divided into four stages (basic job flow)

creation. Typically, a submit script is written to hold all of the job parameters. These parameters could include how long a job should run (i.e., wall time), what resources are necessary to run and what to execute.
submission. A job is submitted with the command qsub. Once submitted, the policies set by the administration and technical staff of the site will dictate the priority of the job and therefore, when it will start executing.
execution. Jobs often spend most of their life cycle executing. While a job is running, its status can be queried with qstat.
finalization. When a job has completed, by default, the stdout and stderr files will be copied to the directory where the job was submitted.

Package:    torque v. 2.5.7 (MAUI v.326p21)
os:             Scientific Linux version 5.6 64 bit
server:        dgiref-batch.fzk.de
manuals: server / client / wn client

Archive links

torque server / client v. 2.3.6
maui server / client v. 3.2.6p19

Information links

Download links

Guidelines links

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

TORQUE server v.2.5.7

Prepare

Operating system: Scientific Linux version 5.6 64 bit

Configure the following settings for the host:

UMD repo
NFS-server
Create Host Based Authentication for Torque clients. See ssh auth

If you don't want to use the maui scheduler instead of the torque scheduler, exclude the maui packets installation and maui configuration from the scripts

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```

inside_network="10.0.171.0/255.255.255.0"

```
# install umd
```
```
#clean old
```

rm /etc/yum.repos.d/UMD* /etc/yum.repos.d/epel*

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

wget http://repository.egi.eu/sw/production/umd/1/sl5/x86_64/updates/umd-release-1.0.2-1.el5.noarch.rpm

rm -f epel-release-5-4.noarch.rpm umd-release-1.0.2-1.el5.noarch.rpm

yum -y install epel-release-5-4.noarch.rpm

```
yum -y install yum-priorities
```

yum -y install umd-release-1.0.2-1.el5.noarch.rpm

sed -i -e "s/priority=.*/priority=5/g" /etc/yum.repos.d/UMD-1-base.repo

sed -i -e "s/priority=.*/priority=4/g" /etc/yum.repos.d/UMD-1-updates.repo

```
 
```
```
#nfs-server
```
```
yum -y instal nfs-utils-lib nfs-utils
```

echo "/var/torque/server_logs $inside_network (rw,async,no_root_squash)" >> /etc/exports

echo "/var/torque/server_priv/accounting $inside_network (rw,async,no_root_squash)" >> /etc/exports

```
/etc/rc.d/init.d/nfslock start
```
```
/etc/rc.d/init.d/nfs start
```

Install

Install Torque-server with yum from umd repo
Install MAUI scheduler

administrator's script: install.sh

```
 
```

yum -y install torque.x86_64 torque-server

yum -y install maui maui-server maui-client

Configure

Create default configuration for torque-server
Configure server name in /var/torque/server_name
Configure allowed WNs with number of CPUs into /var/torque/server_priv/nodes
Customization of Torque qmgr
- Create queue
- Create a server parameters (auth users, timouts, log level, torque managers, etc)
Create munge key for torque communication
Create maui scheduler configuration file

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
/usr/sbin/pbs_server -t create
```
```
 
```

echo `hostname -f` > /var/torque/server_name

```
 
```

echo "wn01 np=2" > /var/torque/server_priv/nodes

echo "wn02 np=2" >> /var/torque/server_priv/nodes

echo "wn03 np=2" >> /var/torque/server_priv/nodes

echo "wn04 np=2" >> /var/torque/server_priv/nodes

echo "wn05 np=2" >> /var/torque/server_priv/nodes

echo "wn06 np=2" >> /var/torque/server_priv/nodes

echo "wn07 np=2" >> /var/torque/server_priv/nodes

echo "wn08 np=2" >> /var/torque/server_priv/nodes

echo "wn09 np=2" >> /var/torque/server_priv/nodes

echo "wn10 np=2" >> /var/torque/server_priv/nodes

```
 
```
```
 
```
```
# Create queues
```
```
echo `qmgr << 
```
```
'create queue dgipar
```
```
set queue dgipar queue_type = Execution
```
```
set queue dgipar max_queuable = 10
```
```
set queue dgipar max_running = 5
```

set queue dgipar acl_host_enable = false

set queue dgipar resources_max.walltime = 72:00:00

set queue dgipar resources_default.walltime = 72:00:00

```
set queue dgipar enabled = True
```
```
set queue dgipar started = True
```
```
#
```
```
# Create and define queue dgiseq
```
```
#
```
```
create queue dgiseq
```
```
set queue dgiseq queue_type = Execution
```
```
set queue dgiseq Priority = 100
```
```
set queue dgiseq max_queuable = 150
```
```
set queue dgiseq max_running = 130
```

set queue dgiseq acl_host_enable = False

set queue dgiseq resources_max.nodect = 6

set queue dgiseq resources_max.walltime = 288:00:00

set queue dgiseq resources_default.nodes = 5:ppn=2

```
set queue dgiseq enabled = True
```
```
set queue dgiseq started = True
```
```
#
```
```
# Set server attributes.
```
```
#
```

set server acl_hosts = dgiref-batch.fzk.de

```
set server managers = root@*
```

set server managers += tomcat@dgiref-glite.fzk.de

```
set server operators = root@*
```
```
set server default_queue = dgiseq
```
```
set server log_events = 511
```
```
set server mail_from = admin
```
```
set server query_other_jobs = True
```
```
set server scheduler_iteration = 600
```
```
set server node_check_rate = 150
```
```
set server tcp_timeout = 12
```
```
set server poll_jobs = False
```
```
set server log_level = 3
```
```
set server log_file_roll_depth = 7
```
```
set server next_job_number = 2387
```

set server authorized_users += *@dgiref-globus.fzk.de

set server authorized_users += *@dgiref-login.fzk.de

set server authorized_users += *@dgiref-ogsadai.fzk.de

set server authorized_users += *@dgiref-batch.fzk.de

set server authorized_users += *@dgiref-glite.fzk.de

set server authorized_users += *@dgiref-unicore.fzk.de

set server authorized_users += *@dgiref-dcache.fzk.de

set server authorized_users += *@dgiref-glite32.fzk.de

set server authorized_users += *@dgiref-globus50.fzk.de

set server authorized_users += *@dgiref-globus40.fzk.de

```
'`
```
```
 
```
```
# configure munge
```
```
/usr/sbin/create-munge-key
```

# Generating a pseudo-random key using /dev/urandom completed.

#chown munge:munge /var/log/munge/munged.log*

```
 
```
```
#configure for maui
```
```
echo"
```
```
UI configuration example
```
```
# @(#)maui.cfg David Groep 20031015.1
```
```
# for MAUI version 3.2.5
```
```
#
```

SERVERHOST              dgiref-batch.fzk.de

```
ADMIN1                  root
```
```
ADMINHOST               sn05
```
```
RMTYPE[0]           PBS
```
```
RMHOST[0]           sn05
```
```
RMSERVER[0]         sn05
```
```
 
```
```
SERVERPORT            40559
```
```
SERVERMODE            NORMAL
```
```
 
```

# Set PBS server polling interval. Since we have many short jobs

# and want fast turn-around, set this to 10 seconds (default: 2 minutes)

```
RMPOLLINTERVAL        00:00:10
```
```
 
```

# a max. 10 MByte log file in a logical location

```
LOGFILE               /var/log/maui.log
```
```
LOGFILEMAXSIZE        10000000
```
```
LOGLEVEL              3
```
```
">/var/spool/maui/maui.cfg
```

Proceed

Make pbs_server, munge, maui a services for OS and add it to autoboot system. So it will be started during boot.

administrator's script: proceed.sh

```
 
```
```
service munge restart
```
```
service maui restart
```
```
service pbs_server restart
```
```
 
```
```
chkconfig maui on
```
```
chkconfig munge on
```
```
chkconfig pbs_server on
```
```
chkconfig pbs_mom off
```

Initial test

From a user account, it should be possible to use a 'Hello World' job submitting, as well as an interactive shell on a WN
The job results are as files STDIN.o<JOBID> (std-output) and STDIN.e<JOBID> (std-error).
Test MAUI
The test on the gLite-CE should work as edginfo user configuration of gLite-packages.
To check the status of the job query, the qstat command is used within the lifetime of submitted jobs.

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```

# verify all queues are properly configured

```
qstat -q
```
```
# view additional server configuration
```
```
qmgr -c 'p s'
```

# verify all nodes are correctly reporting

# If everything Ok, you can see the all WNs details with the WN status "free".

```
pbsnodes -a 
```
```
# submit a basic job
```
```
echo "sleep 30" | qsub
```
```
# verify jobs display
```
```
qstat
```
```
# check queues:
```
```
qstat -Q
```
```
 
```
```
# shutdown server
```
```
qterm -t quick
```
```
#<- end routine
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

TORQUE v.2.5.7 client

Prepare

Operating system: Scientific Linux version 5.6 64 bit

Configure the following settings for the host:

UMD repo
NFS-client
Create Host Based Authentication for Torque server. See ssh auth

Client for torque should be install on server for management torque

administrator's script: prepare.sh

```
 
```
```
# install umd
```
```
#clean old
```

rm /etc/yum.repos.d/UMD* /etc/yum.repos.d/epel*

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

wget http://repository.egi.eu/sw/production/umd/1/sl5/x86_64/updates/umd-release-1.0.2-1.el5.noarch.rpm

rm -f epel-release-5-4.noarch.rpm umd-release-1.0.2-1.el5.noarch.rpm

```
yum install epel-release-5-4.noarch.rpm
```
```
yum install yum-priorities
```

yum install umd-release-1.0.2-1.el5.noarch.rpm

sed -i -e "s/priority=.*/priority=5/g" /etc/yum.repos.d/UMD-1-base.repo

sed -i -e "s/priority=.*/priority=4/g" /etc/yum.repos.d/UMD-1-updates.repo

Install

Install Torque-server with yum from umd repo

administrator's script: install.sh

```
 
```
```
yum -y install torque-client.x86_64
```

# the emi torque package can be used as well

```
# yum install emi-torque-client
```

Configure

Configure server name in /var/torque/server_name
Copy munge key for torque communication from torque-server

administrator's script: configure.sh

```
 
```
```
BATCH_SERVER=dgiref-batch.fzk.de
```

echo -e $BATCH_SERVER > /var/torque/server_name

```
 
```

scp $BATCH_SERVER:/etc/munge/munge.key /etc/munge/munge.key

```
chown munge.munge /etc/munge/munge.key
```

chown munge:munge /var/log/munge/munged.log*

Proceed

Enable munge autoboot

administrator's script: proceed.sh

```
 
```
```
service munge restart
```
```
chkconfig munge on
```

Initial test

Create test script
Test submit job with qsub
Check queue and trace job

administrator's script: test.sh

```
 
```
```
# Try
```
```
echo "
```
```
#!/bin/bash
```
```
sleep 60
```
```
"> test.sh
```
```
qsub -q dgiseq test.sh
```
```
 
```
```
# to check supported queues:
```

# ldapsearch -x -H ldap://<CE_FQDN>:2170 -b mds-vo-name=resource,o=grid

```
 
```
```
#To detect the jobid:
```
```
qstat -Q
```
```
qstat -a
```
```
 
```
```
#To check the job 
```
```
tracejob <jobid>
```

Update

update - is mostly to install new

administrator's script: update.sh

```
 
```
```
yum -y update torque-client.x86_64
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

TORQUE v.2.5.7 wn client

Prepare

Operating system: Scientific Linux version 5.6 64 bit

Configure the following settings for the host:

UMD repo
Create Host Based Authentication for Torque server and Middleware hosts. See ssh auth

administrator's script: prepare.sh

```
 
```
```
# install umd
```
```
#clean old
```

rm /etc/yum.repos.d/UMD* /etc/yum.repos.d/epel*

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

wget http://repository.egi.eu/sw/production/umd/1/sl5/x86_64/updates/umd-release-1.0.2-1.el5.noarch.rpm

rm -f epel-release-5-4.noarch.rpm umd-release-1.0.2-1.el5.noarch.rpm

```
yum install epel-release-5-4.noarch.rpm
```
```
yum install yum-priorities
```

yum install umd-release-1.0.2-1.el5.noarch.rpm

sed -i -e "s/priority=.*/priority=5/g" /etc/yum.repos.d/UMD-1-base.repo

sed -i -e "s/priority=.*/priority=4/g" /etc/yum.repos.d/UMD-1-updates.repo

Install

Install Torque-mom with yum from umd repo

administrator's script: install.sh

```
 
```

# the emi torque package can be used as well

```
yum -y install torque-client torque-mom
```

Configure

Create hostname alias for Bach server in hosts file if use the separate internal network for communication between Batch server and WN

All nodes should be specified in /var/torque/server_priv/nodes on batch server host See configure batch server

Customization of mom config file
Prepare munge service

administrator's script: configure.sh

```
 
```
```
# 
```
```
BATCH_SERVER=dgiref-batch.fzk.de
```

BATCH_SERVER_INTERNAL_IP="10.0.171.205"

echo "$BATCH_SERVER_INTERNAL_IP  $BATCH_SERVER" >> /etc/hosts

echo -e $BATCH_SERVER >> /var/torque/server_name

```
 
```
```
echo "\$pbsserver $BATCH_SERVER
```
```
\$logevent 255
```
```
\$ideal_load 4
```
```
\$max_load 10
```
```
\$usecp *:/home /home
```

\$usecp *:/srv/nfs/home /srv/nfs/home" > /var/torque/mom_priv/config

```
 
```

scp $BATCH_SERVER:/etc/munge/munge.key /etc/munge/munge.key

chown munge:munge /var/log/munge/munged.log*

```
chown munge:munge /etc/munge/munge.key
```

Proceed

Enable autoboot and start services
- munge
- pbs_mom

administrator's script: proceed.sh

```
 
```
```
/etc/init.d/pbs_mom restart
```
```
service munge restart
```
```
chkconfig munge on
```
```
chkconfig pbs_mom on
```

Initial test

Check information about WN from batch server
Start test job for WN

administrator's script: test.sh

```
 
```
```
#
```
```
wn="wn10"
```
```
#show information about wn
```
```
qnodes -q $wn
```
```
 
```
```
# Try
```
```
echo "
```
```
#!/bin/bash
```
```
sleep 60
```
```
"> test.sh
```
```
qsub -q dgiseq test.sh
```
```
 
```
```
# to check supported queues:
```

# ldapsearch -x -H ldap://<CE_FQDN>:2170 -b mds-vo-name=resource,o=grid

```
 
```
```
#To detect the jobid:
```
```
qstat -Q
```
```
qstat -a
```
```
 
```
```
#To check the job 
```
```
tracejob <jobid>
```

cluster:Torque/257

Contents

Introduction

TORQUE server v.2.5.7

Prepare

Install

Configure

Proceed

Initial test

TORQUE v.2.5.7 client

Prepare

Install

Configure

Proceed

Initial test

Update

TORQUE v.2.5.7 wn client

Prepare

Install

Configure

Proceed

Initial test

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox