guide:Common Production Environment

From Dgiref
Jump to: navigation, search

Contents

VO Software Deployment

Information

Geographylogo.png The software deployment tools can be used by VOs to install applications on the site shared VO directory eg. VO_DGTEST_SW_DIR . More details about the software deployment concept implemented can be found here.

VO related software can be installed by authorized user (Software manager) using software deployment tools. The deployment tools consist of:

  • manageSoftware: This tool is used to install/uninstall or validate a VO Software on the VO shared directory provided by the Site. The tool ca be used interactively from the cluster frontend or executed by a grid job (on a cluster Worker Node)
  • manageVOTag: This tool is used to manage the software tags for a specific VO and save them in tag files (one file per VO). This tool is called from manageSoftware after a software installation/uninstallation or validation. It can be used also by autorized V0 users from a grid client.
  • software-provider-glue.pl: This tool publish all the Vo Software tags to the grid middleware information system using the Glue 2.0 as data model

Download

Installation

The software deployment tools must be installed as root on every computing resource into a shared directory <SW_DIR> (e.g /software or /opt-dgrid). The directory must be accessible by authorized grid users from the cluster worker nodes and from midlleware frontends. 

 
## extract  the software deployment package into the shared software parent directory  
$ tar -xzvf software-deployment-tools.tar.gz -C <SW_DIR>
## create a symbolic link in /etc/profile.d to the configuration script sw_env.sh (on every host)
ln -s <SW_DIR>/tools/software-deployment/etc/sw_en.sh /etc/profile.d/

Configuration

common configuation

The software deployment tools can be configured through the script /etc/profile.d/sw_en.sh The script sets the environment Variables of the VO Software directories (VO_<VONAME>SW_DIR) and creates empty software Tag files for every VO (e.g /software/vos/dgtest/dgtest.list> .

The following configuration parameters must be changed and are specific for every site:

  • SW_DIR: The parent directory for VO shared software directory (e.g /software/vos or /opt/d-grid)
  • VOS: List of VOs supported by the site separated by ","
  • ManagerID: Computing Manager id (hostname of the computing resource e.g dgiref-globus.fzk.de)
  • ManagerName: Computing Manager name (resource name e.g fzk-dgiref)
configure the information provider for globus MDS4
<ns1:ResourcePropertyProviderConfigArray xsi:type="ns1:ResourcePropertyProviderConfigArray" xmlns:ns1="http://mds.globus.org/rpprovider/2005/08" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <ns1:configArray xsi:type="ns1:resourcePropertyProviderConfig">
   <ns1:resourcePropertyImpl xsi:type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema">org.globus.mds.usefulrp.rpprovider.SingleValueResourcePropertyProvider</ns1:resourcePropertyImpl>
  <ns1:resourcePropertyName xsi:type="xsd:QName" xmlns:glue2="http://schemas.ogf.org/glue/2008/05/spec_2.0_d42_r01">glue2:Domains</ns1:resourcePropertyName>
   <ns1:resourcePropertyElementProducers xsi:type="ns1:resourcePropertyElementProducerConfig">
      <ns1:className xsi:type="xsd:string">org.globus.mds.usefulrp.rpprovider.producers.ExternalProcessElementProducer</ns1:className>
      <ns1:arguments xsi:type="xsd:string">libexec/softwareprovider/software-provider-glue.pl</ns1:arguments>
      <ns1:period xsi:type="xsd:int">300</ns1:period>
    </ns1:resourcePropertyElementProducers>
</ns1:configArray>
</ns1:ResourcePropertyProviderConfigArray>

Usage

manageSoftware 
manageSoftware [-ivruIh] --vo -V -S -R --host --tag --run_script --valid_script --install_script --unin_script --tar
****Actions to define (no parameters required): 
 -i,--install: performs the VO software installation and publish the installed tag when successful
 -v,--validate: validation of the VO software. It requires a run over a benchmark sample of data. 
                It publish a tag corresponding to the version of the software just validated
 -r,--run: performs a run of the software yet installed
 -u,--uninstall: remove that version of exp. software and the corresponding tag on the Information System
 -I, --iv: performs on the same job both the actions  '--install' and '--validate' . 
****Options 
 --vo: Virtual Organization the users belong 
 -V: Version of software you are installing 
 -S: Software name you are installing
 -R: Number of release for that version of software (default=NULL)
 --host: name of the Computing Element on wich you want install your software (default = localhost))
 --tag: tag name you want to publish in the format name/version-release (not mandatory)
 --run_script: the custom name of the script to run the software (default run_sw)
 --validate_script: the custom name of the script to validate the software (default validation_sw)
 --uninstall_script: the custom name of the script to uninstall the software (default uninstall_sw)
 --install_script: the custom name of the script to install the software (default install_sw)
 --tar tar archive containing install/validate/uninstall scripts (optional)
 -h,--help: this help
manageVOTag
 
General options
       --host Specifies the host name of the Computing Element head node (default localhost) .
       --vo   Specifies the name of the Virtual Organisation. If not specified, the command tries to extract the VO from 
              the user proxy certificate.
Actions
       --add     Adds the specified tags to the existing tags.
       --replace Replaces the existing tags with the specified tags.
       --remove  Removes the specified tags from the existing tags.
       --clean   Deletes all the existing tags.
       --list    Prints the existing tags.
       --help    Prints this help.
Tags
       Tags must follow this syntax: VO-voname-text
       where voname is the name of the Virtual Organisation and text is an arbitrary string containing only letters,
       numbers and the characters .-_/:.  Tags can be specified by the following options:
       --tags tag1,tag2,...
       Specifies the tags to be added or removed as a comma-delimited list (no spaces allowed).
       --tagfile tagfile
         Specifies a file containing a list of tags to be added or removed. The tags can be separated by any number of spaces,      tabs and newlines.
Example
      Add tags: manageVOTag --host remotehost --vo voname --add --tags tags --tagfile tagfile
      Replace tags: manageVOTag --host remotehost --vo voname --replace --tags tags --tagfile tagfile
      Remove tags: manageVOTag --host remotehost --vo voname --remove --tags tags --tagfile tagfile
      Clean tags: manageVOTag --host remotehost --vo voname --clean
      List tags: manageVOTag --host remotehost --vo voname --list
      Prints help: manageVOTag --help
Vo software information

After the configuration of the software information provider in Globus you can view them through WebMDS.

Samples
  • Sample install_sw script for Cosmos 
 
#!/bin/bash
VO=$1
NAME=$2
VER=$3
REL=$4
VONAME=`echo $VO | tr a-z A-Z`
eval MAINPATH=\$VO_${VONAME}_SW_DIR
echo "Running on: $HOSTNAME(`hostname -i`) as `whoami`"
# test if vo-software-directory exists
if [ x$MAINPATH = 'x' ]; then
      echo "VO_${VONAME}_SW_DIR is not set in the environment"
      exit 1
fi
echo "Go to VO software installation root directory"
echo "Installing software in VO_${VONAME}_SW_DIR=$MAINPATH"
echo "Running make install"
INSTALLDIR=${MAINPATH}/${NAME}-${VER}
export INSTALLDIR
make && make install
  • Sample Globus Installation Job 
<?xml version="1.0" encoding="UTF-8"?>
<job>
    <executable>/bin/sh</executable>
    <directory>${GLOBUS_USER_HOME}</directory>
    <argument>/opt/d-grid/tools/software-deployment/bin/manageSoftware</argument>
    <argument>--install</argument>
    <argument>--tag=dummy/4.0</argument>
    <argument>--vo=dgtest</argument>
    <argument>--tar=dummy.tar.gz</argument>
    <stdout>${GLOBUS_USER_HOME}/dummy-install.out</stdout>
    <stderr>${GLOBUS_USER_HOME}/dummy-install.err</stderr>
    <maxWallTime> 30 </maxWallTime> <!-- minutes -->
   <fileStageIn>
        <transfer>
            <sourceUrl>gsiftp://ui.d-gri.de:2811/home/dgdtsgm/dummy.tar.gz</sourceUrl>
            <destinationUrl>file:///${GLOBUS_USER_HOME}/dummy.tar.gz</destinationUrl>
        </transfer>
   </fileStageIn>
 </job>

Initial Tests

Authorization 
 
##############################
# Software Admin Login
############################## 
# Log on into a grid client (eg. with gsissh)
$ gsissh -p 2222 ui.d-grid.de
# Request a VOMS Proxy with software admin role
$ voms-proxy-init -voms dgtest:softwareadmin
Dummy Software Deployment
  • Local deployment on interactive node: 
 
##############################
1- Software Installation
############################## 
## copy the tar archive with installation script from a local repository
$ cp /opt/d-grid/tools/software-deployment/samples/tars/dummuy.tar.gz .
## Call manageSoftware (local)
$ /opt/d-grid/tools/software-deployment/bin/manageSoftware -i --vo dgtest --tar dummuy_software.tar.gz -S dummy -V 1.0
********************************************************************************************************
Starting the installation of software VO-dgtest-dummy/1.0 in /grid/fzk.de/home/dgtest/dgdt0054/software-deployment
********************************************************************************************************
manageSoftware INFO: Preparing to manage the Software VO-dgtest-dummy/1.0
manageSoftware INFO: The dgtest software directory VO_DGTEST_SW_DIR is /opt/d-grid/dgtest
manageSoftware INFO: Extracting the tar archive to temporary directory /opt/d-grid/dgtest/dgtest-dummy-1.0_tmp/
manageSoftware INFO: executing install script install_sw
======================================================================================
Running on: ui.d-grid.de(192.108.45.63) as dgdt0054
Installing software in VO_DGTEST_SW_DIR=/opt/d-grid/dgtest
======================================================================================
manageSoftware INFO: The installation process of software VO-dgtest-dummy/1.0 ran successfully
manageSoftware INFO: Cleaning up the temporary directory dgtest-dummy-1.0_tmp
********************************************************************************************************
Finishing the installation of software VO-dgtest-dummy/1.0
******************************************************************************************************** 
## Show the List of software Tags for the VO using manageVOTag
$ /opt/d-grid/tools/software-deployment/bin/manageVOTag -list --vo dgtest --host gt4-fzk.gridka.de
VO-dgtest-dummy/1.0/to-be-validated
 
### Check the VO Installation directory
$ ls -l /opt/d-grid/dgtest/
drwxr-xr-x  2 dgdtsgm  dgtest 32768 Mar 25 11:43 dummy-1.0
 
##############################
# 2- Software Validation
##############################
## Call manageSoftware (local)
$ /opt/d-grid/tools/software-deployment/bin/manageSoftware --validate --vo dgtest --tar dummuy_software.tar.gz -S dummy -V 1.0
********************************************************************************************************
Starting the validation of Software VO-dgtest-dummy/1.0 in /grid/fzk.de/home/dgtest/dgdt0054/software-deployment
********************************************************************************************************
manageSoftware INFO: Preparing to manage the Software VO-dgtest-dummy/1.0
manageSoftware INFO: The dgtest software directory VO_DGTEST_SW_DIR is /opt/d-grid/dgtest
manageSoftware INFO: Extracting the tar archive to temporary directory /opt/d-grid/dgtest/dgtest-dummy-1.0_tmp/
manageSoftware INFO: executing validation script validation_sw
======================================================================================
Running on: ui.d-grid.de(192.108.45.63) as dgdt0054
Vaildating software dummy 1.0 in VO_DGTEST_SW_DIR=/opt/d-grid/dgtest
-rw-r--r--  1 dgdt0054 dgtest 0 Mar 25 11:43 dummy
======================================================================================
manageSoftware INFO: The validation process of software VO-dgtest-dummy/1.0 ran successfully
manageSoftware INFO: Cleaning up the temporary directory dgtest-dummy-1.0_tmp
********************************************************************************************************
Finishing the validation of software VO-dgtest-dummy/1.0
********************************************************************************************************
##################################
# 3- Remove the installed software
##################################
## Call manageSoftware (local)
$ /opt/d-grid/tools/software-deployment/bin/manageSoftware --uninstall --vo dgtest --tar dummuy_software.tar.gz -S dummy -V 1.0
********************************************************************************************************
Starting the uninstallation of Software VO-dgtest-dummy/1.0 in /grid/fzk.de/home/dgtest/dgdt0054/software-deployment
********************************************************************************************************
manageSoftware INFO: Preparing to manage the Software VO-dgtest-dummy/1.0
manageSoftware INFO: The dgtest software directory VO_DGTEST_SW_DIR is /opt/d-grid/dgtest
manageSoftware INFO: Extracting the tar archive to temporary directory /opt/d-grid/dgtest/dgtest-dummy-1.0_tmp/
manageSoftware INFO: executing uninstall script uninstall_sw
======================================================================================
Running on: ui.d-grid.de(192.108.45.63) as dgdt0054
Removing software dummy 1.0 from VO_DGTEST_SW_DIR=/opt/d-grid/dgtest
======================================================================================
manageSoftware INFO: The removal process of sofware VO-dgtest-dummy/1.0 ran successfully
manageSoftware INFO: Cleaning up the temporary directory dgtest-dummy-1.0_tmp
********************************************************************************************************
Finishing the uninstallation of software VO-dgtest-dummy/1.0
********************************************************************************************************
  • Deployment from Globus Job: 
 
globusrun-ws -submit -S -F gt4-fzk.gridka.de -f dummy-install-job.xml 
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:9617292e-3d19-11df-b35b-0050568d0903
Termination time: 04/01/2010 23:03 GMT
Current job state: StageIn
Current job state: Active
Current job state: StageOut
Current job state: CleanUp
Current job state: Done
Destroying job...Done.
Cleaning up any delegated credentials...Done.
Sample
Cosmos Deployment
  • local deployment on interactive node: 
 
################################
# Validate a Cosmos Installation
################################
## Copy the tar archive containing the validation script and input files from a local repository 
$ cp /opt/d-grid/tools/software-deployment/samples/tars/cosmos_validate.tar.gz .
## Call manageSoftware (local)
$ /opt/d-grid/tools/software-deployment/bin/manageSoftware --validate --vo dgtest --tar cosmos_validate.tar.gz --tag cosmos-backend/5.0
********************************************************************************************************
Starting the validation of Software VO-dgtest-cosmos-backend/5.0 in /grid/fzk.de/home/dgtest/dgdt0054/software-deployment
********************************************************************************************************
manageSoftware INFO: Preparing to manage the Software VO-dgtest-cosmos-backend/5.0
manageSoftware INFO: The dgtest software directory VO_DGTEST_SW_DIR is /opt/d-grid/dgtest
manageSoftware INFO: Extracting the tar archive to temporary directory /opt/d-grid/dgtest/dgtest-cosmos-backend-5.0_tmp/
manageSoftware INFO: executing validation script validation_sw
======================================================================================
Running on: ui.d-grid.de(192.108.45.63) as dgdt0054
Vaildating software cosmos-backend 5.0 in VO_DGTEST_SW_DIR=/opt/d-grid/dgtest
Copy input files to installation directory
Change to cosmos installation directory
Run a test cosmos job and check the output
Clean input/output files
======================================================================================
manageSoftware INFO: The validation process of software VO-dgtest-cosmos-backend/5.0 ran successfully
manageSoftware INFO: Cleaning up the temporary directory dgtest-cosmos-backend-5.0_tmp
********************************************************************************************************
Finishing the validation of software VO-dgtest-cosmos-backend/5.0

Environment Modules concept

The information will be published soon.

Retrieved from "http://dgiref.d-grid.de/wiki/guide:Common_Production_Environment"
Personal tools