middleware:Unicore/63/server
See also troubleshooting for this page.
Contents |
UNICORE server v.6.3
Prepare
- Software
- Scientific Linux version 5.4 64 bit
- Sun Java Runtime Environment version >= 1.6.0
- Perl version >= 5.4
- Torque Client
Optimizing the configuration:
Use minimal operating system installation without firewall. To verify installed packages use the command
-
rpm -qa | grep package_name
Install the following additional packages:
-
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command:
-
chkconfig <SERVICE> off
Configure the following settings for the server:
- Server Certificates for the Gateway service, the XUUDB service, the unicorex service
- Users
- A userid to start and administer the above services (not root!)
- A userid to execute status queries to the local resource management system (qstat) (not root!)
- Others
- NFS-Export of users home directories
- Register your Resource with GRRS: Web Form.
- Firewall configuration
The UNICORE Gateway is the entry to a site, it authenticates the users and forwards all client requests to the corresponding local service. The Gateway's port (8080 by default) needs to be accessible from the Internet. Therefore is must be open in the firewall for incoming https connections.
The unicorex service registers with the central UNICORE Registry which is running on https://dgrid-unic.fz-juelich.de with Port 9110. unicorex must be able to make a connection to this system on the given port, so that the firewall should allow that outgoing connection (how to open port in firewall).
administrator's script: prepare.sh
#!/bin/bash# prepare# Declare the variables section ------------# Please insert your actual configuration# USER_UNICORE=user to install unicore# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~USER_UNICORE=njsadmin# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~# install prerequisitesyum -y install perl python
wget -O /usr/local/src/jdk-6u16-linux-x64.bin http://mirror.scc.kit.edu/downloads/src/misc/jdk-6u16-linux-x64.bin
sh /usr/local/src/jdk-6u16-linux-x64.bin
# assign the JAVA_HOME environment variableecho "export JAVA_HOME=/usr/local/src/jdk1.6.0_16/" > /etc/profile.d/jdk.sh
echo "PATH=/usr/local/src/jdk1.6.0_16/bin:$PATH" >> /etc/profile.d/jdk.sh
chmod +x /etc/profile.d/jdk.sh
source /etc/profile.d/jdk.sh
ln -s /usr/local/src/jdk1.6.0_16 /usr/java/latest
# create non-root user for unicore installationadduser $USER_UNICORE -d /localhome/unicore
# stop local firewallservice iptables stop
chkconfig iptables off
# create directory for the grid host certificatesmkdir /etc/grid-security/
# after copy the host certificate and host key into /etc/grid-security/
Install
| the components Gateway, UNICORE/X, XUUDB, TSI will be used. The component Registry will not be used. |
The installation from rpms packages with the following directory structure:
- /etc/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
- /usr/share/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
- /etc/rc.d/init.d/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
- /var/lib/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
- /var/log/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
- /var/run/
- unicore-gateway/
- unicore-tsi-dgrid/
- unicore-unicorex/
- unicore-xuudb/
administrator's script: install.sh
#!/bin/bash# install unicore 6.3# Declare the variables section ------------BASE_URL=http://mirror.scc.kit.edu/downloads/rpms/unicore/2010.2
PATH_GATEWAY=/etc/unicore-gateway/
PATH_TSI=/etc/unicore-tsi-dgrid/
PATH_UNICOREX=/etc/unicore-unicorex/
PATH_XUUDB=/etc/unicore-xuudb/
USER_UNICORE=unicore# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#-> start routinecd /usr/local/src/
wget ${BASE_URL}/unicore-gateway-6.3.0-1.noarch.rpm
wget ${BASE_URL}/unicore-tsi-dgrid-6.3.0-1.noarch.rpm
wget ${BASE_URL}/unicore-unicorex-6.3.0-1.noarch.rpm
wget ${BASE_URL}/unicore-xuudb-6.3.0-1.noarch.rpm
# install rpmsrpm -ihv *.rpm
chown $USER_UNICORE $PATH_GATEWAY
chown $USER_UNICORE $PATH_TSI
chown $USER_UNICORE $PATH_UNICOREX
chown $USER_UNICORE $PATH_XUUDB
# Optionaly, remove the rpms.rm -f *.rpm
#<- end routine
Configure
| For the integration into UNICORE monitoring (Common Information Service - CIS) you have to configure the information provider (CIP) part of unicorex by defining your site's static values in /etc/unicore-unicorex/site-info.glue. CIP is automatically started together with unicorex.
|
- Configure the Gateway
|
-
security.properties- define the keystore containing the private key of the gateway and the truststore created above, each with its corresponding passphrase:
-
keystore=/path/to/your/gateway/keystore.p12 keystorepassword=****** truststore=/path/to/your/gateway/truststore.jks truststorepassword=******
- Configure the Unicorex and XNJS
-
wsrflite.xml. Set keystore and truststore:
-
<!-- UNICORE/X server identity --> <property name="unicore.wsrflite.ssl.keystore" value="/path/to/your/unicorex/keystore.p12"/> <property name="unicore.wsrflite.ssl.keypass" value="*******"/> <property name="unicore.wsrflite.ssl.keytype" value="PKCS12"/> <!-- UNICORE/X truststore --> <property name="unicore.wsrflite.ssl.truststore" value="/path/to/your/unicorex/truststore.jks"/> <property name="unicore.wsrflite.ssl.truststorepass" value="*******"/> <property name="unicore.wsrflite.ssl.truststoretype" value="JKS"/>
-
simpleidb. It contains all settings to convert the abstract job descriptions received from the Clients into executable jobs for the target system- In sections "Applications" and "Scripts" you have to adapt the paths to applications and script interpreters your site is offering. You can also add your own applications to the list or remove e.g. Perl, Python, CSH or KSH if you don't want to offer them to the users.
- Section "Resources" the base characteristics of the target system are defined, e.g. number of CPUs, amount of memory, architecture, etc. They are used for scheduling purposes and for providing the user with the information through the client. Adapt the settings to define your system.
-
xnjs_legacy.xmlto describe the system where your TSI is running - Edit
site-info.glueto provide static information for the common information provider (CIP) component.
-
- Configure the TSI
- the
tsi/conf/tsi.propertiesis already adapted to Torque, but check once again - The location of Torque commands (qsub, qstat, ...) is defined as
/usr/bin. If you need to change this path adapt in tsi/tsi the following line:my $pbs_bin_dir = "/usr/bin";
- the
- Configure the XUUDB
keytool -import -file /etc/grid-security/hostcert.pem -keystore /path/yo/your/truststore.jks |
- XUUDB distinguishes between Client and Server configuration.
- Edit
xuudb_server.confto define keystore and truststore:
- Edit
xuudb_keystore_file=/path/yo/your/xuudb/keystore.p12
xuudb_keystore_type=PKCS12
xuudb_keystore_password=******
xuudb_truststore_file=/path/to/your/xuudb/truststore.jks
xuudb_truststore_type=JKS
xuudb_truststore_password=******- Edit
xuudb_client.confto define keystore and truststore to use with unicore-xuudb-admin (administrator interface)
- Edit
xuudb_keystore_file=/path/yo/your/xuudb/keystore.p12
xuudb_keystore_type=PKCS12
xuudb_keystore_password=******
xuudb_truststore_file=/path/to/your/xuudb/truststore.jks
xuudb_truststore_type=JKS
xuudb_truststore_password=******- Example: add users to XUUDB:
- manually:
/usr/bin/unicore-xuudb-admin adddn <gcID> <DN> <xlogin> <role>
- manually:
where <gcID> is the gcID as defined in unicorex, <DN> the user's Distinguished Name (formated according to RFC 2253), <XLOGIN> his/her userid on the target system and <role> his/her role, usually it is "User".
- from file:
cd /path/yo/your/xuudb # backup current configuration into the ''NEW'' xuudbBackup.csv /usr/bin/unicore-xuudb-admin export xuudbBackup.csv # generate mapfile for UNICORE 6: dgridmap -cert-path /root/certificates -output-xu /path/yo/your/xuudb/xuudb_in.csv # Import the user records at the same time delete all the old records: /usr/bin/unicore-xuudb-admin import xuudb_in.csv clearDB
| Using the Dgrid Tools Independently from the rpm Unicore packages, the dgrid_extensions.tgz tarball includes some tools. Actually, only two scripts are in use: create_gw_truststore.py together with x509.py to create the trustore repository. |
administrator's script: configure.sh
#!/bin/bash# configure unicore 6.3HOSTNAME=`hostname`
SITENAME="dgiref_scc"
#logpath=/var/log/unicore-*password=t00ltechostcertPath=/etc/grid-security/dgiref-unicore63.fzk.de.p12
truststorePath=/etc/grid-security/truststore.jks
# ========================================================================== truststore configuration# put the hostcert to the $hostcertPath# download dgrid extensionscd /opt
wget http://mirror.scc.kit.edu/downloads/src/unicore/2010.2/dgrid_extensions.tgz
tar xzvf dgrid_extensions.tgz# some directories will be created (only tools will be in use)rm -f dgrid_extensions.tgz
# create trustorecd toolspython create_gw_truststore.py -o $truststorePath
# put the truststore.jks to the $truststorePathkeytoolkeytool -import -file /etc/grid-security/hostcert.pem -alias hostcert -keystore $truststorePath
# Certificate was added to keystore# ========================================================================== Gateway configurationcd /etc/unicore-gateway
echo "$SITENAME = https://$HOSTNAME:7777" > connections.properties
sed -i "s/localhost/$HOSTNAME/g" gateway.properties
echo "\
keystore=$hostcertPathkeystorepassword=$passwordtruststore=$truststorePathtruststorepassword=$password" > security.properties
# ========================================================================== Unicorex/XNJS configurationcd /etc/unicore-unicorex
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure jmxremote.password# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure logging.properties# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure security_policy.xml# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure ucc.logging.properties# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure xacml.config# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure security_policy.xml# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure xnjs.xml# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure xnjs_legacy.xml# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure simpleidb# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure simpleuudbsed -i "s/\${USER_NAME}/njsadmin/g" simpleuudb
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure site-info.gluecat site-info.glue# adapt for site configurations<?xml version="1.0" encoding="UTF-8"?>
<glue:Domains xmlns:glue="http://schemas.ogf.org/glue/2008/05/spec_2.0_d42_r01"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<AdminDomain CreationTime="2009-12-16T10:05:12Z" Validity="500"
BaseType="Domain">
<ID>dgiref</ID>
<!--Name and Description commented because bes's common name and longdescription properties have the same semantics--><Name>DGIREF-SCC</Name><WWW>http://mirror.scc.kit.edu</WWW><OtherInfo>DGIREF-SCC site Part of D-Grid</OtherInfo><Distributed>false</Distributed><Owner>www.scc.kit.edu</Owner><Contact CreationTime="2008-04-21T10:05:12Z" Validity="500"><LocalID>mailto:dgrid-admin@scc.kit.edu</LocalID><URL>mailto:dgrid-admin@scc.kit.edu</URL><Type>general</Type><OtherInfo>working hours: 8-18</OtherInfo></Contact><Services><ComputingService CreationTime="2008-04-21T10:05:12Z"Validity="500" BaseType="Service"><!-- dynamic --><ID>dgiref-unicore63.fzk.de</ID><!-- dynamic --><Name>dgiref-unicore63.fzk.de</Name><Capability>unicore.xnjs.jobexecution.embedded_tsi</Capability><!-- <Capability>unicore.xnjs.jobexecution.legacy_tsi</Capability> --><Type>de.fzj.unicore.tsf</Type><QualityLevel>production</QualityLevel><StatusPage>http://someurlofamonitoringpageshowintthestatusofthisservice</StatusPage>--><OtherInfo>D-Grid Computing Ressource</OtherInfo><Location CreationTime="2008-04-21T10:05:12Z" Validity="500"><LocalID>scc</LocalID><Name>Steinbuch Centre for Computing</Name><Address>Hermann-von-Helmholtz-Platz 1</Address><Place>Karlsruhe</Place><Country>Germany</Country><PostCode>D-76344</PostCode><Latitude>49.09</Latitude><Longitude>8.43</Longitude></Location><ComputingEndpoint><ID>urn:ce1</ID><Name>UNICORE-BES</Name><Technology>webservice</Technology><InterfaceName>TargetSystemFactory</InterfaceName><InterfaceVersion>1.0</InterfaceVersion><SupportedProfile>WS-I 1.0</SupportedProfile><SupportedProfile>WS-RF</SupportedProfile><SupportedProfile>WS-Security X.509 Profile</SupportedProfile><SupportedProfile>WS-Security Username Token Profile</SupportedProfile><SupportedProfile>WS-Security SAML Token Profile</SupportedProfile><Semantics>http://www.ogf.org/documents/GFD.108.pdf</Semantics><Implementor>UNICORE</Implementor><ImplementationName>UNICORE</ImplementationName><ImplementationVersion>6.3.0</ImplementationVersion><!-- development, production, pre-production --><QualityLevel>production</QualityLevel><!-- ok, other, critical, unknown, warning --><HealthState>ok</HealthState><HealthStateInfo>I'm fine, thanks for asking</HealthStateInfo>
<ServingState>production</ServingState>
</ComputingEndpoint>
<ComputingManager BaseType="Manager">
<ID>dgiref.fzk.de</ID>
<Name>PBS/MAUI</Name>
<TotalPhysicalCPUs>40</TotalPhysicalCPUs>
<TotalLogicalCPUs>40</TotalLogicalCPUs>
<ExecutionEnvironment BaseType="Resource">
<ID>dgiref.fzk.de</ID>
<Name>dgiref cluster</Name>
<!-- i386, amd64, itanium, powerpc, sparc, any -->
<Platform>i386</Platform>
</ExecutionEnvironment>
</ComputingManager>
</ComputingService>
</Services>
</AdminDomain>
</glue:Domains>
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure uas.configsed -i "s/DEMO-SITE/$SITENAME/g" uas.config
sed -i "s/localhost/$HOSTNAME/g" uas.config
sed -i "s/localhost:8080/dgrid-unic.fz-juelich.de:9110/g" uas.config
vi uas.config# additional url:#uas.externalregistry.url.2=....uas.externalregistry.url.2=https://iwrunicore.fzk.de:9110/D-Grid-KIT/services/Registry?res=default_registry
sed -i "s/REGISTRY/D-Grid/g" uas.config
mkdir storagesed -i "s/path\/to/etc\/unicore-unicorex/g" uas.config
# New in 6.3:# uas.storagefactory.path=/path/to/storage-factory# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure ucc.preferencesecho "
keystore=$hostcertPathstoretype=pkcs12password=$passwordalias=njs test certificateregistry=https://$HOSTNAME:7777/services/Registry?res=default_registryoutput=/tmpprotocols=BFT RBYTEIO" > ucc.preferences
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure wsrflite.xmlvi wsrflite.xml# find and change<!-- UNICORE/X server identity (keystore definition) -->
<property name="unicore.wsrflite.ssl.keystore" value="$hostcertPath"/>
<property name="unicore.wsrflite.ssl.keypass" value="$password"/> <!-- $password -->
<property name="unicore.wsrflite.ssl.keytype" value="PKCS12"/>
<property name="unicore.wsrflite.ssl.keyalias" value="njs test certificate"/>
<!-- UNICORE/X truststore (can be the same as the keystore) -->
<property name="unicore.wsrflite.ssl.truststore" value="$truststorePath"/>
<property name="unicore.wsrflite.ssl.truststorepass" value="$password"/> <!-- $password -->
<property name="unicore.wsrflite.ssl.truststoretype" value="JKS"/>
# ========================================================================== tsi configurationcd /etc/unicore-tsi-dgrid
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure path2njs# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure tsi.propertiessed -i "s/localhost/$HOSTNAME/g" tsi.properties
# setup path to the qstat into /usr/share/unicore-tsi-dgrid/tsi# my $pbs_bin_dir = "/usr/bin";# ========================================================================== xuudb configurationcd /etc/unicore-xuudb
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure client_logging.properties# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure logging.properties# do not change# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure xuudb_client.confecho "
xuudb_http_host=https://$HOSTNAME#SSL yes/noxuudb_use_ssl=true# Port on which the XUUDB server is listeningxuudb_http_port=34463# When using SSL you have to specify a keystore and a truststore.# It is possible, that they are the same file. In that case, you do not# have to specify xuudb_truststore_* properties.# Possible store types are JKS and PKCS12# The truststore has to contain the CA certificate of the XUUDB server cert.xuudb_keystore_file=$hostcertPathxuudb_keystore_password=$passwordxuudb_keystore_type=PKCS12xuudb_truststore_file=$truststorePathxuudb_truststore_password=$passwordxuudb_truststore_type=JKS" > xuudb_client.conf
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Configure xuudb_server.confmkdir /var/lib/unicore-xuudb/data/
echo "
xuudb_http_host=https://$HOSTNAMExuudb_http_port=34463xuudb_use_ssl=truexuudb_type=dnxuudb_data_file=/var/lib/unicore-xuudb/data/UnicoreUserDB.dataxuudb_keystore_file=$hostcertPathxuudb_keystore_password=$passwordxuudb_keystore_type=PKCS12xuudb_truststore_file=$truststorePathxuudb_truststore_password=$passwordxuudb_truststore_type=JKS" > xuudb_server.conf
# ========================================================================== generate mapfile for UNICORE 6:dgridmap -cert-path /etc/grid-security -output-xu /etc/unicore-xuudb/xuudb_in.csv
# ========================================================================== start Unicore servicesservice unicore-xuudb start
# After starting the XUUDB add the D-Grid users# Import the user records at the same time delete all the old records:unicore-xuudb-admin import /etc/unicore-xuudb/xuudb_in.csv clearDB
# checkunicore-xuudb-admin list
Proceed
| Unicore startup and shutdown services: Please make sure the gateway, unicorex, and xuudb components are not started as root! Please make sure that the TSI is started under userid root, as it has to execute the jobs on behalf of the user! |
- For start, stop UNICORE services, use linux command
service <service name> start|stop
administrator's script: proceed.sh
#!/bin/bash# start unicore 6.3service unicore-gateway start
service unicore-tsi-dgrid start
service unicore-unicorex start
Initial test
For each service you can check startup.log to make sure the component has started properly. You can also use
# to check whether the corresponding services are in the process table. ps -ef | grep <unicore-admin> # you should see: java … eu.unicore.gateway.Gateway java … de.fzj.unicore.xuudb.server.XUUDBServer -start java … de.fzj.unicore.uas.UAS conf/uas.config VsiteName ps -ef | grep tsi # shows whether the TSI is up and running.
administrator's script: test.sh
#!/bin/bash# test unicore 6 installationps -ef | grep tsi
ps -ef | grep unicore
Update
To remove Unicore 6 from your system just use:
administrator's script: update.sh
#!/bin/bash# to remove unicore 6.3rpm -e unicore-gateway-6.3.0-1
rpm -e unicore-tsi-dgrid-6.3.0-1
rpm -e unicore-unicorex-6.3.0-1
rpm -e unicore-xuudb-6.3.0-1