cluster:Nfs/4

From Dgiref

Introduction

A network file system (NFS) is any computer file system that supports files sharing, printers and other resources as persistent storage over a computer network. By using NFS, users and programs can access files on remote systems almost as if they were local files.

NFS consists of two main parts: a server and one or more clients. The client remotely accesses the data that is stored on the server machine. To do the job well, a few processes have to be configured and running. There are some benefits which NFS provides:

Local workstations use less disk space because commonly used data can be stored on a single machine and still remain accessible to others over the network.
There is no need for users to have separate home directories on every network machine. Home directories could be set up on the NFS server and made available throughout the network.
Storage devices can be used by other machines on the network. This may reduce the number of removable media drives throughout the network.

NFSv4 supports several security flavors, including:

AUTH_SYS (also known as AUTH_UNIX) represents the traditional low-security model found in NFSv2/v3. Provides UNIX-style credentials by using UIDs and GIDs to identify the sender and recipient of RPC messages.
RPCSEC_GSS - new security flavor. Introduces secure authentication, integrity, and encryption. It is based on GSS-API. The three required security triples when using RPCSEC_GSS are:
- Kerberos 5 for enterprise/LAN use
- LIPKEY (based on SPKM-3) for Internet use
- SPKM-3

The D-Grid reference installation use the model: one export point pro NFS server. To reduce the number of the mount points for the NFS clients and set the shared structure into one directory. The clients mount (server exports) /srv/nfs/ directory, and allocate links for appropriate shared resource. The AUTH_SYS security model for NFSv4 is choosed.

Monitoring of the current NFS activity can be done with the command: watch nfsstat -r -o net

Package:    NFS v. 4
os:             Scientific Linux version 5.6 64 bit
server:        dgireffs1.fzk.de
manuals: nfs4 server / nfs4 client

Archive links

nfs3 server / nfs3 client

Information links

Download links

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

NFS server v.4

Prepare

Operating system: Scientific Linux version 5.6 64 bit

Optimizing the configuration: 

Use minimal operating system installation without firewall.
To verify installed packages use the command 
rpm -qa | grep package_name
Install the following additional packages:
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command: 
chkconfig <SERVICE> off

Configure the following settings for the server:

Prepare new hard disk

To export a brand new disk through nfs, the corresponding device in the /dev directory must be find (use the boot log messages from /var/log). Lets assume that the new device is /dev/hdg.
Create the new partition on your disk with the command: fdisk /dev/hdg. Follow the instructions and create one big partition /dev/hdg1.
Format the new partition with the ext3 file system with the following command: mkfs -t ext3 -c /dev/hdg1 ("-c" options search for invalid disk sectors).
Create the new export directory (/srv/nfs) and change the permission to 1777 and mount the partition on it.

The new device will be mounted on boot on the selected directory.

Firewall configuration

configure TCP port 2049 for nfs(how to open port in firewall).
configure TCP port 1111 for portmapper (how to open port in firewall).

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare the nfs server installation
```

# Declare the variables section ------------

# Please insert your actual configuration

# NFS_SERVER_DIR="export directory on nfs server"

# RIGHTS="access mode for export directory on the nfs server"

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
echo `su -`
```
```
NFS_SERVER_DIR="/srv/nfs"
```
```
RIGHTS="1777"
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
#-> start routine
```
```
mkdir -m ${RIGHTS} ${NFS_SERVER_DIR}
```
```
#<- end routine
```

Install

Install the actual versions of
- nfs-utils
- portmap
- quota
Create the mount directory

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install nfs server
```
```
source prepare.sh
```
```
 
```
```
#-> start routine
```
```
echo `su -`
```
```
yum -y install nfs-utils portmap quota
```
```
#<- end routine
```

Configure

Configure the following files for the host access:
Configure the following files for the NFS work:
- used on the NFS server
  - /etc/exports -
  - /etc/sysconfig/nfs
- used on the NFS client and server
  - /etc/idmapd.conf
  - /etc/gssapi_mech.conf

The NFS server has to be configured to export filesets, like the users home directories, the system software (etc/profile.d, ogsadai), the VO specific software areas, the UNICORE filespace. Server in D-Grid reference installation exports only one directory, located into /srv/nfs with the following attributes:

	Export	Export options	Node	Mountpoint	Mount options
nfs	/srv/nfs	rw,fsid=0,insecure,no_subtree_check,sync	alle	/srv/nfs	rw,hard,intr,proto=tcp,port=2049,noauto
nfs	/srv/nfs/home	rw,nohide,insecure,no_subtree_check,sync	alle	/srv/nfs	rw,hard,intr,proto=tcp,port=2049,noauto
nfs	/srv/nfs/software	rw,nohide,insecure,no_subtree_check,sync	alle	/srv/nfs	rw,hard,intr,proto=tcp,port=2049,noauto

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# prepare the nfs server installation
```

# Declare the variables section ------------

```
# Private network adresses
```
```
# NETWORK=ip adress for private network
```
```
# NETWORK_MASK=private network mask
```

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
NETWORK=*
```
```
NETWORK_MASK=*
```
```
NFS_SERVER_DIR="/srv/nfs"
```
```
HOME_SERVER_DIR=${NFS_SERVER_DIR}/home
```

SOFTWARE_SERVER_DIR=${NFS_SERVER_DIR}/software

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
#-> start routine
```
```
# Configure /etc/sysconfig/nfs
```

# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 (auth=krb5,krb5i, or krb5p)

```
SECURE_NFS="no"
```
```
 
```
```
# Check the /etc/gssapi_mech.conf
```

[[ -f /etc/gssapi_mech.conf ]] && echo "OK" || exit

```
 
```
```
# configure /etc/exports
```
```
cat >>/etc/exports << EOF 
```

${NFS_SERVER_DIR} ${NETWORK}/${NETWORK_MASK}(rw,fsid=0,insecure,no_subtree_check,async,no_root_squash)

${HOME_SERVER_DIR} ${NETWORK}/${NETWORK_MASK}(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

${SOFTWARE_SERVER_DIR} ${NETWORK}/${NETWORK_MASK}(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

```
 
```
```
EOF
```
```
#<- end routine
```

Proceed

/etc/init.d/portmap - used on the client and server
/etc/init.d/nfs - required on the server
/etc/init.d/rpcidmapd - required on both client and server
/etc/init.d/rpcsvcgssd - required on the server when RPCSEC_GSS is used

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```
```
# start script
```
```
chkconfig --level 0123456 portmap off
```
```
chkconfig --level 345 portmap on
```
```
chkconfig --level 0123456 rpcidmapd off
```
```
chkconfig --level 345 rpcidmapd on
```
```
chkconfig --level 0123456 nfslock off
```
```
chkconfig --level 345 nfslock on
```
```
chkconfig --level 0123456 nfs off
```
```
chkconfig --level 345 nfs on
```
```
 
```
```
chkconfig --level 0123456 rpcgssd off
```

chkconfig --level 0123456 rpcsvcgssd off

```
 
```
```
# stop services
```
```
/etc/init.d/rpcgssd stop
```
```
/etc/init.d/rpcsvcgssd stop
```
```
 
```
```
# start services
```
```
/etc/init.d/portmap restart
```
```
/etc/init.d/rpcidmapd restart
```
```
/etc/init.d/nfslock restart
```
```
/etc/init.d/nfs restart
```
```
 
```
```
#<- end routine
```

Initial test

Try to examine the output from the following commands:

rpcinfo -p to check the rpc
netstat -tupa to display the open ports for the applications

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```
```
rpcinfo -p
```
```
#   program vers proto   port
```

#    100000    2   tcp    111  portmapper

#    100000    2   udp    111  portmapper

```
#    100024    1   udp    975  status
```
```
#    100024    1   tcp    978  status
```
```
#    100011    1   udp    615  rquotad
```
```
#    100011    2   udp    615  rquotad
```
```
#    100011    1   tcp    618  rquotad
```
```
#    100011    2   tcp    618  rquotad
```
```
#    100003    2   udp   2049  nfs
```
```
#    100003    3   udp   2049  nfs
```
```
#    100003    4   udp   2049  nfs
```
```
#    100021    1   udp  32770  nlockmgr
```
```
#    100021    3   udp  32770  nlockmgr
```
```
#    100021    4   udp  32770  nlockmgr
```
```
#    100003    2   tcp   2049  nfs
```
```
#    100003    3   tcp   2049  nfs
```
```
#    100003    4   tcp   2049  nfs
```
```
#    100021    1   tcp  50432  nlockmgr
```
```
#    100021    3   tcp  50432  nlockmgr
```
```
#    100021    4   tcp  50432  nlockmgr
```
```
#    100005    1   udp    652  mountd
```
```
#    100005    1   tcp    655  mountd
```
```
#    100005    2   udp    652  mountd
```
```
#    100005    2   tcp    655  mountd
```
```
#    100005    3   udp    652  mountd
```
```
#    100005    3   tcp    655  mountd
```
```
 
```
```
netstat -tupa
```

#Active Internet connections (servers and established)

#Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name

#tcp        0      0 *:nfs                       *:*                         LISTEN      -

#tcp        0      0 *:dei-icda                  *:*                         LISTEN      3830/rpc.rquotad

#tcp        0      0 *:tinc                      *:*                         LISTEN      3868/rpc.mountd

#tcp        0      0 *:sunrpc                    *:*                         LISTEN      3697/portmap

#tcp        0      0 *:978                       *:*                         LISTEN      3764/rpc.statd

```
 
```

#udp        0      0 *:hello-port                *:*                                     3868/rpc.mountd

#udp        0      0 *:972                       *:*                                     3764/rpc.statd

#udp        0      0 *:975                       *:*                                     3764/rpc.statd

#udp        0      0 *:sco-inetmgr               *:*                                     3830/rpc.rquotad

#udp        0      0 *:sunrpc                    *:*                                     3697/portmap

```
 
```
```
exportfs -rv
```

#exporting 10.0.171.0/255.255.255.0:/srv/nfs/software

#exporting 10.0.171.0/255.255.255.0:/srv/nfs/home

#exporting 10.0.171.0/255.255.255.0:/srv/nfs

```
 
```
```
showmount -e
```
```
#Export list for dgireffs1.fzk.de:
```

#/srv/nfs          10.0.171.0/255.255.255.0

#/srv/nfs/home     10.0.171.0/255.255.255.0

#/srv/nfs/software 10.0.171.0/255.255.255.0

```
 
```
```
#<- end routine
```

Update

The direct way to update or delete the installed nfs rpm software package is to use the yum or rpm.

administrator's script: update.sh

```
 
```
```
#!/bin/bash
```
```
# update
```
```
#-> start routine
```
```
su -c 'yum update nfs-utils'
```
```
#<- end routine
```

Please open a NGI-DE ticket if you experience any Installation or Configuration problem.

NFS client v.4

Prepare

Operating system: Scientific Linux version 5.6 64 bit

Optimizing the configuration: 

Use minimal operating system installation without firewall.
To verify installed packages use the command 
rpm -qa | grep package_name
Install the following additional packages:
yum -y install wget yum rpm make gcc gcc-c++ tar sed zlib openssl
After the installation is complete, turn off any unnecessary services (like gpm, sendmail, cups, haldaemon, messagebus, pcmcia, anacron, atd) with the following command: 
chkconfig <SERVICE> off

Configure the following settings for the server:

Prepare new hard disk

To export a brand new disk through nfs, the corresponding device in the /dev directory must be find (use the boot log messages from /var/log). Lets assume that the new device is /dev/hdg.
Create the new partition on your disk with the command: fdisk /dev/hdg. Follow the instructions and create one big partition /dev/hdg1.
Format the new partition with the ext3 file system with the following command: mkfs -t ext3 -c /dev/hdg1 ("-c" options search for invalid disk sectors).
Create the new export directory (/srv/nfs) and change the permission to 1777 and mount the partition on it.

The new device will be mounted on boot on the selected directory.

Firewall configuration

configure TCP port 2049 for nfs(how to open port in firewall).
configure TCP port 1111 for portmapper (how to open port in firewall).

administrator's script: prepare.sh

```
 
```
```
#!/bin/bash
```
```
# prepare the nfs client installation
```

# Declare the variables section ------------

# Please insert your actual configuration

# NFS_SERVER_DIR="export directory on nfs server"

# RIGHTS="access mode for export directory on the nfs server"

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
echo `su -`
```
```
NFS_SERVER_DIR="/srv/nfs"
```
```
RIGHTS="755"
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
#-> start routine
```
```
mkdir -m ${RIGHTS} ${NFS_SERVER_DIR}
```
```
#<- end routine
```

Install

Install the actual versions of
- nfs-utils
- portmap
- quota
Create the mount directory

administrator's script: install.sh

```
 
```
```
#!/bin/bash
```
```
# install nfs client
```
```
source prepare.sh
```
```
 
```
```
#-> start routine
```
```
echo `su -`
```
```
yum -y install nfs-utils portmap quota
```
```
#<- end routine
```

Configure

Configure the following files for the host access:
Configure the following files for the NFS work:
- used on the NFS client
  - /etc/fstab
  - /etc/auto.master
- used on the NFS client and server
  - /etc/idmapd.conf
  - /etc/gssapi_mech.conf

NFS client should mount the /srv/nfs directory, but some softlinks should point to the /srv/nfs subdirectories. For example:

for all hosts in cluster
/home -> /srv/nfs/home
/software -> /srv/nfs/software

for WNs
/opt/glite-wn -> /srv/nfs/software/glite-wn/
/opt/d-grid -> /srv/nfs/software/vos/
/etc/profile.d/dgrid_env.sh -> /opt/d-grid/etc/profile.d/dgrid_env.sh
/uspace -> /srv/nfs/uspace

for gLite CE
/opt/glite-wn -> /srv/nfs/software/glite-wn/
/opt/d-grid -> /srv/nfs/software/vos/
/etc/profile.d/dgrid_env.sh -> /opt/d-grid/etc/profile.d/dgrid_env.sh
/etc/profile.d/grid-env.sh -> /opt/glite/etc/profile.d/grid-env.sh
/etc/profile.d/grid-env.csh -> /opt/glite/etc/profile.d/grid-env.csh

for Unicore
- /uspace -> /srv/nfs/uspace

administrator's script: configure.sh

```
 
```
```
#!/bin/bash
```
```
# prepare the nfs client installation
```

# Declare the variables section ------------

# from here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
NFS_SERVER=dgireffs1
```

# till here ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```
#-> start routine
```
```
# Configure /etc/sysconfig/nfs
```

# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 (auth=krb5,krb5i, or krb5p)

```
SECURE_NFS="no"
```
```
 
```
```
# Check the /etc/gssapi_mech.conf
```

[[ -f /etc/gssapi_mech.conf ]] && echo "OK" || exit

```
 
```
```
cat << EOF >> /etc/fstab
```

${NFS_SERVER}.${DOMAIN}:/     /srv/nfs    nfs4    rw,hard,intr,proto=tcp,port=2049,noauto  0 0

```
 
```
```
EOF
```
```
#<- end routine
```

Proceed

/etc/init.d/portmap - used on the client and server
/etc/init.d/rpcidmapd - required on both client and server
/etc/init.d/rpcgssd - required on the client when RPCSEC_GSS is used

administrator's script: proceed.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```
```
chkconfig --level 0123456 portmap off
```
```
chkconfig --level 345 portmap on
```
```
chkconfig --level 0123456 rpcidmapd off
```
```
chkconfig --level 345 rpcidmapd on
```
```
chkconfig --level 0123456 nfslock off
```
```
chkconfig --level 0123456 nfs off
```
```
chkconfig --level 0123456 rpcgssd off
```

chkconfig --level 0123456 rpcsvcgssd off

```
# Stop local firewall
```
```
service iptables stop
```
```
chkconfig --level 0123456 iptables off
```
```
 
```
```
# stop services
```
```
/etc/init.d/nfslock stop
```
```
/etc/init.d/nfs stop
```
```
/etc/init.d/rpcgssd stop
```
```
/etc/init.d/rpcsvcgssd stop
```
```
 
```
```
# start services
```
```
/etc/init.d/portmap restart
```
```
/etc/init.d/rpcidmapd restart
```
```
 
```
```
# mount the nfs directory
```
```
mount -v /srv/nfs
```
```
 
```
```
#<- end routine
```

Initial test

Try to examine the output from the following commands:

rpcinfo -p to check the rpc
netstat -tunap to display the open ports for the applications

administrator's script: test.sh

```
 
```
```
#!/bin/bash
```
```
#-> start routine
```
```
rpcinfo -p
```
```
   program vers proto   port
```

    100000    2   tcp    111  portmapper

    100000    2   udp    111  portmapper

```
    100024    1   udp    727  status
```
```
    100024    1   tcp    730  status
```
```
1073741824    1   tcp  48909
```
```
    100011    1   udp    965  rquotad
```
```
    100011    2   udp    965  rquotad
```
```
    100011    1   tcp    968  rquotad
```
```
    100011    2   tcp    968  rquotad
```
```
    100003    2   udp   2049  nfs
```
```
    100003    3   udp   2049  nfs
```
```
    100003    4   udp   2049  nfs
```
```
    100021    1   udp  32771  nlockmgr
```
```
    100021    3   udp  32771  nlockmgr
```
```
    100021    4   udp  32771  nlockmgr
```
```
    100003    2   tcp   2049  nfs
```
```
    100003    3   tcp   2049  nfs
```
```
    100003    4   tcp   2049  nfs
```
```
    100021    1   tcp  39151  nlockmgr
```
```
    100021    3   tcp  39151  nlockmgr
```
```
    100021    4   tcp  39151  nlockmgr
```
```
    100005    1   udp    981  mountd
```
```
    100005    1   tcp    984  mountd
```
```
    100005    2   udp    981  mountd
```
```
    100005    2   tcp    984  mountd
```
```
    100005    3   udp    981  mountd
```
```
    100005    3   tcp    984  mountd
```
```
 
```
```
 
```
```
netstat -tunap
```

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name

tcp        0      0 0.0.0.0:2049                0.0.0.0:*                   LISTEN      -

tcp        0      0 0.0.0.0:49126               0.0.0.0:*                   LISTEN      -

tcp        0      0 0.0.0.0:968                 0.0.0.0:*                   LISTEN      32164/rpc.rquotad

tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      31182/portmap

tcp        0      0 0.0.0.0:984                 0.0.0.0:*                   LISTEN      32181/rpc.mountd

udp        0      0 0.0.0.0:2049                0.0.0.0:*                               -

udp        0      0 0.0.0.0:32771               0.0.0.0:*                               -

udp        0      0 0.0.0.0:965                 0.0.0.0:*                               32164/rpc.rquotad

udp        0      0 0.0.0.0:981                 0.0.0.0:*                               32181/rpc.mountd

udp        0      0 0.0.0.0:111                 0.0.0.0:*                               31182/portmap

```
#<- end routine
```

Update

The direct way to update or delete the installed nfs rpm software package is to use the yum or rpm.

administrator's script: update.sh

```
 
```
```
#!/bin/bash
```
```
# update
```
```
#-> start routine
```
```
su -c 'yum update nfs-utils'
```
```
#<- end routine
```

cluster:Nfs/4

Contents

Introduction

NFS server v.4

Prepare

Install

Configure

Proceed

Initial test

Update

NFS client v.4

Prepare

Install

Configure

Proceed

Initial test

Update

Views

Personal tools

general

guidelines

reference installation

Search

internal

Toolbox