PVFS version 2, first try, part 2

The next step is to propagate the config on the nodes (compute-00-00, compute-00-01, compute-00-02). The installer is not part of the server pool (will copy the fs-conf file on clients only later).

We want a rc script to start the server, so just edit the one provided in the example directory:

--------------------------------------------------------
cp examples/pvfs2-server.rc examples/pvfs2-server
--------------------------------------------------------

and the diff is:

--------------------------------------------------------
[mbozzore@stakhanov pvfs-2.6.3]$ diff -U4 examples/pvfs2-server.rc examples/pvfs2-server
--- examples/pvfs2-server.rc 2008-08-19 10:08:26.000000000 -0400
+++ examples/pvfs2-server 2008-08-19 10:11:17.000000000 -0400
@@ -13,9 +13,9 @@
# override this if your server binary resides elsewhere
PVFS2SERVER=/home/mbozzore/pvfs-2.6.3/sbin/pvfs2-server
# override this if you want servers to automatically pick a conf file,
# but you just need to specify what directory they are in
-PVFS2_CONF_PATH=/etc
+PVFS2_CONF_PATH=/etc/pvfs2

# the server will record its PID in this file
PVFS2_PIDFILE=/var/run/pvfs2.pid
--------------------------------------------------------

Propagating the rc script is quite easy (don't forget to change the permissions):

--------------------------------------------------------

• chown root.root /VM/mbozzore/PVFS/pvfs-2.6.3/examples/pvfs2-server ; chmod u+x /VM/mbozzore/PVFS/pvfs-2.6.3/examples/pvfs2-server
  
• for i in $liste; do scp /VM/mbozzore/PVFS/pvfs-2.6.3/examples/pvfs2-server $i:/etc/rc.d/init.d/;done

--------------------------------------------------------

Of course IO and metadata servers are in $liste

Ouch, I forgot something: pvfs2-server is dynamically linked with libdb-4.6 so we need to create a new entry under /etc/ld.so.conf.d for BerkeleyDB (I mean ... on the nodes):

1. create the file in a shared location (under a user account)
   
   --------------------------------------------------------
   [root@stakhanov ~]# cat /home/mbozzore/tmp/BerkeleyDB.conf
   
   /home/mbozzore/db-4.6.21/lib/
   --------------------------------------------------------
   
   
2. propagate the file (using pdsh this time)
   
   --------------------------------------------------------
   pdsh -a "cp /home/mbozzore/tmp/BerkeleyDB.conf /etc/ld.so.conf.d/BerkeleyDB.conf
   --------------------------------------------------------
   
3. run ldconfig on the nodes

--------------------------------------------------------
pdsh -a "ldconfig"
--------------------------------------------------------

Done !!

It is now time for initialization:

--------------------------------------------------------
[root@stakhanov ~]# for i in $liste; do ssh -x $i "/home/mbozzore/pvfs-2.6.3/sbin/pvfs2-server /etc/pvfs2/pvfs2-fs.conf /etc/pvfs2/pvfs2-server.conf-$i -f";done
[D 07:02:04.125320] PVFS2 Server version 2.6.3 starting.
[D 08/25 07:02] PVFS2 Server: storage space created. Exiting.
[D 20:27:29.628509] PVFS2 Server version 2.6.3 starting.
[D 08/25 20:27] PVFS2 Server: storage space created. Exiting.
[D 22:58:03.677750] PVFS2 Server version 2.6.3 starting.
[D 08/19 22:58] PVFS2 Server: storage space created. Exiting.
--------------------------------------------------------


Then, we can start the server:

--------------------------------------------------------
[root@stakhanov ~]# for i in $liste; do ssh -x $i "service pvfs2-server start" ; done
Starting PVFS2 server: [D 07:07:50.038126] PVFS2 Server version 2.6.3 starting.
[ OK ]
Starting PVFS2 server: [D 20:36:08.930395] PVFS2 Server version 2.6.3 starting.
[ OK ]
Starting PVFS2 server: [D 23:06:40.730802] PVFS2 Server version 2.6.3 starting.
[ OK ]
--------------------------------------------------------

Looks fine (after ps on the nodes and log check).


Let's take care of the clients now.

Hmmm ... pvfs2-server is not dynamically linked with the pvfs2 library, but pvfs2-client is. We need to create a new entry under /etc/ld.so.conf.d/ on the nodes. Same story again:

1. create the file
   
   --------------------------------------------------------
   [root@stakhanov ~]# cat /home/mbozzore/tmp/pvfs-2.6.3.conf
   /home/mbozzore/pvfs-2.6.3/lib
   --------------------------------------------------------
   
   
2. propagate the file
   
   --------------------------------------------------------
   [root@stakhanov ~]#pdsh -a "cp /home/mbozzore/tmp/pvfs-2.6.3.conf /etc/ld.so.conf.d/pvfs-2.6.3.conf"
   --------------------------------------------------------
   

   • check the cache on one node before updating it

   --------------------------------------------------------
   [root@compute-00-01 ~]# ldconfig -p | grep pvfs
   [root@compute-00-01 ~]#
   --------------------------------------------------------

3. update the cache:
   
   --------------------------------------------------------
   [root@stakhanov ~]#pdsh -a "ldconfig"
   --------------------------------------------------------
   

   • check the cache again

   --------------------------------------------------------
   [root@compute-00-01 ~]# ldconfig -p | grep pvfs libpvfs2.so (libc6,x86-64) => /home/mbozzore/pvfs-2.6.3/lib/libpvfs2.so
   [root@compute-00-01 ~]#
   --------------------------------------------------------

Good, we can load the kernel module and start the client:
--------------------------------------------------------[/
[root@stakhanov ~]pdsh -a "insmod /VM/mbozzore/PVFS/pvfs-2.6.3/src/kernel/linux-2.6/pvfs2.ko"

[root@stakhanov ~]pdsh -a "/VM/mbozzore/PVFS/pvfs-2.6.3/src/apps/kernel/linux/pvfs2-client -p /VM/mbozzore/PVFS/pvfs-2.6.3/src/apps/kernel/linux/pvfs2-client-core -L /var/log/pvfs2-client.log"
--------------------------------------------------------

Note: if you forget to insert the pvfs2 kernel module before starting the pvfs2-client, you will get something like that in the client logs:

###############################################
[root@compute-00-03 ~]# cat /var/log/pvfs2-client.log
[E 14:56:33.296570] Error: could not setup device /dev/pvfs2-req.
[E 14:56:33.296740] Error: did you remember to load the kernel module?
[E 14:56:33.300203] pvfs2-client-core with pid 14238 exited with value 254
[E 14:56:34.331904] Error: could not setup device /dev/pvfs2-req.
[E 14:56:34.331972] Error: did you remember to load the kernel module?
###############################################

Ok, ready to mount the filesystem then:

• create an entry under /mnt
  
  --------------------------------------------------------
  [root@stakhanov ~]# pdsh -a "mkdir /mnt/pvfs2"
  --------------------------------------------------------
  
• append what is needed to /etc/fstab
  
  --------------------------------------------------------
  [root@stakhanov ~]# pdsh -a "echo "tcp://compute-00-01:3334/pvfs2-fs /mnt/pvfs2 pvfs2 defaults,noauto 0 0" >> /etc/fstab"
  --------------------------------------------------------
  
• mount
  
  --------------------------------------------------------
  [root@stakhanov ~]# pdsh -a "mount /mnt/pvfs2"
  --------------------------------------------------------

Looks good, we are now able to run basic tests.
On any compute node:

--------------------------------------------------------
[root@compute-00-02 ~]# /home/mbozzore/pvfs-2.6.3/bin/pvfs2-ping -m /mnt/pvfs2

(1) Parsing tab file...

(2) Initializing system interface...

(3) Initializing each file system found in tab file: /etc/fstab...

PVFS2 servers: tcp://compute-00-01:3334
Storage name: pvfs2-fs
Local mount point: /mnt/pvfs2
/mnt/pvfs2: Ok

(4) Searching for /mnt/pvfs2 in pvfstab...

PVFS2 servers: tcp://compute-00-01:3334
Storage name: pvfs2-fs
Local mount point: /mnt/pvfs2

meta servers:
tcp://compute-00-02:3334

data servers:
tcp://compute-00-00:3334
tcp://compute-00-01:3334

(5) Verifying that all servers are responding...

meta servers:
tcp://compute-00-02:3334 Ok

data servers:
tcp://compute-00-00:3334 Ok
tcp://compute-00-01:3334 Ok

(6) Verifying that fsid 398972398 is acceptable to all servers...

Ok; all servers understand fs_id 398972398

(7) Verifying that root handle is owned by one server...

Root handle: 1048576
Ok; root handle is owned by exactly one server.

================================================== ===========

The PVFS2 filesystem at /mnt/pvfs2 appears to be correctly configured.
--------------------------------------------------------

Wonderful, it is now time to play with the test suite and MPI-IO.

Let's check what is available in the test directory. There is a configure script available and the (partial) output of ./configure --help is:

--------------------------------------------------------
Optional Packages:
--with-PACKAGE[=ARG] use PACKAGE [ARG=yes]
--without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no)
--with-efence=<path> Use electric fence for malloc debugging.
--with-mpi=<dir> Location of the MPI installation
--with-pvfs2-src=<dir> Location of the PVFS2 src directory
--with-pvfs2-build=<dir> Location of the PVFS2 build dir (if different from src dir)
--with-db=<dir> Location of installed DB package (default=/usr)
--with-openssl=<dir> Location of installed openssl package (default=/usr)
--without-openssl Don't build with openssl.

--with-libaio=<dir> Location of installed libaio package (default=/usr)
--without-libaio Don't build with libaio.
--------------------------------------------------------

We will see in part 3 how painful can QA be :-)

Mehdi Bozzo-Rey