Thread: Using MPI on Platform OCS

Originally posted by: bill, Tue Apr 03, 2007 7:30 pm

Hi All,

This topic is for questions related to using MPI on Platform OCS.... There are many different MPI's included with Platform OCS, but we usually focus on OpenMPI, MPICH2 and Intel MPI......

So if you have questions on MPI and Platform OCS please post them here.

Bill.

Originally posted by: bill, Tue Apr 03, 2007 7:43 pm

Okay so you are using the modules command and you want to configure Open MPI to work with the Portland Compilers, and you want to make sure that Portland Compilers is loaded before you use OpenMPI here is how you do it:

1) using sudo create the directory: /share/apps/modulefiles.custom

2) Create the module file using sudo (included in -- cut here -- below) and save it as ompi<version>.pgf (we called ours ompi114.pgf ) You'll have to modify the module file so it points to your version of Open MPI.

3) create the following symlink...

ln -s /share/apps/modulefiles.custom/ompi114.pgf /opt/modules/modulefiles/hpc/ompi114.pgf

(replacing the verion 114 with your version)

4) Test it...The module uses the modules command 'prereq' which requires the hpc/portland module to be loaded before the openmpi114.pgf
module is loaded.

For example...

$ module list
Currently Loaded Modulefiles:
1) hpc/portland
$ module unload hpc/portland
$ module load hpc/ompi114.pgf
hpc/ompi114.pgf(30):ERROR:151: Module 'hpc/ompi114.pgf' depends on one of the module(s) 'hpc/portland '
hpc/ompi114.pgf(30):ERROR:102: Tcl command execution failed: prereq hpc/portland


$ module load hpc/portland

portland compilers module
(deleted output from command)

$ module load hpc/ompi114.pgf

ompi114-pgf module
************************************************** **

(deleted output from command)

And then check what are the backend compilers and the environment :

$ which mpicc
/share/apps/openmpi-1.1.4.pgf/bin/mpicc
$ which mpif90
/share/apps/openmpi-1.1.4.pgf/bin/mpif90
$ which mpif77
/share/apps/openmpi-1.1.4.pgf/bin/mpif77
$ which mpicxx
/share/apps/openmpi-1.1.4.pgf/bin/mpicxx
$ echo $MPIHOME
/share/apps/openmpi-1.1.4.pgf

$ mpicc -v
Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-java-awt=gtk --host=x86_64-redhat-linux
Thread model: posix
gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)

$ mpicxx -v
Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-java-awt=gtk --host=x86_64-redhat-linux
Thread model: posix
gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)




-- Cut Here --

#%Module1.0####################################### ##############################
##
## openmpi modulefile
##
proc ModulesHelp { } {
global version

puts stderr "\n\tompi114-pgf module"
puts stderr "\t*********************************************** *****"
puts stderr "\n\t This module sets up the following environment"
puts stderr "\t variables for openmpi 1.1.4 (gnu / portland pgf77/90):"
puts stderr "\t PATH"
puts stderr "\t LD_LIBRARY_PATH"
puts stderr "\t MPIHOME"
puts stderr "\t MANPATH"
puts stderr "\n\t Version $version\n"
puts stderr "\t*********************************************** *****\n"
}

module-whatis "Set up environment for openmpi 1.1.4 (gnu / portland pgf77/90)"

# for Tcl script use only
set version "3.1.6"

# Make sure no other hpc modulefiles are loaded before loading this module
eval set [ array get env MODULESHOME ]
source $MODULESHOME/modulefiles/hpc/utils.tcl
isModuleInListLoaded $hpc_module_list

prereq hpc/portland

setenv MPIHOME /share/apps/openmpi-1.1.4.pgf

prepend-path PATH /share/apps/openmpi-1.1.4.pgf/bin
prepend-path LD_LIBRARY_PATH /share/apps/openmpi-1.1.4.pgf/lib
prepend-path MANPATH /share/apps/openmpi-1.1.4.pgf/man

if [ module-info mode load ] {
ModulesHelp
}

-- Cut Here --

Originally posted by: mbozzore, Sun Apr 08, 2007 3:27 am

Lava is installed on your cluster and you want to submit an openmpi job. Lava is not designed to handle mpi jobs; as soon as the job is launched, you rely on mpi for signal propagation, proper cleanup ...

You can use the following procedure :

1- create the script

sudo mkdir /share/apps/lava.custom
sudo vim /share/apps/lava.custom/openmpi-mpirun
sudo chmod a+x /share/apps/lava.custom/openmpi-mpirun


2- create the symlink on the frontend

sudo ln -s /share/apps/lava.custom/openmpi-mpirun /opt/lava/6.1/linux2.6-glibc2.3-ia32e/bin


3- propagate the script

sudo cluster-fork "ln -s /share/apps/lava.custom/openmpi-mpirun /opt/lava/6.1/linux2.6-glibc2.3-ia32e/bin"


4- test the script

You can use for example the cpi example (from the mpich distribution)

bsub -o%J.out -n 8 openmpi-mpirun -np 8 ./cpi


And the script is here :

-----Cut here----
#!/bin/sh
#
# Copyright (c) 2006 Platform Computing
#
# This script is a wrapper for openmpi mpirun
# it generates the machine file based on the hosts
# given to it by Lava.
#

usage() {
cat <<USEEOF
USAGE: $0
This command is a wrapper for mpirun (openmpi). It can
only be run within Lava using bsub e.g.
bsub -n # "$0 -np # {my mpi command and args}"

The wrapper will automatically generate the
machinefile used by mpirun.

NOTE: The list of hosts cannot exceed 4KBytes.
USEEOF
}

if [ x"${LSB_JOBFILENAME}" = x -o x"${LSB_HOSTS}" = x ]; then
usage
exit -1
fi

MYARGS=$*
WORKDIR=`dirname ${LSB_JOBFILENAME}`
MACHFILE=${WORKDIR}/mpi_machines
ARGLIST=${WORKDIR}/mpi_args

# Check if mpirun is in the PATH
T=`which mpirun`
if [ $? -ne 0 ]; then
echo "Error: mpirun is not in your PATH."
exit -2
fi

echo "${MYARGS}" > ${ARGLIST}
T=`grep -- -machinefile ${ARGLIST} |wc -l`
if [ $T -gt 0 ]; then
echo "Error: Do not provide the machinefile for mpirun."
echo " It is generated automatically for you."
exit -3
fi

# Make the MPICH machine file
echo "${LSB_HOSTS}" > ${MACHFILE}.lst
tr '\/ ' '\r\n' < ${MACHFILE}.lst > ${MACHFILE}

MPIRUN=`which --skip-alias mpirun`
${MPIRUN} -machinefile ${MACHFILE} ${MYARGS}

exit $?
-----Cut here----
_________________
Mehdi Bozzo-Rey

Originally posted by: mbozzore, Sun Apr 08, 2007 4:02 am

Now, you would like to recompile openmpi using the portland group compilers, create the associated module file and run a test outside and inside lava.
I assume you already have the portland compilers module written and published on the frontend.

Here is the procedure :


1- load the portland module :

module load hpc/portland


2- export environment for the compilers :

export F77=pgf77
export FC=pgf90
export CC=pgcc
export CXX=pgCC


3- configure :

./configure --prefix=/share/apps/openmpi-1.1.4.allpgi


4- make and make install :

make ; sudo make install


5- create the associated module file and publish it on the frontend

------Cut here------
#%Module1.0####################################### ##############################
##
## openmpi modulefile
##
proc ModulesHelp { } {
global version

puts stderr "\n\tompi114-allpgi module"
puts stderr "\t*********************************************** *****"
puts stderr "\n\t This module sets up the following environment"
puts stderr "\t variables for openmpi 1.1.4 (all portland compilers):"
puts stderr "\t PATH"
puts stderr "\t LD_LIBRARY_PATH"
puts stderr "\t MPIHOME"
puts stderr "\t MANPATH"
puts stderr "\n\t Version $version\n"
puts stderr "\t*********************************************** *****\n"
}

module-whatis "Set up environment for openmpi 1.1.4 (all portland compilers)"

# for Tcl script use only
set version "3.1.6"

# Make sure no other hpc modulefiles are loaded before loading this module
eval set [ array get env MODULESHOME ]
source $MODULESHOME/modulefiles/hpc/utils.tcl
isModuleInListLoaded $hpc_module_list

prereq hpc/portland

setenv MPIHOME /share/apps/openmpi-1.1.4.allpgi

prepend-path PATH /share/apps/openmpi-1.1.4.allpgi/bin
prepend-path LD_LIBRARY_PATH /share/apps/openmpi-1.1.4.allpgi/lib:/share/apps/openmpi-1.1.4.allpgi/lib/openmpi
prepend-path MANPATH /share/apps/openmpi-1.1.4.allpgi/man

if [ module-info mode load ] {
ModulesHelp
}
------Cut here------


6- load the module file :

module load hpc/ompi114.allpgi

Check that everything should be ok :

which mpicc

/share/apps/openmpi-1.1.4.allpgi/bin/mpicc

mpicc -V

pgcc 6.2-5 64-bit target on x86-64 Linux
Copyright 1989-2000, The Portland Group, Inc. All Rights Reserved.
Copyright 2000-2006, STMicroelectronics, Inc. All Rights Reserved.


7- recompile and run a basic example (outside of lava):

7.1- Recompile the example :

mpicc -o cpi.ompiallpg cpi.c

7.2- Build the machine file

cat /etc/hosts | grep compute | awk '{print $3}' > hosts.12

7.3- Run the example

mpirun -np 12 --prefix $MPIHOME --machinefile hosts.12 ./cpi.ompiallpg
Process 1 of 12 on compute-0-1.local
Process 2 of 12 on compute-0-2.local
Process 8 of 12 on compute-0-8.local
Process 10 of 12 on compute-0-10.local
Process 0 of 12 on compute-0-0.local
Process 6 of 12 on compute-0-6.local
Process 3 of 12 on compute-0-3.local
Process 7 of 12 on compute-0-7.local
Process 9 of 12 on compute-0-9.local
Process 11 of 12 on compute-0-11.local
Process 5 of 12 on compute-0-5.local
Process 4 of 12 on compute-0-4.local
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.008779

8- run the same example using lava (and for example 44 cores)

bsub -o%J.out -n 44 openmpi-mpirun -np 44 ./cpi.ompiallpg
_________________
Mehdi Bozzo-Rey

Originally posted by: mbozzore, Sun Apr 08, 2007 4:33 am

Ok, cpi is pretty nice, but nothing is better than a good linpack to check that everything is ok.

This time we will use openmpi 1.1.4 and gcc, pgf77 as backend compilers for mpicc and mpif77.

You can find information about linpack on the top500 website : http://top500.org

They have a specific page for linpack : http://top500.org/about/linpack

And the official page for linpack is on netlib : http://www.netlib.org/benchmark/hpl/

A lot of useful information is available : algorithm used for the calculation, tuning, what size you should use for your linpack run (N)...

As I said before, linpack is _really_ good for the final bench (just after deployment) as it is memory / cpu and network intensive.

Also, in order to achieve the best result, you will have to tune (sometimes a lot) the HPL.dat and also use the best library. mp_linpack from intel comes with a subset of MKL, but you can also use the Goto BLAS library (http://www.tacc.utexas.edu/resources/software/). Another common library is ATLAS (auto tuned) but usually, it is less efficient than the Goto library, or MKL from intel or ACML from AMD, available here : http://developer.amd.com/downloads.jsp (you will have to register).
The ATLAS library is installed by default on your cluster. The ACML library from AMD is also part of the portland group. Please note this is not the latest version. It is easy to upgrade your ACML libs to the latest version : http://www.pgroup.com/support/libacml.htm

So, as soon as you have to call any BLAS / LAPACK function, this is not anymore a game. If your code is 10 or 20% slower than it should be using the appropriate libraries, you can calculate the time "lost" after one year and then (taking in account electricity / AC / room / cluster depreciation ...) convert it (money or node equivalent).

The parallel versions for BLAS/LAPACK are BLACS and ScaLapack. They are included in the intel MKL , and are also part of the portland compiler suite.

To be more precise, they are here (assuming a standard path for the portland suite) :

/opt/pgi/linux86-64/6.2/mpi/mpich/lib (mpi1 version)

/opt/pgi/linux86-64/6.2/mpi2/mpich/lib (mpi2 version)

You will have to recompile everything (I mean, BLACS/ScaLapack) for openmpi.

Another thing, for linpack (and also any mpi application with calls to this kind of libraries) is that a lot of these libraries are multithreaded so you can run one mpi instance per node and then use advantage of multithreads. This is good if the program spends most of its time performing calculations using these libraries because everything will be done in the memory (no network involved at this time). This is why usually linpack is run using one mpi instance per node (and a specific value for OMP_NUM_THREADS). It also allows you to run bigger cases (no memory overhead due to ssh or a specific interconnect (InfiniBand for example).

For the ACML library, the multithreaded version is under the "mp" directory.

Finally, the procedure is :

1- download linpack :

wget http://www.netlib.org/benchmark/hpl/hpl.tgz


2- copy a makefile from the setup directory (and rename it):

cp setup/Make.Linux_ATHLON_FBLAS Make.amd_pgi_ompi_acml


3- edit and modify the Makefile (end of this message) :

Please note that I used the ACML library (non threaded) provided by portland.


4- make the distribution :

make arch=amd_pgi_ompi_acml

A binary named xhpl and the configuration file for linpack HPL.dat will be created here (under the hpll directory):

bin/amd_pgi_ompi_acml

5- edit the HPL.dat and run a basic example


Here is the Makefile (please modify TOPdir according your environment)


------Cut here------
#
# -- High Performance Computing Linpack Benchmark (HPL)
# HPL - 1.0a - January 20, 2004
# Antoine P. Petitet
# University of Tennessee, Knoxville
# Innovative Computing Laboratories
# (C) Copyright 2000-2004 All Rights Reserved
#
# -- Copyright notice and Licensing terms:
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions, and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# 3. All advertising materials mentioning features or use of this
# software must display the following acknowledgement:
# This product includes software developed at the University of
# Tennessee, Knoxville, Innovative Computing Laboratories.
#
# 4. The name of the University, the name of the Laboratory, or the
# names of its contributors may not be used to endorse or promote
# products derived from this software without specific written
# permission.
#
# -- Disclaimer:
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
# OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# ################################################## ####################
#
# ----------------------------------------------------------------------
# - shell --------------------------------------------------------------
# ----------------------------------------------------------------------
#
SHELL = /bin/sh
#
CD = cd
CP = cp
LN_S = ln -s
MKDIR = mkdir
RM = /bin/rm -f
TOUCH = touch
#
# ----------------------------------------------------------------------
# - Platform identifier ------------------------------------------------
# ----------------------------------------------------------------------
#
ARCH = amd_pgi_ompi_acml
#
# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir = $(HOME)/compile_temp/hpl
INCdir = $(TOPdir)/include
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)
#
HPLlib = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
# MPinc tells the C compiler where to find the Message Passing library
# header files, MPlib is defined to be the name of the library to be
# used. The variable MPdir is only used for defining MPinc and MPlib.
#
MPdir = /share/apps/openmpi-1.1.4.pgf
MPinc = -I$(MPdir)/include
MPlib = $(MPdir)/lib/libmpi.a $(MPdir)/lib/openmpi/mca_io_romio.so
#$(MPdir)/lib/openmpi/mca_io_romio.so
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
# LAinc tells the C compiler where to find the Linear Algebra library
# header files, LAlib is defined to be the name of the library to be
# used. The variable LAdir is only used for defining LAinc and LAlib.
#
LAdir = /opt/pgi/linux86-64/6.2/lib/
LAinc =
LAlib = $(LAdir)/libacml.a
#
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_ : all lower case and a suffixed underscore (Suns,
# Intel, ...), [default]
# -DNoChange : all lower case (IBM RS6000),
# -DUpCase : all upper case (Cray),
# -DAdd__ : the FORTRAN compiler in use is f2c.
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int : Fortran 77 INTEGER is a C int, [default]
# -DF77_INTEGER=long : Fortran 77 INTEGER is a C long,
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short.
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle : The string address is passed at the string loca-
# tion on the stack, and the string length is then
# passed as an F77_INTEGER after all explicit
# stack arguments, [default]
# -DStringStructPtr : The address of a structure is passed by a
# Fortran 77 string, and the structure is of the
# form: struct {char *cp; F77_INTEGER len;},
# -DStringStructVal : A structure is passed by value for each Fortran
# 77 string, and the structure is of the form:
# struct {char *cp; F77_INTEGER len;},
# -DStringCrayStyle : Special option for Cray machines, which uses
# Cray fcd (fortran character descriptor) for
# interoperation.
#
F2CDEFS =
#-DAdd__ -DF77_INTEGER=int -DStringSunStyle
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
# -DHPL_COPY_L force the copy of the panel L before bcast;
# -DHPL_CALL_CBLAS call the cblas interface;
# -DHPL_CALL_VSIPL call the vsip library;
# -DHPL_DETAILED_TIMING enable detailed timers;
#
# By default HPL will:
# *) not copy L before broadcast,
# *) call the BLAS Fortran 77 interface,
# *) not display detailed timing information.
#
HPL_OPTS =
#
# ----------------------------------------------------------------------
#
HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC = mpicc
CCNOOPT = $(HPL_DEFS)
CCFLAGS = $(HPL_DEFS) -O3
#-fomit-frame-pointer -O3 -funroll-loops -W -Wall
#
LINKER = mpif77
LINKFLAGS = $(CCFLAGS) -Mnomain
#
ARCHIVER = ar
ARFLAGS = r
RANLIB = echo
#
# ----------------------------------------------------------------------
------Cut here------
_________________
Mehdi Bozzo-Rey