Reaching Others University at Buffalo - The State University of New York
Skip to Content

OPEN MPI

Open MPI is an open source implementation of MPI (v2.x).

Category:  MPI

Compiler "wrappers" around the compiler suite (mpif90, mpicc, mpicxx).

Usage Notes:

Show the software versions:  module avail openmpi

Loading the module will set the path and any necessary variables:  module load openmpi/version

 

Java Support

Version 1.7.4 of OpenMPI has support for java-based applications.

An example that includes a SLURM sbatch script (i.e. slurm-java-ompi)  may be found at:

/util/openmpi/java-example

Job Startup

At UB CCR Open MPI has been tested using mpirun to launch tasks under SLURM. An example application and sbatch script may be found at:

/util/openmpi/example

To work properly in the SLURM environment, a few preparatory steps are required to ensure tasks are assigned to correct processors.  

The following sample code for launching OpenMPI tasks is written in bash:


RANKFILE=ranks.$$
HOSTFILE=nodes.$$
uid=`id -u`
jid=$SLURM_JOB_ID
nodes=`nodeset -e $SLURM_NODELIST`
NP=${SLURM_NPROCS}

# trigger creation of cpuset information and save to working dir
CGROUP_INFO=/cgroup/cpuset/slurm/uid_${uid}/job_${jid}/cpuset.cpus
srun bash -c "cat ${CGROUP_INFO} > cpus.\`hostname\`.$SLURM_JOB_ID"

# assemble HOSTFILE and RANKFILE
rank=0
for i in ${nodes}; do
  # extract space-separated list of assigned cpus
  cpus=`cat cpus.${i}.${SLURM_JOB_ID}`
  cpus=`nodeset -Re $cpus`
  # add cpu assignments to the rank file
  for j in ${cpus}; do
    echo "rank ${rank}=$i slot=$j" >> $RANKFILE
    echo "$i" >> $HOSTFILE
    rank=`expr $rank + 1`
    if [ "$rank" == "$NP" ]; then
      break;
    fi
  done
  if [ "$rank" == "$NP" ]; then
    break;
  fi
done

# use ssh as the launcher, the rankfile will enforce the cpusets
export OMPI_MCA_plm=rsh

# Now launch the code using mpirun
mpirun -hostfile $HOSTFILE -rf $RANKFILE --prefix $OMPI -n $NP ./a.out