---Description:
This is an entirely new ch_gm device written for the MPICH 1.2.1 release by Patrick Geoffray at
Myricom, in close coordination with the systems group at AHPCC. It is architecturally different from
previous implementations of GM devices for MPICH, and is believed to be both faster and more reliable.

---Locations and Compilers:
Currently, only Gnu and PGI versions of this MPICH are available.

The standard, most-tested version of MPICH-GM will always be installed in:
/usr/parallel/ll/mpich-gm.<compiler>

The following information is for those wanting to use specific versions of the software.

For each of the compilers, there is both a highly optimized version and a version with debugging symbols and no optimization.

All versions are in /usr/parallel/ll/<version>

GNU Optimized:
<version> = mpich-gm.gnu-4
GNU Debugging:
<version> = mpich-gm.gnu-4g

PGI Optimized:
<version> = mpich-gm.pgi-4
PGI Debugging:
<version> = mpich-gm.pgi-4g

For example, to use the optimized pgcc version, set:

MPIHOME=/usr/parallel/ll/mpich-gm.pgi-4
PATH=$MPIHOME/bin:$PATH
CC=$MPIHOME/bin/mpicc

---Running Programs:
There are several specialized options for the ch_gm device which are not accessible through mpirun; for this reason, we recommend running with:

{gmpiconf or gmpiconf2}
$MPIHOME/bin/mpirun.ch_gm <args> <program> <program-args>

Important arguments to the ch_gm device follow, along with information on their functionality.

GMPI_EAGER_SIZE=<number of bytes>
This sets both the threshold for small messages, and the buffer size for messages.
There are two ways that GM sends packets to another node - Eager(small packets) and Rendez-vous(large packets). The small packet mode simply puts the packet on the card and sends it, w/o any negotiation - the receiving end must buffer the packet if it is not expected. The large packet mode first buffers the packet, then queries the receiving end - the receiving end then allocates a buffer for the packet, and signals the sender to go ahead. By default, the threshold for large packets is set at 16K - if your application send many small packets, it is more efficient to set the threshold lower, so that MPICH_GM allocates smaller buffers for these packets.

--gm-recv <mode> {blocking, polling, hybrid}
This selects the mode used for all MPI calls which receive messages. Polling is the default mode, and is the most reliable. You would only want to change this option if you are running with more than one process per CPU, in which case the blocking mode will allow other processes to take CPU time while one waits for communication.

-np <number of processors> (This is unneccesary with gmpiconf)

--gm-v (Turns on verbose operation)

Note: Because we are working so closely with Myricom on this release, bugs in the device code are being addressed with unusual speed. Therefore we request that you copy all MPI or GM errors to the help desk, along with your MPIHOME and PBS_NODEFILE.

---- gmpiconf usage

Before running an mpich-gm program, you must run the "gmpiconf" program to prepare a process group file for mpich. The standard versions of gmpiconf are:

/usr/local/bin/gmpiconf for one processor per node
/usr/local/bin/gmpicnf2 for 2 processors per node

However, there may be situations, for example when processes spend a lot of time waiting on communication, in which you would want to run more than one process per CPU. In this case, you can use:

/usr/local/bin/gmpiconf3 for 3 processes per node
/usr/local/bin/gmpiconf4 for 4 processes per node

In any situation, the command:

gmpiconf[n] $PBS_NODEFILE

Will work within PBS to prepare your process group file.

**********************************************************************
* Please report errors in this document to: *
* tensile@alliance.unm.edu *
**********************************************************************