CARC web archive · Archived copy of ahpcc.unm.edu/Systems/Documentation/LinuxSupercluster/RefCard/index.html as captured by the Internet Archive on 28 January 2001. Links to pages that were not saved here go to the Wayback Machine. Wayback copy · All pages in this snapshot · Web archive index
Linux Supercluster Reference Card
News
Accounts-Allocations
All Machines
Help Desk
Documentation
Training
Research
Downloads
Ask Users

Linux Supercluster Reference Card

The University of New Mexico & National Computational Science Alliance

This page includes:

For assistance, please contact the Help Desk at help@ahpcc.unm.edu or (505) 277-8249 (Monday through Friday, 9 a.m. to 5 p.m. Mountain Time)

For additional documentation, please see: http://www.ncsa.uiuc.edu/SCD/Info/MoreHelp.html

 

Access

Depending on your allocation, login to the appropriate Linux Supercluster using secure-shell (ssh):

% ssh roadrunner.alliance.unm.edu

% ssh loslobos.alliance.unm.edu

 

Transfer files to the Linux Superclusters from your home machine using secure copy protocol (scp):

% scp <file> <user>@<Linux_Superluster>.alliance.unm.edu:dest

 

Your Account

Change password on the Linux Superclusters:

% yppasswd

 

Change default shell on the Linux Superclusters:

% ypchsh

 

Mail

You need to forward your mail to your local system. To forward mail, create or modify the .forward file with:

<username@host.domain>

 

Compilers

If you are interested in debugging your Fortran program, try the NAG compiler first. If you are interested in performance, try all compilers since each are optimized differently. MPI C++ is only supported on Ethernet and may work on Myrinet.

C

% gcc –o exec [options] filename(s).c

GNU

% pgcc –o exec [options] filename(s).c

PGI

C++

% g++ –o exec [options] filename(s).C

GNU

% pgCC –o exec [options] filename(s).C

PGI

Fortran77

% f77 –o exec [options] filename(s).f

GNU

% pgf77 –o exec [options] filename(s).f

PGI

Fortran90/95

% f90 –o exec [options] filename(s).f90

VAST

% pgf90 –o exec [options] filename(s).f90

PGI

% pghpf –o exec [options] filename(s).hpf

PGI

% f95 –o exec [options] filename(s).f95

NAG

% f90 –YEXT_NAMES=LCS –YEXT_SFX=_ –o exec [options] filename(s).hpf

Absoft

 

Programming Tools

Source code debuggers

% gdb

GNU

% pgdbg

PGI

% xxgdb

GNU

Performance profiler

% pgprof

PGI

TotalView MPI Debugger

% totalview

Etnus

 

 

Selecting a MPI Library

There are 16 MPI libraries on the Linux Superclusters, a set of four using Myrinet and four using Ethernet for each Linux Supercluster. Select a library by setting the MPIHOME environment variable. You can set this variable by removing the comment sentinel (#) from the beginning of the line specifying the library that you want to use. Be sure that the beginning of the lines for the other libraries start with the comment sentinel (#). Refer to your .cshrc file for examples. Use the "which" command to verify your compile library.

For example:

% which mpicc

/usr/parallel/ll/mpich-gm.pgi/bin/mpicc

shows that you are using GM with PGI compilers.

 

 

Compiling with MPI

C

% mpicc –o exec [options] filename(s).c

C++

% mpiCC –o exec [options] filename(s).C

Fortran77

% mpif77 –o exec [options] filename(s).f

Fortran90

% mpif90 -o exec [options] filename(s).f90

 

Job Managers

PBS and LRM/Maui support two types of modes for running your job – batch and interactive. With interactive, a session is automatically started on one of the compute nodes assigned to you through the Job Manager. In the following scripts, ppn = process(es) per node and n = number of nodes requested.

Portable Batch System (PBS)

PBS Batch Mode Sample Scripts:

 

Myrinet

Ethernet

1 ppn

#PBS –l nodes=n

gmpiconf $PBS_NODEFILE

mpirun -np n .exec

#PBS –l nodes=n

mpirun -np n .exec

 

2 ppn

#PBS –l nodes=n

gmpiconf2 $PBS_NODEFILE

mpirun –np 2*n <prog>.exec

#PBS –l nodes=n

mpirun –np 2*n <prog>.exec

Standard output and error files will be written to your working directory. To submit the script to PBS, use the following command:

% qsub <pbs_script>

 

PBS Interactive Mode Sample Session:
Request an interactive session with n nodes:

% qsub –I –l nodes=n <pbs_script>

 

Change directory to:

$PBS_O_WORKDIR (your working directory).

 

Run your program as follows:
Myrinet Ethernet
1 ppn If Myrinet 1 ppn:

% gmpiconf $PBS_NODEFILE

% mpirun -np n .exec

If Ethernet 1 ppn

% mpirun -np n .exec

 
2 ppn

If Myrinet 2 ppn:

% gmpiconf2 $PBS_NODEFILE

% mpirun –np 2*n <prog>.exec

If Ethernet 2 ppn

% mpirun –np 2*n <prog>.exec

Release your nodes when finished by typing:

% exit

 

Display contents and status of queue:

% qstat

 

Remove job from queue:

% qdel <job_id>

 

Linux Resource Manager (LRM) and Maui Scheduler - LosLobos
If you want to run an executable only, compose a script like the following table:

LRM Batch Mode Sample Scripts:

 

Myrinet

Ethernet

1 ppn

WCLimit == <WCLimit>

JobType == mpi.ch_gm

Tasks == n

Nodes == n

TaskPerNode == 1

Output == <script>.out

Err == <script>.err

Log == <script>.log

Exec == <prog>.exec

WCLimit == <WCLimit>

JobType == mpi.ch_p4

Tasks == n

Nodes == n

TaskPerNode == 1

Output == <script>.out

Err == <script>.err

Log == <script>.log

Exec == <prog>.exec

2 ppn

WCLimit == <WCLimit>

JobType == mpi.ch_gm

Tasks == 2*n

Nodes == n

TaskPerNode == 2

Output == <script>.out

Err == <script>.err

Log == <script>.log

Exec == <prog>.exec

WCLimit == <WCLimit>

JobType == mpi.ch_p4

Tasks == 2*n

Nodes == n

TaskPerNode == 2

Output == <script>.out

Err == <script>.err

Log == <script>.log

Exec == <prog>.exec

*WCLimit (Wall Clock Limit) is in seconds.

Standard output and error files will be written to your working directory. To submit the script to LRM/Maui, use the following command:

% submitjob <lrm_script>

 

*If you want to execute your program inside of a secondary script (that would allow running of multiple executables, setup other environmental variables, etc.), in your LRM script use the following:

JobType == mpi

Exec == <secondary_lrm_script>

 

To submit your job:

% submitjob <lrm_script>

 

Sample secondary LRM script:

#!/bin/csh

# gmpiconf ${ NODE_FILE} # uncomment this line for 1 ppn

gmpiconf2 ${ NODE_FILE} # 2 ppn

mpirun –np ${ WIKI_TASKCOUNT} -machinefile ${ NODE_FILE} <prog>.exe

 

 

LRM/Maui Interactive Mode Sample Session:
Request an interactive session with n nodes:

% submitjob –I nodes = n

 

Run your program as follows:

 

Myrinet

Ethernet

1 ppn

% gmpiconf $NODEFILE
% mpirun -np n .exec

% mpirun -np n .exec

2 ppn

% gmpiconf2 $NODEFILE
% mpirun –np 2*n <prog>.exec

% mpirun –np 2*n <prog>.exec

 

Release your nodes when finished by typing:

% exit

*You can submit jobs to LRM with a LRM script or with a PBS script and specify a PBS job type. Submit PBS scripts to LRM with the following command:

% submitjob –P <pbs_script>

 

Display contents and status of queue:

% showq

 

Remove job from queue:

% canceljob <job_id>
The University of New Mexico
Albuquerque High Performance Computing Center
Galles Building - 1601 Central Avenue, NE
Albuquerque, NM 87131UNM
Phone # 505.277.8249000000Fax # 505.277.8235
Last Modified: December 1, 2000