AHPCC Computing Systems

CARC web archive · Archived copy of alliance.unm.edu/Systems/Documentation/LinuxSupercluster/RefCard/index.html as captured by the Internet Archive on 4 March 2001. Links to pages that were not saved here go to the Wayback Machine. Wayback copy · All pages in this snapshot · Web archive index

Linux Supercluster Reference Card

Linux Supercluster Reference Card

The University of New Mexico & National Computational Science Alliance

This page includes:

Access
Mail
Compilers
Your Account
Programming Tools
Selecting a MPI Library
Compiling with MPI
Job Managers

For assistance, please contact the Help Desk at help@ahpcc.unm.edu or (505) 277-8249 (Monday through Friday, 9 a.m. to 5 p.m. Mountain Time)

For additional documentation, please see: http://www.ncsa.uiuc.edu/SCD/Info/MoreHelp.html

Access

Depending on your allocation, login to the appropriate Linux Supercluster using secure-shell (ssh):

% ssh roadrunner.alliance.unm.edu

% ssh loslobos.alliance.unm.edu

Transfer files to the Linux Superclusters from your home machine using secure copy protocol (scp):

% scp <file> <user>@<Linux_Superluster>.alliance.unm.edu:dest

Your Account

Change password on the Linux Superclusters:

% yppasswd

Change default shell on the Linux Superclusters:

% ypchsh

Mail

You need to forward your mail to your local system. To forward mail, create or modify the .forward file with:

<username@host.domain>

Compilers

If you are interested in debugging your Fortran program, try the NAG compiler first. If you are interested in performance, try all compilers since each are optimized differently. MPI C++ is only supported on Ethernet and may work on Myrinet.

C	% gcc –o exec [options] filename(s).c	GNU
C	% pgcc –o exec [options] filename(s).c	PGI
C++	% g++ –o exec [options] filename(s).C	GNU
C++	% pgCC –o exec [options] filename(s).C	PGI
Fortran77	% f77 –o exec [options] filename(s).f	GNU
Fortran77	% pgf77 –o exec [options] filename(s).f	PGI
Fortran90/95	% f90 –o exec [options] filename(s).f90	VAST
	% pgf90 –o exec [options] filename(s).f90	PGI
	% pghpf –o exec [options] filename(s).hpf	PGI
	% f95 –o exec [options] filename(s).f95	NAG
	% f90 –YEXT_NAMES=LCS –YEXT_SFX=_ –o exec [options] filename(s).hpf	Absoft

Programming Tools

Source code debuggers	% gdb	GNU
	% pgdbg	PGI
	% xxgdb	GNU
Performance profiler	% pgprof	PGI
TotalView MPI Debugger	% totalview	Etnus

Selecting a MPI Library

There are 16 MPI libraries on the Linux Superclusters, a set of four using Myrinet and four using Ethernet for each Linux Supercluster. Select a library by setting the MPIHOME environment variable. You can set this variable by removing the comment sentinel (#) from the beginning of the line specifying the library that you want to use. Be sure that the beginning of the lines for the other libraries start with the comment sentinel (#). Refer to your .cshrc file for examples. Use the "which" command to verify your compile library.

For example:

% which mpicc

/usr/parallel/ll/mpich-gm.pgi/bin/mpicc

shows that you are using GM with PGI compilers.

Compiling with MPI

C	% mpicc –o exec [options] filename(s).c
C++	% mpiCC –o exec [options] filename(s).C
Fortran77	% mpif77 –o exec [options] filename(s).f
Fortran90	% mpif90 -o exec [options] filename(s).f90

Job Managers

PBS and LRM/Maui support two types of modes for running your job – batch and interactive. With interactive, a session is automatically started on one of the compute nodes assigned to you through the Job Manager. In the following scripts, ppn = process(es) per node and n = number of nodes requested.

Portable Batch System (PBS)

PBS Batch Mode Sample Scripts:
	Myrinet	Ethernet
1 ppn	#PBS –l nodes=n gmpiconf $PBS_NODEFILE mpirun -np n .exec	#PBS –l nodes=n mpirun -np n .exec
2 ppn	#PBS –l nodes=n gmpiconf2 $PBS_NODEFILE mpirun –np 2n* <prog>.exec	#PBS –l nodes=n mpirun –np 2n* <prog>.exec

Standard output and error files will be written to your working directory. To submit the script to PBS, use the following command:

% qsub <pbs_script>

PBS Interactive Mode Sample Session:
Request an interactive session with n nodes:

% qsub –I –l nodes=n <pbs_script>

Change directory to:

$PBS_O_WORKDIR (your working directory).

Run your program as follows:

Myrinet Ethernet

1 ppn If Myrinet 1 ppn:

% gmpiconf $PBS_NODEFILE

% mpirun -np n .exec

If Ethernet 1 ppn

% mpirun -np n .exec

2 ppn
If Myrinet 2 ppn:

% gmpiconf2 $PBS_NODEFILE

% mpirun –np 2*n <prog>.exec

If Ethernet 2 ppn

% mpirun –np 2*n <prog>.exec

Release your nodes when finished by typing:

% exit

Display contents and status of queue:

% qstat

Remove job from queue:

% qdel <job_id>

Linux Resource Manager (LRM) and Maui Scheduler - LosLobos
If you want to run an executable only, compose a script like the following table:

LRM Batch Mode Sample Scripts:
	Myrinet	Ethernet
1 ppn	WCLimit == <WCLimit> JobType == "mpi.ch_gm" Tasks == n Nodes == n TaskPerNode == 1 Output == "<script>.out" Err == "<script>.err" Log == "<script>.log" Exec == "<prog>.exec"	WCLimit == <WCLimit> JobType == "mpi.ch_p4" Tasks == n Nodes == n TaskPerNode == 1 Output == "<script>.out" Err == "<script>.err" Log == "<script>.log" Exec == "<prog>.exec"
2 ppn	WCLimit == <WCLimit> JobType == "mpi.ch_gm" Tasks == 2n* Nodes == n TaskPerNode == 2 Output == "<script>.out" Err == "<script>.err" Log == "<script>.log" Exec == "<prog>.exec"	WCLimit == <WCLimit> JobType == "mpi.ch_p4" Tasks == 2n* Nodes == n TaskPerNode == 2 Output == "<script>.out" Err == "<script>.err" Log == "<script>.log" Exec == "<prog>.exec"

*WCLimit (Wall Clock Limit) is in seconds.

Standard output and error files will be written to your working directory. To submit the script to LRM/Maui, use the following command:

% submitjob <lrm_script>

*If you want to execute your program inside of a secondary script (that would allow running of multiple executables, setup other environmental variables, etc.), in your LRM script use the following:

JobType == "mpi"

Exec == <secondary_lrm_script>

To submit your job:

% submitjob <lrm_script>

Sample secondary LRM script:

#!/bin/csh

# gmpiconf ${ NODE_FILE} # uncomment this line for 1 ppn

gmpiconf2 ${ NODE_FILE} # 2 ppn

mpirun –np ${ WIKI_TASKCOUNT} -machinefile ${ NODE_FILE} <prog>.exe

LRM/Maui Interactive Mode Sample Session:
Request an interactive session with n nodes:

% submitjob –I nodes = n

Run your program as follows:

	Myrinet	Ethernet
1 ppn	% gmpiconf $NODE_FILE % mpirun -np n .exec	% mpirun -np n .exec
2 ppn	% gmpiconf2 $NODE_FILE % mpirun –np 2n* <prog>.exec	% mpirun –np 2n* <prog>.exec

Release your nodes when finished by typing:

% exit

*You can submit jobs to LRM with a LRM script or with a PBS script and specify a PBS job type. Submit PBS scripts to LRM with the following command:

% submitjob –P <pbs_script>

Display contents and status of queue:

% showq

Remove job from queue:

% canceljob <job_id>

The University of New Mexico
Albuquerque High Performance Computing Center
Galles Building - 1601 Central Avenue, NE
Albuquerque, NM 87131
Phone # 505.277.8249000000Fax # 505.277.8235

Last Modified: December 1, 2000