Access

The Linux Supercluster is accessed through one machine. This front-end system, named roadrunner.alliance.unm.edu, is intended for code development and submitting jobs to the Roadrunner compute nodes.

Logging in:
Use a secure login utility such as secure shell (ssh) to login to the Roadrunner systems.
ssh roadrunner.alliance.unm.edu

FTP:
Use a secure file transfer utility such as secure copy (scp) to transfer files to the Roadrunner systems.
scp file rr_user@roadrunner.alliance.unm.edu:dest_file        -> uploads a local file to Roadrunner

Users can FTP their files from Roadrunner.
 

Your Account

You can change your password using the "yppasswd" command. You will be prompted for your current password in order to change it.
yppasswd

Changing your default shell is similar to changing your password. Use the "ypchsh" command and you will be prompted for your password and then the new shell you wish to use (including full path - e.g., /bin/tcsh)
ypchsh
 

Environment

Many environment variables needed for compilers and libraries are defined in your .cshrc. Users can copy the current default files from /etc/skel.

• .cshrc (for Csh and Tcsh users)

Software

Included in the Portland Group suite are two useful programming tools. The source code debugger and performance profiler is documented in the on-line PGI manual.

Compile scripts are available for compiling MPI code. These scripts automatically link in all MPI libraries and include files.

Running Jobs

Selecting MPI Libraries

There are four MPI libraries on the Roadrunner Supercluster, two using Myrinet (for PGI and GNU compilers) and two using Ethernet (for PGI and GNU). Select a library by setting the MPIHOME environment variable. Refer to your .cshrc file for examples.

Use the "which" command to verify your compile library. For example,

% which mpicc
/usr/parallel/mpich-tcp.gnu/bin/mpicc
shows that you are using Ethernet with GNU compilers. And,

% which mpicc
/usr/parallel/mpich-gm.pgi/bin/mpicc
shows that you are using Myrinet with PGI compilers.

Compiling

Compile and link C code with MPI libraries
% mpicc -o exec [options] filename(s).c

Compile and link C++ code with MPI libraries
% mpiCC -o exec [options] filename(s).C

Compile and link Fortran77 code with MPI libraries
% mpif77 -o exec [options] filename(s).f

Compile and link Fortran90 code with MPI libraries
% mpif90 -o exec [options] filename(s).f90

Running MPI jobs

MPI can only be run on the Roadrunner compute nodes through PBS. You can use the "mpirun" script to run an MPI job. The script has the following syntax

mpirun -np n full_path/executable

-np n  - Number of nodes to start program on
full_path/executable  - Executable including full path name

Using the Batch Scheduler (PBS)

Roadrunner uses the Portable Batch System (PBS) to manage parallel job scheduling. Users can run batch or interactive jobs on the Roadrunner Supercluster through PBS. Batch jobs are described in command files which are submitted to PBS for execution.

* Job length can be extended for special requests.
Please contact help@alliance.unm.edu

PBS Command Files

The PBS job command file defines the number of nodes you are requesting and contains the shell commands you wish to be executed. One or two processes per node can be started. Some samples are listed below for a job to run the executable complete_path/test_prog on N nodes: 

For Myrinet jobs (one process per node)

#PBS -l nodes=N
gmpiconf $PBS_NODEFILE
mpirun complete_path/test_prog

For MPI jobs to run using Myrinet, the "gmpiconf" command must be used to create a configuration file ($HOME/.gmpi/conf). To run two processes per node, use the gmpiconf2 command and double the number of processes (-np 2*n) for the mpirun command as shown below:

For Myrinet jobs (two processes per node)

#PBS -l nodes=N
gmpiconf2 $PBS_NODEFILE
mpirun -np 2*N complete_path/test_prog

For Ethernet jobs (one process per node)

#PBS -l nodes=N
mpirun complete_path/test_prog

For Ethernet jobs (two processes per node)

#PBS -l nodes=N
mpirun -np 2*N complete_path/test_prog

The "#PBS -l nodes=N" line defines how many nodes are being requested. The PBS environment variables which are used in the script above are defined below. These variables are set when PBS allocates nodes for a user.

Example script (sample.cmd): 32-node Myrinet using two processes per node:

#PBS -l nodes=32
gmpiconf2 $PBS_NODEFILE
mpirun -np 64 /users/wilson/solve_run.exe

PBS Environment Variables

• PBS_O_WORKDIR - user's current directory when "qsub" command was issued
• PBS_NODEFILE - file containing the list of nodes the user was allocated

Submiting jobs to PBS
qsub [flags] (script_name)

Submit a batch job listed in scriptname.pbs file
qsub scriptname.pbs

Displaying the queue contents and status
qstat [flags]

i.e., qstat -s (displays status and elapsed time of job)

Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2534.rr         nestor12 rr       STDIN       23483  16  --    --  06:00 R 01:04
   Job started on Fri Jul 09 at 15:53
2554.rr         corona   rr       STDIN       19704   2  --    --  06:00 R 00:44
   Job started on Fri Jul 09 at 16:14
2557.rr         corona   rr       STDIN       22012   2  --    --  06:00 R 00:42
   Job started on Fri Jul 09 at 16:15
2559.rr         nestor12 rr       alltoall.p    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available
2560.rr         nestor12 rr       alltoall12    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available

• Job ID: ID for users' job in queue
• Username: job owner
• Jobname: name of script submitted to PBS or STDIN if user is in an interactive session
• NDS: number of nodes allocated
• Time: maximum wall clock run time
• S (State): R - Running; Q - Waiting in queue; E - Job is in error state
• Time (last column): job's elapsed time

Remove job from queue
qdel (job_id)

where job_id can be obtained from the queue listing using "qstat".

Output and Error files

Standard error and output are contained in two files with the names:

      Jobname.eJob_ID
      Jobname.oJob_ID

using the Jobname and Job_ID from your job in the queue.

Running Interactive Jobs

You can initiate an interactive session on Roadrunner nodes through PBS. This is the only case where a user has direct access to a compute node.

Initiate an interactive session using n nodes
qsub -I -l nodes=n

Once an interactive session is granted, you are placed in your home directory on one of the nodes you were allocated. Execute the code as described in MPIRUN section.

Storage

Approximately 4 GB of scratch space is available on each Roadrunner compute node in /localscratch.