Blackbear General User's Guide

1. Architecture and System Configurations
2. Allocations and Gaining Access
1. AHPCC Request  Form
2. Encryption software requirement
3. Connecting
3. Account Management
1. Understanding your account
2. Adding users
3. Charging algorithm
4. File Systems and Storage
1. Local file space
2. System file space
3. Archival storage options
5. Compiling and Linking
1. Available compilers
2. Compiling a job
3. Programming  Tools
4. Programming  for MPI
5. Mathematic  and Scientific Libraries
6. Jobs
1. Selecting MPI Libraries
2. Running MPI programs
3. Using the Batch Scheduler (PBS)
4. Submitting Jobs to PBS
5. Running Interactive Jobs
7. Debugging and Optimization
1. Message passing and interconnects
2. Debuggers
3. Performance analysis tools
8. Resources
1. Getting help
2. Training
3. References

Access
The Black Bear Supercluster is accessed through one machine. This  front-end system, named blackbear.ahpcc.unm.edu,  is intended for code development and submitting jobs to the Black  Bear compute nodes. 

Logging in:
Use a secure login utility such as secure shell (ssh) to login  to the Black Bear systems. 
ssh blackbear.ahpcc.unm.edu
ssh bb.ahpcc.unm.edu
ssh osonegro.ahpcc.unm.edu

Your Account
You can change your password using the "yppasswd" command. You  will be prompted for your current password in order to change  it. 
yppasswd

Changing your default shell is similar to changing your password.  Use the "ypchsh" command and you will be prompted for your password  and then the new shell you wish to use (including full path  - e.g., /usr/local/bin/tcsh) 
ypchsh

C gcc pgcc C++ g++ pgCC Fortran77 g77 pgf77 Fortran90/95 pgf90 f95 High. Perf. Fortran (HPF) pghpf

• setup_env.bb (script to setup your environment to run on Black Bear)
• test.pbs (sample  submit file) 
• rhosts (contains the hostnames of the Black Bear nodes)
• cshrc_bb (for  csh and tcsh users)
• bashrc_bb (for ksh and bash users)

Included in the Portland Group suite are two useful programming  tools. The source code debugger and performance profiler is documented  in the on-line PGI manual. 

Programming Tools Source Code Debugger pgdbg Performance Profiler pgprof

 Compile scripts are available for compiling MPI code. These scripts  automatically link in all MPI libraries and include files. 

Compiling for MPI C mpicc C++ mpiCC Fortran77 mpif77 Fortran90 mpif90

Mathematic and Scientific  Libraries Linear Algebra BLAS  (Basic Linear Algebra Subroutines) LAPACK /usr/lib/libblas.a /usr/pgi/linux86/lib/libblas.a*  * (for PGI compilers only) /usr/lib/liblapack.a /usr/pgi/linux86/lib/liblapack.a*  * (for PGI compilers only) FFT FFTW /usr/local/lib/libfftw_mpi.a /usr/local/lib/librfftw_mpi.a

There are four MPI libraries on the Black Bear Supercluster, two  using Myrinet (for PGI and GNU compilers) and two using Ethernet  (for PGI and GNU). Select a library by setting the MPIHOME environment  variable. Refer to your .cshrc file for examples. 

Use the "which" command to verify your compile library. For  example:

% source $HOME/BLACKBEAR/cshrc_bb

% which mpicc
/usr/parallel/mpich-tcp.gnu/bin/mpicc
shows that you are using Ethernet with GNU compilers. And, 

% which mpicc
/usr/parallel/mpich-gm.pgi/bin/mpicc
shows that you are using Myrinet with PGI compilers. 

Compiling

Compile and link C code with MPI libraries 
% mpicc -o exec [options]  filename(s).c

Compile and link C++ code with MPI libraries 
% mpiCC -o exec [options] filename(s).C

Compile and link Fortran77 code with MPI libraries 
% mpif77 -o exec [options] filename(s).f

Compile and link Fortran90 code with MPI libraries 
% mpif90 -o exec [options] filename(s).f90

MPI can only be run on the Black Bear compute nodes through PBS.  You can use the "mpirun" script to run an MPI job. The script  has the following syntax 

mpirun -np n -machinefile $PBS_NODESFILE full_path/executable

-np n  - Number of nodes to start  program on 
full_path/executable  - Executable  including full path name

Using the Batch Scheduler  (PBS)

Black Bear uses the Portable Batch System (PBS) to manage parallel  job scheduling. Users can run batch or interactive jobs on the  Black Bear Supercluster through PBS. Batch jobs are described  in command files which are submitted to PBS for execution. 

PBS Queue Limits Maximum Job Length*: 6 hours

* Job length can be extended for special requests.
Please contact help@ahpcc.unm.edu

PBS Command Files

The PBS job command file defines the number of nodes you are requesting  and contains the shell commands you wish to be executed. One or  two processes per node can be started. Some samples are listed  below for a job to run the executable complete_path/test_progon N nodes 
(for a user using a (t)csh style shell): 

For Myrinet jobs (one process per node)

#PBS -l nodes=N
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users 
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf $PBS_NODEFILE
mpirun -np N -machinefile $PBS_NODEFILE <full path to your executable> For MPI jobs to run using Myrinet, the "gmpiconf" command must  be used to create a configuration file ($HOME/.gmpi/conf). To  run two processes per node, use the gmpiconf2 command  and double the number of processes (-np 2*n) for the  mpirun command as shown below: 

For Myrinet jobs (two processes per node)

gmpiconf2 $PBS_NODEFILE
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_progFor Ethernet jobs (one process per node)

mpirun -np N -machinefile $PBS_NODEFILE complete_path/test_progFor Ethernet jobs (two processes per node)

#PBS -l nodes=N
mpirun -np 2*N -machinefile $PBS_NODEFILE complete_path/test_prog

The "#PBS -l nodes=N" line defines how many nodes are being  requested. The PBS environment variables which are used in the  script above are defined below. These variables are set when PBS  allocates nodes for a user. 

Example script (sample.cmd): 32-node Myrinet using  two processes per node: 
#PBS -l nodes=16
# source to get environment variables
# cshrc_bb - for (t)csh users
# bashrc_bb - for (ba)(k)sh users 
# uncomment the proper line based on your shell
source $HOME/BLACKBEAR/cshrc_bb
#. $HOME/BLACKBEAR/bashrc_bb
gmpiconf2 $PBS_NODEFILE
mpirun -np 32 /home/cashmont/test.exe

PBS Environment Variables

• PBS_O_WORKDIR - user's current directory when "qsub"  command was issued
• PBS_NODEFILE - file containing the list of nodes  the user was allocated

Submiting jobs to PBS
qsub [flags] (script_name) 

Submit a batch job listed in scriptname.pbs file 
qsub scriptname.pbs 

Displaying the queue contents and status 
qstat [flags]

i.e., qstat  -s (displays status and elapsed time of job) 

Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
2534.bb         nestor12 rr       STDIN       23483  16  --    --  06:00 R 01:04
   Job started on Fri Jul 09 at 15:53
2554.bb         corona   rr       STDIN       19704   2  --    --  06:00 R 00:44
   Job started on Fri Jul 09 at 16:14
2557.bb         corona   rr       STDIN       22012   2  --    --  06:00 R 00:42
   Job started on Fri Jul 09 at 16:15
2559.bb         nestor12 rr       alltoall.p    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available
2560.bb         nestor12 rr       alltoall12    --   64  --    --  06:00 Q   --
   Not Running: Not enough of the right type of nodes are available

• Job ID: ID for users' job in queue
• Username: job owner
• Jobname: name of script submitted to PBS or STDIN  if user is in an interactive session
• NDS: number of nodes allocated
• Time: maximum wall clock run time 
• S (State): R - Running; Q - Waiting in queue; E  - Job is in error state
• Time (last column): job's elapsed time

Remove job from queue
qdel (job_id)

where job_id can be obtained from the queue listing using "qstat". 

Output and Error files

Standard error and output are contained in two files with the  names: 

      Jobname.eJob_ID
      Jobname.oJob_ID

using the Jobname and Job_ID from your job in the queue. 

Running Interactive  Jobs

You can initiate an interactive session on Black Bear nodes through  PBS. This is the only case where a user has direct access to a  compute node. 

Initiate an interactive session using n nodes 
qsub -I -l nodes=n

Once an interactive session is granted, you are placed in your  home directory on one of the nodes you were allocated. Execute  the code as described in MPIRUN section. 

Approximately 3 GB of scratch space is available on each Black  Bear compute node in/scratch. 
/scratch is for TEMPORARY storage only - All files in /scratch  will be deleted when the job has finished.

Please contact our Webmaster with questions or comments.
© Copyright 2001 Albuquerque High Performance Computing Center.  All rights reserved.