CARC web archive · Archived copy of ahpcc.unm.edu/Workshop/System-subroutine/pvm/pvm.html as captured by the Internet Archive on 8 July 2000. Links to pages that were not saved here go to the Wayback Machine. Wayback copy · All pages in this snapshot · Web archive index

AHPCC Logo


Introduction to PVM

John Greenfield green@arc.unm.edu
Amy Stevenson amy@arc.unm.edu
Georgi Yanakiev yanakiev@arc.unm.edu
Anish Malanthara anishm@arc.unm.edu


Bullet Click here for a frame enabled version of this document.

Contents

Overview

Parallel Environments

Message Passing

Coordination of tasks in a distributed memory MIMD environment involves

  1. synchronizing tasks - do the right thing at the right time
  2. sharing data - a task may need data assigned to or produced by another task, or may need to give data to other tasks

Message passing is the mechanism to achieve these requirements

( Click here for more information on message passing in PVM. )

Message Passing Systems

Message passing library Software package to initiate and coordinate tasks on a distributed memory machine.

PVM Terms

Host : a physical machine (or nodes of a parallel machine)

Virtual Machine : meta-machine composed of one or more hosts

Task : a PVM process - smallest unit of computation

TID : unique identifier associated with each task

PVMD : the PVM daemon, 'pvm3/lib/ARCH/pvmd3'

Message : ordered list of data sent between tasks

Group : set of tasks assigned a symbolic name, each task has a unique index in the group and a task can belong to zero or more groups

PVM Overview

PVM (Parallel Virtual Machine)

A software package that permits a heterogeneous collection of computers on a network to appear as one parallel machine.

Heterogeneity

PVM exploits heterogeneity at three levels: Application, Machine and Network

Application

subtasks can be written for the architecture best suited to their solution (serial, vector, massively parallel)

Machine

machines with different architectures, operating systems and data formats can be collected together to form a single virtual machine

Network

the machines can be connected by different networks (FDDI, token ring, ethernet, high performance switch)

Major Features

Portability

PVM System

The PVM System is composed of:

a daemon program pvmd3

A console ( either GUI based or character mode) is available for controlling the virtual machine. It may be started up by typing pvm or xpvm at the unix prompt ( assuming the paths have been set correctly ). The console maybe used to add hosts, spawn tasks, delete hosts etc. ( more details to follow)
( Click here for more information. )

A virtual machine is configured by starting the daemon program on all of the intended hosts, one daemon runs on each host for each user.

The daemon program:

The libpvm programming library:

Applications:

PVM Library

Fortran Library

all programs should have

Additionally, PVM programs perform process control by starting and stopping tasks

Fortran Library - Message Passing

Sending is a three step process:

  1. pvmfinitsend() - initialize send buffer
    Equivalent C function: pvm_initsend(int encoding)
    encoding     specifies the message encoding scheme to be used. An integer id to the buffer for packing the data into is returned.
  2. pvmfpack() - pack data into buffer (any number of times)
    Equivalent C function: pvm_pk*()C has many functions according to the data type to be packed in. The functions, in Fortran or C, take a number of arguments, and return a status code. A negative return value would indicate an error.
  3. pvmfsend() - send message to desired process
    Equivalent C function: pvm_send() The function takes in a tid to the destination process and a positive integer message tag supplied by the user,allowing the program to distinguish between different messages. The function returns an integer status code, and a negative value indicates an error.

Note: pvmfsend can be replaced by any other send routine such as pvmfmcast, see User Guide

Receiving: for every send there should be a receive, and for every pack there should be a matching unpack

  1. pvmfrecv() - receive a message (blocking receive) Equivalent C function: pvm_recv()The function takes in as input a tid of the sending process, and a positive integer message tag supplied by the user. The function returns an id for the new active receive buffer. A negative value signals an error.
  2. pvmfunpack() - unpack data from buffer Equivalent C function: pvm_upk*()C has a number of functions to match the type of data to be unpacked. The function takes in a number of arguments, and returns a status code, which, if negative, signals an error.

Note:there are nonblocking as well as blocking receives, see User Guide

Once composed, a message can be:

  1. sent to another task, multicast to a list of tasks or broadcast to a group
  2. sent any number of times without repacking

Fortran Library - Group Routines

PVM Console

Visualization with Upshot

A Basic Parallel Problem

Problem: Compute the entries of a matrix where

Goal: parallelize this problem, along the way, consider the issues of load balancing and task granularity

A Basic Parallel Problem

The serial code sermat.f contains the lines

do 10 i = 1,n
do 10 j = 1,n
mat(i,j) = setmat(i,j)
10   continue

To parallelize, we will divide up this calculation into tasks, and assign the computation to the various processors

Parallel Strategies

in this strategy, the speedup is limited by the slowest processor we want the faster processors to get more jobs

still have a problem here, the task granularity is too small, the speedup is limited by the communications

PVM setup

New accounts should already be set up correctly. Older accounts may need to be updated by one of the following methods:

in your home directory

check that file .cshrc contains the following lines

setenv PVM_ROOT    /usr/local/pvm3
setenv XPVM_ROOT   $PVM_ROOT/xpvm
setenv PVM_LOCAL   $HOME/pvm3
setenv TCL_LIBRARY $XPVM_ROOT/src/tcl
setenv TK_LIBRARY  $XPVM_ROOT/src/tcl
if ( $OSNAME == AIX ) then
setenv PVM_ARCH RS6K
else if ( $OSNAME == IRIX ) then
setenv PVM_ARCH SGIMP
else if ( $OSNAME == HP-UX ) then
setenv PVM_ARCH HPPA
else if ( $OSNAME == SunOS ) then
setenv PVM_ARCH SUNMP
endif

check that file .cshrc contains the following lines {tiny

set path=($path $PVM_LOCAL/bin/$PVM_ARCH)
set path=($path $PVM_ROOT/lib/$PVM_ARCH)
set path=($path $PVM_ROOT/bin/$PVM_ARCH)
set path=($path $XPVM_ROOT/src/$PVM_ARCH)

create your PVM directories and files {small

% cd
% mkdir pvm3
% mkdir pvm3/bin pvm3/src
% mkdir pvm3/bin/RS6K pvm3/bin/SGIMP pvm3/bin/HPPA pvm3/bin/SUNMP
% cd pvm3

PVM hostfile

Your PVM hostfile defines your parallel virtual machine. It contains the names of all desired machines, one per line, and only needs to reside on the machine where you start PVM

 

########################################################
# FILENAME:  hostfile.2
# DESCRIPTION: PVM hostfile with a few options
########################################################

fr29s01.mhpcc.edu
fr29s02.mhpcc.edu
fr29s03.mhpcc.edu
fr29s04.mhpcc.edu

#
# 	next two hosts will be may be added dynamically later
#

&fr10n12.mhpcc.edu
&fr12n14.mhpcc.edu

#
# next host has different path for its PVM executables
#

beech.tc.cornell.edu  ep=/u/user02/pvm/testing/bin

#
# next line applies to all hosts which follow it
#

* lo=user02 pw

kanaha.mhpcc.edu
makena.mhpcc.edu
littleb.mhpcc.edu

PVM examples

first login into taos or acoma (set DISPLAY variable) copy examples, compile and run

% mkdir -p $HOME/pvm3/src
% cp -R $PVM_ROOT/examples/workshop/* $HOME/pvm3/src
% cd $HOME/pvm3/src
% make all
% cp dlb1 dlb2 slb $HOME/pvm3/bin/RS6K
% xpvm

once xpvm window appears, configure virtual machine consisting of taos and acoma, then spawn the process slb

DO NOT spawn the process `dlb1'

Troubleshooting PVM3.3.x

The following is a troubleshooting list for the setup of PVM:

  1. pvmgs file from earlier version still in $HOME/pvm3/bin/RS6K. Delete it. (Causes mksocs connect error?)
  2. .rhost file write able by others. rw-r--r-- is correct. To identify this problem, try rlogin to another AHPCC machine. If it prompts you for a password, .rhosts is setup incorrectly.
  3. Path not set before "not interactive shell" exit in the .cshrc file. Test with rsh pvmd3. If it cannot find pvmd3, the path is not set before the "not interactive shell" exit.
  4. Path not set up or environment not set up properly in .cshrc file. Use printenv to check path and environment. They should all show pvm3 (or the version of pvm being used).
  5. pvmd.xxx files. Pvmd's cannot startup if a pvmd.xxx file exists in the /tmp directory. Since /tmp filesystems exist on each machine, you should check on each machine.
  6. Xpvm does not come up with list of hosts. .xpvm_hosts file does not exist or is set up improperly.
  7. Old programs don't run. Programs must be recompiled under the new pvm to run. Note that the include files have also changed.
  8. Korn shell. Korn shell does not read the .cshrc file on startup. In order to run under the korn shell the following statement is necessary in your .profile:
    export PVM_DPATH=$PVM_ROOT/lib/pvmd
  9. Interactive statements in the .cshrc. The interactive statements block the running of the .cshrc on the remote system when it is started up. If interactive commands are needed, put them in the .login or after the "if not interactive shell" exit in the .cshrc.
  10. Be careful of concurrent I/O to the same files in a shared filesystem! For concurrent I/O (writing) in a shared filesystem, you may want to implement a scheme to assign unique names to output files.
  11. For the most recent PVM versions, stdout and stderr from spawned processes are written to your /tmp/pvml.uid file. Only stdout and stderr from parent process will appear on your screen. This will be "fixed" in future releases of PVM. A workaround is to use the PVM interactive console and the "->" option with the spawn command, or to use XPVM.
  12. On independent file systems, you must maintain all of the different versions of your executables on multiple machines.
  13. Make sure all pvmd3s and PVM libraries are the same PVM version.
  14. Always use full domain names in .rhosts and PVM hostfiles.

Parallel Strategies II

Adjust the comm/comp ratio in strategy 2 by computing rows instead of elements

  1. strategy 3, dynamic load balance: each processor computes a row of the matrix at a time, when it completes this task, it determines the next row to be computed and repeats until all rows are completed

Example II

Implement strategy 3
code and makefile should be in working directory

% cd
% cd pvm3/src
% ls

you should see the Fortran files, sermat.f, spmddlb2.f and matdlb2.f along with a makefile
We will go over the code found in spmddlb2.f and matdlb2.f first run this code under xpvm and process through upshot

% make dlb2
% xpvm

spawn the process dlb2 after completion, use upshot

% cd
% cp /usr/local/ftp/pub/pvm3/states.dlb2
% mergelogs alogfile.p* > alogfile.dlb2
% upshot -l alogfile.dlb2 -s states.dlb2

SPMD driver

part 1 include file fpvm3.h

	include 'fpvm3.h'

part 2 declare PVM variables

	integer mytid,ptid,me,info,inst,igrpsize
	integer numt,nhosts,njobs,narch,dtid,speed
	character host,arch
	parameter (mnproc = 4)
	integer tids(0:mnproc)

part 3 enroll in PVM

	call pvmfparent(ptid)
	call pvmfmytid(mytid)

part 4 determine number of jobs running

	call pvmfconfig(nhosts,narch,dtid,host,arch,speed,info)
	njobs = nhosts + 1

part 5 find out if parent or child

	call pvmfjoingroup('strat3grp',inst)
	call pvmfgsize('strat3grp',igrpsize)

part 6 parent process

	tids(0) = mytid
	me = 0
c start copies ...
	call pvmfspawn('strat3',PVMDEFAULT,'*',njobs-1,tids(1),numt)

part 7 child process

	me = inst
	tids(me) = mytid
	tids(0) = ptid

part 8 synchronize processes

	call pvmfbarrier('strat3grp',njobs,info)
	call pvmflvgroup('strat3grp',info)

part 9 exit pvm

	call pvmfexit()

work subroutine

part 1 subroutine header and include

	subroutine parmat(me,tids,mnproc,njobs)
	include 'fpvm3.h'

part 2 declare PVM variables

	integer me,mnproc,njobs,mytid,ctid,irow
	integer tids(0:mnproc)

part 3 child computes rows

	if (me .gt. 0) then
  	   mytid = tids(me)
 	   irow = me
5      	continue

part 4 send results to parent

	call pvmfinitsend(0,info)
	call pvmfpack(INTEGER4,mytid,1,1,info)
	call pvmfpack(INTEGER4,irow,1,1,info)
	call pvmfpack(REAL4,mat(irow,1),n,nmax,info)
	call pvmfsend(tids(0),2,info)

part 5 receive next job from parent

	call pvmfrecv(tids(0),3,info)
	call pvmfunpack(INTEGER4,irow,1,1,info)
	if (irow .le. n) goto 5

part 6 receive results from child

	else
	do 6 icnt = 1,n
 	  call pvmfrecv(-1,2,info)
	  call pvmfunpack(INTEGER4,ctid,1,1,info)
	  call pvmfunpack(INTEGER4,irow,1,1,info)
	  call pvmfunpack(REAL4,mat(irow,1),n,nmax,info)

part 7 send next job to child

	  irow = icnt + njobs - 1
 	  call pvmfinitsend(0,info)
 	  call pvmfpack(INTEGER4,irow,1,1,info)
 	  call pvmfsend(ctid,3,info)
6 	continue

Other Examples

Matrix Multiply

This example is a simple matrix multiply program. In the parallel version, the data is distributed among the worker tasks who perform the actual multiplication and send back their respective results to the master task.

pi Calculation

This program calculates pi using a "dart board" algorithm. See Fox et al.(1988) Solving Problems on Concurrent Processors, vol.1 page 207. In the parallel version, all processes contribute to the calculation, with the master averaging the values for pi.

Concurrent Wave Equation

This program implements the concurrent wave equation described in Chapter 5 of Fox et al., 1988, Solving Problems on Concurrent Processors, vol 1.

A vibrating string is decomposed into points. Each processor is responsible for updating the amplitude of a number of points over time. At each iteration, each processor exchanges boundary points with their nearest neighbors.

An X based display of the final wave is provided.

PVM Files:

2D Heat Equation

This example is based on a simplified two-dimensional heat equation domain decomposition. The initial temperature is computed to be high in the middle of the domain and zero at the boundaries. The boundaries are held at zero throughout the simulation. During the time-stepping, an array containing two domains is used; these domains alternate between old data and new data.

Timing Example

This example is a communication timing test for PVM.

A set number of messages are sent between the master and worker tasks. Before and after timings are made for each message and an average is calculated when completed.

PVM Files:

Load Balancing

This example demonstrates simple load balancing for PVM.

A matrix is sent by the master to each worker task. Each worker task computes the inverse of its matrix and returns the result to the master. As the master receives each result it sends a new matrix to that worker.

PVM Files:

Closing


Copyright 1996, The High Performance Computing Education and Research Center
July 1998, editor@arc.unm.edu