AHPCC Condor Flock Information

• Running Jobs on the AHPCC Condor Flock
  

• Current Status (updated every 15 minutes)

What is Condor?

Condor is a job submission facility that manages and schedules a pool of computing resources in a manner that focuses on High Throughput Computing (HTC). High Throughput Computing supports a large amount of processing capacity sustained over a very long period of time. Condor, in focusing HTC, allows users to submit jobs which are then executed using idle cycles on machines otherwise not being used. Whenever the load on the machine exceeds a certain pre-defined level or the mouse or keyboard is activated, Condor will remove any processes on that machine and not schedule it further. Condor programs are linked with checkpoint code that allows them to be unloaded from one machine and restarted, from where they left off, on another. Probably the best way to describe how Condor implements HTC is to use the terms "bottom feeder" or "cycle stealing" system. Condor supports both parallel as well as serial jobs, but currently the only parallel paradigm available is PVM.

Whe have also been experimenting with combining this AHPCC pool with other pools, namely the Department of Computer Science at UNM and the University of Wisconsin.

The idea behind condor is to allow users to submit jobs which are then executed using idle cycles on machines otherwise not being used. These machines are usually assumed to be "owned" by individuals who have the machines on a local area network. The "owners" whish to share and pool these machines together to provide in effect a large computing resource which is made available when the owners are not using the machines. Thus, whenever the load on the machine exceeds a certain pre-defined level or the mouse or keyboard is activated, Condor system will remove any processes on that machine and not schedule it further.

Take the Condor Tutorial

Learn how to test and operate Condor.

Download Condor

Condor is publicly available in binary for several different platforms at http://www.cs.wisc.edu/condor.

Projects

"Flocking Pools" Recently, we have been experimenting with connecting the AHPCC pool with other pools. The first and very successful experiment was to combine the AHPCC pool and UNM CS department's pool of machines, each running Condor. Jobs submitted to either pool can run on available resources in the other pool. The pools are set up so that if a job submitted to one pool has no available resources and the other pool does, the jobs will migrate to the second pool. Preemptive criterion have been established so that jobs submitted to a particular pool from accounts in that pool have preference to run in that pool over jobs from "outside" pools. For example, if a job is submitted to the CS department's pool and that pool is busy with jobs with accounts from the AHPCC pool, the AHPCC jobs are check-pointed and migrated out of the CS department pool, allowing the newly entered job from the CS department's pool to run. The check-pointed jobs may then run later on any available machine of the same architecture when one becomes available.

The flocking of pools is not limited to pools on the same "campus." We have experimented very successfully by joining with one of the pools at the University of Wisconsin, in addition to the AHPCC pool.

Recently, the AHPCC installed a development version of Condor for the NT operating system. This version is being managed from the Linux-based pool manager and is on the same AHPCC network of Intel PCs.

Experience with Condor

Condor has been very reliable and robust and has provided an effective computational service to AHPCC users that complements and supplements our relatively large parallel systems. Computations that do not require specialized high performance networking capabilities but run for a long time or represent a large number of jobs are best done on the Condor pool.