CARC web archive · Archived copy of hpc.unm.edu/systems/downtime.html as captured by the Internet Archive on 28 December 2005. Links to pages that were not saved here go to the Wayback Machine. Wayback copy · All pages in this snapshot · Web archive index
  

Shutdown schedule to
bring up New Machine Room

Updated:  
Mon Nov 28 13:23:12 MST 2005


Contact:   Tim Thomas (505-277-8245)
        or Main Desk  (505-277-8249)


(Click HERE to skip to the latest updated info...)



=============
Nov 7, Monday
=============

8 AM:

  In preparation for the power shutdown this afternoon (see below),
  all systems - except for Ristra and Hammer - will be shut down.

  (Ristra and Hammer, presently located outside the Gallas building,
  will be brought down on Tuesday.)

NOON: 

  The Gallas building will close.  It will remain closed to the public
  for the rest of the day, re-opening at 8 AM on Tuesday morning.
  All HPC staff will remain on hand for the remainder of the day.

  (Resident researchers and HPC Users possessing keys may enter
  with caution...)

1 PM: 

  All power in the Gallas building will be turned off until between 3
  and 4 PM, in order to perform one final test of the new building
  electrical systems supporting the new machine room.

  (This will be the ONLY PERIOD DURING THE WEEK when power will be off
  and the Gallas building closed.)

  During this period, there will be no heating, cooling, or electric
  lighting operational anywhere in the building.  The fire and security
  alarm systems WILL continue to operate via generator power.

3-4 PM: 

  Power, heating/cooling return.  Building will remain closed for the day.

By 5 PM: 

  The following Center Services will return to operation, for both
  the ERC and Alliance sides of the center:

    . Building Network and connection to campus
    . Domain Name Service (DNS)
    . Directory Services (NIS)
    . NFS/NAS File Services
    . Full Email Services
    . Full Web Services

  All other computing systems will remain down.


===============
Nov 8, Tuesday
===============

11 AM:

  Ristra will be shut down.

2 PM: 

  Hammer will be shut down.

3:30 PM: 

  HPC Email Service will transition to a REDUCED STATE of operation:
  messages will be queued within the mail server, but they will not
  be forwarded.

  The HPC Web Server will transition to a REDUCED STATE: only certain
  pages - including this status page - will be available.

4:15 PM: 

  All NFS/NAS File Services (both ERC- and Alliance-side) will be shut down.

  (The building network and connection to campus, as well as the NIS
  system, will remain available.)


=================
Nov 9,  Wednesday
=================

8 AM - 7 PM: 

  All physical equipment movement will happen today.

  Equipment will be moved in the following order:

  From outside locations:

    1) IBM JS20 (Army:"Ristra") - 2 racks
    2) Apple G5 cluster #1 (Army:"Hammer") - 1 rack

  From the LosLobos server room:

    3) IBM/Oracle cluster + SAN (SRBCB:"Delphi") - 1 rack
    4) Appro/RaidInc Alliance + ERC NFS servers rack - 1 rack

  From the RoadRunner room:

    5) Vista/Azul Switch - 1 rack
    6) Apple G5 cluster #2 + SAN (ARTSLab:"Yenta") - 1 rack
    7) IBM p570/p550 SMP systems (MRI:"Poblano"/"Pequin") - 1 rack
    8) SeaSpace cluster + NAS (CREATE) - 1 rack
    9) IBM NetFinity 8x4-way Xeon ("Azul") - 2 racks
   10) Infrastructure servers ("BlackBear") - 1 rack

The LosLobos and Vista systems will not be moved.


================
Nov 10, Thursday
================

8 AM:

OUR PLAN for this day had been:

  We will begin bringing machines back online and intend to have all
  systems available by 5 PM, by which time file system, Email, and
  web services will have returned to full operation.

WHAT ACTUALLY HAPPENED:

  The electricians completed their planned work very late.  In addition,
  there were the 'predictably unexpected' glitches in the electrical
  system, which delayed things further.  We could not roll the machines
  into their final positions until the electrical problems were resolved.

  We thereby ended up experiencing an overall slip of about half a day in
  the schedule.  By the end of the day, we had the Alliance and ERC-side
  NFS service back up and email flowing again.  Hammer was also powered
  on, in a testing mode, but because there are still some issues with the
  A/C units, we deemed it unsafe to bring anything else up so late in the
  day.
  

 
==============
Nov 11, Friday
============== 

OUR PLAN for this day had been: 

  Veteran's Day.  All machines should be operational.  With help from our
  expert users, HPC will identify and correct any lingering issues during
  the course of the day.

WHAT ACTUALLY HAPPENED:

  Most electrical system issues were resolved.  IBM gave it a final "OK",
  vetting the continuation of our warranties and maintenance agreements.
  Then the A/C system experienced some difficulties.  Technicians resolved
  this issue while we continued our re-cabling work.  A final unplanned
  A/C and electrical shutdown took place during the lunch hour to resolve
  one final electrical issue.  All these activities set us back another
  half day, resulting in an overall one day delay, which has bumped us
  up against the weekend.

CURRENT MACHINE STATUS:

  Alliance- and ERC-side NFS:   UP
  Full Email and web services:  UP

  CREATE:   UP
  Poblano:  UP
  Delphi:   UP

  Yenta:    UP
  Hammer:   UP

  LosLobos: UP
  Ristra:   UP
  Azul:     UP
  Pequin:   UP
  Vista:    UP