A SuperCluster typically includes a high-performance communication network to support the interprocess communication needed by the applications running on the system along with one or two networks with significantly lower performance requirements for administrative tasks. There is no fundamental reason that the administrative traffic could not use the high-performance network; however, the separation is frequently exploited in the communication protocols supported by each network. For example, the Roadrunner SuperCluster uses a Myrinet network for application level interprocess communication and a 100 BaseT Ethernet network for administrative tasks. The Myrinet network supports MPI which is used for interprocess communication in most scientific applications, but does not support TCP/IP which is for most administrative tasks. TCP/IP is supported on the 100 BaseT network. In the SUR Netfinity cluster, we propose to use the Gigabit ethernet network to support MPI traffic and the 100 BaseT network to support TCP/IP traffic.
By separating the classes of traffic, the performance of each network can be tuned to the expected traffic patterns. In particular, it has been our experience that we can gain significant advantages by tuning the network to support MPI traffic. In particular, MPI is easily layered on most of the current OS Bypass protocols (e.g., Myricom's GM, VIA and Scheduled Transfer). In developing the Portals API with Sandia National Laboratories, we have found that we can gain additional performance benefits using techniques that enable application bypass. That is, techniques that allow large message transfers using remote puts and gets without the need for any intervention on the part of the application or an application level thread. Taking advantage of these techniques requires message matching on the network interface card (NIC) and, as such, requires a programmable interface for the NIC.
The IBM Netfinity cluster will provide us with the equipment needed to compare a commodity (e.g. TCP/IP over Gigabit ethernet) network similar to the next generation of Beowulf clusters with a faster implementation (using OS Bypass or Portals) as well as with the networking speeds of Myrinet (from RoadRunner) and the SP switch.