New Infiniband Fabric Notes

Owned by Stephen M. Rumble
Last updated: Nov 07, 2011

The new IB fabric for all 80 nodes looks as follows:

Notes

  • Nov 7th, 2011: Not all SX switches are working properly at FDR14 (56Gbps). One core and one leaf appear to, the other two SX switches are not.
  • Nov 7th, 2011: RAMCloud recovery appears to be broken by internal development MLNX OFED 1.5.3-1.1.1 (a very specific reliable message nevers get through to the recipient, although the sender appears to issue a completion queue for that message). Reverting rc20-rc40 to 1.5.3-1.0.0 appears to make things work again.