Cluster Issues
- Ankita Kejriwal
- Stephen Yang
- Seo Jin Park
Owned by Ankita Kejriwal
Central documentation of issues we see in the cluster – those that we want to solve right away, as well as those that we don't. This is hopefully help us see patterns if any, apart from making it easy for the cluster custodian to keep track of issues that currently need attention.
| Observed on | Component affected | Description | Reporter | Resolution | Resolver |
|---|---|---|---|---|---|
| 2014/10/?? | Infiniband switch that connects to rc01 | The status light has a color between yellow and red. Might mean firmware issue based on the manual | Ankita | ||
| May 19, 2015 | rcmaster | "BMC" failed | Henry | ||
| May 19, 2015 | rc cluster machines | 11 power supplies have failed. | Jonathan | 1 power supply was resurrected, 10 new units were purchased and installed. | Collin |
| May 21, 2015 | Port 20 on right rack ethernet | Network port assumed to have failed; rc58 which was connected to it was not able to connect to network through it. | Collin | ||
| May 21, 2015 | rc20 | Unable connect to network. No Ping; No IPMI; | Collin | Some combination of wire jiggling and rebooting seems to have solved the issue. | Collin |
| May 21, 2015 | rc50 | Network connection issues | Collin | rczap | Collin |
| Jun 24, 2015 | Port 15 on right rack ethernet | Network port assumed to have failed; rc58 which was connected to it was not able to connect to network through it. | Ankita | ||
| Dec 11, 2015 | rc52 | showing only 12 GB of memory | John | Open the box and reconditioned all sticks. | Seo Jin |
| Jan 6, 2016 | rc46, rc70 | No IB devices found (nothing reported from ibv_devinfo.) | Seo Jin | Replugging wire & manual power off & on. (IPMI reboot didn't resolve the issue.) | Seo Jin |
| Feb 20, 2016 | Ports 8,9,13,15,16,20 on right rack ethernet | These ports don't appear to be functioning. | Jonathan | Avoid using them. | Jonathan |
Dec 23, 2016 | rc47 | Networking issue, bad network switch port/cable. Wasn't able to connect to the host. | Behnam | Resolved! | Behnam |
Jan 10, 2016 | rc57 | Host unreachable ssh error. bad network switch port. | Behnam | Resolved | Behnam |
| Nov 27, 2017 | rc80 | Host unreachable. | Seo Jin | Resolved | Behnam |
| Dec. 06, 2017 | rc74 | Host unreachable | Behnam | ||
Aug. 16, | rcmaster | libc was updated to unstable (debian sid) on rcmaster; causes some apps built on rcmaster to crash on rcXX | Stephen | Resolved | Stephen |
| Apr. 6, 2019 | rc35, rc36, rc69 | Host unreachable | Yilong | ||
| Dec 9, 2019 | rc50 | Host unreachable and pings to machine/ipmi interface sees massive drops | Stephen | ||