GPU node nyc-gpu32 in New York (NYC1) reporting Xid errors

Affected:

GPU nodes · NYC1

Duration

33 min

Reference

inc-20240414-1

Timeline

1. Resolved14 Apr 2024, 23:37 UTC

   Resolved. The customer workload was restarted on the repaired node; the faulty GPU is returned to the vendor.

2. Monitoring14 Apr 2024, 23:30 UTC

   The card has been replaced from the spare pool and the node passed burn-in.

3. Identified14 Apr 2024, 23:14 UTC

   The GPU has fallen off the bus. The node is being drained for a card swap.

4. Investigating14 Apr 2024, 23:04 UTC

   Node nyc-gpu32 in New York (NYC1) is logging NVIDIA Xid errors on one GPU. The affected customer has been notified.

Were you affected? Open a ticket within 7 days and we credit your balance under the SLA.