GPU node nyc-gpu15 in New York (NYC1) reporting Xid errors

Affected:

GPU nodes · NYC1

Duration

1 h 37

Reference

inc-20250522-1

Timeline

1. Resolved22 May 2025, 22:02 UTC

   Resolved. The customer workload was restarted on the repaired node; the faulty GPU is returned to the vendor.

2. Monitoring22 May 2025, 21:41 UTC

   The card has been replaced from the spare pool and the node passed burn-in.

3. Identified22 May 2025, 20:48 UTC

   The GPU has fallen off the bus. The node is being drained for a card swap.

4. Investigating22 May 2025, 20:25 UTC

   Node nyc-gpu15 in New York (NYC1) is logging NVIDIA Xid errors on one GPU. The affected customer has been notified.

Were you affected? Open a ticket within 7 days and we credit your balance under the SLA.