Driver rollout paused on GPU nodes in Amsterdam (AMS1)

Affected:

GPU nodes · AMS1

Duration

47 min

Reference

inc-20240811-1

Timeline

1. Resolved11 Aug 2024, 09:05 UTC

   Resolved. The driver version has been pulled from our images until NVIDIA publishes a fix.

2. Monitoring11 Aug 2024, 08:57 UTC

   All nodes are healthy; workloads restarted.

3. Identified11 Aug 2024, 08:33 UTC

   The affected nodes have been rolled back to the previous driver and rebooted with customer consent.

4. Investigating11 Aug 2024, 08:18 UTC

   A driver update caused CUDA initialisation failures on two nodes in Amsterdam (AMS1). The rollout is paused.

Were you affected? Open a ticket within 7 days and we credit your balance under the SLA.