Elevated NVMe latency on a storage pool in New York (NYC1)

Affected:

Storage & snapshots · NYC1, VPS hosts · NYC1

Duration

1 h 21

Reference

inc-20230829-2

Timeline

1. Resolved29 Aug 2023, 21:55 UTC

   Resolved. Resilver complete. No data was at risk: the mirror stayed redundant throughout.

2. Monitoring29 Aug 2023, 21:42 UTC

   The replacement drive is resilvering; latency is back to normal.

3. Identified29 Aug 2023, 20:47 UTC

   A drive in the mirror is failing and slowing the pool. It is being detached and replaced.

4. Investigating29 Aug 2023, 20:34 UTC

   One NVMe pool in New York (NYC1) shows write latency above 5 ms. Some VPS on this pool may see slower disk I/O.

Were you affected? Open a ticket within 7 days and we credit your balance under the SLA.