On March 6, 2024, at approximately 18:25 UTC, our admins were made aware of a series of Managed Database backup failures in the Fremont data center. Investigation began immediately.
On March 6, 2024 at approximately 22:27 UTC, our admins were able to determine that these backup failures were caused by Block Storage volumes failing to attach to their respective Linodes. Investigation into why these failures were occurring continued.
On March 7, 2024 at approximately 21:07, our admins identified the cause of the Block Storage attaching failures to be a routing issue in the Fremont data center. At 21:28 UTC on March 7, 2024, a fix was implemented to address this routing issue. It was then confirmed on March 7, 2024 at 22:28 UTC that Block Storage volumes were attaching properly, and Managed Database backups were successfully taken.
Moving forward, we will be making improvements to our monitoring systems to increase the resiliency of our Block Storage service and ensure issues like this do not reoccur in the future.