Service Issue - Block Storage (Fremont)
Incident Report for Linode
Postmortem

On March 6, 2024, at approximately 18:25 UTC, our admins were made aware of a series of Managed Database backup failures in the Fremont data center. Investigation began immediately. 

On March 6, 2024 at approximately 22:27 UTC, our admins were able to determine that these backup failures were caused by Block Storage volumes failing to attach to their respective Linodes. Investigation into why these failures were occurring continued.

On March 7, 2024 at approximately 21:07, our admins identified the cause of the Block Storage attaching failures to be a routing issue in the Fremont data center. At 21:28 UTC on March 7, 2024, a fix was implemented to address this routing issue. It was then confirmed on March 7, 2024 at 22:28 UTC that Block Storage volumes were attaching properly, and Managed Database backups were successfully taken.

Moving forward, we will be making improvements to our monitoring systems to increase the resiliency of our Block Storage service and ensure issues like this do not reoccur in the future.

Posted Mar 13, 2024 - 13:33 UTC

Resolved
We haven’t observed any additional issues with the Block Storage service in Fremont, and will now consider this incident resolved.

If you continue to experience issues, please open a Support ticket for assistance.
Posted Mar 08, 2024 - 18:25 UTC
Monitoring
Our team has identified an issue affecting Block Storage service in our Fremont data center, also affecting our Manage Databases Service. A fix has been implemented, and we will now be monitoring for any further issues.

If you continue to experience problems, please open a Support ticket for assistance.
Posted Mar 07, 2024 - 23:07 UTC
This incident affected: Block Storage (US-West (Fremont) Block Storage).