Linode Status - Service Issue

Service Issue - Block Storage

Incident Report for Linode

Postmortem

At 1647 UTC, March 4, 2024 a new release to the Linode API was published. This release contained a new method of validating limits for Block Storage Volumes, erroneously preventing Customers with more than 100TB of storage in a single region from deploying further Storage Volumes regardless of their actual limit.

On the morning of March 5, our engineering team detected errors when attempting to scale Block Storage Volumes. At 1101 UTC, the root cause was determined to be the new limits implemented by the API change.

At 1626 UTC, March 5, 2024 our team successfully rolled back the change that caused this limitation to be applied. We have revised the deployment process for the BlockStorage API to ensure that no unchecked changes are made to software accessible by customers, safeguarding them against unintended limits.

Posted Mar 18, 2024 - 18:42 UTC

Resolved

We haven’t observed any additional issues with the Block Storage service during our monitoring, and will now consider this incident resolved.

If you continue to experience problems, please open a Support ticket for assistance.

Posted Mar 05, 2024 - 23:25 UTC

Monitoring

Testing is now complete. Our team deployed a fix to address the impact caused by our previous change. At this time we have been able to correct the issues affecting the Block Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.

Posted Mar 05, 2024 - 18:04 UTC

Identified

We have identified the cause for unexpected limitations that were put in place that resulted in the prevention of adding additional services. Our team is currently working on addressing this, which will include testing. Due to the time required for this testing, we do not anticipate an update for another two hours. That said, we will update this page once we have more information to share.

Posted Mar 05, 2024 - 16:52 UTC

Investigating

Our team is investigating an issue affecting the Block Storage service on accounts with large data pools (100TB or greater). During this time, users may experience errors creating further volumes. We will share additional updates as we have more information.

Posted Mar 05, 2024 - 13:19 UTC

This incident affected: Block Storage (US-East (Newark) Block Storage, US-Central (Dallas) Block Storage, US-West (Fremont) Block Storage, US-Southeast (Atlanta) Block Storage, US-IAD (Washington) Block Storage, US-ORD (Chicago) Block Storage, CA-Central (Toronto) Block Storage, EU-West (London) Block Storage, EU-Central (Frankfurt) Block Storage, FR-PAR (Paris) Block Storage, AP-South (Singapore) Block Storage, AP-Northeast-2 (Tokyo 2) Block Storage, AP-West (Mumbai) Block Storage, AP-Southeast (Sydney) Block Storage, SE-STO (Stockholm) Block Storage, US-SEA (Seattle) Block Storage, JP-OSA (Osaka) Block Storage, IN-MAA (Chennai) Block Storage).