Linode Status - Service Issue

Service Issue - Block Storage

Incident Report for Linode

Postmortem

On November 6, 2024, at 5:30 UTC, we started upgrading an internal LKE component. At approximately 10:30 UTC, customers started reporting that they could not attach block storage volumes to LKE nodes.

‌

Investigation revealed that the issue was caused by a software malfunction introduced during an internal LKE component upgrade. This resulted in errors due to a bad check intended to confirm volumes and Linodes were in the same region.

‌

We began rolling back the internal upgrade region by region at 14:55 UTC and completed it at approximately 15:25 UTC. After monitoring our systems post-rollback, we confirmed that the issue was fully mitigated.

‌

This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

Posted Nov 12, 2024 - 14:03 UTC

Resolved

We haven’t observed any additional issues with the Block Storage service in all our data centers, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.

Posted Nov 06, 2024 - 16:27 UTC

Monitoring

At this time we have been able to correct the issues affecting the Block Storage service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.

Posted Nov 06, 2024 - 15:37 UTC

Identified

Our team has identified the issue affecting the Block Storage service in all our data centers. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.

Posted Nov 06, 2024 - 15:05 UTC

Investigating

We are investigating an issue affecting the Block Storage service which appears constraint to LKE nodes in all data centers. During this time, users may be unable to attach Block Storage volumes to LKE nodes. We will share additional updates as we have more information.

Posted Nov 06, 2024 - 14:54 UTC

This incident affected: Block Storage (US-East (Newark) Block Storage, US-Central (Dallas) Block Storage, US-West (Fremont) Block Storage, US-Southeast (Atlanta) Block Storage, US-IAD (Washington) Block Storage, US-ORD (Chicago) Block Storage, CA-Central (Toronto) Block Storage, EU-West (London) Block Storage, EU-Central (Frankfurt) Block Storage, FR-PAR (Paris) Block Storage, AP-South (Singapore) Block Storage, AP-Northeast-2 (Tokyo 2) Block Storage, AP-West (Mumbai) Block Storage, AP-Southeast (Sydney) Block Storage, SE-STO (Stockholm) Block Storage, US-SEA (Seattle) Block Storage, JP-OSA (Osaka) Block Storage, IN-MAA (Chennai) Block Storage).