Linode Status - Cloud Manager and API Issue

Cloud Manager and API Issue - AP-West (Mumbai)

Incident Report for Linode

Postmortem

On 8 November 2021 at 07:00 UTC, a Block Storage host in our Mumbai datacenter underwent routine maintenance. After this maintenance, the device came back online, but was missing a critical networking route. As a result, a portion of Linodes in Mumbai with attached Block Storage volumes became unresponsive.

Linode’s Operations Center became aware of this issue at 08:45 UTC and engaged our incident response protocols. We investigated and concluded that the route was not permanently set on the Block Storage device. The route was subsequently added back at 12:26 UTC and service was restored.

As a result of this outage, we have verified that all remaining Block Storage devices have fully persistent routes. Additionally, we’ve added procedures to prevent future incidents of this type.

If you have any further questions regarding this incident, please reach out to Linode Support.

Posted Nov 22, 2021 - 20:56 UTC

Resolved

This incident has been resolved.

Posted Nov 08, 2021 - 15:00 UTC

Monitoring

Our team has implemented a fix and we are currently monitoring things. We will keep you posted and provide an update as soon as possible.

Posted Nov 08, 2021 - 13:39 UTC

Identified

Our team has identified the issue, which is related to our Block Storage service. We are currently working towards a fix for this and we will provide updates as soon as possible.

Posted Nov 08, 2021 - 12:43 UTC

Update

Our team is still investigating this issue and we will provide updates as soon as possible.

Posted Nov 08, 2021 - 11:35 UTC

Update

We are continuing to investigate this issue and we will provide updates as soon as possible.

Posted Nov 08, 2021 - 10:25 UTC

Investigating

Our team is investigating a service issue affecting the Cloud Manager and API for some Linodes located in our Mumbai data center. During this time, users may experience issues when attempting to use these systems with their Mumbai Linodes. We will share additional updates as we have more information.

Posted Nov 08, 2021 - 09:24 UTC

This incident affected: Regions (AP-West (Mumbai)).