Linode Status - Service Issue - Linode Provisioning

Service Issue - Linode Provisioning

Incident Report for Linode

Postmortem

On January 6th at 16:07 UTC we were alerted to an issue with the Linode API causing intermittent issues with Linode creation, deletion, and other actions. Due to the way the API and certain endpoints were handling calls, delays in processing the API calls resulted in intermittent failures. Running services such as Linodes would not have experienced downtime as a result of this incident.

Following an investigation, an emergency maintenance of the API was started at 19:15 UTC. After this maintenance was complete, the API was brought fully online at 19:42 UTC, with services restored at 19:49 UTC.

As a result of this incident we are reviewing ways to improve the efficiency of the API to help prevent issues like this from happening.

For any further questions about this incident, please open a ticket with the Linode Support Team.

Posted Jan 31, 2023 - 22:23 UTC

Resolved

As of 20:50 UTC Friday, all affected services have been fully restored. If you are still experiencing issues, please open a Support ticket for assistance.

Posted Jan 06, 2023 - 20:51 UTC

Monitoring

Our emergency maintenance is now complete, and the issue has now been resolved. All services have now been restored, and we are continuing to monitor for any lingering issues.

Please contact our Support team with any further issues.

Posted Jan 06, 2023 - 19:49 UTC

Update

At 19:15 UTC today, Jan 6th, we will be performing emergency maintenance that will result in a full outage for a period of up to 1 hour.

During this maintenance, your Linodes and other services will still be available, however you will not be able to access the Cloud Manager or API during this maintenance period, and jobs like reboots, backups, etc. will not be able to process during this period.

Posted Jan 06, 2023 - 18:58 UTC

Update

We are continuing to investigate the incident and are taking steps to address the performance issues detected by our administrators. Please see the bottom of this page for a list of services that this incident is affecting. As stated before, you may experience Gateway Timeout errors in Cloud Manager and API when performing actions, and you may also experience difficulties with logging into your account.

Posted Jan 06, 2023 - 18:08 UTC

Update

Correction: The impact appears to be wider scope than just provisions. At this time, you may experience Gateway Timeout errors in Cloud Manager and API when performing any actions.

Posted Jan 06, 2023 - 16:52 UTC

Investigating

We are currently investigating an issue where Linode creation is resulting in 502 gateway timeout errors. At this time, you may experience issues deploying new Linodes, Managed Database clusters, and LKE clusters.

Posted Jan 06, 2023 - 16:36 UTC

This incident affected: NodeBalancers (US-East (Newark) NodeBalancers, US-Central (Dallas) NodeBalancers, US-West (Fremont) NodeBalancers, US-Southeast (Atlanta) NodeBalancers, CA-Central (Toronto) NodeBalancers, EU-West (London) NodeBalancers, EU-Central (Frankfurt) NodeBalancers, AP-South (Singapore) NodeBalancers, AP-Northeast-2 (Tokyo 2) NodeBalancers, AP-West (Mumbai) NodeBalancers, AP-Southeast (Sydney) NodeBalancers), Linode Kubernetes Engine (US-East (Newark) Linode Kubernetes Engine, US-Central (Dallas) Linode Kubernetes Engine, US-West (Fremont) Linode Kubernetes Engine, US-Southeast (Atlanta) Linode Kubernetes Engine, CA-Central (Toronto) Linode Kubernetes Engine, EU-West (London) Linode Kubernetes Engine, EU-Central (Frankfurt) Linode Kubernetes Engine, AP-South (Singapore) Linode Kubernetes Engine, AP-Northeast-2 (Tokyo 2) Linode Kubernetes Engine, AP-West (Mumbai) Linode Kubernetes Engine, AP-Southeast (Sydney) Linode Kubernetes Engine), and Linode Manager and API, Hosted DNS Service, Managed Databases.