Emerging Service Issue - Connectivity Across Multiple Regions

Incident Report for Linode

Postmortem

On April 16, 2025, at approximately 20:10 UTC, we noticed numerous internal alerts fired across multiple data centers, indicating that various servers were unreachable for an extended period. At the same time, customers started reporting issues with multiple products.

During the impact window, customers may have experienced issues with all Compute Products, which prevented them from deploying new Linodes, booting Linodes, and performing other host-level jobs. There was intermittent and/or total loss of network connectivity, as well as an inability to access services such as Object Storage or Linode Kubernetes Engine (LKE).

The investigation has indicated that the issue can be attributed to a configuration anomaly on BGP route reflector infrastructure providing routes for Compute services; however, it has been determined that only those route servers that had undergone a restart at some point were experiencing this impact.

Akamai implemented the correct new configuration across all affected route servers at approximately 22:15 UTC to alleviate the residual impact. Nevertheless, a degree of impact reemerged in specific locations during the mitigation efforts.

This process was undertaken in phases, commencing with the regions most adversely affected, and was concluded at approximately 03:45 UTC on April 17, 2025. Following an extensive monitoring period of our systems, we verified that the issue has been resolved.

This summary provides an overview of our current understanding of the incident, given the information available. Our investigation is ongoing, and any information herein is subject to change.

Posted Apr 18, 2025 - 00:15 UTC

Resolved

We haven’t observed any additional connectivity issues in our multiple data centers, and will now consider this incident resolved. During the impact window, we did not experience issues with Cloud Manager and API. If you continue to experience problems, please open a Support ticket for assistance.
Posted Apr 17, 2025 - 14:44 UTC

Monitoring

At this time we have been able to correct the issue affecting the Cloud Manager and API. We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues and unable to open a Support ticket, please call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Posted Apr 17, 2025 - 03:31 UTC

Update

We are continuing to work on a resolution for this issue. Subsequent updates around mitigation status will be posted as progress is made.
Posted Apr 17, 2025 - 01:02 UTC

Identified

After performing mitigation actions we became aware the issue is still occurring; the issue is identified and a fix is being implemented.
Posted Apr 16, 2025 - 23:08 UTC

Monitoring

At this time we have been able to correct the issues affecting connectivity. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Apr 16, 2025 - 22:39 UTC

Identified

We've identified a root cause and are applying a mitigation as quickly as possible. We'll continue to post updates as this incident develops.
Posted Apr 16, 2025 - 22:16 UTC

Update

We are working quickly on mitigating this issue. We will share additional updates as we have more information.
Posted Apr 16, 2025 - 21:25 UTC

Investigating

Our team is investigating an emerging service issue affecting connectivity in multiple regions. We will share additional updates as we have more information.
Posted Apr 16, 2025 - 20:35 UTC
This incident affected: Linode Manager and API and Regions (US-IAD (Washington), US-ORD (Chicago), EU-West (London), EU-Central (Frankfurt), FR-PAR (Paris), SE-STO (Stockholm), US-SEA (Seattle), US-LAX (Los Angeles), DE-FRA-2 (Frankfurt 2)).