Connectivity Issue - EU-West (London)
Incident Report for Linode
Postmortem

Starting at approximately October 30, 2022 at 06:30 UTC, Linode first became aware of connectivity issues impacting customers in the London data center. After investigating the customer reports, Linode’s system administrators could not identify the cause of this impact and paged out to the on-call Network Engineering team member at 7:00 UTC.

Network Engineering was able to determine the presence of a network issue starting at approximately 02:00 UTC. Upon investigating the issue, Linode’s system administrators and Network Engineering team were able to provide small improvements to the situation, but could not completely resolve it. As a result, Linode formally began its incident response procedure at approximately 09:11 UTC, with a status page going live at 10:36 UTC.

Linode’s engineering teams implemented various measures that were expected to alleviate the issue. The issue appeared to be resolved by 13:20 UTC, and the status page was updated at this time.

However, connectivity issues returned to London on a fluctuating basis at approximately 15:44 UTC. Due to renewed customer impact, Linode posted a new status page at 16:27 UTC. Linode immediately started reviewing the situation for resolution pathways, making a change to the network configuration at 16:35 UTC.

During this time, an unrelated issue occurred for certain customers trying to access the London data center. This issue received a separate status page:

Similar recurrences of this issue occurred throughout the week, including at the following times:

  • November 3 09:57 UTC to November 4 02:00 UTC
  • November 6 06:30 UTC to November 6 08:15 UTC
  • November 6 12:00 UTC to November 6 13:05 UTC
  • November 6 14:35 UTC to November 6 15:45 UTC

Linode will be reviewing the data gathered during these incident periods to implement longer-term changes into its network.

Posted Dec 28, 2022 - 00:55 UTC

Resolved
We haven’t observed any additional connectivity issues in our London data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Nov 08, 2022 - 13:20 UTC
Update
We are continuing to monitor connectivity issues affecting our London data center. Brief periods of increased latency or packet loss may still occur. If you are still experiencing issues, please open a Support ticket with as much detail as possible, including MTR outputs for assistance.
Posted Nov 04, 2022 - 05:23 UTC
Update
We are continuing to monitor connectivity issues affecting our London data center. Brief periods of increased latency or packet loss may still occur. If you are still experiencing issues, please open a Support ticket with as much detail as possible, including MTR outputs for assistance.
Posted Nov 04, 2022 - 01:12 UTC
Update
We are continuing to monitor connectivity issues affecting our London data center. Brief periods of increased latency or packet loss may still occur. If you are still experiencing issues, please open a Support ticket with as much detail as possible, including MTR outputs for assistance.
Posted Nov 03, 2022 - 20:40 UTC
Monitoring
At this time we have been able to correct the issues affecting connectivity in our London data center. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Nov 03, 2022 - 16:34 UTC
Update
Our team has identified the issue affecting connectivity in our London data center. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Posted Nov 03, 2022 - 12:36 UTC
Investigating
Our team is investigating connectivity issues in our London data center. During this time users may experience intermittent connection timeouts, packet loss, and other errors for all services deployed in this DC. We will share additional updates as we gather more information.
Posted Nov 03, 2022 - 10:36 UTC
This incident affected: Regions (EU-West (London)).