Connectivity Issue - Dallas
Incident Report for Linode
Postmortem

On November 22, 2022 at approximately 4:45 UTC, monitoring systems alerted to reachability issues in the Dallas data center. The impact of this incident occurred on Dallas’s private network, but would have affected public services that depend upon private communications such as NodeBalancers. At 5:28 UTC an incident was raised, the incident response team was deployed and investigation for the root cause of the issue began. After initial work on the situation, a status page was created at 5:43 UTC. Upon troubleshooting by several members of the Network Operations team team, it was determined that a network loop was occurring in a portion of the Dallas network, resulting in increased resource usage in several of the network devices in that segment. During troubleshooting, the Network Operations team determined that the loop was a result of a software bug and took steps to isolate the faulty software. Due to the nature of the bug, several systems needed to be restarted consecutively in order to properly isolate the network. Our Network Operations team began this process at 11:00 UTC and continued to roll out the fix for the next 90 minutes. Once all of the systems were restarted, connectivity was restored at approximately 12:30 UTC.

Investigation is ongoing regarding the recent networking issues that have been occurring within our Dallas data center. As updates to our global infrastructure are being released, we are continuously implementing processes to strengthen the reliability and stability of our network in the future. These upgrades are being placed on an expedited timeline, which will help to prevent incidents such as these from occurring as we move forward.

Posted Dec 21, 2022 - 18:37 UTC

Resolved
We haven’t observed any additional connectivity issues, and will now consider this incident resolved. Please reference Emergency Network Maintenance - US-Central (Dallas) for further updates. After we have completed maintenance and have more information, we will provide a detailed post-mortem for this incident.
Posted Nov 22, 2022 - 17:02 UTC
Monitoring
At this time, connectivity remains stable and we are continuing to monitor for any additional issues. Further maintenance is necessary to fully resolve this issue. For more information about the upcoming maintenance, please reference Emergency Network Maintenance - US-Central (Dallas).
Posted Nov 22, 2022 - 14:46 UTC
Identified
Our engineers have completed emergency maintenance in Dallas and connectivity is currently stable. We will be performing additional maintenance to fully resolve this issue. A timeline for this maintenance will be communicated in a separate post.
Posted Nov 22, 2022 - 13:54 UTC
Update
Our Engineers are still performing emergency maintenance in Dallas within our routing environment. We will provide additional updates as soon as possible.
Posted Nov 22, 2022 - 12:14 UTC
Update
Our Engineers will be performing emergency maintenance within our routing environment to address the connection issues observed in our Dallas Data Center. During this time some customers may observe additional networking traffic issues.
Posted Nov 22, 2022 - 11:02 UTC
Update
We are continuing to investigate this issue.
Posted Nov 22, 2022 - 10:00 UTC
Update
We are continuing to investigate this issue.
Posted Nov 22, 2022 - 08:52 UTC
Update
We are continuing to investigate the connectivity issue affecting our Dallas data center. We will provide additional updates as soon as possible.
Posted Nov 22, 2022 - 07:52 UTC
Update
We are continuing to investigate this issue.
Posted Nov 22, 2022 - 06:49 UTC
Investigating
We have identified connectivity issues affecting our Dallas data center. Our team is working as quickly as possible to have connectivity restored. We will provide additional updates as the situation develops.
Posted Nov 22, 2022 - 05:42 UTC
This incident affected: Regions (US-Central (Dallas)).