Connectivity Issue - (Frankfurt)
Incident Report for Linode
Postmortem

At 18:40 UTC on 1/25/21, the Linode Network Operations team began receiving alerts and reports from the Frankfurt data center indicating a possible network outage.   After some investigation, we were able to determine a core switch was missing active sections of its interface configuration which connects this switch to its downstream hosts.  Once identified, the missing configuration was restored from a backup which repaired network connectivity to affected customers.  The incident was confirmed to be fully resolved for all customers at 20:46 UTC on 1/25/21.

The post-incident investigation revealed a vendor bug in this core switch that caused corruption while writing the device configuration.  A routine automated job accepted a configuration rollback to an incomplete snapshot created by this vendor bug. This caused the device to enter an inconsistent state which disrupted networking services in Frankfurt.  

With this bug identified, the Network Operations team has made sure no rollbacks are performed on this switch, and will be conducting a network maintenance to address this bug.

NOTE: This incident, while close in time to the previous outage in Frankfurt, has proven to be unrelated.

Posted Jan 27, 2021 - 22:05 UTC

Resolved
We haven’t observed any additional connectivity issues in our Frankfurt data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Jan 25, 2021 - 21:30 UTC
Monitoring
At this time we have been able to correct the issues affecting connectivity in our Frankfurt data center. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Jan 25, 2021 - 20:33 UTC
Identified
Our team has identified the issue affecting connectivity in our Frankfurt data center. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Posted Jan 25, 2021 - 19:27 UTC
Investigating
Our team is investigating a connectivity issue in our Frankfurt data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information.
Posted Jan 25, 2021 - 18:57 UTC
This incident affected: Regions (EU-Central (Frankfurt)).