Connectivity Issue - Newark
Incident Report for Linode
Postmortem

On November 10th, 2020, at 21:20 UTC, Linode Network Operations Team responded to alerts of networking issues occurring in our Newark data center. Upon investigation, it appeared a secondary core switch had rebooted unexpectedly. These switches are in a redundant pair. Unfortunately, this reboot resulted in the primary switch entering a suspended state to protect itself from a split-brain scenario. This was not expected nor desired behavior, and we are currently working with the switch vendor to better understand and explain why this occurred. The normal behavior during a failure like this should be the continued operation of the primary switch. During the outage, the NetOps team identified the issue and monitored the recovery of the primary switch. A deeper investigation revealed a failed line card on the secondary switch. This card has been replaced.

Posted Nov 18, 2020 - 20:02 UTC

Resolved
We haven’t observed any additional connectivity issues in our Newark data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Nov 10, 2020 - 23:33 UTC
Monitoring
At this time we have been able to correct the issues affecting connectivity in our Newark data center. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Nov 10, 2020 - 22:18 UTC
Identified
Our team has identified the issue affecting connectivity in our Newark data center. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Posted Nov 10, 2020 - 21:52 UTC
Investigating
Our team is investigating a connectivity issue in our Newark data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information.
Posted Nov 10, 2020 - 21:22 UTC
This incident affected: Regions (US-East (Newark)) and Linode Manager and API.