Connectivity issues in Newark
Incident Report for Linode
Postmortem

At approximately 10:05pm EDT on Monday July 18th our upstream provider made a requested update to a prefix-list which controls the IP blocks Linode announces to the upstream provider (then to the internet). A typo was made on the prefix-list and one of Linode's IP blocks ceased to be routed to the internet for a period of several minutes. Linode engineers had the upstream quickly remedy the error and routing to that one specific prefix was restored. The prefix was still being announced to Linode's hundreds of settlement free peers on the NYIIX and DE-CIX meaning customers with IPs within that larger IP block would have seen loss of connectivity to random destinations for several minutes until the prefix was fully internet reachable again.

Linode plans to turn up several IP transit providers within the next week over our already lit dark fiber which will make errors like this transparent to our customers as we will have many full internet views from each provider. The providers we expect to have up in the next week are Telia, Cogent, Zayo and NTT to follow closely after. We are also constantly looking to bring more settlement-free peers into our blend everyday.

Posted Jul 19, 2016 - 19:37 UTC

Resolved
The connectivity issues within our Newark data center have been resolved at this time.
Posted Jul 19, 2016 - 05:47 UTC
Monitoring
At this time our engineers have rectified the connectivity issue, and are monitoring to ensure the issue does not reoccur.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Posted Jul 19, 2016 - 02:48 UTC
Investigating
We're aware of connectivity issues affecting some hosts in our Newark datacenter, and our networking engineers are currently investigating. We'll update this post with more information on the incident as soon as possible.
Posted Jul 19, 2016 - 02:30 UTC
This incident affected: Regions (US-East (Newark)).