Connectivity Issues - Fremont
Incident Report for Linode
Postmortem

At approximately 12:50:00 PM EST on February 23, 2018, utility power was interrupted to our Fremont data center. The UPS systems provided power for their respective areas while 7 of 8 generators started and began to come up to operating temperature. During this phase, one of the generators servicing Linode’s infrastructure failed to start. UPS power for our area was exhausted at 13:01:00 EST causing a subset of Linode hardware (and the customer instances on them) to be powered off unexpectedly and become unreachable.

Utility power was restored at approximately 13:18:00 EST. At that time, Linode engineers worked to bring our infrastructure and affected customer instances back online. All Linode infrastructure and customer instances were online by approximately 16:50:00 EST, and the incident was deemed resolved.

Further diagnostics determined that the generator’s autostart controller had failed, preventing a successful start and subsequent transfer of generator power. The colocation site’s vendor was engaged and a replacement card was installed.

We do not anticipate any further issues with the Fremont facility at this time. Thank you for your patience and understanding. We apologize for any inconvenience this outage has caused.

Posted Feb 28, 2018 - 22:16 UTC

Resolved
Because we have not experienced additional connectivity issues affecting our Fremont data center, this matter is now resolved.
If you are still experiencing connectivity issues, please reach out to our Customer Support Team for assistance.
Posted Feb 23, 2018 - 21:50 UTC
Monitoring
Connectivity issues affecting our Fremont data center have been resolved. The connectivity issues have been identified as being the result of a power failure. Additional details regarding the power failure will be provided in a post-mortem. We will be monitoring this issue to ensure connectivity remains stable.
Posted Feb 23, 2018 - 20:04 UTC
Identified
We have identified the connectivity issues affecting our Fremont data center, and our team is working as quickly as possible to have full connectivity restored. We will provide additional updates as they develop.
Posted Feb 23, 2018 - 18:51 UTC
Investigating
We are aware of connectivity issues affecting Linodes and NodeBalancers in our Fremont data center and are currently investigating. We will continue to provide additional updates as this incident develops.
Posted Feb 23, 2018 - 18:12 UTC