DHCP issue - Dallas, Fremont, Atlanta, Newark, Toronto
Incident Report for Linode
Postmortem

At 1:23 AM EDT our team was alerted to connectivity issues affecting services for a subset of customers in our Dallas, Fremont, Atlanta, Newark and Toronto data centers. An investigation determined that invalid DHCP lease configurations were the issue, and only customers using DHCP were affected. Additional resources were engaged to work on a fix. This work continued until approximately 10:45 AM EDT, when the fix was merged and DHCP configurations were regenerated, restoring service to all Linode infrastructure.

This incident was triggered by a series of changes introduced in preparation for Linode’s next data center launch. The latest deployment did not include a necessary update to our codebase which is partly responsible for the generation of our DHCP configurations. This resulted in the service that generates DHCP configurations to fail due to the error not being properly handled, which prevented Linodes from renewing their leases. We mitigated the event by adding the necessary changes to our codebase.

To prevent this from occurring again, additional monitoring has been put in place to alert us if the service that generates our DHCP configurations fails. Our team’s runbooks have been updated with additional preventative measures and tooling has been updated to better handle upcoming changes to Linode’s infrastructure.

This DHCP issue was not related to the other ongoing network incident affecting Linode’s London data center.

Thank you for your patience as we worked to resolve this. If you are still experiencing issues, enabling network helper and rebooting your Linode should resolve this issue for most customers. Please open a Support ticket if you need any further assistance.

Posted Nov 04, 2022 - 22:23 UTC

Resolved
We haven’t observed any additional connectivity issues in our Dallas, Fremont, Atlanta, Newark and Toronto data centers, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Posted Nov 04, 2022 - 15:02 UTC
Monitoring
At this time we have been able to correct the issues affecting connectivity in our Dallas, Fremont, Atlanta, Newark and Toronto data centers. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Posted Nov 04, 2022 - 14:59 UTC
Identified
Our team has identified the issue affecting connectivity in our Dallas, Fremont, Atlanta, Newark and Toronto data center. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Posted Nov 04, 2022 - 14:54 UTC
Update
Our team is still working to correct the DHCP issues affecting Linodes in Dallas, Fremont, Atlanta, Newark, and Toronto. For most customers, enabling network helper and rebooting your Linode should resolve this issue.
Posted Nov 04, 2022 - 12:52 UTC
Update
We have identified issues affecting the DHCP service in our Dallas, Fremont, Atlanta, Newark, and Toronto data centers. Our team is working as quickly as possible to have the service fully restored. We will provide additional updates as the situation develops.
Posted Nov 04, 2022 - 09:33 UTC
Investigating
We have identified issues affecting the DHCP service in our Dallas, Fremont, Atlanta, and Newark data centers. Our team is working as quickly as possible to have the service fully restored. We will provide additional updates as the situation develops.
Posted Nov 04, 2022 - 09:18 UTC
This incident affected: Regions (US-East (Newark), US-Central (Dallas), US-West (Fremont), US-Southeast (Atlanta), CA-Central (Toronto)) and Linode Manager and API.