Service Issue - Cloud Manager and API in Toronto
Incident Report for Linode
Postmortem

On October 6, 2024, at approximately 11:33 UTC, our administrators received multiple alerts notifying us of database connectivity issues in Toronto. During the impact window, customers were intermittently unable to create new services, delete services, or reboot their Linodes in our CA-Central (Toronto) data center.

Investigation revealed that a faulty router was causing the issue, and the device was promptly taken offline at 17:15 UTC. This quickly mitigated the impact on customers.

After addressing the immediate issue, we initiated the process of replacing the faulty router to restore capacity and redundancy. We conducted an emergency network maintenance at our CA-Central (Toronto) data center from October 7, 2024, 22:00 UTC to October 8, 2024, 02:00 UTC, to install the new router. After encountering some unexpected complications with this process, an additional round of maintenance was required. This second maintenance started on October 8, 2024, at 22:00 UTC and was successfully completed on October 9, 2024, at 00:15 UTC.

This summary provides an overview of our current understanding of the incident given the information available. Our investigation is ongoing and any information herein is subject to change.

Posted Oct 10, 2024 - 18:27 UTC

Resolved
We haven't observed any additional issues with the Cloud Manager or API, and will now consider this incident resolved. If you continue to experience issues, please contact us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com for assistance.
Posted Oct 06, 2024 - 18:22 UTC
Monitoring
At this time we have been able to correct the issue affecting the Cloud Manager and API. We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues and unable to open a Support ticket, please call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Posted Oct 06, 2024 - 15:56 UTC
Update
We are continuing to investigate the issue. We will share additional updates as we have more information.
Posted Oct 06, 2024 - 14:50 UTC
Investigating
Our team is investigating a service issue that affects the Linode Cloud Manager and API. During that time, some users may experience issues when attempting to take any actions like creating and/or deleting any service, rebooting linodes, etc. We will share additional updates as we have more information.
Posted Oct 06, 2024 - 14:08 UTC
This incident affected: Linode Manager and API.