Service Issue: Lassie

Incident Report for Linode

Postmortem

On April 20th, 2022 an update to our hypervisors prevented Lassie reboots from being issued to customer Linodes. At around 00:46 UTC we began receiving reports of Linodes that were not being rebooted by our kernel watchdog service (Lassie). The issue was brought to the attention of our administrators and a fix was implemented at approximately 04:30 UTC.

The issue was initially resolved by rolling back the previous update across our fleet. Following the roll back, administrators identified and fixed the bug that caused this issue. We’ve now added checks to test for similar issues in future updates, and we have not observed any related problems following resolution of this incident.

Posted Apr 28, 2022 - 20:00 UTC

Resolved

We haven’t observed any additional issues affecting the Lassie service, and now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.

Posted Apr 20, 2022 - 06:35 UTC

Monitoring

At this time we have been able to correct the issue affecting the Lassie service. We will be monitoring this to ensure that it remains stable.

Posted Apr 20, 2022 - 05:22 UTC

Identified

Our team has identified the issue affecting the Lassie service. We are working to implement a fix and will provide an update as soon as the solution is in place.

Posted Apr 20, 2022 - 04:21 UTC

Investigating

We are currently investigating an issue with Lassie, our reboot helper. At this time if your Linode experiences an unexpected shutdown, it will need to be manually resolved within Cloud Manager or Linode CLI.

Posted Apr 20, 2022 - 03:45 UTC

This incident affected: Regions (US-East (Newark), US-Central (Dallas), US-West (Fremont), US-Southeast (Atlanta), CA-Central (Toronto), EU-West (London), EU-Central (Frankfurt), AP-South (Singapore), AP-Northeast-2 (Tokyo 2), AP-West (Mumbai), AP-Southeast (Sydney)).