Starting around 15:45 UTC on May 17, 2023, a change was pushed to a backend service on the Linode platform that as a result prevented customers from being able to provision or fully delete Managed Databases on the platform. The errors presented as customers receiving errors in attempting to provision a new cluster, or when existing clusters were deleted, the Managed Database cluster would disappear from Cloud Manager and become inaccessible, however the cluster data itself would not be immediately deleted.

This issue was first detected via tickets to the Linode Support team. At 17:24 UTC, the ability for customers to provision new Managed Database clusters was restored. At 20:00 UTC, the ability to fully delete Managed Database clusters was restored, and the remaining delete jobs that were entered into the queue finished, completely deleting the data.

The root cause of this incident was a hotfix released by a team to a backend service that was not expected to have a direct impact related to the Managed Database service. An additional hotfix to the code was released to address this immediate issue.

To help prevent future recurrences of similar incidents, we are seeking to implement greater communication between the related teams and generate tooling and policies to test changes in these areas more frequently.