On April 22, 2024 at approximately 14:05 UTC, our team noticed new LKE clusters failing to provision in the Toronto data center, and began to investigate. At 17:32 UTC our team was able to narrow the issue down to a specific section of infrastructure in the Toronto data center.
On April 23, 2024 at 15:03 UTC, the problematic infrastructure in question was disabled from provisioning new LKE clusters while our team continued investigating the root cause.
On April 25, 2024 at approximately 16:45 UTC, a fix was implemented to address the root cause determined during our investigation: A previous API update that did not work as intended, leading to timeouts when attempting to provision new LKE clusters. By 18:05 UTC the ability to create new LKE clusters was confirmed to be fully restored.
After a period of monitoring for further issues, the incident was considered resolved on April 26, 2024 at 17:09 UTC.