On March 24, 2022 at 14:45 UTC, a bug in our route distribution software resulted in a partial networking outage for NodeBalancers, elastic IP sharing, and routed IPv6 ranges within the Newark data center. Services affected by the issue experienced a total loss of networking at the host level due to missing routes.
At 14:51 UTC our infrastructure monitoring alerted our administrators to the outage. The routing issue was identified around 15:02 UTC and connectivity was restored as of 15:39 UTC. Following resolution the software bug was identified and patched, and we have not seen any further problems related to this bug.