The post-incident report reads like a thriller. A routine change to routing policy triggered a feedback loop, control-plane services lost quorum and the dashboards engineers needed were hosted on the failing platform. Recovery meant finding people with physical access.
What is changing afterwards
Out-of-band management that does not depend on the production network, staged rollouts with automatic rollback and a rule that monitoring must live somewhere else. None of this is novel; all of it is expensive until the day it is not.
Key takeaways
- Control-plane dependencies caused the cascade
- Monitoring lived on the failing platform
- Out-of-band access is now mandatory
This story is part of our ongoing coverage of cloud & infrastructure. We will update it as more details emerge.
