How we built zero-downtime API key rotation at 50B requests/month

API key rotation is one of those problems that seems trivial on paper but becomes incredibly complex at scale. When you're handling 50 billion requests per month, every millisecond of downtime translates to thousands of failed requests and frustrated customers.

In this post, we walk through the architecture we built to rotate API keys with zero observable downtime, the mistakes we made along the way (there were several), and what we would do differently if we had to rebuild it today.

The key insight was separating the concept of a key's identity from its secret material. By maintaining a keyring of active secrets and using a gradual rollout strategy, we were able to rotate keys across our entire fleet without any single coordinated cutover.