Operations / Incident reviews
2026-09-28 · API latency spike after a deploy (#4)
Published from the team's review in InfraNest. Change the review there: publishing again updates this page.
Open the incident in InfraNest
Summary
- What failed: API health
- Impact: Major
- Started: 2026-09-28 14:54 UTC
- Resolved: 2026-09-28 15:16 UTC
- Lasted: 22 minutes
- Nobody acknowledged it.
What happened
API response times climbed from about 120 ms to over 4 seconds eight minutes after release 2.31 reached production. Sam rolled the release back, and latency was normal again 22 minutes after it started.
Impact
For 22 minutes most API requests took several seconds and about 6% timed out. Checkout and the dashboard were slow but stayed up; no data was lost.
Why it happened
The release included a migration that added an index without a lock timeout. On the primary database it waited behind long-running reads and held its connections open, so the pool ran out and requests queued behind it.
What we change
- Run schema migrations with a lock timeout, so a slow one fails instead of holding the primary database.
- Alert when the connection pool on db-01 passes 80%, before latency reaches users.
- Add a pre-deploy check that flags migrations expected to run longer than a minute.