P0 INCIDENTLet's Encrypt Rate Limit Blocks Certificate Renewal#ENG-WAR-020VercelNetlify
Junior~20 min15:00
API CPU Usage99.2%↑ 42%
P99 Latency2450 ms↑ 400%
5xx Error Rate12.4%↑ 12%
DB Connections14,492↑ 800%
bastion-prod-1.internal — bash
[SYSTEM] War-Room terminal initialised. Bastion host connection established.
[SYSTEM] Active incident: Let's Encrypt Rate Limit Blocks Certificate Renewal
[SYSTEM] Type "help" for a list of investigation commands.
user@bastion:~$
Execute Remediation⚠ PROD
Your platform provisions Let's Encrypt certificates automatically for customer custom domains. An engineer deployed a bug that caused the certificate provisioning service to retry failed ACME requests in a tight loop (no exponential backoff). Within 4 hours, the service burned through all 50 certificates for your wildcard domain for the week. New customers adding custom domains get HTTPS errors. Existing customers are unaffected.

What is your first action?