P0 INCIDENTIstio mTLS Breaks Service After Policy Change#ENG-WAR-058GoogleLyft
Senior~25 min15:00
API CPU Usage99.2%↑ 42%
P99 Latency2450 ms↑ 400%
5xx Error Rate12.4%↑ 12%
DB Connections14,492↑ 800%
bastion-prod-1.internal — bash
[SYSTEM] War-Room terminal initialised. Bastion host connection established.
[SYSTEM] Active incident: Istio mTLS Breaks Service After Policy Change
[SYSTEM] Type "help" for a list of investigation commands.
user@bastion:~$
Execute Remediation⚠ PROD
Your security team changed the Istio PeerAuthentication policy from PERMISSIVE to STRICT across all namespaces to enforce mTLS. Within minutes, your legacy batch job service (which doesn't have Envoy sidecar injection enabled — it has sidecar.istio.io/inject: "false") started failing all outbound HTTP calls to other microservices with: TLS handshake error from remote address. The batch job is stuck.

What is your first action?