P0 INCIDENTDual-Write Inconsistency During Database Migration#ENG-WAR-056FacebookAirbnb
Senior~35 min15:00
API CPU Usage99.2%↑ 42%
P99 Latency2450 ms↑ 400%
5xx Error Rate12.4%↑ 12%
DB Connections14,492↑ 800%
bastion-prod-1.internal — bash
[SYSTEM] War-Room terminal initialised. Bastion host connection established.
[SYSTEM] Active incident: Dual-Write Inconsistency During Database Migration
[SYSTEM] Type "help" for a list of investigation commands.
user@bastion:~$
Execute Remediation⚠ PROD
You're migrating from MySQL to Cassandra. During the dual-write phase, writes are sent to both MySQL and Cassandra. A Cassandra network partition caused 2 hours of write failures — all writes during this period went only to MySQL. The migration team didn't detect this. Now, Cassandra is missing 2 hours of data. If you cut over to Cassandra, users will see missing records. You cannot afford extended downtime for a full re-sync.

What is your first action?