Our MySQL primary hung and then aborted with the data volume at 100%. Binary logs had grown to 410 GB because a backfill tripled write volume while retention had been raised for a lagging replica. How we recovered, purged safely and stopped it recurring.
A one-second ALTER TABLE ... ALGORITHM=INSTANT hung for 11 minutes and every query on the table queued behind it. The blocker was a reporting job holding an open transaction. How we found it with performance_schema and made migrations fail fast.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
A slow accounting API pushed our PayPal webhook handler past the nginx timeout. PayPal saw a 504 and retried, while the first request was still running, and 41 wallets were topped up twice. Postmortem plus the idempotent design we moved to.
Our job board listing took 1.8 seconds and ran 312 queries per page. Blade was lazy loading company, tags and application counts for every row. We fixed it with with(), withCount() and made lazy loading throw in development.
Stock updates from supplier feeds started failing with Deadlock found when trying to get lock during peak hours. Concurrent upsert batches touched the same unique keys in different orders under InnoDB next-key locking. Sorting, smaller batches and a retry fixed it.
Product reviews containing emoji failed with error 1366 after we upgraded MySQL to 8.0, while a nightly job quietly turned them into question marks. Legacy utf8mb3 tables, a wrong connection charset and an index length limit were all involved.