MySQL stopped when binary logs filled the disk: retention, purging and alerts

At 02:47 on a Sunday our order-processing workers started timing out on every write. Reads still worked, which made the first few minutes confusing: the storefront loaded, product pages rendered, but checkouts spun until they hit the 30-second application timeout. Twenty minutes later the MySQL primary stopped entirely. The error log on the primary showed two phases. First, writes blocked:

2025-09-14T02:47:19.304812Z 18842 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.004871' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.

Then, after an engineer killed a few long-running sessions to "unstick" things, the server aborted. The log said binary logging was not possible because an error occurred during the flush stage of the commit, and that because binlog_error_action was set to ABORT_SERVER, the server was being stopped.

df -h told us the obvious part:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    500G  500G     0 100% /var/lib/mysql

Reproducing it

The mechanism is simple to reproduce on a throwaway VM with a small volume: put the data directory on a 2 GB filesystem, enable binary logging (on by default in MySQL 8), and run a loop of large UPDATE statements. Once the filesystem fills, commits block with the same MY-000035 message, and killing the blocked session produces the abort.

What we ruled out

• Table data growth. du on the data directory showed the InnoDB tablespaces at 71 GB, roughly where they had been a month earlier.
• Temporary files from a large query. Nothing big in tmpdir, and the disk did not free itself when sessions ended.
• Slow query or general log left on. Both were off; the error log was a few megabytes.

The rest of the disk was binary logs:

ls -1 /var/lib/mysql/binlog.* | wc -l
du -ch /var/lib/mysql/binlog.0* | tail -n 1

4872
410G    total

Root cause

Two changes, months apart, combined. In April, after a reporting replica fell several days behind during a network partition and could not catch up because the source had already purged the files it needed, someone raised binlog_expire_logs_seconds from the 30-day default to 45 days "to give replicas room". At normal write volume that cost about 4 GB a day, so 180 GB, which fit. In September a data migration started backfilling a new JSON column on our 240-million-row order_items table in batches. With row-based logging and full row images, every updated row was written to the binary log with its entire before and after image. Binary log volume went from about 4 GB to roughly 21 GB a day. Disk usage climbed by 17 GB a day for ten days. Our disk alert existed, at 90%, but it was routed to a channel that had been muted during a noisy migration of the monitoring stack.

What is happening under the hood

The binary log records every change committed on the server so that replicas can apply it and so you can do point-in-time recovery. MySQL rotates binlog files when they reach max_binlog_size (1 GB by default) and removes old files automatically once they are older than binlog_expire_logs_seconds. Expiry is checked at startup and when the binary log is flushed or rotated. Nothing about expiry looks at free disk space.

With binlog_format=ROW (the default) and binlog_row_image=FULL (also the default), an UPDATE that touches one column of a wide row logs the full row twice: once as it was and once as it is. A backfill that touches every row of a large table therefore writes something close to twice the table size into the binlog, on top of normal traffic.

When the filesystem fills, MySQL does not crash immediately. For many write paths it waits and retries, logging the Disk is full message, which is why our writes hung instead of failing. If the binary log cannot be written at commit time, binlog_error_action decides what happens. The default, ABORT_SERVER, stops the server rather than continuing without a binary log, because continuing would silently break replication and point-in-time recovery. That is the right default, but it means a full disk ends in an outage either way.

Environment

Minimal reproduction

-- Watch binlog growth from a single large update.
SHOW BINARY LOGS;
UPDATE order_items SET attrs = JSON_OBJECT('legacy', TRUE) WHERE id BETWEEN 1 AND 2000000;
SHOW BINARY LOGS;

On our data, two million rows produced about 1.6 GB of new binary log.

Who is affected

Any MySQL or MariaDB server with binary logging enabled and binlogs on the same filesystem as data, especially when retention is long, row images are full, or a bulk job runs. Managed services handle some of this for you but have their own retention settings; check them rather than assuming.

The fix

1. Get the server running again without deleting files by hand. PURGE BINARY LOGS needs a running server, and deleting binlog files with rm desynchronises the binlog index file. Our volume was a cloud block device, so the fastest safe option was to grow it online and extend the filesystem:

# after resizing the volume from 500 GB to 800 GB in the cloud console
sudo xfs_growfs /var/lib/mysql
sudo systemctl start mysql

MySQL ran crash recovery and came up in about four minutes.

2. Find out what the replicas still need. On each replica:

SHOW REPLICA STATUS\G

We noted Source_Log_File (what the receiver thread is reading) and Relay_Source_Log_File (what the applier has executed up to). The reporting replica was the furthest behind; it had executed up to binlog.004702. We kept everything from that file on, so that even if the replica had to re-fetch its relay logs it would find what it needed.

3. Purge on the source, using the server, not the shell.

SHOW BINARY LOGS;
PURGE BINARY LOGS TO 'binlog.004702';

This freed 228 GB in a few seconds. MySQL also refuses to delete a file a connected replica is currently reading, but we did not want to rely on that alone, because a disconnected replica gets no such protection.

4. Fix retention to match reality. We set retention back to a value derived from what we actually need: our maximum tolerated replica lag plus margin, and the window covered by our backups for point-in-time recovery. Older history now lives in the backup system, not on the primary's disk.

SET PERSIST binlog_expire_logs_seconds = 604800;  -- 7 days

5. Separate binlogs from data. We moved binary logs to their own volume with log_bin pointing at it. A runaway binlog can now fill its own volume without taking the tablespaces and redo log down with it, and the alert for that volume is specific.

6. Make backfills cheaper to log. For the remaining backfill batches we set binlog_row_image=MINIMAL at session level, which logs only the columns needed to identify the row plus the changed columns. We kept the global default at FULL because some of our change-data-capture consumers expect full images.

Alternatives and trade-offs

• Shorter retention without a backup system that streams binlogs shrinks your point-in-time recovery window. Decide that explicitly with whoever owns recovery objectives.
• binlog_transaction_compression=ON compresses transaction payloads with zstd. It saved us about 60% in a test, at a small CPU cost; we turned it on after confirming our CDC tooling supports compressed payloads.
• Turning off binary logging is not an option for us: replicas and recovery depend on it.

How to test the fix

We checked the effective settings and a manual flush-triggered expiry:

SELECT @@binlog_expire_logs_seconds, @@log_bin_basename, @@binlog_transaction_compression;
FLUSH BINARY LOGS;
SHOW BINARY LOGS;

After the flush, files older than seven days disappeared from the list and from disk.

Confirming in production

• Binlog volume usage settled at about 31% of its 300 GB size during the rest of the backfill and dropped to 12% after it finished.
• Daily binlog growth went from about 21 GB to about 7 GB with minimal row images and compression on the backfill sessions.
• Both replicas stayed connected throughout. The reporting replica caught up within 40 minutes after the primary restarted.

Regression guard

• Disk alerts now fire on rate of growth as well as level: a forecast that the volume will be full within 72 hours pages on-call, independent of the absolute percentage.
• A dedicated alert tracks total binlog size from SHOW BINARY LOGS and warns if it exceeds 3x the trailing 30-day daily average times retention days.
• Our migration runbook requires an estimate of binlog volume for any backfill over 10 million rows, and the backfill tool pauses itself when free space on the binlog volume falls below 25%.
• Alert routing is checked weekly by a synthetic alert; a muted channel now shows up as a failed check.

Security implications

Binary logs contain full row data, including personal data and anything sensitive in the tables, so copying them off the server during an incident to "make room" creates an untracked copy of production data. We did not do that, and the runbook now says so explicitly. Shortening retention also reduces how long that data persists on the host. We enabled binlog_encryption for the new volume as part of the move. A full disk is also a denial-of-service condition: any user able to trigger very large writes can contribute to it, which is another reason for per-volume alerts and growth forecasting.

Prevention checklist

• Put binary logs on their own volume and alert on it separately.
• Derive binlog_expire_logs_seconds from replica lag tolerance and backup coverage, and write the reason down next to the setting.
• Always purge with PURGE BINARY LOGS after checking every replica, never with rm.
• Estimate binlog volume before backfills and consider MINIMAL row images for those sessions.
• Alert on disk growth rate, and test that alerts actually reach a human.

References

• MySQL 8.4 Reference Manual: PURGE BINARY LOGS Statement — https://dev.mysql.com/doc/refman/8.4/en/purge-binary-logs.html
• MySQL 8.4 Reference Manual: Binary Logging Options and Variables — https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html
• MySQL 8.4 Reference Manual: The Binary Log — https://dev.mysql.com/doc/refman/8.4/en/binary-log.html
• MySQL 8.4 Error Reference: Global error messages — https://dev.mysql.com/doc/mysql-errors/8.4/en/global-error-reference.html