One RabbitMQ consumer hoarded 40,000 unacked messages while the others sat idle

Our image processing pipeline takes uploaded photos from a RabbitMQ queue, generates thumbnails, and writes them to object storage. A marketing campaign produced a spike of uploads, and the queue grew to about 45,000 messages. We scaled the consumer deployment from one replica to eight. The queue depth barely changed for the next half hour.

The RabbitMQ management UI showed why:

Queue: images.process
  Ready:     3,104
  Unacked:  41,922
  Total:    45,026
  Consumers: 8

Almost all messages were "unacked": delivered to a consumer but not yet acknowledged. And the per-consumer view showed where they were:

Consumer tag        Channel                       Prefetch  Unacked
amq.ctag-1A...      10.2.4.11:51520 (1)           0         41,880
amq.ctag-7F...      10.2.4.38:40114 (1)           0         6
amq.ctag-C2...      10.2.4.40:33852 (1)           0         7
...

The first replica, which had been running alone before we scaled out, held 41,880 messages. The seven new replicas had a handful each. Prefetch was 0, which in RabbitMQ means unlimited.

What the consumer code did

def on_message(channel, method, properties, body):
    image = json.loads(body)
    make_thumbnails(image)          # 200 ms to 2 s per image
    channel.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue="images.process", on_message_callback=on_message)
channel.start_consuming()

Manual acknowledgement after processing, which is correct. But no basic_qos call, so the channel had no prefetch limit.

How RabbitMQ pushes messages

RabbitMQ consumers using basic.consume receive messages by push. The broker sends messages to a consumer as fast as it can, until the number of unacknowledged messages on that channel reaches the prefetch count. With no limit, it sends everything available.

When the single consumer started, the queue had 45,000 messages and one consumer, so the broker pushed nearly all of them to it. They sat in that consumer's client-side buffer, already counted as delivered. The consumer processed them one by one, at about two per second. When new consumers joined, there was almost nothing left in the queue to give them. The messages they needed were stuck in the first consumer's memory.

Round-robin dispatch only works with messages still in the queue. Once delivered, a message stays with its consumer until it is acked, rejected, or the consumer's channel closes.

This is also why the problem looked so confusing from the outside. The queue metric most dashboards show first, total messages, was high, which suggested we simply needed more consumers. The more useful split, ready versus unacked, showed that the queue itself was nearly empty and the backlog lived inside one process. Adding replicas could never help while that process held the work.

Other costs of unlimited prefetch

• Memory. The first consumer's memory grew to 2.3 GB holding message bodies it would not touch for hours. Kubernetes eventually OOM-killed it.
• Redelivery storms. When that consumer died, its channel closed and all 41,880 unacked messages returned to the queue at once, marked as redelivered. Several had already been half processed.
• Head-of-line latency. A message delivered to a busy consumer waits for everything ahead of it in that consumer's buffer, even if another consumer is idle.

The opposite mistake: auto-ack

A common "fix" seen online is auto_ack=True. It makes the broker consider a message done as soon as it is sent. Distribution still goes wrong (prefetch does not apply to auto-ack consumers, so the broker pushes as fast as the network allows), and every message buffered in a consumer that crashes is lost. We kept manual acks.

Environment

1. Set a prefetch count

channel.basic_qos(prefetch_count=10)
channel.basic_consume(queue="images.process", on_message_callback=on_message)

With a prefetch of 10, each consumer holds at most ten unacked messages. The rest stay in the queue, where any consumer with capacity can take them. Scaling out now immediately shares the work.

How to choose the number: the prefetch needs to be large enough to keep the consumer busy while the next message travels from the broker, and small enough that messages do not wait in one consumer while others idle. A rough starting point is the processing time divided by the round-trip time to the broker, plus a little headroom. With 200 ms minimum processing and a 1 ms round trip, even a prefetch of 2 or 3 would keep the consumer busy. We chose 10 to smooth out variation, and measured.

For very fast consumers (sub-millisecond work, such as routing or counting), higher prefetch values like 100 to 300 improve throughput significantly. The RabbitMQ documentation suggests values in the 100 to 300 range as a starting point for throughput-focused consumers. Slow, variable consumers like ours want small values.

2. Keep acking after the work, and handle failures explicitly

def on_message(channel, method, properties, body):
    try:
        image = json.loads(body)
        make_thumbnails(image)
    except InvalidImage as exc:
        log.warning("rejecting %s: %s", properties.message_id, exc)
        channel.basic_reject(delivery_tag=method.delivery_tag, requeue=False)  # to DLX
        return
    except Exception:
        log.exception("transient failure, requeueing %s", properties.message_id)
        channel.basic_nack(delivery_tag=method.delivery_tag, requeue=True)
        return
    channel.basic_ack(delivery_tag=method.delivery_tag)

requeue=False sends the message to the queue's dead-letter exchange, configured with a policy, so a broken image does not cycle forever. Thumbnail generation is idempotent (it overwrites the same object keys), so a redelivery after a crash is harmless.

3. Consider quorum queues for delivery limits

We later moved the queue to a quorum queue. Quorum queues track redeliveries and support a delivery-limit, after which a message is dead-lettered. That protects against a message that crashes the consumer process itself, where our try block never gets a chance to reject it.

4. Scale on the right signal

Our autoscaler had been scaling on CPU. With unlimited prefetch, new replicas had no work and low CPU, so the autoscaler concluded more replicas were not needed. It now scales on queue depth (ready messages) per consumer, using a metrics exporter for RabbitMQ.

1. Distribution under load. In staging, we filled the queue with 50,000 messages, started one consumer, waited a minute, then scaled to eight:

With prefetch 10, the management UI showed each of the eight consumers holding exactly ten unacked messages, and the ready count falling steadily.

2. Memory. Consumer memory stayed under 180 MB throughout, compared with 2.3 GB before the OOM kill.

3. Crash test. We killed a consumer mid-batch. Ten messages returned to the queue, were redelivered to other consumers, and processed once more without errors.

4. Guardrail. Our shared consumer library now refuses to start a consumer without an explicit prefetch value, and a dashboard panel shows unacked messages per consumer so imbalance is visible at a glance.

5. Security implications. Large unacked buffers also meant thousands of user-uploaded images sat in a consumer's memory longer than needed. Smaller buffers reduce how much user data any one crashed or compromised worker holds at a time.

Takeaways

• Without basic.qos, RabbitMQ pushes every available message to the first consumer.
• Delivered but unacked messages stay with that consumer; new consumers cannot take them.
• Set a prefetch count sized to processing time, and ack after the work is done.
• Use dead-lettering and delivery limits for messages that keep failing, and scale on queue depth.

References: RabbitMQ consumer prefetch, RabbitMQ consumer acknowledgements, RabbitMQ dead letter exchanges, RabbitMQ quorum queues.