Kafka consumer group stuck in a rebalance loop: max.poll.interval.ms and slow processing

Our order enrichment service consumes the orders.created topic (48 partitions), calls an internal pricing API and a third-party tax API for each order, and writes enriched records to another topic. On a Monday after a promotion launch, consumer lag climbed from near zero to over 2.1 million messages in three hours. The consumer group's state flapped between Stable and PreparingRebalance every few minutes, and the application logs on every instance were full of the same warning:

WARN [Consumer clientId=enricher-7, groupId=order-enricher] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

followed shortly by commit failures:

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.

Downstream, the finance team noticed duplicate enriched records: the same order ID appearing two or three times.

Reproducing it

In staging we pointed the consumer at a stub tax API that responds in 700 ms instead of the usual 80 ms. With the default max.poll.records of 500 and sequential processing, one batch took about 500 x (0.7 + 0.05) seconds, roughly 375 seconds, which is longer than the default max.poll.interval.ms of 300,000 ms. The first rebalance happened about five minutes after start, every time.

What we ruled out

• Broker problems. Broker CPU, request latency and ISR counts were normal, and other consumer groups on the same cluster were healthy.
• Heartbeat or session timeout. session.timeout.ms failures look different: the coordinator removes the member because heartbeats stop. Our heartbeats were fine; the warning was explicitly about the poll interval.
• Network partitions between pods and brokers. No connection errors, and the same pods produced to other topics without trouble.

Root cause

The tax provider had degraded during the promotion: p50 latency rose from 80 ms to about 650 ms. Our consumer processed each polled batch sequentially, calling the API for every record, before calling poll() again. When a batch took longer than max.poll.interval.ms, the consumer's background thread proactively left the group. The group rebalanced, partitions moved to other instances, which were in the same state, and the cycle repeated. Each time, the batch in progress could not commit its offsets, so whichever consumer took over the partition reprocessed records that had already been enriched and produced. That is where the duplicates came from.

What is happening under the hood

A Kafka consumer has two liveness checks:

The second check exists so that a consumer stuck in processing does not hold partitions forever. If poll() is not called within the interval, the client leaves the group and the coordinator triggers a rebalance. With the classic eager protocol, a rebalance revokes all partitions from all members before reassigning them, so every rebalance pauses the whole group. The default assignment strategy list in the Java client includes the cooperative sticky assignor, which only moves partitions that need to move, but a group only uses it once all members agree on it.

max.poll.records (default 500) caps how many records one poll() returns. It does not change fetch sizes; it just bounds how much work the application takes on between polls. The worst case time between polls is therefore roughly max.poll.records times the slowest per-record processing time, and that is the number to compare with max.poll.interval.ms.

Environment

Minimal reproduction

// Excerpt: the original poll loop.
while (running) {
    ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, Order> r : records) {
        Enriched e = enrich(r.value());          // pricing + tax API, ~700 ms when tax is slow
        producer.send(new ProducerRecord<>("orders.enriched", r.key(), e));
    }
    consumer.commitSync();                       // fails after a rebalance kicked us out
}

Who is affected

Any consumer that does slow or variable-latency work inside the poll loop: external HTTP calls, database writes with lock waits, large file processing, or ML inference. It shows up when a dependency slows down, after increasing batch sizes, or after a deploy that adds a new call per record.

The fix

We changed four things, from quickest to most structural.

1. Bound the batch by time, not by hope. We lowered max.poll.records to 50 and set an explicit HTTP timeout and circuit breaker on the tax client (2 seconds per call). The worst case batch is now about 50 x 2 seconds = 100 seconds, comfortably inside the poll interval, and normal batches take a few seconds.

max.poll.records=50
max.poll.interval.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id=${POD_NAME}-${THREAD_ID}
session.timeout.ms=45000
enable.auto.commit=false

We deliberately did not just raise max.poll.interval.ms to an hour. That would have hidden the problem and made genuinely stuck consumers hold partitions for an hour.

2. Process records concurrently within a batch, but per key in order. Records for the same order key must be processed in order, but different keys can run in parallel. We group the batch by key and process groups on a bounded executor, then wait for all of them before committing:

Map<String, List<ConsumerRecord<String, Order>>> byKey = new LinkedHashMap<>();
for (ConsumerRecord<String, Order> r : records) {
    byKey.computeIfAbsent(r.key(), k -> new ArrayList<>()).add(r);
}
List<CompletableFuture<Void>> futures = byKey.values().stream()
    .map(group -> CompletableFuture.runAsync(() -> group.forEach(this::enrichAndSend), executor))
    .toList();
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
producer.flush();
consumer.commitSync();

With 16 threads per consumer, a 50-record batch completes in roughly the time of its slowest few calls.

3. Pause partitions instead of blocking when a dependency is down. When the tax client's circuit breaker is open, we call consumer.pause(consumer.assignment()), keep calling poll() (which returns nothing for paused partitions but keeps the member alive), and resume() when the breaker closes. The group stays stable while the dependency recovers, instead of rebalancing in a loop.

4. Make rebalances cheaper and output idempotent. The cooperative sticky assignor means a rebalance no longer stops the whole group, and static membership (group.instance.id) means a pod restart within the session timeout does not trigger a rebalance at all. To handle the duplicates that any at-least-once consumer can produce, the enriched topic is keyed by order ID and the downstream sink upserts by order ID and source offset.

To move the group from eager to cooperative, we followed the documented two-step rolling upgrade: first deploy with both assignors listed (cooperative first), then a second rolling deploy with only the cooperative assignor.

Alternatives and trade-offs

• Raise max.poll.interval.ms only. Acceptable when processing time is long but predictable, such as a batch job. Not for variable external latency.
• Hand records to a separate worker pool and commit asynchronously. Highest throughput, but you must track completed offsets per partition yourself and commit only contiguous ranges. We considered the Confluent parallel consumer library for this; our simpler per-batch approach was enough.
• Kafka Streams handles poll loop mechanics for you but has its own version of the same constraint for slow processors, and would have been a larger rewrite.
• The new consumer group protocol (KIP-848) moves assignment to the broker and avoids group-wide synchronisation barriers. It was available as early access in Kafka 3.7 and later when we did this, so we did not use it in production yet.

How to test the fix

The staging test with the slow tax stub now runs as an automated soak test for 30 minutes:

./gradlew :enricher:soakTest -DtaxStubLatencyMs=700 -DdurationMinutes=30

It asserts zero rebalances after the initial join (read from the consumer's rebalance-total metric on the coordinator metrics group), zero CommitFailedException, and that lag returns to zero after a burst of 200,000 messages.

Confirming in production

• The consumer group has stayed Stable since the deploy; rebalances now happen only on scale-out events, about 3 per week, each completing in under 2 seconds.
• During a later tax API slowdown (p50 of 900 ms for 40 minutes), the circuit breaker paused partitions, lag rose to 180,000, and drained within 12 minutes of recovery with no rebalances.
• Duplicate enriched records dropped to zero at the sink thanks to the upsert, and the producer-side duplicate rate (measured by order ID per hour) fell from 4.1% during the incident to below 0.01%.

Regression guard

• Alerts on the consumer metrics rebalance-rate-per-hour and on the age of the last successful poll() per consumer, which shows trouble before the interval is exceeded.
• A dashboard panel plots p99 batch processing time against max.poll.interval.ms so the margin is visible.
• The soak test runs in CI on any change to the enricher or its HTTP clients.

Security implications

None beyond availability. A slow or malicious upstream dependency could previously stall our whole consumer group, which was effectively a denial-of-service path through a third party; timeouts, circuit breakers and pausing remove it. Duplicate records had financial impact rather than security impact, and idempotent writes at the sink now protect against them regardless of cause.

Prevention checklist

• Keep max.poll.records times worst-case per-record time well under max.poll.interval.ms.
• Put strict timeouts and circuit breakers on every external call inside the poll loop.
• Use pause() and resume() to wait out dependencies without leaving the group.
• Prefer the cooperative sticky assignor and static membership for long-running consumers.
• Assume at-least-once delivery and make downstream writes idempotent.

References

• Apache Kafka 3.9 documentation: Consumer configs — https://kafka.apache.org/39/documentation.html
• Apache Kafka Javadoc: KafkaConsumer (poll, pause, resume) — https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
• KIP-429: Kafka Consumer Incremental Rebalance Protocol — https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
• KIP-345: Introduce static membership protocol to reduce consumer rebalances — https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
• KIP-848: The Next Generation of the Consumer Rebalance Protocol — https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol