Data Engineering
Bug Report
Kafka consumer group stuck in a rebalance loop: max.poll.interval.ms and slow processing
Our enrichment consumers rebalanced every few minutes, lag climbed past two million messages and some records were processed twice. A slow external API pushed batch processing past max.poll.interval.ms. How we proved it, fixed the poll loop and made rebalances cheap.