Kafka consumer lag: the first alert to set up, and how to read it

If you run Kafka, one metric tells you most of what you need: consumer lag, the distance between the newest message in a partition and the last one a consumer group committed.

A big number isn’t automatically bad. A nightly backfill produces huge lag that drains on its own. A small number isn’t automatically fine either: a consumer stuck on one poison message can sit at lag 40 forever.

Alert on the trend

Two signals are more useful than a fixed threshold:

• Lag growing for N minutes while producers are healthy. The consumers aren’t keeping up.
• Committed offset not moving while lag is above zero. The consumer is stuck or dead.

In Prometheus terms, the second one is a rate on the committed offset compared with the log end offset:

increase(kafka_consumergroup_current_offset[10m]) == 0
  and kafka_consumergroup_lag > 0

Convert lag to time

“Two million messages behind” means nothing to a product owner. Estimate time lag: lag divided by the recent consume rate. “We are 14 minutes behind” is something you can put in an SLO.

What usually causes it

• A slow downstream call inside the consumer loop, not Kafka itself.
• A poison message that fails repeatedly, with no dead-letter topic.
• Too few partitions to scale consumers further.
• A rebalance storm from long processing between polls.

Fix the cause, then keep the alert. Pair it with a dead-letter topic and replayable, idempotent consumers, so a bad message costs you one record instead of an incident.