How to reset the consumer offset in Apache Kafka topic

In Kafka, we can easily reset the consumer group offset to any value we want. It allows us to reprocess some messages or skip the messages we don’t want to process. All we need is the kafka-consumer-groups.sh.

For example, if I want to reset the offset of the topic my_topic accessed by the consumer group called the_consumers to the earliest available offset, I have to run the following command:

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-earliest --topic my_topic --execute

Note that I must add the --execute parameter at the end of the command. Otherwise, it will print the result of the operation, but it will not perform it!

There are a few more options regarding resetting the offsets. In my opinion, the most useful ones are:

  • to the latest, which makes the consumers skip all of the existing messages and start with the last one added to the topic:
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-latest --topic my_topic --execute
  • to datetime, which sets the offset to the offset of the first message added to the topic after the datetime:
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    the_consumers --reset-offsets --to-datetime 2020-12-20T00:00:00.000 --topic my_topic --execute

Note that it uses the date in the ISO8601 format.

Older post

How to purge a Kafka topic

How to remove all messages from a Kafka topic

Newer post

How to add dependencies as jar files or Python scripts to PySpark

How to add a jar file or a Python file as a Pyspark dependency

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>