In Kafka, we can easily reset the consumer group offset to any value we want. It allows us to reprocess some messages or skip the messages we don’t want to process. All we need is the kafka-consumer-groups.sh.
For example, if I want to reset the offset of the topic my_topic accessed by the consumer group called the_consumers to the earliest available offset, I have to run the following command:
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
the_consumers --reset-offsets --to-earliest --topic my_topic --execute
Note that I must add the --execute parameter at the end of the command. Otherwise, it will print the result of the operation, but it will not perform it!
There are a few more options regarding resetting the offsets. In my opinion, the most useful ones are:
- to the latest, which makes the consumers skip all of the existing messages and start with the last one added to the topic:
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
the_consumers --reset-offsets --to-latest --topic my_topic --execute
- to datetime, which sets the offset to the offset of the first message added to the topic after the datetime:
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
the_consumers --reset-offsets --to-datetime 2020-12-20T00:00:00.000 --topic my_topic --execute
Note that it uses the date in the ISO8601 format.
Want to build AI systems that actually work?
Download my expert-crafted GenAI Transformation Guide for Data Teams and discover how to properly measure AI performance, set up guardrails, and continuously improve your AI solutions like the pros.