1
0
Fork 0
Kafka consumer offset tracking tool. Parsing an internal binary topic that should have been public and in plain text
Go to file
Gregory Eremin 693c9765e8
Add readme
2018-07-29 19:06:25 +02:00
cmd Initial commit 2018-07-29 16:29:04 +02:00
LICENSE Initial commit 2018-07-29 16:29:04 +02:00
README.md Add readme 2018-07-29 19:06:25 +02:00
consumer.go Add context for logging and cancellation 2018-07-29 18:53:36 +02:00
decoder.go Add context for logging and cancellation 2018-07-29 18:53:36 +02:00

README.md

Koff

Kafka consumer (group) offset tracker.

WIP

Work in progress, there's plenty of stuff left to figure out. You have been warned.

Usage

So far it is only cable of printing parsed messages. For usage eample take a look at the main command.

go run cmd/main.go -brokers 127.0.0.1:9092

Design

Starting with Kafka version 0.9 consumer offsets are stored and managed by the Kafka server. Internally offsets are stored in the __consumer_offsets topic. It is not designed to be used by third party software but nothing stops us from doing that really.

This topic is not only used to store individual consumer offsets, it also contains consumer group metadata: list of group members and their subscriptions and assignments, leader details and plenty of other things. Given that the topic provides realtime updates on consumer offsets and consumer group structure and state, it makes it a very convenient foundation for consumer or group tracking and monitoring.

Licence

MIT