koff

Kafka consumer offset tracking tool. Parsing an internal binary topic that should have been public and in plain text

Go to file

Gregory Eremin 693c9765e8 Add readme		2018-07-29 19:06:25 +02:00
cmd	Initial commit	2018-07-29 16:29:04 +02:00
LICENSE	Initial commit	2018-07-29 16:29:04 +02:00
README.md	Add readme	2018-07-29 19:06:25 +02:00
consumer.go	Add context for logging and cancellation	2018-07-29 18:53:36 +02:00
decoder.go	Add context for logging and cancellation	2018-07-29 18:53:36 +02:00

README.md

Koff

Kafka consumer (group) offset tracker.

WIP

Work in progress, there's plenty of stuff left to figure out. You have been warned.

Usage

So far it is only cable of printing parsed messages. For usage eample take a look at the main command.

go run cmd/main.go -brokers 127.0.0.1:9092

Design

Starting with Kafka version 0.9 consumer offsets are stored and managed by the Kafka server. Internally offsets are stored in the __consumer_offsets topic. It is not designed to be used by third party software but nothing stops us from doing that really.

This topic is not only used to store individual consumer offsets, it also contains consumer group metadata: list of group members and their subscriptions and assignments, leader details and plenty of other things. Given that the topic provides realtime updates on consumer offsets and consumer group structure and state, it makes it a very convenient foundation for consumer or group tracking and monitoring.

Licence

MIT