Kafka on Docker

In this post we setup Confluent Kafka, Zookeeper and Confluent Schema registry on docker and try to access from outside the container. Install docker and make sure you have access access to run docker commands like docker ps etc. Create a docker compose file(kafka_docker_compose.yml) like below which contains images, properties Read more…

Reset Kafka offsets

Status of offsets of consumer groups are stored on Kafka broker. If that should be reset, for ex based on a duration ./bin/kafka-consumer-groups –bootstrap-server kafkaserver:9092 –group consumergroupname –reset-offsets –by-duration 1522760000 –execute If that should be reset, for ex based on a timestamp ./bin/kafka-consumer-groups –bootstrap-server kafkaserver:9092 –group consumergroupname –reset-offsets –to-datetime 2019-08-28T19:45:00.000 Read more…

Zookeeper quorum

Zookeeper plays the role of coordinating with servers in fail over scenarios, stores naming and configuration information, meta info. For more info visit : https://zookeeper.apache.org/ Kafka needs Zookeeper to maintain its state and store topics, acls cluster information. To maintain high availability, it is required maintain a quorum – to Read more…

Commands for SchemaRegistry

With curl commands, schemas can be retrieved/updated to Confluent SchemaRegistry. Get the latest schema available on the topic: curl -X GET http://localhost:8081/subjects/testtopic-value/versions/latest Get Schema compatibility: curl -X GET http://localhost:8081/config/testtopic-value Get versions of all registered schemas on a topic: curl -X GET http://localhost:8081/subjects/testtopic-value/versions [1,2,3] List all subjects – registered schemas on Read more…