Kafka on Docker

Published by Murali Basani on

In this post we setup Confluent Kafka, Zookeeper and Confluent Schema registry on docker and try to access from outside the container.

  1. Install docker and make sure you have access access to run docker commands like docker ps etc.
  2. Create a docker compose file(kafka_docker_compose.yml) like below which contains images, properties of Confluent kafka, Zookeeper and Schema registry
version: '3.3'
services:
  zookeeper-1:
    image: confluentinc/cp-zookeeper:5.3.3
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
    -   "22181:22181"
    networks: 
      - kafka
    extra_hosts:
      - "moby:127.0.0.1"

  kafka-1:
    image: confluentinc/cp-kafka:5.3.3
    hostname: localhost
    networks: 
      - kafka
    depends_on:
      - zookeeper-1
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:22181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1      
    ports:
    -   "19092:19092"
    -   "29092:29092"
    extra_hosts:
      - "moby:127.0.0.1"
      
  schema-registry-1:
    image: confluentinc/cp-schema-registry:5.3.3
    networks: 
      - kafka
    depends_on:
      - zookeeper-1
      - kafka-1
    environment:
      SCHEMA_REGISTRY_HOST_NAME: localhost
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper-1:22181
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:18081   
      SCHEMA_REGISTRY_DEBUG: "true"
    ports:
    -   "18081:18081"
    links:
      - zookeeper-1
      - kafka-1
    extra_hosts:
      - "moby:127.0.0.1"      
networks: 
    kafka:

Above file one instance each of those components. If you would like to add more, just copy paste those block, and increment the ids.

Start servers (start kafka, zookeeper and schema registry)

To run the docker compose file, run the below command where the above file is saved.

docker-compose -f .\kafka_docker_compose.yml up

Verify processes

docker ps

Connect urls of Kafka, Schema registry and Zookeeper

Kafka bootstrap servers :  localhost:29092
Zookeeper : zookeeper-1:22181
Schema Registry : http://localhost:18081

How to add more properties

If you know the property already from kafka documentation, then prefix the property with KAFKA, all in caps, and replace dots with _ (underscore)

Ex : number.partitions will be KAFKA_NUMBER_PARTITIONS

Connect to docker shell

docker exec -ti <docker_container_id> /bin/bash

Conclusion

So in this post we learnt how to run kafka, zookeeper and schema registry in a docker container and connect from out side container.


Leave a Reply

Your email address will not be published. Required fields are marked *