K A F K A W I Z E

Loading

In this post we setup Confluent Kafka, Zookeeper and Confluent Schema registry on docker and try to access from outside the container.

  1. Install docker and make sure you have access access to run docker commands like docker ps etc.
  2. Create a docker compose file(kafka_docker_compose.yml) like below which contains images, properties
  3. of Confluent kafka, Zookeeper and Schema registry

version: '3.3'
services:
  zookeeper-1:
    image: confluentinc/cp-zookeeper:5.3.3
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
    -   "22181:22181"
    networks: 
      - kafka
    extra_hosts:
      - "moby:127.0.0.1"

  kafka-1:
    image: confluentinc/cp-kafka:5.3.3
    hostname: localhost
    networks: 
      - kafka
    depends_on:
      - zookeeper-1
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:22181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1      
    ports:
    -   "19092:19092"
    -   "29092:29092"
    extra_hosts:
      - "moby:127.0.0.1"
      
  schema-registry-1:
    image: confluentinc/cp-schema-registry:5.3.3
    networks: 
      - kafka
    depends_on:
      - zookeeper-1
      - kafka-1
    environment:
      SCHEMA_REGISTRY_HOST_NAME: localhost
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper-1:22181
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:18081   
      SCHEMA_REGISTRY_DEBUG: "true"
    ports:
    -   "18081:18081"
    links:
      - zookeeper-1
      - kafka-1
    extra_hosts:
      - "moby:127.0.0.1"      
networks: 
    kafka:

Above file one instance each of those components. If you would like to add more, just copy paste those block, and increment the ids.

Start servers (start kafka, zookeeper and schema registry)

To run the docker compose file, run the below command where the above file is saved.

docker-compose -f .\kafka_docker_compose.yml up

 

 


Verify processes


 
docker ps

Connect urls of Kafka, Schema registry and Zookeeper


 

Kafka bootstrap servers :  localhost:29092
Zookeeper : zookeeper-1:22181
Schema Registry : http://localhost:18081


 

How to add more properties


If you know the property already from kafka documentation, then prefix the property with KAFKA, all in caps, and replace dots with _ (underscore)

Ex : number.partitions will be KAFKA_NUMBER_PARTITIONS


 

Connect to docker shell


 

docker exec -ti <docker_container_id> /bin/bash

Conclusion

So in this post we learnt how to run kafka, zookeeper and schema registry in a docker container and connect from out side container.

Automate processes around your Kafka clusters with workflows, reduced risk full audit, governance.

About Us

Follow Us

Email: info@klaw-project.io

© 2021 Kafkawize Technology Solutions. All rights reserved. Registered company in The Netherlands.