In this post we setup Confluent Kafka, Zookeeper and Confluent Schema registry on docker and try to access from outside the container.
Install docker and make sure you have access access to run docker commands like docker ps etc.
Create a docker compose file(kafka_docker_compose.yml) like below which contains images, properties
of Confluent kafka, Zookeeper and Schema registry
version: '3.3'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.3.3
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 22181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2
ports:
- "22181:22181"
networks:
- kafka
extra_hosts:
- "moby:127.0.0.1"
kafka-1:
image: confluentinc/cp-kafka:5.3.3
hostname: localhost
networks:
- kafka
depends_on:
- zookeeper-1
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:22181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092,PLAINTEXT_HOST://localhost:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
ports:
- "19092:19092"
- "29092:29092"
extra_hosts:
- "moby:127.0.0.1"
schema-registry-1:
image: confluentinc/cp-schema-registry:5.3.3
networks:
- kafka
depends_on:
- zookeeper-1
- kafka-1
environment:
SCHEMA_REGISTRY_HOST_NAME: localhost
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper-1:22181
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:18081
SCHEMA_REGISTRY_DEBUG: "true"
ports:
- "18081:18081"
links:
- zookeeper-1
- kafka-1
extra_hosts:
- "moby:127.0.0.1"
networks:
kafka:
Above file one instance each of those components. If you would like to add more, just copy paste those block, and increment the ids.
Start servers (start kafka, zookeeper and schema registry)
To run the docker compose file, run the below command where the above file is saved.
docker-compose -f .\kafka_docker_compose.yml up
Verify processes
docker ps
Connect urls of Kafka, Schema registry and Zookeeper
Kafka bootstrap servers : localhost:29092 Zookeeper : zookeeper-1:22181 Schema Registry : http://localhost:18081
How to add more properties
If you know the property already from kafka documentation, then prefix the property with KAFKA, all in caps, and replace dots with _ (underscore)
Ex : number.partitions will be KAFKA_NUMBER_PARTITIONS
Connect to docker shell
docker exec -ti <docker_container_id> /bin/bash
Conclusion
So in this post we learnt how to run kafka, zookeeper and schema registry in a docker container and connect from out side container.