Let me give some background. Apache Kafka is widely being adapted in organizations irrespective of the scale. It’s unique features like scalability, retention and reliability unlike the traditional messaging platforms, makes it stand out.

While adapting Kafka, you would notice there are few manual activities like creating topics, acls, updating configuration etc.
There have been few topic management tools in the market which provide automation to certain extent, however its not complete. Assume there is a team requesting for a kafka topic with in an organization. They would either request for it through email with a template or any user interface. The actual kafka team in the organization executes the commands to create topics. With Development, Test, Acceptance and Production environments, managing these configurations, by the Kafka team and handling all the requests from various teams would become a hassle as you see the growth of teams using Kafka. It is just not manual activity, but Audit, time and effort, the team has to put in.

Apart from that, this meta data (Kafka topics, acls) which is stored in ZK could be easily stored in a good readable format and can be used as source of truth.

Hence I saw the need of Kafkawize – a self service topic management tool., to keep ZERO interaction by other Teams with Kafka team for topics, acls or schemas creation by introducing roles and authorizations to users of teams. It is developed with a simple front end, and backend with Spring boot. It saves time and effort, governs information, avoids risk in kafka config loss, avoids manual activities and mistakes, maintains up to date SOT (source of truth). Current User Interface is not so great but it serves the purpose for now. It would be updated with Angular 7 material design soon.
See the Roadmap
. Key features below:

  • 4 eyed principle – Requesting and Approving topics/acls/schemas
  • Auditing
  • Spring Security
  • Synchronize Source of truth with Meta store
  • Support for Cassandra or an Rdbms

Kafkawize : A Self service Apache Kafka Topic Management tool/portal. A Web application which automates the process of creating and browsing Kafka topics, acls, schemas by introducing roles/authorizations to users of various teams of an organization.

Download : https://kafkawize.com/download-kafkawize/

previous arrow
next arrow
Slider

Kafkawize contains two main APIs. (User Interface API and Cluster management API)



  • User Interface API directly communicates between users and Cluster API.
  • Cluster API acts as middle layer between Kafka brokers and UserInterface API.
  • Cluster API creates Kafka Admin Client and executes the requests for Topic, Acls or Schema registry.

Front end is built with AngularJs, HTML, and Java script.
Apache Cassandra Or RDBMS(MySql for ex) datastore stores all users, teams, topicRequests, request and execution data from all the users., and to maintain source of truth.
Spring Security, Spring Boot frameworks, Hibernate are used to develop this tool.

Functionalities:

(Broadly divided into two categories based on user roles.)
ROLE : USER can request for creation of kafka components (topics, acls, schemas), and browse kafka components.
ROLE : ADMIN can approve and execute the requests of users to create kafka components.
ROLE : SUPERUSER can synchronize topicRequests meta information with Cassandra/Rdbms Datastore from Kafka Brokers (Source of Truth.)

Browse: (ROLE : USER, ADMIN, SUPERUSER)
All users can Browse Topics
All users can Browse Acls
All users can view the producers and consumers of all topicRequests. 

Requests:(ROLE : USER)
Users can request for Kafka Topics 
Users can request for Kafka Acl 
Users can request for Schemas to be registered on Confluent Schema registry. 
Users can view all the requests from his team. 

Clusters:(ROLE : USER)
Users can view the available environments 

Approve – Execute :(ROLE : ADMIN)
Users can appprove requests for creating Kafka Topics 
Users can appprove requests for creating Kafka Acls 
Users can appprove requests for uploading schemas on topicRequests

Users :(ROLE : ADMIN)
Users can view all user details 
Users can add new users 

Teams :(ROLE : ADMIN, SUPERUSER)
Users can view all team details 
Users can add new team 

Clusters:(ROLE : SUPERUSER)
Users can add a new environment environments 

Synchronize Metadata :(ROLE : SUPERUSER)
Users can synchronize topicRequest information from Brokers with Cassandra/Rdbms datastore. (Update team info.) 
Users can synchronize acls information from Brokers with Cassandra/Rdbms datastore. (Update team info.) 

My Profile :(ROLE : USER, ADMIN)
Users can view their profile. 

Change Password :(ROLE : USER, ADMIN)
Users can change their passwords. 

Logout :(ROLE : USER, ADMIN)
Users can logout. 

How to Run the application

KafkaWize needs the following applications to be up and running.

  1. Spring boot application KafkaWize
  2. Spring boot application KafkaWize ClusterApi
  3. Apache Cassandra OR Rdbms(MySql/Oracle/..)

Steps to run with Cassandra as Metastore:

  1. Install Apache Cassandra
  2. Setup project KafkawizeClusterApi and update server.port if necessary in application properties
  3. Start KafkaClusterApi
  4. Setup project KafkaWize, and configure Cassandra running host, Cluster api host, in application properties
  5. Set db.storetype=cassandra in application.properties
  6. Start KafkaWize
  7. Cassandra db setup will be done on the startup of the application. We do not have to create manually.

Steps to run with Rdbms as Metastore:

  1. Install an Rdbms database (Mysql or Oracle or ..)
  2. Setup project KafkawizeClusterApi and update server.port if necessary in application properties
  3. Start KafkaClusterApi
  4. Setup project KafkaWize, and configure Cassandra running host, Cluster api host, in application properties
  5. Set db.storetype=rdbms and few other datasource properties in application.properties
  6. Start KafkaWize
  7. Run the ddl and insert scripts available in src/main/resources/scripts/base/rdbms.

By default KafkaWize runs on port 9097. Access it by http://localhost:9097

Default Teams Team1 Team2 Team3

Default Users (with pwds)

uiuser1/user from Team1
uiuser2/user from Team2
uiuser4/user from Team1 Admin
uiuser5/user from Team2 Admin
superuser/user from Team2 Superuser

Demo of Kafkawize :

If you have any questions about installation or suggestions, please contact me.