VOOZH about

URL: https://dev.to/vultr/deploying-kafka-distributed-event-streaming-on-ubuntu-2404-48ak

⇱ Deploying Kafka Distributed Event Streaming on Ubuntu 24.04 - DEV Community

Apache Kafka is an open-source distributed event streaming platform used for real-time pipelines, log aggregation, and event-driven architectures. This guide deploys a single-node Kafka 4.x cluster in KRaft mode (no ZooKeeper) using Docker Compose with persistent storage, then verifies producer/consumer flow with the Kafka CLI tools. By the end, you'll have Kafka accepting produced events and serving them to consumers on your server.

Set Up the Directory Structure

1. Create the project directory:

$mkdir -p ~/kafka-stack/kafka-data
$sudo chown 1000:1000 ~/kafka-stack/kafka-data
$cd ~/kafka-stack

2. Generate a Kafka cluster ID:

$uuidgen | tr -d '-'

Note the value — it goes into the .env file below.

3. Create the environment file:

$nano .env

KAFKA_NODE_ID=1
KAFKA_CLUSTER_ID=YOUR_KAFKA_CLUSTER_ID
KAFKA_PORT=9092
KAFKA_ADVERTISED_HOST=localhost

Deploy with Docker Compose

1. Create the Docker Compose manifest:

$nano docker-compose.yaml

services:
 kafka:
 image: apache/kafka:4.1.1
 container_name: kafka
 ports:
 - "${KAFKA_PORT}:9092"
 environment:
 KAFKA_NODE_ID: ${KAFKA_NODE_ID}
 KAFKA_PROCESS_ROLES: broker,controller
 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${KAFKA_ADVERTISED_HOST}:9092
 KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
 KAFKA_LOG_DIRS: /kafka/data
 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
 KAFKA_CLUSTER_ID: ${KAFKA_CLUSTER_ID}
 volumes:
 - "./kafka-data:/kafka/data"
 restart: unless-stopped
 healthcheck:
 test: ["CMD", "kafka-broker-api-versions", "--bootstrap-server=localhost:9092"]
 interval: 10s
 timeout: 5s
 retries: 5

2. Start the service:

$docker compose up -d

3. Verify the service is running:

$docker compose ps
$docker compose logs

Install the Kafka CLI Tools

1. Install Java and download the Kafka tools:

$sudo apt update && sudo apt install -y default-jdk
$wget https://downloads.apache.org/kafka/4.1.1/kafka_2.13-4.1.1.tgz
$sudo tar -xzf kafka_2.13-4.1.1.tgz -C /opt

2. Put the Kafka CLI on your PATH:

$echo 'export PATH=$PATH:/opt/kafka_2.13-4.1.1/bin' | sudo tee /etc/profile.d/kafka.sh
$source /etc/profile.d/kafka.sh

Test Message Production and Consumption

1. Create a topic:

$kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 3 --replication-factor 1

2. Produce messages:

$kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

Type a few lines and press Ctrl+D to send.

3. Consume from the beginning:

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

4. Describe the topic:

$kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic

Next Steps

Kafka is running in KRaft mode with persistent storage. From here you can:

  • Add more brokers and set replication factors > 1 for production durability
  • Enable SASL/SSL listeners to authenticate and encrypt client traffic
  • Wire in Kafka Connect, Schema Registry, or ksqlDB for richer pipelines

For the full guide with additional tips, visit the original article on Vultr Docs.