Оффлайн
- Регистрация
- 1 Мар 2015
- Сообщения
- 1,481
- Баллы
- 155
Apache Kafka is a powerful distributed event streaming platform widely used for building real-time data pipelines and applications. However, setting up Kafka can be daunting, especially when integrating it with a Node.js application. This article explores a GitHub project that simplifies this process by using Docker Compose to set up a Kafka environment and a Node.js Express application to test Kafka's functionality.
Overview of the Project
This project provides a ready-to-use Kafka environment using Docker Compose and a Node.js Express application (kafka-express-app) to test Kafka by producing and consuming messages. It is an excellent starting point for developers looking to integrate Kafka into their applications.
Project Structure
clone this github repo:
The repository is well-organized, with the following structure:
kafka-docker/
├── docker-compose.yml # Docker Compose file for Kafka setup
├── kafka-express-app/ # Node.js Express application
│ ├── src/
│ │ ├── kafka/
│ │ │ ├── producer.ts # Kafka producer logic
│ │ │ └── consumer.ts # Kafka consumer logic
│ │ ├── routes/
│ │ │ └── index.ts # Express routes
│ │ └── app.ts # Express app entry point
│ ├── package.json # Node.js dependencies and scripts
│ ├── tsconfig.json # TypeScript configuration
│ └── .env # Environment variables
└── README.md # Project documentation
Kafka Setup with Docker Compose
The docker-compose.yml file defines the services required to run Kafka, including:
Steps to Start Kafka
docker-compose up -d
docker ps
(If you don't see anything, then wait for 1 min. Let the containers spin up and connect with each other.)
On the Click of this page. It will open the cluster details page.
here you can see numbers of broker running, topics, partitions and other details.
Node.js Application Setup
The kafka-express-app is a Node.js Express application that interacts with Kafka. It includes producer and consumer logic written in TypeScript.
Environment Variables
Create a .env file in the kafka-express-app directory with the following content:
KAFKA_CLIENT_ID=my-app
KAFKA_BROKERS=localhost:9092
KAFKA_GROUP_ID=test-group
KAFKA_TOPIC=test-topic
Install Dependencies
Navigate to the kafka-express-app directory and install the required dependencies:
cd kafka-express-app
npm install
Build the Application
Compile the TypeScript code to JavaScript:
npm run build
Run the Application
Start the application in development mode:
npm run dev
Or start the compiled application:
npm start
Testing Kafka
The Node.js application provides endpoints to test Kafka's producer and consumer functionality.
Produce a Message
Use a tool like curl or Postman to send a POST request to the /api/publish endpoint:
curl -X POST \
-H "Content-Type: application/json" \
-d '{"topic": "test-topic", "message": "Hello Kafka!"}'
Consume Messages
Send a GET request to the /api/subscribe endpoint to start consuming messages:
curl
You should see the consumed messages logged in the console.
Stopping Services
To stop all Docker services, run:
docker-compose down
Troubleshooting
Conclusion
This project simplifies the process of setting up Kafka with Docker Compose and integrating it with a Node.js application. Whether you're a beginner or an experienced developer, this repository provides a solid foundation for building real-time data streaming applications.
If you're looking to dive into Kafka and Node.js, this project is a great place to start. Try it out, and let us know your thoughts in the comments below!
Overview of the Project
This project provides a ready-to-use Kafka environment using Docker Compose and a Node.js Express application (kafka-express-app) to test Kafka by producing and consuming messages. It is an excellent starting point for developers looking to integrate Kafka into their applications.
Project Structure
clone this github repo:
The repository is well-organized, with the following structure:
kafka-docker/
├── docker-compose.yml # Docker Compose file for Kafka setup
├── kafka-express-app/ # Node.js Express application
│ ├── src/
│ │ ├── kafka/
│ │ │ ├── producer.ts # Kafka producer logic
│ │ │ └── consumer.ts # Kafka consumer logic
│ │ ├── routes/
│ │ │ └── index.ts # Express routes
│ │ └── app.ts # Express app entry point
│ ├── package.json # Node.js dependencies and scripts
│ ├── tsconfig.json # TypeScript configuration
│ └── .env # Environment variables
└── README.md # Project documentation
Kafka Setup with Docker Compose
The docker-compose.yml file defines the services required to run Kafka, including:
- Zookeeper: Manages Kafka brokers.
- Kafka Broker: The Kafka server.
- Schema Registry: Manages Avro schemas for Kafka topics.
- Kafka Connect: Connects Kafka to external systems.
- Control Center: A web UI for managing Kafka.
- kafka-express-app: The Node.js application for testing Kafka.
Steps to Start Kafka
- Start Docker Services: Run the following command to start all services:
docker-compose up -d
- Verify Services: Check if all services are running:
docker ps
- Access Control Center: Open the Confluent Control Center in your browser:
(If you don't see anything, then wait for 1 min. Let the containers spin up and connect with each other.)
On the Click of this page. It will open the cluster details page.
here you can see numbers of broker running, topics, partitions and other details.
Node.js Application Setup
The kafka-express-app is a Node.js Express application that interacts with Kafka. It includes producer and consumer logic written in TypeScript.
Environment Variables
Create a .env file in the kafka-express-app directory with the following content:
KAFKA_CLIENT_ID=my-app
KAFKA_BROKERS=localhost:9092
KAFKA_GROUP_ID=test-group
KAFKA_TOPIC=test-topic
Install Dependencies
Navigate to the kafka-express-app directory and install the required dependencies:
cd kafka-express-app
npm install
Build the Application
Compile the TypeScript code to JavaScript:
npm run build
Run the Application
Start the application in development mode:
npm run dev
Or start the compiled application:
npm start
Testing Kafka
The Node.js application provides endpoints to test Kafka's producer and consumer functionality.
Produce a Message
Use a tool like curl or Postman to send a POST request to the /api/publish endpoint:
curl -X POST \
-H "Content-Type: application/json" \
-d '{"topic": "test-topic", "message": "Hello Kafka!"}'
Consume Messages
Send a GET request to the /api/subscribe endpoint to start consuming messages:
curl
You should see the consumed messages logged in the console.
Stopping Services
To stop all Docker services, run:
docker-compose down
Troubleshooting
- Kafka Broker Connection Issues: Ensure the KAFKA_BROKERS in .env matches the broker address in docker-compose.yml (e.g., broker:29092).
- Port Conflicts: Ensure the ports defined in docker-compose.yml (e.g., 9092, 29092, 9021) are not in use by other applications.
Conclusion
This project simplifies the process of setting up Kafka with Docker Compose and integrating it with a Node.js application. Whether you're a beginner or an experienced developer, this repository provides a solid foundation for building real-time data streaming applications.
If you're looking to dive into Kafka and Node.js, this project is a great place to start. Try it out, and let us know your thoughts in the comments below!