How to install Apache Kafka 2.4.0 on a CentOS 8 x64 server

This guide explains

  1. How to install Apache Kafka 2.4.0 on a CentOS 8 x64 server from Vultr.

Prerequisites

Step 1 — Creating a User for Kafka

We will create a dedicated kafka user to perform our tasks in this tutorial.

Logged in as your non-root sudo user, create a user called kafka with the useradd command:

Set the password using passwd:

Add the kafka user to the wheel group with the adduser command, so that it has the privileges required to install Kafka’s dependencies:

Your kafka user is now ready. Log into this account using su:

Now that we’ve created the Kafka-specific user, we can move on to downloading and extracting the Kafka binaries.

Step 2 — Downloading and Extracting the Kafka Binaries

Let's go to our kafka user's home directory

Creating a new folder called downloads

Use curl to download the Kafka binaries:

Let's create a directory kafka under /home/kafka

Using the tar command to extract the downloaded file:

Now that we’ve downloaded and extracted the binaries successfully, we can move on the next step

Step 3 — Configuring the Kafka Server

Kafka’s configuration options are specified in server.properties. Open this file with nano

Add the following line to the bottom of the file

This setting allows user to delete the Kafka topics.

Step 4 — Creating Systemd Unit Files and Starting the Kafka Server

Create the unit file for zookeeper:

Enter the following unit definition into the file:

Next, create the unit file for kafka:

Enter the following unit definition into the file:

Zookeeper is a service that Kafka uses to manage its cluster state and configurations. For further information about it, please visit the official page [Zookeeper][https://zookeeper.apache.org/doc/current/index.html]

Now that the units have been defined, start Kafka with the following command:

Check our Kafka server status:

It may shows the error as below:

The error happens because there is no rule in SELinux policy to allow the specific access: allow non-transitioning execution of a file with context config_home_t by a process running in init_t domain.

You can generate a local policy module to allow this access by doing the following steps:

Logged in with root user account

Now, let's try to start our Kafka server again:

You will see the server has started successfully

Step 5 — Testing the Installation

Let’s publish and consume a Hello, Vultr, Kafka message to make sure the Kafka server is behaving correctly. Publishing messages in Kafka requires:

First, create a topic named HelloVultrTopic by typing:

You will see the following output:

You can create a producer from the command line using the kafka-console-producer.sh script. It expects the Kafka server’s hostname, port, and a topic name as arguments.

Publish the string "Hello, Vultr, Kafka" to the HelloVultrTopic topic by typing:

Next, you can create a Kafka consumer using the kafka-console-consumer.sh script. It expects the ZooKeeper server’s hostname and port, along with a topic name as arguments.

The following command consumes messages from HelloVultrTopic. Note the use of the --from-beginning flag, which allows the consumption of messages that were published before the consumer was started:

If there are no configuration issues, you should see Hello, Vultr, Kafka in your terminal:

The script will continue to run, waiting for more messages to be published to the topic. Feel free to open a new terminal and start a producer to publish a few more messages. You should be able to see them all in the consumer’s output.

When you are done testing, press CTRL+C to stop the consumer script. Now that we have tested the installation.

Congratulations, you have successfully installed Apache Kafka 2.4.0 on a CentOS 8 x64 server.

Tags

What do you think?
2 Responses
up vote for this article
Upvote
 
funny emotion for this article
Funny
 
love emotion for this article
Love
 
surprised emotion for this article
Surprised
 
Angry
 
sad emotion for this article
Sad
 

Related Articles

This is related article 2

We all know how hard it can be to make a site look like the demo, so to make your start into the world of X as easy as possible we have included the demo content from our showcase site.

This is related article 2

We all know how hard it can be to make a site look like the demo, so to make your start into the world of X as easy as possible we have included the demo content from our showcase site.

Comments