How To Test a Kafka Consumer with Cerberus Testing

Welcome to this Cerberus Testing step-by-step tutorial for testing Apache Kafka® producers and consumers.

This guide will help you understand the test automation workflow and good practices supported by the Cerberus Testing open-source framework. 

At the end of this article, you will be able to automate a Kafka Consumer use-case. You will learn how to configure your service library, create and execute the test case. 

We will perform the test using a Kafka broker on a sample topic, producer and consumer. You can use this tutorial to implement your own Kafka consumer test relying on JSON messages.

If you are new to Cerberus Testing, we recommend that you start by having a look at our GitHub page, website and overview video. This tutorial is also available as a video.

Prerequisites for implementing a Kafka Consumer Test

The following elements are required in order to successfully complete this tutorial:

• A Cerberus Testing instance available in version 3.8 minimum, self-hosted or SaaS.
• Access to a Cerberus Testing account granted with a “Test” role.
• Setup of a Cerberus Testing environment (by default “QA” and “UAT” are set up).
• A configured SSL certificate between your Cerberus instance and the Kafka broker – if this is required.

If you decided to implement your own Test, you need to:

• Have access to the specific Kafka broker, topic and environment.
• Define the Test action and controls you want to perform.

This scheme summarizes the steps we will go through.

If you have any questions on those prerequisites, have a look at our documentation, videos or contact your administrator. Feel free to interact with the community on our slack channel or submit a GitHub issue.

Step 1 – Configure Kafka Producer and Consumer in Cerberus Testing

This section describes the Kafka Producer and Consumer service definition.

You need to know your Kafka broker endpoint. This information is usually provided by your provider or internal team. 

The Service URL contains the Host and the Port. The certificate information is also available there if you use SSL.

1.1. Setup the Kafka Producer

Let’s start by configuring the Kafka Producer.

Navigate to the Service Library, and click on “Add Service”.

You can now define the Kafka Producer information. Make sure to select the “Kafka Service” typology, select the “PRODUCE” option and fill in the endpoint.

Switch to the “Request” tab to confirm the Kafka Topic and Service Request to be performed in your test.

In our example, the data is fixed. However, you can implement more complex test data scenarios in Cerberus Testing.

The last tab to configure is the “Header” one, that requires the connection details.

1.2. Setup the Kafka Consumer

The definition of a Kafka Consumer is very similar.

The first difference is to pick the “SEARCH” method.

Then, to specify the Kafka Topic, the consumer will need to search the message.

Once done, both your Kafka Producer and Consumer are ready to be used in your test case definition.

Step 2 – Define the Test Case Actions and Controls

This section describes how to configure the Kafka Consumer Test.

The test case design relies on producing a message and then searching for it inside the Kafka topic. We can implement various actions and controls over this process.

2.1. How to produce a Kafka message

We find here the main Cerberus Testing elements to define: Step, Action and Control.

To produce a message to the topic, we will use the Kafka Producer service of the previous section. The Action to use is the “callService” referencing our library.

We do not need to specify the other parameters for the Producer case.

2.2. How to consume a Kafka Message

Once the message is on the topic, we want to verify its presence.

We can achieve this by using the same “callService” action, specifying the relevant search criteria: number of events to fetch and the search timeout.

2.3. How to test the Kafka consumer

The previous actions will already verify that the producer and consumer are tied together.

We can implement more advanced controls on the message content, for example. This can be achieved by adding typical controls with the “verifyElement” and a corresponding path, such as “$.data.messageHeader”.

The various use-cases and controls are available in the online documentation.

Step 3 – Execute and Verify the Test Results

The last part is to execute and access the test case reporting.

3.1. Launch the Kafka Automated Test

Navigate to the Run page of Cerberus Testing.

Select the Kafka test with its corresponding test environment. You can pick any robot for the execution.

Click on “Run Test Case (and See Result)” to follow-up the execution.

3.2. See the Execution in Real-Time

Once started, you can visualize the test case execution live.

We can see the request and service call content. In blue, we also see the consumer searching for the message we just produced.

Then, the consumer found the message we were expecting.

3.3. Verify the Test Case Results and Traceability

You can access various test traceability elements, including the robots in the various tabs.

For example, the details of the request are available by clicking on the “Service Call” image.

We can also confirm that the message has been produced in our Kafka Topic.

Expanding your Kafka Automated Testing

You should have successfully implemented your Kafka consumer test by reaching the end of this tutorial.

We hope to have clarified the main steps involved in implementing automated tests for your Kafka applications.

You can scale this approach by using the various test data, property and execution capabilities. 

A typical use-case is to configure those tests in CI/CD, leveraging the parallel execution and campaign management.

We wish you a nice test automation journey!