How To Test a Kafka Consumer with Cerberus Testing

Welcome to this Cerberus Testing step-by-step tutorial for testing Apache Kafka® producers and consumers.

This guide will help you understand the test automation workflow and good practices supported by the Cerberus Testing open-source framework. 

At the end of this article, you will be able to automate a Kafka Consumer use-case. You will learn how to configure your service library, create and execute the test case. 

We will perform the test using a Kafka broker on a sample topic, producer and consumer. You can use this tutorial to implement your own Kafka consumer test relying on JSON messages.

If you are new to Cerberus Testing, we recommend that you start by having a look at our GitHub page, website and overview video. This tutorial is also available as a video.

Prerequisites for implementing a Kafka Consumer Test

The following elements are required in order to successfully complete this tutorial:

  • A Cerberus Testing instance available in version 3.8 minimum, self-hosted or SaaS.
  • Access to a Cerberus Testing account granted with a “Test” role.
  • Setup of a Cerberus Testing environment (by default “QA” and “UAT” are set up).
  • A configured SSL certificate between your Cerberus instance and the Kafka broker – if this is required.

If you decided to implement your own Test, you need to:

  • Have access to the specific Kafka broker, topic and environment.
  • Define the Test action and controls you want to perform.

This scheme summarizes the steps we will go through.

Figure 1: The Kafka Integration Steps in Cerberus Testing

If you have any questions on those prerequisites, have a look at our documentation, videos or contact your administrator. Feel free to interact with the community on our slack channel or submit a GitHub issue.

Step 1 – Configure Kafka Producer and Consumer in Cerberus Testing

This section describes the Kafka Producer and Consumer service definition.

You need to know your Kafka broker endpoint. This information is usually provided by your provider or internal team. 

Figure 2: The Connectivity Information Required of the Kafka Broker

The Service URL contains the Host and the Port. The certificate information is also available there if you use SSL.

1.1. Setup the Kafka Producer

Let’s start by configuring the Kafka Producer.

Navigate to the Service Library, and click on “Add Service”.

Figure 3: The Cerberus Testing Screen to a Service Library for the Kafka Calls

You can now define the Kafka Producer information. Make sure to select the “Kafka Service” typology, select the “PRODUCE” option and fill in the endpoint.

Figure 4: The Kafka Producer Definition

Switch to the “Request” tab to confirm the Kafka Topic and Service Request to be performed in your test.

Figure 5: The Information Required to produce to a Kafka Topic with Cerberus Testing

In our example, the data is fixed. However, you can implement more complex test data scenarios in Cerberus Testing.

The last tab to configure is the “Header” one, that requires the connection details.

Figure 6: The Detailed Connectivity Information of the “Header” Tab

1.2. Setup the Kafka Consumer

The definition of a Kafka Consumer is very similar.

The first difference is to pick the “SEARCH” method.

Figure 7: The Configuration of a Kafka Consumer in Cerberus Testing

Then, to specify the Kafka Topic, the consumer will need to search the message.

Figure 8: Only the Topic is required for a Consumer Definition

Once done, both your Kafka Producer and Consumer are ready to be used in your test case definition.

Step 2 – Define the Test Case Actions and Controls

This section describes how to configure the Kafka Consumer Test.

The test case design relies on producing a message and then searching for it inside the Kafka topic. We can implement various actions and controls over this process.

2.1. How to produce a Kafka message

We find here the main Cerberus Testing elements to define: Step, Action and Control.

To produce a message to the topic, we will use the Kafka Producer service of the previous section. The Action to use is the “callService” referencing our library.

Figure 9: The Test Case Action and Control for a Kafka Producer and Consumer Test

We do not need to specify the other parameters for the Producer case.

2.2. How to consume a Kafka Message

Once the message is on the topic, we want to verify its presence.

We can achieve this by using the same “callService” action, specifying the relevant search criteria: number of events to fetch and the search timeout.

Figure 10: The additional parameters for the Search of a Consumer

2.3. How to test the Kafka consumer

The previous actions will already verify that the producer and consumer are tied together.

We can implement more advanced controls on the message content, for example. This can be achieved by adding typical controls with the “verifyElement” and a corresponding path, such as “$.data.messageHeader”.

The various use-cases and controls are available in the online documentation.

Step 3 – Execute and Verify the Test Results

The last part is to execute and access the test case reporting.

3.1. Launch the Kafka Automated Test

Navigate to the Run page of Cerberus Testing.

Select the Kafka test with its corresponding test environment. You can pick any robot for the execution.

Figure 11: The Various Execution Parameters and Traceability Elements are available

Click on “Run Test Case (and See Result)” to follow-up the execution.

3.2. See the Execution in Real-Time

Once started, you can visualize the test case execution live.

Figure 12: The Events are correctly generated in Cerberus Testing Producer

We can see the request and service call content. In blue, we also see the consumer searching for the message we just produced.

Then, the consumer found the message we were expecting.

Figure 13: The Consumer is able to retrieve the Events in Cerberus Testing

3.3. Verify the Test Case Results and Traceability

You can access various test traceability elements, including the robots in the various tabs.

For example, the details of the request are available by clicking on the “Service Call” image.

Figure 14: Detailed Trace and Event Log are available in Cerberus Testing

We can also confirm that the message has been produced in our Kafka Topic.

Figure 15: The Message is available in the original Kafka Broker

Expanding your Kafka Automated Testing

You should have successfully implemented your Kafka consumer test by reaching the end of this tutorial.

We hope to have clarified the main steps involved in implementing automated tests for your Kafka applications.

You can scale this approach by using the various test data, property and execution capabilities

A typical use-case is to configure those tests in CI/CD, leveraging the parallel execution and campaign management.

We wish you a nice test automation journey!

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.