The Test Reporting You Need To Accelerate with Quality

We define the test strategy, implement the test framework and start building up our test suite. These tasks are already time-consuming. Our stakeholders are waiting for results where reporting is essential.

A mature test automation framework providing native reporting capabilities is of added value to the team; they can work more on quality than coding a test automation framework. 

But test reporting is not only about one test execution report. We must answer different questions for different stakeholders and perspectives promptly to keep fast feedback loops.

This article shares the why and what of test reporting features, exemplifying its implementation within Cerberus Testing.

First things first with individual Test Report

The first thing our team will ask is the individual test reporting. They need to know the status of the test through its lifecycle, from its first iterations until it is fully implemented, ready for use to the rest of the team.

Traceability is a fundamental requirement for fast implementation and analysis of test results. As an individual test report, it means providing the details of all actions and controls executed, as well as the context. Live execution traceability is very useful during the implementation cycles or for replay purposes to help developers correct bugs.

In Cerberus Testing, individual test reporting is a set of native features available through the web interface. A test reporting with each action and control detail is available with the robot logs, execution context, and environment variables. Screenshots and videos are available; they need a configuration within the test or at execution time. A one-click bug opener is also available to JIRA to ease the debugging lifecycle.

Once satisfied with one test, the team starts to build up a test suite. They also come up with their reporting challenge.

Test Campaign Reporting with test suite execution

A suite of tests is a way to group particular requirements to be verified against a specific context. We can perform a non-regression test suite in CI/CD for each environment or execute a set of tests post-deployment in production. The difference with a single test is that the overall status gives additional information on the application quality.

A test suite reporting has a global status that can be customized based on the underlying test results. For example, we can consider a suite validated only with priority 1 tests confirmed and priority 2 as additional controls. Complementarily, a test suite will provide additional views based on the test meta-data like the features they cover, the personas, or the screen involved. That way, we can more easily report specific areas to answer our stakeholders’ questions, like “is the customer area under control?”.

Cerberus Testing implements test suites with the concept of test campaigns. They are a logical way to group tests with flexibility; by test case references, application, system, among others. Once defined, the test campaign can be triggered within the web interface by scheduling or manually. A campaign reporting then enables to know the overall and status per meta-data available. These statuses can be used within CI/CD pipelines for go/nogo on the deployment stage. To save you time, we also compute the necessary ratio like flakiness, percentage of success.

Over time we will run many individual tests and test suites where analytics is a game-changer.

Test Analytics

Having the possibility to collect, store and report on the test data is test analytics. It enables the stakeholders to access insightful data analysis and decision-making that is otherwise not possible. It becomes a requirement for teams with more tests and more maturity to keep a view over the entire practice.

Test analytics is about aggregating the views of various test executions over a period of time. We need the same type of visibility of Google Analytics for our tests: a dashboard to compare previous execution, dashboards we can drill down, perform trends analysis. Test analytics is, therefore, an entire reporting and exploration suite to create.

Cerberus Testing natively provides the test analytics within the same web interface. Reporting of tests or test campaigns is available over a more significant period of time. The filters and dashboards enable the users to analyze global indicators and trends and easily detect spikes. Additionally, the same report can be customized dynamically, adding or removing data to explore specific questions. The overall flakiness ratio is also available, being a key metric to accelerate with quality.

The analytical capability closes our loop of test reporting requirements.

Accelerate with Quality requires fast feedback loops

Performing fast feedback loops in testing are about iterating fast with test design, execution, and reporting. Test reporting is therefore an essential part of a test automation framework from individual test reports, test suites, and test analytics. 

We can only answer our stakeholders’ requirements by providing this whole set of features. Choosing a test automation framework natively providing this entire set of features is a critical decision to truly accelerate with quality. Else, the quality teams lose valuable time implementing reporting features.

Cerberus Testing is the open-source test automation framework to enable testing over coding. You can start and use it for free to simplify and accelerate your test automation journey. Start Testing with Cerberus Testing.

See it in video here at https://www.youtube.com/watch?v=pxVjMoZSQ6Y