The Test Reporting You Need To Accelerate with Quality

cerberus-testing-test-reporting-need-to-accelerate-with-quality-featured

We define the test strategy, implement the test framework and start building up our test suite. These tasks are already time-consuming. Our stakeholders are waiting for results where reporting is essential.

A mature test automation framework providing native reporting capabilities is of added value to the team; they can work more on quality than coding a test automation framework. 

But test reporting is not only about one test execution report. We must answer different questions for different stakeholders and perspectives promptly to keep fast feedback loops.

This article shares the why and what of test reporting features, exemplifying its implementation within Cerberus Testing.

First things first with individual Test Report

The first thing our team will ask is the individual test reporting. They need to know the status of the test through its lifecycle, from its first iterations until it is fully implemented, ready for use to the rest of the team.

Traceability is a fundamental requirement for fast implementation and analysis of test results. As an individual test report, it means providing the details of all actions and controls executed, as well as the context. Live execution traceability is very useful during the implementation cycles or for replay purposes to help developers correct bugs.

Figure 1: The native individual test report available in Cerberus Testing

In Cerberus Testing, individual test reporting is a set of native features available through the web interface. A test reporting with each action and control detail is available with the robot logs, execution context, and environment variables. Screenshots and videos are available; they need a configuration within the test or at execution time. A one-click bug opener is also available to JIRA to ease the debugging lifecycle.

Once satisfied with one test, the team starts to build up a test suite. They also come up with their reporting challenge.

Test Campaign Reporting with test suite execution

A suite of tests is a way to group particular requirements to be verified against a specific context. We can perform a non-regression test suite in CI/CD for each environment or execute a set of tests post-deployment in production. The difference with a single test is that the overall status gives additional information on the application quality.

A test suite reporting has a global status that can be customized based on the underlying test results. For example, we can consider a suite validated only with priority 1 tests confirmed and priority 2 as additional controls. Complementarily, a test suite will provide additional views based on the test meta-data like the features they cover, the personas, or the screen involved. That way, we can more easily report specific areas to answer our stakeholders’ questions, like “is the customer area under control?”.

Figure 2: The test suite reporting natively available in Cerberus Testing

Cerberus Testing implements test suites with the concept of test campaigns. They are a logical way to group tests with flexibility; by test case references, application, system, among others. Once defined, the test campaign can be triggered within the web interface by scheduling or manually. A campaign reporting then enables to know the overall and status per meta-data available. These statuses can be used within CI/CD pipelines for go/nogo on the deployment stage. To save you time, we also compute the necessary ratio like flakiness, percentage of success.

Over time we will run many individual tests and test suites where analytics is a game-changer.

Test Analytics

Having the possibility to collect, store and report on the test data is test analytics. It enables the stakeholders to access insightful data analysis and decision-making that is otherwise not possible. It becomes a requirement for teams with more tests and more maturity to keep a view over the entire practice.

Test analytics is about aggregating the views of various test executions over a period of time. We need the same type of visibility of Google Analytics for our tests: a dashboard to compare previous execution, dashboards we can drill down, perform trends analysis. Test analytics is, therefore, an entire reporting and exploration suite to create.

Figure 3: The test analytics natively available in Cerberus Testing

Cerberus Testing natively provides the test analytics within the same web interface. Reporting of tests or test campaigns is available over a more significant period of time. The filters and dashboards enable the users to analyze global indicators and trends and easily detect spikes. Additionally, the same report can be customized dynamically, adding or removing data to explore specific questions. The overall flakiness ratio is also available, being a key metric to accelerate with quality.

The analytical capability closes our loop of test reporting requirements.

Accelerate with Quality requires fast feedback loops

Performing fast feedback loops in testing are about iterating fast with test design, execution, and reporting. Test reporting is therefore an essential part of a test automation framework from individual test reports, test suites, and test analytics. 

We can only answer our stakeholders’ requirements by providing this whole set of features. Choosing a test automation framework natively providing this entire set of features is a critical decision to truly accelerate with quality. Else, the quality teams lose valuable time implementing reporting features.

Cerberus Testing is the open-source test automation framework to enable testing over coding. You can start and use it for free to simplify and accelerate your test automation journey. Start Testing with Cerberus Testing.

See it in video here at https://www.youtube.com/watch?v=pxVjMoZSQ6Y

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.