The Most Valuable Elements Of One Test Automation Framework

cerberus-testing-elements-valuable-test-automation-framework-featured

Test automation can be associated with a technical activity implemented after the software development is complete. Valuable test automation requires to enable the various actors to solve their problems; this is where a framework is practical.

A framework aims to structure the way particular activities are performed. It aims both to accelerate the implementation as well as containing the complexity over time. A test automation framework must provide these capabilities to deliver automated tests.

This article shares the most valuable elements of one test automation framework, Cerberus Testing. It is based on the experience of building the framework since 2010,   iterating on the value creation.

A framework is not only about technical aspects. “Quality is value to someone who matters”, meaning that our framework must support the collaboration between the various actors involved in the test automation processes.

Favor a shared understanding and collaboration

Creating value through test automation requires an alignment of the different actors. By default, the actors are of different backgrounds, not necessarily knowing each other. Hence, an initial and continuous alignment is required.

In test automation, we achieve alignment by defining the product’s quality attributes, to then decline how it translates for test automation. The critical point is to avoid the gap between the business and technical test repository, which have a natural tendency to local optimization over time.

Figure 1: The shared interface of Cerberus Testing for collaboration https://cerberus-testing.com/features/

We need to maintain the alignment through the implementation that requires a shared vocabulary, processes, and formulation. The first element is a shared user interface, easily accessible within a web browser. The complementary element is the structure of the tests. The formalization in BDD style in plain text favors a shared understanding and provides a repeatable structure. Additionally, keyword-driven testing enables a coherent and more accessible implementation.

An improved collaboration unlocks other benefits, such as faster feedback loops.

Enable fast feedback loop for different test typologies

Software changes require faster iterations to adapt the offer to the customers. It is a requirement in a global competition where new actors can emerge and disrupt existing businesses. A supporting test automation framework must provide fast feedback loops.

Our framework must be an enabler rather than a limiting factor. The test automation feedback loops are to define, design, implement, execute, report, and improve. These activities can require the orchestration of various activities and tools. A valuable test automation framework must therefore support a unified experience.

Figure 2: Cerberus Testing provides a unified experience https://cerberus-testing.com 

In Cerberus Testing, three main areas support the lifecycle through a unified interface: Test Repository, Test Execution, and Test Reporting. The test repository enables the test definition, design, and implementation of manual or different automated tests (web, mobile, APIs A library of actions and controls is natively available to accelerate, while a custom library of modules can be built.

The test execution experience is also centralized inside Cerberus Testing. A variety of robots can be configured to execute different types of tests. Once the execution ends, we want instant visibility and reactivity to the results. This is where the built-in reporting and analytics are valuable, being available by default. A set of notifications and messages can also be customized.

Achieving fast feedback loops within our framework requires integration capabilities. 

Integrate within the environment to accelerate

Digital transformation increases the collaboration of actors through the use of technology. The need for real-time reactivity and collaboration increases the interdependence of systems. The ease of integration became a requirement of survival in this context.

The first area to integrate is the test repository. Cerberus Testing can import selenium IDE scenarios and Erratum self-healing tests while keeping the ID of JIRA tickets. To ease the execution experience, various internal and external farms can be natively connected, such as Browserstack or LambdaTest. Cerberus Testing supports parallel web, mobile, APIs, Kafka, databases, or even desktop tests.

Complementarily, test suites can be launched, scheduled or triggered manually or via CI/CD. A series of native connectors are provided for Jenkins, Gitlab, Bitbucket or Azure DevOps. In addition, public APIs are available to support flexible integrations. Bugs reports, alerts, and notifications are available during or after the test execution through various tools like Slack or email. 

Fast feedback loops with native integrations are the foundations to support a continuous testing approach.

Support end-to-end continuous testing approaches

Continuous testing is inspired by the TQM methodologies to support a repeatable process of Plan, Do, Check, Act. We find similar techniques in the agile rituals of planning and retrospective. One model we can inspire is the Holistic Testing of Janet Grégory.

Continuous testing is about acting over the overall value chain of the software lifecycle. To create value for the product users, one must prioritize the most valuable testing practices of Shift-Up, Shift-Left, and Shift-Right. For a test automation framework, that means executing seamlessly across the various environments and supporting the requirements definition and continuous testing processes.

Figure 4: Cerberus Testing provides a variety of solutions and use-cases https://cerberus-testing.com/blog/testing-automation-framework-open-source/ 

Cerberus Testing natively provides an application framework management. We can define for a single application across its various environments, simplifying an end-to-end execution. From the middle of the chain, we can leverage the scheduling and CI/CD capabilities. In the end, the user journey monitoring is an example of a continuous testing approach supported by Cerberus Testing.

We close the tour of our framework with continuous improvement.

Provide measurement and visibility

We can’t improve what we can’t measure. Measurement is fundamental to measure the value of our test automation effort. By default, a framework will lack or provide a poor reporting capability, not being the core of the engine in the first place. The thing is, improvement and usability are requirements of a valuable test automation framework.

The typical metrics are the coverage, flakiness ratio, and detailed tests traceability. The value measurement should go above the test automation area, measuring key user experience satisfaction like NPS, engagement-time, and activity. Additionally, supporting measures of reliability are necessary at the core of a successful customer experience.

Figure 5: The native reports available in Cerberus Testing https://sourceforge.net/software/product/Cerberus-Testing/ 

Concretely, the users of the framework are familiar with web interfaces or even mobile applications for dashboarding. It means that our framework should provide the various metrics in a graphical way. Reporting starts with the test repository to know how many tests we have, which areas, which status, and internal composition.

The next area is to measure the execution phase, providing traceability of individual tests as well as a global perspective. The test and campaign reporting are natively available within Cerberus Testing, removing the need to implement custom dashboards. Additionally, test analytics goes a step further by providing trends analysis over a more extended period of time with new insights.

A test automation framework is a long journey when starting from scratch.

Value is required across the entire process and actors

A valuable test automation framework lies in being of value to its actors. It starts from automating the right tests to then implementing the tests right. That means supporting collaboration and providing a reliable framework are two essential requirements.

The ability to cover the end-to-end processes with a unified interface enables fast feedback loops. This allows us to start with experimentation before expanding and optimizing the test automation effort. This is why reusable modules, measurement, and rework functionalities make a difference.

Building a test automation framework is not a one-night nor a one-man effort. It requires time, dedication, and iterations. We started Cerberus Testing in 2010, releasing it in open-source in 2013. Having similar needs to solve, we are more than happy to share on its usage, capabilities, and even possible contributions.

Whatever your choice is, a valuable test automation framework is our common goal.

You can use Cerberus Testing for free, start now.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.