The Most Valuable Elements Of One Test Automation Framework

Test automation can be associated with a technical activity implemented after the software development is complete. Valuable test automation requires to enable the various actors to solve their problems; this is where a framework is practical.

A framework aims to structure the way particular activities are performed. It aims both to accelerate the implementation as well as containing the complexity over time. A test automation framework must provide these capabilities to deliver automated tests.

This article shares the most valuable elements of one test automation framework, Cerberus Testing. It is based on the experience of building the framework since 2010,   iterating on the value creation.

A framework is not only about technical aspects. “Quality is value to someone who matters”, meaning that our framework must support the collaboration between the various actors involved in the test automation processes.

Favor a shared understanding and collaboration

Creating value through test automation requires an alignment of the different actors. By default, the actors are of different backgrounds, not necessarily knowing each other. Hence, an initial and continuous alignment is required.

In test automation, we achieve alignment by defining the product’s quality attributes, to then decline how it translates for test automation. The critical point is to avoid the gap between the business and technical test repository, which have a natural tendency to local optimization over time.

We need to maintain the alignment through the implementation that requires a shared vocabulary, processes, and formulation. The first element is a shared user interface, easily accessible within a web browser. The complementary element is the structure of the tests. The formalization in BDD style in plain text favors a shared understanding and provides a repeatable structure. Additionally, keyword-driven testing enables a coherent and more accessible implementation.

An improved collaboration unlocks other benefits, such as faster feedback loops.

Enable fast feedback loop for different test typologies

Software changes require faster iterations to adapt the offer to the customers. It is a requirement in a global competition where new actors can emerge and disrupt existing businesses. A supporting test automation framework must provide fast feedback loops.

Our framework must be an enabler rather than a limiting factor. The test automation feedback loops are to define, design, implement, execute, report, and improve. These activities can require the orchestration of various activities and tools. A valuable test automation framework must therefore support a unified experience.

In Cerberus Testing, three main areas support the lifecycle through a unified interface: Test Repository, Test Execution, and Test Reporting. The test repository enables the test definition, design, and implementation of manual or different automated tests (web, mobile, APIs A library of actions and controls is natively available to accelerate, while a custom library of modules can be built.

The test execution experience is also centralized inside Cerberus Testing. A variety of robots can be configured to execute different types of tests. Once the execution ends, we want instant visibility and reactivity to the results. This is where the built-in reporting and analytics are valuable, being available by default. A set of notifications and messages can also be customized.

Achieving fast feedback loops within our framework requires integration capabilities. 

Integrate within the environment to accelerate

Digital transformation increases the collaboration of actors through the use of technology. The need for real-time reactivity and collaboration increases the interdependence of systems. The ease of integration became a requirement of survival in this context.

The first area to integrate is the test repository. Cerberus Testing can import selenium IDE scenarios and Erratum self-healing tests while keeping the ID of JIRA tickets. To ease the execution experience, various internal and external farms can be natively connected, such as Browserstack or LambdaTest. Cerberus Testing supports parallel web, mobile, APIs, Kafka, databases, or even desktop tests.

Complementarily, test suites can be launched, scheduled or triggered manually or via CI/CD. A series of native connectors are provided for Jenkins, Gitlab, Bitbucket or Azure DevOps. In addition, public APIs are available to support flexible integrations. Bugs reports, alerts, and notifications are available during or after the test execution through various tools like Slack or email. 

Fast feedback loops with native integrations are the foundations to support a continuous testing approach.

Support end-to-end continuous testing approaches

Continuous testing is inspired by the TQM methodologies to support a repeatable process of Plan, Do, Check, Act. We find similar techniques in the agile rituals of planning and retrospective. One model we can inspire is the Holistic Testing of Janet Grégory.

Continuous testing is about acting over the overall value chain of the software lifecycle. To create value for the product users, one must prioritize the most valuable testing practices of Shift-Up, Shift-Left, and Shift-Right. For a test automation framework, that means executing seamlessly across the various environments and supporting the requirements definition and continuous testing processes.

Cerberus Testing natively provides an application framework management. We can define for a single application across its various environments, simplifying an end-to-end execution. From the middle of the chain, we can leverage the scheduling and CI/CD capabilities. In the end, the user journey monitoring is an example of a continuous testing approach supported by Cerberus Testing.

We close the tour of our framework with continuous improvement.

Provide measurement and visibility

We can’t improve what we can’t measure. Measurement is fundamental to measure the value of our test automation effort. By default, a framework will lack or provide a poor reporting capability, not being the core of the engine in the first place. The thing is, improvement and usability are requirements of a valuable test automation framework.

The typical metrics are the coverage, flakiness ratio, and detailed tests traceability. The value measurement should go above the test automation area, measuring key user experience satisfaction like NPS, engagement-time, and activity. Additionally, supporting measures of reliability are necessary at the core of a successful customer experience.

Concretely, the users of the framework are familiar with web interfaces or even mobile applications for dashboarding. It means that our framework should provide the various metrics in a graphical way. Reporting starts with the test repository to know how many tests we have, which areas, which status, and internal composition.

The next area is to measure the execution phase, providing traceability of individual tests as well as a global perspective. The test and campaign reporting are natively available within Cerberus Testing, removing the need to implement custom dashboards. Additionally, test analytics goes a step further by providing trends analysis over a more extended period of time with new insights.

A test automation framework is a long journey when starting from scratch.

Value is required across the entire process and actors

A valuable test automation framework lies in being of value to its actors. It starts from automating the right tests to then implementing the tests right. That means supporting collaboration and providing a reliable framework are two essential requirements.

The ability to cover the end-to-end processes with a unified interface enables fast feedback loops. This allows us to start with experimentation before expanding and optimizing the test automation effort. This is why reusable modules, measurement, and rework functionalities make a difference.

Building a test automation framework is not a one-night nor a one-man effort. It requires time, dedication, and iterations. We started Cerberus Testing in 2010, releasing it in open-source in 2013. Having similar needs to solve, we are more than happy to share on its usage, capabilities, and even possible contributions.

Whatever your choice is, a valuable test automation framework is our common goal.

You can use Cerberus Testing for free, start now.