Building an Open Source Hybrid Test Automation Framework

When starting test automation, it seems reasonable to reuse our existing manual tests to perform a record & replay model, also known as a Linear automation framework.

Afterward, we need to compensate for the limitations of the linear model, adding reusable functions, modules, and components. Finally, after a while, we identify that we can even reach an additional level of abstraction with a library, data, and keywords.

We will start to build modular, library, data-driven capabilities to our test framework. Problems come with the increasing maintenance of our frameworks; the team coding the solution has less time to perform the initial testing. Developing a user interface is even another challenge teams do not have sufficient time available.

Test automation frameworks available in open-source exist to support transversal collaboration, reuse, and maintenance across organizations having similar problems. This article shares the various test framework typology, how Cerberus Testing incorporates them, and which evolutions are coming.

Linear Automation Framework

The linear approach relies on a “record and playback” model, mainly used when moving from manual to automated testing. The goal is to reproduce exactly the recorded actions done by a human operator.

Cerberus Testing implements the linear framework with an import functionality of Selenium IDE scenarios. Cerberus Testing is designed to scale test automation, combining other framework types with built-in collaboration. Let’s look at the pros and cons of the linear framework to understand why.

This replay model offers the advantage of a simple way to create a test suite with a manual testing team, requiring few technical skills. The recording is usually available on a graphical interface with a live recorder or an import functionality, simplifying the collaboration of the various actors.

A linear automation framework falls short in scaling our test automation effort. We will reach various limitations in reusability, maintainability, and coverage. We have to complement our approach with modular, library, or keyword-driven testing frameworks. 

Modular Based Testing Framework

Modularity is one of the key design concepts of software engineering. A suitable identification of function, perimeter, and interactions supports a better understanding, decoupling, and reuse of components.

Modular test frameworks are helpful when growing our test suite. Cerberus Testing incorporated from the start the concept of “Step” allowing us to define clear modules within our test. That way, we can compose other tests based on reusable parts we can maintain more easily. You can look at this article to scale test automation with modular testing.

A modular testing framework helps to solve the limitation of the linear one, providing better foundations for expansion. However, modularity comes with the cost of setup and maintenance that someone must manage. We made the choice to have modules with both functional and technical vision to foster collaboration.

Modules are useful but still lack essential elements of scaling: flexible library, data-driven, and keyword-driven. Let’s explore each of them.

Library Architecture Testing Framework

A library takes the modules approach up to the next level, providing reusable components on the different elements composing a test. Concretely, instead of only reusing actions inside a particular test, we can compose larger components.

The library architecture testing framework is available in Cerberus Testing on various test elements. It is present at the test level, allowing to reuse action and controls using the notion of “Step”. From that point, a “Step” marked as a Library one is reusable in other tests.

We can also manage our library of an entire test case or visual elements such as buttons. The various libraries are beneficial with a growing number of test cases and teams collaborating on automation. Proper design becomes a requirement to scale.

The library architecture is an advanced form of a modular test framework, still lacking the scalability of data and keywords.

Data-Driven Framework

We manage to have correctly structured test cases with reusable modules and libraries with previous frameworks. Some scenarios have a similar structure but require different data sets from boundary analysis or other techniques of test identification.

We can duplicate our test cases affecting a different data-set to each one, but this approach lacks scalability. Data-driven frameworks come into play, decoupling test actions from test data.

Pragmatically, we have one test structure containing the various steps, actions, and controls. Then, we can execute the same test across multiple data contexts, usually iterating on a data perimeter that is either fixed or dynamic. We use flat files such as CSV and XLS for a fixed perimeter, whereas dynamic perimeter will rely on a Database, API, or random access within a static perimeter.

Cerberus Testing supports data-driven testing to retrieve data in a fixed or dynamic perimeter. It relies on a “Data Library” concept across all tests, completed by “Properties” inside each test case. A property can be of various types (fixed, dynamic, retrieved in HTML, with javascript, etc) and connected to the library.

There is no capability to iterate over a dynamic dataset with the same test case inside Cerberus Testing. This design decision supports better visibility of the test suite, execution, and management. Having one test definition that can create multiple execution contexts with different data does complexify the analysis; if it’s failing, is it due to the test structure or one particular data set? How do I know that for one test, all test data were executed?

Today we rely exclusively on the library architecture that does scale with the cost of duplicating the test structure. The library enables to keep the whole test suite maintainable with a much more coherent library. Cerberus Testing users maintain a suite of 1000 to 5000 test cases without problem that way.

Providing a keyword-driven test framework is the last important piece of a robust test automation framework.

Keyword-Driven Framework

The previous test frameworks focus mainly on the external structure and perimeter of the test case. On the contrary, a keyword-driven framework aims to provide a reusable internal test case structure for actions and controls.

Without a keyword-driven framework, each test case will have its set of actions and control. The steps and library can help, but they will not ensure the coherence of our entire automation test suite cross-applications. Keyword-driven frameworks provide reusable, transversal, and shared action plus control structures.

Keywords are generally of action or control typology. Each typology can then be divided per type of testing such as web, mobile, or API, even if not a strict requirement. In Cerberus Testing, both keywords are available to provide a whole set of capabilities natively. We can talk about a library of actions and controls.

The main advantage of keywords is providing a reusable transversal structure while simplifying more complex action and control. In addition, each keyword has a particular parameter that will support easier use. For instance, we can specify an XPath when using a click action, whereas to recognize an image, we need to set an image in a parameter.

A proper keyword-driven testing framework allows the team to validate the application and grow a robust test suite instead of coding their test library.

We Need a Hybrid Test Automation Framework

Each test framework typology is complementary to each other; there is no single test framework to use. Therefore, we can talk about “Hybrid Test Automation Framework” to support the different requirements of test automation and applications under test.

Emerging frameworks of model-based and process-driven testing appear, providing an additional level of abstraction on top of the library and test automation architecture. These models are promising to bridge the gap between the business behavior and application changes to reflect in our test suite.

Test Automation Frameworks must evolve according to this whole landscape to provide a reusable test platform to our teams. Our motto “Stop Coding, Start Testing” is driven by the open-source product and culture. By a transversal collaboration, we can solve once a problem to the benefit of the whole community.

Join the Cerberus Testing community for free now.