Building an Open Source Hybrid Test Automation Framework

cerberus-building-open-source-hybrid-test-automation-framework-featured

When starting test automation, it seems reasonable to reuse our existing manual tests to perform a record & replay model, also known as a Linear automation framework.

Afterward, we need to compensate for the limitations of the linear model, adding reusable functions, modules, and components. Finally, after a while, we identify that we can even reach an additional level of abstraction with a library, data, and keywords.

We will start to build modular, library, data-driven capabilities to our test framework. Problems come with the increasing maintenance of our frameworks; the team coding the solution has less time to perform the initial testing. Developing a user interface is even another challenge teams do not have sufficient time available.

Test automation frameworks available in open-source exist to support transversal collaboration, reuse, and maintenance across organizations having similar problems. This article shares the various test framework typology, how Cerberus Testing incorporates them, and which evolutions are coming.

Linear Automation Framework

The linear approach relies on a “record and playback” model, mainly used when moving from manual to automated testing. The goal is to reproduce exactly the recorded actions done by a human operator.

Cerberus Testing implements the linear framework with an import functionality of Selenium IDE scenarios. Cerberus Testing is designed to scale test automation, combining other framework types with built-in collaboration. Let’s look at the pros and cons of the linear framework to understand why.

Figure 1: The Import of Selenium IDE inside Cerberus Testing

This replay model offers the advantage of a simple way to create a test suite with a manual testing team, requiring few technical skills. The recording is usually available on a graphical interface with a live recorder or an import functionality, simplifying the collaboration of the various actors.

A linear automation framework falls short in scaling our test automation effort. We will reach various limitations in reusability, maintainability, and coverage. We have to complement our approach with modular, library, or keyword-driven testing frameworks. 

Modular Based Testing Framework

Modularity is one of the key design concepts of software engineering. A suitable identification of function, perimeter, and interactions supports a better understanding, decoupling, and reuse of components.

Modular test frameworks are helpful when growing our test suite. Cerberus Testing incorporated from the start the concept of “Step” allowing us to define clear modules within our test. That way, we can compose other tests based on reusable parts we can maintain more easily. You can look at this article to scale test automation with modular testing.

A modular testing framework helps to solve the limitation of the linear one, providing better foundations for expansion. However, modularity comes with the cost of setup and maintenance that someone must manage. We made the choice to have modules with both functional and technical vision to foster collaboration.

Modules are useful but still lack essential elements of scaling: flexible library, data-driven, and keyword-driven. Let’s explore each of them.

Library Architecture Testing Framework

A library takes the modules approach up to the next level, providing reusable components on the different elements composing a test. Concretely, instead of only reusing actions inside a particular test, we can compose larger components.

The library architecture testing framework is available in Cerberus Testing on various test elements. It is present at the test level, allowing to reuse action and controls using the notion of “Step”. From that point, a “Step” marked as a Library one is reusable in other tests.

Figure 2: The Concept of Step inside Cerberus Testing

We can also manage our library of an entire test case or visual elements such as buttons. The various libraries are beneficial with a growing number of test cases and teams collaborating on automation. Proper design becomes a requirement to scale.

The library architecture is an advanced form of a modular test framework, still lacking the scalability of data and keywords.

Data-Driven Framework

We manage to have correctly structured test cases with reusable modules and libraries with previous frameworks. Some scenarios have a similar structure but require different data sets from boundary analysis or other techniques of test identification.

We can duplicate our test cases affecting a different data-set to each one, but this approach lacks scalability. Data-driven frameworks come into play, decoupling test actions from test data.

Pragmatically, we have one test structure containing the various steps, actions, and controls. Then, we can execute the same test across multiple data contexts, usually iterating on a data perimeter that is either fixed or dynamic. We use flat files such as CSV and XLS for a fixed perimeter, whereas dynamic perimeter will rely on a Database, API, or random access within a static perimeter.

Cerberus Testing supports data-driven testing to retrieve data in a fixed or dynamic perimeter. It relies on a “Data Library” concept across all tests, completed by “Properties” inside each test case. A property can be of various types (fixed, dynamic, retrieved in HTML, with javascript, etc) and connected to the library.

Figure 3: The Different Data Library available in Cerberus Testing

There is no capability to iterate over a dynamic dataset with the same test case inside Cerberus Testing. This design decision supports better visibility of the test suite, execution, and management. Having one test definition that can create multiple execution contexts with different data does complexify the analysis; if it’s failing, is it due to the test structure or one particular data set? How do I know that for one test, all test data were executed?

Today we rely exclusively on the library architecture that does scale with the cost of duplicating the test structure. The library enables to keep the whole test suite maintainable with a much more coherent library. Cerberus Testing users maintain a suite of 1000 to 5000 test cases without problem that way.

Providing a keyword-driven test framework is the last important piece of a robust test automation framework.

Keyword-Driven Framework

The previous test frameworks focus mainly on the external structure and perimeter of the test case. On the contrary, a keyword-driven framework aims to provide a reusable internal test case structure for actions and controls.

Without a keyword-driven framework, each test case will have its set of actions and control. The steps and library can help, but they will not ensure the coherence of our entire automation test suite cross-applications. Keyword-driven frameworks provide reusable, transversal, and shared action plus control structures.

Keywords are generally of action or control typology. Each typology can then be divided per type of testing such as web, mobile, or API, even if not a strict requirement. In Cerberus Testing, both keywords are available to provide a whole set of capabilities natively. We can talk about a library of actions and controls.

Figure 4: An example of Cerberus Testing Action Library

The main advantage of keywords is providing a reusable transversal structure while simplifying more complex action and control. In addition, each keyword has a particular parameter that will support easier use. For instance, we can specify an XPath when using a click action, whereas to recognize an image, we need to set an image in a parameter.

A proper keyword-driven testing framework allows the team to validate the application and grow a robust test suite instead of coding their test library.

We Need a Hybrid Test Automation Framework

Each test framework typology is complementary to each other; there is no single test framework to use. Therefore, we can talk about “Hybrid Test Automation Framework” to support the different requirements of test automation and applications under test.

Emerging frameworks of model-based and process-driven testing appear, providing an additional level of abstraction on top of the library and test automation architecture. These models are promising to bridge the gap between the business behavior and application changes to reflect in our test suite.

Test Automation Frameworks must evolve according to this whole landscape to provide a reusable test platform to our teams. Our motto “Stop Coding, Start Testing” is driven by the open-source product and culture. By a transversal collaboration, we can solve once a problem to the benefit of the whole community.

Join the Cerberus Testing community for free now.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.