The Tested Way to Build a UI Test Automation Framework


Delivering successful user interfaces is one of the main challenges of digital transformation. We must constantly iterate to improve the experience while regularly providing novelty to maintain user engagement.

This acceleration of user interface changes creates a challenge for engineering teams: how to ensure our core functionalities are still working? Are the key use-cases working as expected? Is it working on the various web browsers?

QA teams will implement automated tests to help the team keep confidence in their changes. The team will want to build a UI automation framework to have a repeatable, reliable, and expandable solution.

This article shares the key requirements of a UI automation framework and how we implemented it with an open-source approach.

What are the requirements for a UI automation framework?

Preparation is fundamental to ensure the delivery afterward. Building a UI automation framework is no exception. We can quickly implement a POC, but a proper framework requires a set of core features.

We have to start with why clarifying what we are trying to achieve with our UI automation framework. Trivial features come natively to our mind, like defining the test structuring and executing it. Other ones come over time like reporting, reusable components or parallel execution.

We can summarize the features in the following list :

  • Define a test case structure for the application under test
  • Interact with the various UI locators with an abstraction layer
  • Execute in parallel and across multiple platforms
  • Visualize the results with test reporting and analytics
  • Support maintainability by creating modules and reusable components

Here’s an example from InfoQ on Functional GUI Testing Automation Patterns.

Figure 1: A functional GUI Framework 

We will need to keep our initial goal in mind through iterations. Our goal is to support fast verification of the UI interface while ensuring the non-functional requirements of usability, portability, or maintainability. 

Let’s start with the first part, defining the test layer.

First things first, define the test case structure

A test case structure has essential elements. We will need a description, a series of actions, and controls. Key design decisions lie in the relationship and attributes of the various objects. For instance, we can logically group actions and controls inside a step.

Automated tests then have specificities compared to manual testing. For instance, we will want to define the various application environments applicable to our test. In the case we want dynamic data, we can also use specific datasets retrieving data automatically.

Figure 2: The Step, Action and Control Structure of Cerberus Testing 

In Cerberus, we choose to use a test case structure compatible with UI testing and other typologies inspired by the BDD structure. Each action and control can be grouped into a step, all part of a test case. Each action is then defined with a description, a specific action type, and parameters.

A test case is linked to an application and can be enabled for particular environments.

Interact with the elements, simplifying their usage

Building a UI framework naturally requires interacting with user interface elements. The various interfaces must be considered like web, mobile application, or desktop. Conversational ones are not directly considered part of the UI category.

A traditional approach uses standard readily available drivers, such as Selenium Driver or Appium for mobile applications. A framework’s key design consideration is to build or not an abstraction layer, independent from the typology of tests to perform.

Figure 3: The unified test automation framework approach of Cerberus Testing

Explicit trade-offs are a valuable practice to balance our perspective of particular choices. In our case, we wanted to build a cross-platform test automation framework, so an abstraction layer was a meaningful choice. The cost comes with managing the complexity of the UI and engine of this abstraction layer.

The engine can perform web, mobile, and desktop UI testing based on the same test case structure. That way, business and engineering teams can collaborate more easily on the plain language BDD-style format, where implementation details can be hidden.

Our tests are defined and ready to interact with the UI layers; they now need execution.

Execute UI tests in various contexts, fast

User interface tests have the particularity of a large set of execution contexts. We have to execute them across different operating systems, devices, and browsers. Like with the driver, we need a runtime abstraction layer.

Execution of UI tests can be done on physical or virtual devices using an API to launch a test and execute a series of actions and controls. Selenium Grid is a common choice for web testing, whereas Appium Hub is an option for mobile applications.

Figure 4: The various robots can be combined for execution, in this example with LambdaTest 

We relied on open solutions for Cerberus Testing, aligned with our open-source philosophy. In addition, we introduced the notion of “Robots” to leave flexibility of connecting to a local or remote platform. The key requirement is to be compatible with the integrated open drivers.

You can notice a series of execution options natively available like verbosity level, snapshots, and timeout parameters. These features are all increments a UI automation will require overtime to answer the different requirements.

Reporting is a key aspect of test automation. We can only improve what we can measure, and we don’t have time to parse logs or construct a new dashboard every time we need one.

Visualize the results with reporting and analytics

The first need that arises is to visualize the test reporting status and key metrics. We need to answer the common questions of overall campaign status, duration, and features health in the particular execution context.

Our reporting layer must provide these dashboards off the shelf for a successful UI framework. Various zooms are required, starting with an overall picture of the execution within our platform, then being able to zoom from campaign to individual test case reports.

Figure 5: An example of the Cerberus Testing overall execution dashboard 

Test case reporting requires a specific level of traceability for various purposes. For understandability, we require to have a clear understanding of each action and control performed, with associated data. The execution is also essential: robot logs, screenshot on error, timings of each step. Ideally, these features are provided by the framework, freeing up the test case from those requirements.

Over time we will accumulate a series of test execution data. It is important to note that keeping history is also a critical requirement in that aspect, data is valuable. At that point, analytics requirements can make sense to analyze over-time our automated test data. We can, for instance, look at our stability over time, identifying if specific contexts are more prone to errors.

Figure 6: The Cerberus Testing analytics dashboard 

With more tests also comes the requirement of expanding on stable foundations. The framework must provide scalability and maintainability.

Provide modules and reusable components for maintainability

Maintainability has various attributes but essentially means adapting our test quickly with changing application behavior or requirements. Like with any software, we have to do it fast, with confidence and little effort over time.

Test automation maintainability is covered in some areas outside of the framework. Prioritizing and selecting the practical test to automate is a best practice to avoid unnecessary work in the first place. For the one implemented, we can rely on several suitable methods as well.

Our test automation framework must provide the reusability of elements to optimize our implementation effort. The modularity can be applied to test cases, steps, actions, controls, and properties, providing reusable components on their definition. A concrete example in Cerberus Testing is the step, action, and control library you can compose.

Figure 7: The Cerberus Testing action and control library in action implementation 

Tooling is not everything. While we can have a very good UI test automation framework, our design and implementation decisions remain key. Balance and trade-offs are key points to secure; we cannot always have perfect or extreme solutions.

The case of modularity is a good one. We need “just-enough modularity” for efficient test automation. Not following this principle will result in highly coupled tests we cannot change, contrary to our initial goal.

A UI test automation framework is a long journey.

An automated test UI platform, more than a framework

We covered the key elements of a UI automation framework, identifying the core components to support fast feedback loops in our test automation journey. The identified complexities materialize the need for a platform rather than a framework.

A UI automation framework will provide the key features to implement our test in a defined structure. We will also execute test cases under certain limitations and contexts, depending on the time we have available under time pressure constraints.

A UI Automation platform makes more sense in our current context of acceleration and diversification of user interfaces. We must increase our quality and test automation for the growing number of applications and changes happening on digital products. 

This pressure of time requires fast feedback loops where our UI tests cannot last for hours before giving their results. We need a scalable solution to get fast results on the various environments, hence the value of a ready-to-use platform providing native integration, parallel execution, and simplified usage.

Cerberus Testing has years of development invested with the open-source community to provide a valuable test automation framework and platform for UI and other use-cases. We would be happy if you decide to contribute and use it.

You can easily start with a Free plan at Cerberus Testing.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service

Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.