The Tested Way to Build a UI Test Automation Framework

Delivering successful user interfaces is one of the main challenges of digital transformation. We must constantly iterate to improve the experience while regularly providing novelty to maintain user engagement.

This acceleration of user interface changes creates a challenge for engineering teams: how to ensure our core functionalities are still working? Are the key use-cases working as expected? Is it working on the various web browsers?

QA teams will implement automated tests to help the team keep confidence in their changes. The team will want to build a UI automation framework to have a repeatable, reliable, and expandable solution.

This article shares the key requirements of a UI automation framework and how we implemented it with an open-source approach.

What are the requirements for a UI automation framework?

Preparation is fundamental to ensure the delivery afterward. Building a UI automation framework is no exception. We can quickly implement a POC, but a proper framework requires a set of core features.

We have to start with why clarifying what we are trying to achieve with our UI automation framework. Trivial features come natively to our mind, like defining the test structuring and executing it. Other ones come over time like reporting, reusable components or parallel execution.

We can summarize the features in the following list :

• Define a test case structure for the application under test
• Interact with the various UI locators with an abstraction layer
• Execute in parallel and across multiple platforms
• Visualize the results with test reporting and analytics
• Support maintainability by creating modules and reusable components

Here’s an example from InfoQ on Functional GUI Testing Automation Patterns.

We will need to keep our initial goal in mind through iterations. Our goal is to support fast verification of the UI interface while ensuring the non-functional requirements of usability, portability, or maintainability. 

Let’s start with the first part, defining the test layer.

First things first, define the test case structure

A test case structure has essential elements. We will need a description, a series of actions, and controls. Key design decisions lie in the relationship and attributes of the various objects. For instance, we can logically group actions and controls inside a step.

Automated tests then have specificities compared to manual testing. For instance, we will want to define the various application environments applicable to our test. In the case we want dynamic data, we can also use specific datasets retrieving data automatically.

In Cerberus, we choose to use a test case structure compatible with UI testing and other typologies inspired by the BDD structure. Each action and control can be grouped into a step, all part of a test case. Each action is then defined with a description, a specific action type, and parameters.

A test case is linked to an application and can be enabled for particular environments.

Interact with the elements, simplifying their usage

Building a UI framework naturally requires interacting with user interface elements. The various interfaces must be considered like web, mobile application, or desktop. Conversational ones are not directly considered part of the UI category.

A traditional approach uses standard readily available drivers, such as Selenium Driver or Appium for mobile applications. A framework’s key design consideration is to build or not an abstraction layer, independent from the typology of tests to perform.

Explicit trade-offs are a valuable practice to balance our perspective of particular choices. In our case, we wanted to build a cross-platform test automation framework, so an abstraction layer was a meaningful choice. The cost comes with managing the complexity of the UI and engine of this abstraction layer.

The engine can perform web, mobile, and desktop UI testing based on the same test case structure. That way, business and engineering teams can collaborate more easily on the plain language BDD-style format, where implementation details can be hidden.

Our tests are defined and ready to interact with the UI layers; they now need execution.

Execute UI tests in various contexts, fast

User interface tests have the particularity of a large set of execution contexts. We have to execute them across different operating systems, devices, and browsers. Like with the driver, we need a runtime abstraction layer.

Execution of UI tests can be done on physical or virtual devices using an API to launch a test and execute a series of actions and controls. Selenium Grid is a common choice for web testing, whereas Appium Hub is an option for mobile applications.

We relied on open solutions for Cerberus Testing, aligned with our open-source philosophy. In addition, we introduced the notion of “Robots” to leave flexibility of connecting to a local or remote platform. The key requirement is to be compatible with the integrated open drivers.

You can notice a series of execution options natively available like verbosity level, snapshots, and timeout parameters. These features are all increments a UI automation will require overtime to answer the different requirements.

Reporting is a key aspect of test automation. We can only improve what we can measure, and we don’t have time to parse logs or construct a new dashboard every time we need one.

Visualize the results with reporting and analytics

The first need that arises is to visualize the test reporting status and key metrics. We need to answer the common questions of overall campaign status, duration, and features health in the particular execution context.

Our reporting layer must provide these dashboards off the shelf for a successful UI framework. Various zooms are required, starting with an overall picture of the execution within our platform, then being able to zoom from campaign to individual test case reports.

Test case reporting requires a specific level of traceability for various purposes. For understandability, we require to have a clear understanding of each action and control performed, with associated data. The execution is also essential: robot logs, screenshot on error, timings of each step. Ideally, these features are provided by the framework, freeing up the test case from those requirements.

Over time we will accumulate a series of test execution data. It is important to note that keeping history is also a critical requirement in that aspect, data is valuable. At that point, analytics requirements can make sense to analyze over-time our automated test data. We can, for instance, look at our stability over time, identifying if specific contexts are more prone to errors.

With more tests also comes the requirement of expanding on stable foundations. The framework must provide scalability and maintainability.

Provide modules and reusable components for maintainability

Maintainability has various attributes but essentially means adapting our test quickly with changing application behavior or requirements. Like with any software, we have to do it fast, with confidence and little effort over time.

Test automation maintainability is covered in some areas outside of the framework. Prioritizing and selecting the practical test to automate is a best practice to avoid unnecessary work in the first place. For the one implemented, we can rely on several suitable methods as well.

Our test automation framework must provide the reusability of elements to optimize our implementation effort. The modularity can be applied to test cases, steps, actions, controls, and properties, providing reusable components on their definition. A concrete example in Cerberus Testing is the step, action, and control library you can compose.

Tooling is not everything. While we can have a very good UI test automation framework, our design and implementation decisions remain key. Balance and trade-offs are key points to secure; we cannot always have perfect or extreme solutions.

The case of modularity is a good one. We need “just-enough modularity” for efficient test automation. Not following this principle will result in highly coupled tests we cannot change, contrary to our initial goal.

A UI test automation framework is a long journey.

An automated test UI platform, more than a framework

We covered the key elements of a UI automation framework, identifying the core components to support fast feedback loops in our test automation journey. The identified complexities materialize the need for a platform rather than a framework.

A UI automation framework will provide the key features to implement our test in a defined structure. We will also execute test cases under certain limitations and contexts, depending on the time we have available under time pressure constraints.

A UI Automation platform makes more sense in our current context of acceleration and diversification of user interfaces. We must increase our quality and test automation for the growing number of applications and changes happening on digital products. 

This pressure of time requires fast feedback loops where our UI tests cannot last for hours before giving their results. We need a scalable solution to get fast results on the various environments, hence the value of a ready-to-use platform providing native integration, parallel execution, and simplified usage.

Cerberus Testing has years of development invested with the open-source community to provide a valuable test automation framework and platform for UI and other use-cases. We would be happy if you decide to contribute and use it.

You can easily start with a Free plan at Cerberus Testing.