Companies are in search of quality at speed to remain competitive.

This creates a strong drive of accelerating the iterations of software increments that are useful and reliable.

Verifications that are good candidates for automation are largely implemented using test automation, increasing adoption by the various actors looking for agility.

The test platform and its parallelization capability is then a key supporting capability you need in your automation journey, at the risk of failing at supporting the product teams.

In this article, we share the key concepts of parallel test execution and the implementation capabilities inside Cerberus Testing.

The need for parallel testing in test automation

The requirements for requiring test execution parallelization are more and more relevant in our ecosystem.

We observed a significant investment for test automation, increasing the number of test suites to be executed, in reports such as the World Quality Report 2020-21.

Our journey is richer in the variety of devices we need to deal with, from the various web browsers, smartphones and different IoT devices.

Additionally, the need for fast feedback loops in a setup such as CI/CD is pushing for maximizing the parallelization in the shortest execution time.

Those various factors are pressuring the quality assurance ecosystem for fast and reliable test execution if we want to keep our promised value to the team.

The requirements for automated parallel execution

Proper parallelization of tests is easier said than done.

If you did some programming or tried to implement multi-threaded applications, you can normally remember that pain.

For test automation, the key functional requirements are :

• Launch test in parallel at the desired level of parallelism
• Be able to test the various devices for the application under test
• Have the capacity to prioritize between different tests execution
• Define a different parallel threshold per test context

The non-functional requirements can make the difference, namely :

• Performance solution to support fast execution 
• Portability for internal and external infrastructures
• Compatibility with the various devices requirements
• Usability for ease of integration and deployment
• High-availability always-on of the parallel execution service
• Reliability of the execution to avoid retries and delayed feedback
• Secured execution from external sources
• Traceability of the execution and test context (pictures, videos, …)
• Monitorability to be able to alert and recover the service if needed 
• Cost optimization scaling-up and down with minimum boot time

The aligned open-source test framework architecture 

Our implementation in the Cerberus Testing platform had to answer those various requirements.

A key design decision we took was to keep decoupling between the test repository and its execution :

• Test Repository
  • Store the various test, actions and controls
  • Maintain the robots, test suite and pre-defined execution parameters
  • Keep the desired test execution contexts
• Test Execution
  • Perform and secure the execution mechanisms
  • Deal with the parallelization execution over the various devices
  • Collect the test, robot and execution data 

That way, we have a clear separation of responsibilities and concerns allowing abstraction and independent scaling of each area.

The concept and definition of the Robot

We introduced the concept of the Robot to abstract the configuration of the execution environment.

A Robot needs to be defined especially for web, mobile and desktop testing, the APIs and database one being handled automatically within Cerberus Testing.

Each robot contains the definition of test typology, execution capabilities, connection information such as login, urls and password that is then automatically hidden.

The interesting part of the Robot is in its configuration flexibility :

• A test can run on any number of robots 
• A robot can run in any test environment
• A robot can be load-balanced across different farms (internal or external)
• The high-availability mechanism can be chosen (active/passive, round-robin)

The introduction of the Execution Queue

The next key element we introduced was the concept of the Execution Queue Management to deal with the prioritization requirement.

A traditional use-case is the execution of regular functional tests monitoring the customer journeys in production, while during the day you will have a non-regression campaign in a test environment.

You want your customer journey tests to run as expected on schedule, without waiting for the UAT tests to end, especially if they are not yet fast enough.

This layer enables the prioritization of various tests launched in parallel in different contexts to solve this.

You can also administer your execution queue to cancel specific ones, update the backlog and other standard tasks. 

The types of parallel tests available with Cerberus Testing

All the test typologies supported by Cerberus Testing are available for parallelization.

As of today, this covers web, mobile, APIs, desktop and databases, on the variety of devices and vendors supported.

You can mix different types of parallel execution Inside the same instance of Cerberus as desired.

The main constraints we have to deal with are the execution infrastructure and the devices available.

By default, the Cerberus Testing Free Cloud comes with the capability to execute web on firefox, chrome, APIs, database.

Then if you need more browsers, mobile devices or desktop additional setup are required.

The various parallel testing integrations in Cerberus Testing

You can integrate with different farms: local ones on your laptop, private cloud, public cloud and even SaaS services.

Figure X: The main Testing Farms supported by Cerberus Testing

The main standard market integrations are supported by Cerberus Testing :

• Selenium Hub and Zalenium for web
• Appium Hub for mobile
• Sikuli for desktop
• Standard APIs and JDBC drivers for APIs and database

You have the flexibility to combine the different typologies of internal and external as desired, to optimize your infrastructures.

A traditional setup is to use each piece on its strengths :

• A private farm or via the Cerberus SaaS the main robots needed, for speed of execution, lower costs and control
• Public SaaS services such as LambdaTest, Browserstack, Kobiton, Saucelabs for more diverse devices and browsers

The various ways you can perform parallel testing

Parallel execution requires availability in a variety of contexts supported by Cerberus Testing.

The main use-cases are to use the Cerberus web interface while setting up the test case, to then automate them with CI/CD or scheduling mechanisms.

Here are the three typologies you can launch Cerberus Testing parallel test :

• Via the web interface
• Using the public APIs
• Using the campaign scheduler

Based on those possibilities, you have the flexibility to integrate it as you desired. For example, using the native bitbucket CI/CD integration or your own scheduler calling the APIs.

The campaigns are very useful to define your test suites that you then execute in CI/CD during the development lifecycle, to then schedule them in production.

The best practices we recommend for parallel test automation

The value of automated tests is influenced by their performance, reliability and maintainability.

Regular stories share the abandonment of test suites that are too slow to execute, with flaky tests and that become hard to evolve.

This is why you need to secure your investment in your automated test suite.

As we saw, the subject is a real matter of integrations, processes and design.

Our recommendation is to focus on keeping your test suite fast and reliable first. 

Only then, add gradually other tests while keeping an eye on the overall campaign performance and complexity of maintainability.

You can fine-tune larger campaigns afterward with automated retries and queue management parameters, even if we recommend keeping things simple in the first steps.

A series of iterations is normally required to improve your test and parallel architecture, a bit like doing a TDD of your automated test implementation.

Interested to experiment and know more? 

Get the Free version allowing the Free Trial version here: Start Now.