cerberus-parallel-exec-featured

Companies are in search of quality at speed to remain competitive.

This creates a strong drive of accelerating the iterations of software increments that are useful and reliable.

Verifications that are good candidates for automation are largely implemented using test automation, increasing adoption by the various actors looking for agility.

The test platform and its parallelization capability is then a key supporting capability you need in your automation journey, at the risk of failing at supporting the product teams.

In this article, we share the key concepts of parallel test execution and the implementation capabilities inside Cerberus Testing.

The need for parallel testing in test automation

The requirements for requiring test execution parallelization are more and more relevant in our ecosystem.

We observed a significant investment for test automation, increasing the number of test suites to be executed, in reports such as the World Quality Report 2020-21.

Our journey is richer in the variety of devices we need to deal with, from the various web browsers, smartphones and different IoT devices.

Additionally, the need for fast feedback loops in a setup such as CI/CD is pushing for maximizing the parallelization in the shortest execution time.

Those various factors are pressuring the quality assurance ecosystem for fast and reliable test execution if we want to keep our promised value to the team.

The requirements for automated parallel execution

Proper parallelization of tests is easier said than done.

If you did some programming or tried to implement multi-threaded applications, you can normally remember that pain.

For test automation, the key functional requirements are :

  • Launch test in parallel at the desired level of parallelism
  • Be able to test the various devices for the application under test
  • Have the capacity to prioritize between different tests execution
  • Define a different parallel threshold per test context

The non-functional requirements can make the difference, namely :

  • Performance solution to support fast execution 
  • Portability for internal and external infrastructures
  • Compatibility with the various devices requirements
  • Usability for ease of integration and deployment
  • High-availability always-on of the parallel execution service
  • Reliability of the execution to avoid retries and delayed feedback
  • Secured execution from external sources
  • Traceability of the execution and test context (pictures, videos, …)
  • Monitorability to be able to alert and recover the service if needed 
  • Cost optimization scaling-up and down with minimum boot time

The aligned open-source test framework architecture 

Our implementation in the Cerberus Testing platform had to answer those various requirements.

A key design decision we took was to keep decoupling between the test repository and its execution :

  • Test Repository
    • Store the various test, actions and controls
    • Maintain the robots, test suite and pre-defined execution parameters
    • Keep the desired test execution contexts
  • Test Execution
    • Perform and secure the execution mechanisms
    • Deal with the parallelization execution over the various devices
    • Collect the test, robot and execution data 

That way, we have a clear separation of responsibilities and concerns allowing abstraction and independent scaling of each area.

The concept and definition of the Robot

We introduced the concept of the Robot to abstract the configuration of the execution environment.

A Robot needs to be defined especially for web, mobile and desktop testing, the APIs and database one being handled automatically within Cerberus Testing.

Each robot contains the definition of test typology, execution capabilities, connection information such as login, urls and password that is then automatically hidden.

Figure 1: The Robot Definition inside Cerberus Testing

The interesting part of the Robot is in its configuration flexibility :

  • A test can run on any number of robots 
  • A robot can run in any test environment
  • A robot can be load-balanced across different farms (internal or external)
  • The high-availability mechanism can be chosen (active/passive, round-robin)
Figure 2: The load-balancing parallelization capability of Cerberus Testing

The introduction of the Execution Queue

The next key element we introduced was the concept of the Execution Queue Management to deal with the prioritization requirement.

A traditional use-case is the execution of regular functional tests monitoring the customer journeys in production, while during the day you will have a non-regression campaign in a test environment.

Figure 3: The Cerberus Testing Execution Queue Dashboard

You want your customer journey tests to run as expected on schedule, without waiting for the UAT tests to end, especially if they are not yet fast enough.

This layer enables the prioritization of various tests launched in parallel in different contexts to solve this.

You can also administer your execution queue to cancel specific ones, update the backlog and other standard tasks. 

The types of parallel tests available with Cerberus Testing

All the test typologies supported by Cerberus Testing are available for parallelization.

As of today, this covers web, mobile, APIs, desktop and databases, on the variety of devices and vendors supported.

You can mix different types of parallel execution Inside the same instance of Cerberus as desired.

The main constraints we have to deal with are the execution infrastructure and the devices available.

By default, the Cerberus Testing Free Cloud comes with the capability to execute web on firefox, chrome, APIs, database.

Then if you need more browsers, mobile devices or desktop additional setup are required.

The various parallel testing integrations in Cerberus Testing

You can integrate with different farms: local ones on your laptop, private cloud, public cloud and even SaaS services.

Figure X: The main Testing Farms supported by Cerberus Testing

The main standard market integrations are supported by Cerberus Testing :

  • Selenium Hub and Zalenium for web
  • Appium Hub for mobile
  • Sikuli for desktop
  • Standard APIs and JDBC drivers for APIs and database

You have the flexibility to combine the different typologies of internal and external as desired, to optimize your infrastructures.

A traditional setup is to use each piece on its strengths :

  • A private farm or via the Cerberus SaaS the main robots needed, for speed of execution, lower costs and control
  • Public SaaS services such as LambdaTest, Browserstack, Kobiton, Saucelabs for more diverse devices and browsers

The various ways you can perform parallel testing

Parallel execution requires availability in a variety of contexts supported by Cerberus Testing.

The main use-cases are to use the Cerberus web interface while setting up the test case, to then automate them with CI/CD or scheduling mechanisms.

Here are the three typologies you can launch Cerberus Testing parallel test :

  • Via the web interface
  • Using the public APIs
  • Using the campaign scheduler

Based on those possibilities, you have the flexibility to integrate it as you desired. For example, using the native bitbucket CI/CD integration or your own scheduler calling the APIs.

The campaigns are very useful to define your test suites that you then execute in CI/CD during the development lifecycle, to then schedule them in production.

The best practices we recommend for parallel test automation

The value of automated tests is influenced by their performance, reliability and maintainability.

Regular stories share the abandonment of test suites that are too slow to execute, with flaky tests and that become hard to evolve.

This is why you need to secure your investment in your automated test suite.

As we saw, the subject is a real matter of integrations, processes and design.

Our recommendation is to focus on keeping your test suite fast and reliable first. 

Only then, add gradually other tests while keeping an eye on the overall campaign performance and complexity of maintainability.

You can fine-tune larger campaigns afterward with automated retries and queue management parameters, even if we recommend keeping things simple in the first steps.

A series of iterations is normally required to improve your test and parallel architecture, a bit like doing a TDD of your automated test implementation.

Interested to experiment and know more? 

Get the Free version allowing the Free Trial version here: Start Now.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://www.cerberus-testing.com, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.