How To Make Your Test Automation Framework Scalable by Design

Scalability is a challenge.

The definition of scalability is the capacity to be changed in size or scale, and to support an increasing number of capabilities. In the case of an automation framework, scalability translates in the number of users, application under test, executions and use-cases it can support over-time.

However, scalability is too often a missed requirement. Teams rush to implementation as there is no need for scale on a fresh project.

The problem is that scalability problems are caused by underlying design issues that are hard to solve when the whole framework has been built. That’s why you should care about scalability by design to set the right structure from the beginning, sustaining the lifetime of your test automation framework.

Follow Cerberus Testing for more open-source test automation.

Enable an increasing value through test automation

The need for scalability will materialize if the framework provides an increasing value to a set of stakeholders. For test automation, that’s about giving confidence to accelerate software delivery to cross-functional teams.

The first design consideration for scalability is to actually provide value to the different stakeholders of test automation. The more value you will deliver, the more scalability your framework will have to support.

A Product Owner, QA, or Software Engineer have different backgrounds and technical sensibility. An important design consideration for your test automation framework is therefore to foster the collaboration of these different profiles.

Providing them with a common user interface supporting back and forth sharing on the test automation lifecycle is essential. A restrictive interface reserved to technicians for scripting will miss the point of overall value creation.

Decouple the key areas of test automation

Test automation requires a different set of functions to be performed on different objects. Clear boundaries are necessary to avoid accumulating complexity that will first make it hard to change the system, until it is not possible to evolve anymore.

Separation of concerns enable you to draw clear boundaries between contexts of high cohesion and low coupling. Properly defined lanes will let you evolve independent parts with limited impacts on the overall systems, improving its scalability.

You can split the key areas of test automation in the main domains of test repository, execution and reporting. Each of these areas can focus on solving its core problem, relying on explicit interfaces when it needs sharing with another one.

Another required separation is on a technical layer. Clearly separating your front-end and back-end layers through APIs will support the addition of a mobile interface later on, and ease the connectivity of external solutions. But first, let’s cover internal scalability.

Separate applications, test cases and data

Test cases are pivotal objects in test automation. They are used in every domain with different facets depending on the cases at hand. Still, key elements revolving around test cases must be explicitly decoupled for scalability.

The key objects to decouple from test case are:

• The application under test, including the environments and countries.
• The test data can be different per context, sources and variable.
• The test objects that can be reused across in multiple places.

The separation enables us to scale each part independently without impacting the whole system. You can add application environments to existing test cases, variabilize test data for multiple test cases and applications, and change one object used in various test cases.

Your framework must therefore be able to cope with object references and variabilization. This additional work is valuable in allowing software teams to scale their test automation effort, and let you evolve your framework more easily later on.

Enable parallel execution with flexibility

One main challenge of test automation is execution: a broad set of devices must be supported for fast and reliable execution. Slow or unstable execution will make your team doubts about test cases until they just stop using them.

The first element of execution requires horizontal scalability of the test robots to execute a large number of tests in parallel, keeping the total test execution time in a timing acceptable for the team.

One common way to provide test environments scalability is to let users configure with flexibility local and remote test agents. That way, they can spread the execution in their own context, and more easily ensure a stable environment will be available.

The second element of execution scalability is to be available in different contexts. While it is impossible to plan for all cases, you can provide test execution as a service. You have to provide a scheduler and APIs that enable tests to be triggered within CI/CD pipelines.

Offer key endpoints for extensibility

One solution cannot provide all functionalities for all use-cases. There is a limit to the complexity one product can provide. And even if that would be possible, it would be too risky to have all eggs in the same bag.

Offering endpoints on your main domain and use-cases ensure the extensibility of your framework. This by-design consideration will enable partners to connect to your solution, solving existing problems that were unsolved or with more efficiency.

APIs will usually give access to referential objects such as test repositories or test reporting. But they can also provide more flexible connectors in the form of webhooks, actions and notification services.

In a world of digital acceleration where software must be rapidly composed, the scalability of your framework depends on its extensibility. By connecting more users and partners, you increase your likelihood of survival, and also of becoming a platform. 

Scalability of a test automation framework is a platform

A scalable test automation framework is not the result of chance. The key elements we covered are essential design considerations that make the difference in the product evolution. “We should build something from scratch” is the sign of failure.

In addition to supporting more users, a test automation framework aiming to provide more use-cases has to consider scalability as a must-have requirement, at the same of functionality or security of other requirements.

We applied these scalability-by-design principles on Cerberus Testing to transform an emerging test automation framework into a test automation platform for web, mobile, APIs, database and desktop testing with continuous testing capabilities.

The joy of scalability is when you actually need it and can provide it. The value of your investment will materialize by accelerating the value you can provide to an increasing number of people.

Start now with Cerberus Testing, the 100% open-source test automation platform.