How To Build An Awesome APIs Test Automation Framework

Your team has started to implement a whole set of APIs. While they seem powerful for the business, you have a new challenge to solve: testing them. Manual testing is fine for some cases, but you quickly need automation.

Implementing the right API test automation right requires structure to ensure its implementation. This is what a framework is about, providing a repeatable frame of reference to solve a set of problems.

In our case, we want to implement end-to-end API test automation aligned with the business requirements, additionally ensuring its accessibility, maintainability, and scalability.

This article shares the common characteristics of an end-to-end API test automation framework, highlighting its implementation within Cerberus Testing. The following points are covered:

• Why building an API Testing Framework?
• What is a valuable API Testing Framework?
• The personas and use-cases of an API Testing framework
• Build a reusable and scalable API library
• Implement tests in low-code with shared keywords
• Execute our APIs tests with speed and reliability
• Provide the necessary traceability and visibility
• Scale your API Testing leveraging the framework

Let’s start by aligning the why before entering in implementation details.

Why building an API Testing Framework?

APIs stand for Application Programming Interfaces. They are the bridge to perform data exchanges and perform functions. The advent of distributed systems has drastically expanded the maturity and usage of APIs within organizations.

APIs are nowadays the cornerstone of business digitalization and digitization. They enable to build fast a user experience relying on ready-to-use backend services published as APIs. Similarly, they accelerated the capability of back-end services to be reused by composition.

The maturity of APIs has significantly progressed since its first file, RPC, and SOAP protocols. Lightweight, more open, and standard protocols have emerged to fulfill specific requirements like REST, GraphQL, and gRPC.

APIs are now mission-critical to the survivability of companies. They enable a business to integrate with other partners and are core in the business operations execution. These critical APIs can leverage test automation to ensure their delivery.

We can now clarify the elements of an API test automation framework.

What is a valuable API Testing framework?

We face various questions once we decided to perform automated tests of our APIs. What to test? How to define the test? How to execute the tests? How to reuse and expand our approach? The framework is there to answer these elements.

Our framework needs to bring value to its users, not to the software under test in itself. That means our first focus should be on defining clearly the use-cases for the actors of API test automation. Traditionally, product, development, and operations are involved.

“Our framework needs to bring value to its users, not to the software under test in itself.”

Antoine Craske

An essential requirement is therefore the accessibility and collaboration of our framework. We easily forgot it if we rush into coding a specific API test. Based on the use-case of each actor, we can then drill down to the functions to provide.

Our overall objective of building a framework is to solve the initial business problem: ensure the repeatable fast delivery of valuable APIs. Our test automation must therefore support fast feedback loops for the team to iterate on the value delivery.

The business value of our framework will be contextual. Nevertheless, the DORA metrics of lead-time, change fail rate, MTTA and MTTR make all sense to measure our impact at the end of the day.

Let’s explore the different personas requirements.

The personas and use-cases of an API Testing framework

The product team will usually want to ensure particular end-to-end behavior of the user experience or business operations. That means they need a shared test repository, understandable, with access to the reporting of the various APIs tests.

The developers will be the ones fixing the APIs issues. Their main requirement is about fast execution, traceability, and repeatability of the tests. Once they got a bug affected, they have to quickly find the APIs, the problem, deliver a change, and verify if it’s working.

A side-note, developers helping setting up the APIs is a warning sign in the accessibility of the APIs. Alas, not understanding an API manually is a form of testing not requiring automation.

Operations have similar requirements of overall visibility like the product team. Additionally, like developers, they need to drill down on the technical view and report to speed up the resolution.

Let’s now move to the implementation details. The example used is based on REST and GraphQL for a single API test but also for an end-to-end test, testing web and API elements within the same test.

The first foundation is to have reusable API components.

Build a reusable and scalable API library

An API library will help us cope with the inherent complexity that will come. We have to start testing one API in one environment, and then it’s for all environments, different countries. The stories repeat for validating other use-cases and other APIs. 

An API library is there to contains the complexity of the application to test. It will include the various APIs definition such as endpoints, parameters, and specific connection information.

The value of the library is to abstract our tests from the configuration details. We manage to decouple the API definition for its testing. That way, any change to an API definition requires a single change, not impacting the test cases.

For example, if we need to add a new environment to the API, it’s in a single place. The same to add a country. The same to change an authentication with a fixed token like a Bearer one.

We can now move to the API test definition.

Implement tests in low-code with shared keywords

Our API test structure framework can follow the “given, when, then” structure. That way, they are more accessible, being a standard practice and also integrable within other tests for the end-to-end requirement. Before coding, don’t forget our personas. 

We identified that accessibility and usability are vital requirements. They translate by building a keyword-driven test structure leveraging low-code, available in a web interface to foster collaboration.

Once our test case structure is defined, we need to enrich it to support more advanced use-cases. The first one is to support dynamic properties, to retrieve a dataset from another system, for example. This is also requires to implement data-driven techniques.

A usual pain-point is the definition of the controls, more painful with APIs; we have to find the path, try to find some elements, parsing the whole response. A good practice is to provide a parser to that effect. In Cerberus Testing, it is possible using a “$.<element>” with a getFromJson action.

Our next goal is to provide the execution capability.

Execute our APIs tests with speed and reliability

The execution of our APIs tests must support the fast and reliable feedback loops. We can achieve that by providing test suites we can launch using a parallel execution engine with built-in reliability features.

A test suite is a logical way of grouping tests. We find value when our number of tests starts to grow and we need to execute various tests to know the particular state of a system. Another advantage is to run what we want for specific purposes.

Once know what to execute, we have to deal with the execution. A practical approach would be to run the tests one after the other. In reality, our users won’t accept slow feedback loops; parallel APIs execution is a mandatory requirement.

Executing a test is not an easy journey. Like for our business software, the network can fail or the API can be temporarily available. We have to cope with these potential issues by design, providing reliability features like retries with back-off. 

Our job is not done yet. Our users want to know what is happening.

Provide the necessary traceability and visibility

The challenge of observability also applies to our automated tests. Our users want to know the overall status of the campaign while being able to zoom at each action and control performed. The appealing of the results is also important to consider.

Would you accept a csv extraction to look at your bank account? Not anymore. Our users have the same exigence for the test automation visualization. We cannot provide a static web page. They want a beautiful graph that lets them explore the data.

The first level of information to provide is at the test case level. We must keep a trace and logs of the action and controls perform, plus contextual elements like the environment, timing, etc. That way, our users know directly the status of the involved steps.

Taking a step back, the team quickly wants to answer: “Can we deploy our changes to our final users in production?”. They need to know the test campaign’s overall health with major indicators and the status of each test cases part of the suite executed.

A standard ratio to provide is flakiness. It measures the stability of the tests in a percentage, allowing to identify the trend and specific unstable tests. It is essential to aims for stable tests to maintain fast feedback loops the team can trust. The team will start to discard a campaign with frequent failures and a long execution time.

Assuming we have done the previous steps right, we are ready to expand our APIs test automation.

Scale your API Testing leveraging the framework

We went through the process of building an API test framework through the prism of one application. The enterprise reality has many APIs across different teams to solve different business use-cases. A framework brings additional value here.

We can reverse the situation. What would happen without a shared framework? Each team, or even worse, every tester will implement the tests in its own way. While they would reach partially similar requirements, their implementation will be different, lacking integration and interoperability. 

Therefore, a framework brings value by first being valuable to its initial users and being shareable across the company. A true framework is a product we can expand like the business services to our end-users.

Building a valuable framework does not happen overnight, It is the fruit of continuous improvement over various iterations. Cerberus Testing is one product available natively  providing the features we covered, including the end-to-end test.

Interested to know more? You can use it for free, start now.

See the full video here

https://www.youtube.com/watch?v=j1qnFaHr59I

References

Alexandre Mohr (2018), How to Achieve Collaboration as a Key Driver for Continuous Testing. 2018.

Antoine Craske (2021), Automate REST APIs Test with Cerberus Testing Framework. Cerberus Testing.

Antoine Craske (2021), Outil de Test : Automatisation avec Cerberus Testing. La Taverne Du Testeur.