How To Shift Right With a Continuous Testing Framework

Shifting right enables an organization to focus on the actual user experience in production. This is a true practice of quality, orienting the team towards the common objective of delivering value to our users.

But shifting right is complex. We need to combine various practices in the world of operations without falling into technical details, hard to maintain, and costly solutions. A different approach from “let’s build something new” is possible.

A Continuous Testing initiative leads us to implement systematic quality steps through the software delivery lifecycle. We usually started from the left part of the requirements to secure the overall CI/CD chain with quality gates.

Cerberus Testing is an open-source test automation framework aiming at achieving a scale and collaborative quality effort in organizations. In this article, we will share how to implement shift-right with maximum value and minimal effort :

• Define the shift-right objectives from a user perspective
• Build up upon existing functional and non-regression assets
• Use reporting in push and pull for a transversal collaboration
• Measure the user experience performance and reliability
• Close the feedback loop by improving the initial flow of work

Delivering value through quality also requires us to follow systematic and structured processes. Let’s start by clarifying what we want to achieve.

Define the shift-right objectives from a user perspective

The general purposes of shift-right are mostly known as improving the actual user experience in production. It involves a set of existing and new practices like monitoring, observability, SRE (Site Reliability Engineering), among others. It can be hard to know where to start from our Continuous Testing perspective.

We will work with the context of a web and API application deployed in two countries, with production and non-production environment. Our first focus is to define what we can do to improve the measurement of quality in production. The best way is to change our perspective to the user one, thinking about what matters for them while using our product.

Our website provides a way to buy specific products our users may be looking for. Therefore, our shift right approach will focus on the key use-cases required to achieve that goal: visualize the product, add them to the basket, and pay. We additionally need to consider the non-functional cases necessary like accessing the homepage, logging in, and accepting the consent.

From this list of use-cases, we can then drill down the necessary quality dimensions to verify. In our case, we can consider the availability, functional behavior, and response time as key criteria for our users to remain connected. We can then look at what we can reuse from our previous Continuous Testing work.

Build up upon existing functional and non-regression assets 

We created a series of automated tests to use in our CI/CD quality gates. Their goal was to ensure the selected use-cases to remain functional while performing software changes. A non-regression campaign is usually larger than a production one and does not need a regular execution; we need to pick the most valuable ones.

We can rely on the useful functionality and practice of tagging the tests to quickly achieve our goal. Each test has a specific label telling the user experience area covered by the test case. The additional meta-data of the country and environment let us confirm the scope of execution meets our needs; in our case to be executed on the Web application, the production environment, and the two countries.

Our selection can then be added to a Cerberus Testing campaign specific to the shift-right requirements. Instead of being triggered within the CI/CD pipeline, we will schedule it to run periodically in production on the most used devices and browsers. That way, we can systematically verify if the user experience is meeting the quality requirements.

We will start to collect data that becomes valuable when used to improve the user experience from that work.

Use reporting in push and pull for a transversal collaboration

The added value of using a test automation framework is to access native functionalities that else we would need to implement. Test traceability and reporting are two requirements for test automation and shift-right we truly need but have no interest in developing. In our case, let’s start using them directly.

Cerberus Testing natively provides the traceability, reporting, and analytics of all the tests executed within the platform. The reporting can then be accessed in pull mode through the web interface, easily accessible even to non-technical users. Additionally, we can push notifications like slack or email on campaign executions. Soon, we will also allow the sharing of reporting data with test management solutions like Xray. 

The added value of reporting for our shift-right effort is in the measurement and associated decision-making. A first value is to have common objectives for different actors like Product Owner, Quality Engineer, Software Engineering, or an SRE. The next stage is to work collaboratively to implement the most valuable improvements based on our data.

In our case, we can start by acting on the various operational criteria of the experience.

Measure the user experience performance and reliability 

The selected use-cases to shift-right are oriented towards the user experience. On top of verifying the functional behavior, we added specific non-functional requirements control helpful to verify the user requirements are met. Then, the existing reporting allows us to look at the reliability aspects.

Cerberus Testing allows you to make non-functional requirements verifications like response time, browser log content, third-party API calls made. Concretely, we added control to ensure every page loads under 2 seconds and that the consentment partner API call is successful when accepting them on the homepage. Any degradation of a specific experience is then available via the notification of the reporting.

We can for example detect a response time problem only for one country on a specific browser. This is the power of continuous testing in shifting-right, localizing the problem fast. Using the reports available, we can verify reliability indicators like the flakiness ratio of our campaigns, stability of the execution, and the actual scheduling coverage.

The power of shifting right comes when taking actions based on quality attributes.

Close the feedback loop by improving the initial flow of work

We cover the way to reuse Continuous Testing assets for a shift-right approach. We rely on the test automation framework Cerberus Testing to accelerate our implementation. We saw that objective definition and its measurements are two key pillars.

Data is everything. It can drive much more factual discussion between various stakeholders. It helps to focus all the team on standard indicators, focused on the user experience. 

The Graal of Continuous Testing is to combine the most valuable practices to continuously improve the user experience. Shifting right is one piece of the puzzle we must master combining with other practices to achieve bottom-line results.

You can freely access Continuous Testing in the Cloud, start now.