How To Shift Right With a Continuous Testing Framework

cerberus-testing-shift-right-continuous-testing-framework

Shifting right enables an organization to focus on the actual user experience in production. This is a true practice of quality, orienting the team towards the common objective of delivering value to our users.

But shifting right is complex. We need to combine various practices in the world of operations without falling into technical details, hard to maintain, and costly solutions. A different approach from “let’s build something new” is possible.

A Continuous Testing initiative leads us to implement systematic quality steps through the software delivery lifecycle. We usually started from the left part of the requirements to secure the overall CI/CD chain with quality gates.

Cerberus Testing is an open-source test automation framework aiming at achieving a scale and collaborative quality effort in organizations. In this article, we will share how to implement shift-right with maximum value and minimal effort :

  • Define the shift-right objectives from a user perspective
  • Build up upon existing functional and non-regression assets
  • Use reporting in push and pull for a transversal collaboration
  • Measure the user experience performance and reliability
  • Close the feedback loop by improving the initial flow of work

Delivering value through quality also requires us to follow systematic and structured processes. Let’s start by clarifying what we want to achieve.

Define the shift-right objectives from a user perspective

The general purposes of shift-right are mostly known as improving the actual user experience in production. It involves a set of existing and new practices like monitoring, observability, SRE (Site Reliability Engineering), among others. It can be hard to know where to start from our Continuous Testing perspective.

We will work with the context of a web and API application deployed in two countries, with production and non-production environment. Our first focus is to define what we can do to improve the measurement of quality in production. The best way is to change our perspective to the user one, thinking about what matters for them while using our product.

Figure 1: The application under test and its environments, Cerberus Testing.

Our website provides a way to buy specific products our users may be looking for. Therefore, our shift right approach will focus on the key use-cases required to achieve that goal: visualize the product, add them to the basket, and pay. We additionally need to consider the non-functional cases necessary like accessing the homepage, logging in, and accepting the consent.

From this list of use-cases, we can then drill down the necessary quality dimensions to verify. In our case, we can consider the availability, functional behavior, and response time as key criteria for our users to remain connected. We can then look at what we can reuse from our previous Continuous Testing work.

Build up upon existing functional and non-regression assets 

We created a series of automated tests to use in our CI/CD quality gates. Their goal was to ensure the selected use-cases to remain functional while performing software changes. A non-regression campaign is usually larger than a production one and does not need a regular execution; we need to pick the most valuable ones.

We can rely on the useful functionality and practice of tagging the tests to quickly achieve our goal. Each test has a specific label telling the user experience area covered by the test case. The additional meta-data of the country and environment let us confirm the scope of execution meets our needs; in our case to be executed on the Web application, the production environment, and the two countries.

Figure 2: The test case we can select using the tags, Cerberus Testing.

Our selection can then be added to a Cerberus Testing campaign specific to the shift-right requirements. Instead of being triggered within the CI/CD pipeline, we will schedule it to run periodically in production on the most used devices and browsers. That way, we can systematically verify if the user experience is meeting the quality requirements.

We will start to collect data that becomes valuable when used to improve the user experience from that work.

Use reporting in push and pull for a transversal collaboration

The added value of using a test automation framework is to access native functionalities that else we would need to implement. Test traceability and reporting are two requirements for test automation and shift-right we truly need but have no interest in developing. In our case, let’s start using them directly.

Cerberus Testing natively provides the traceability, reporting, and analytics of all the tests executed within the platform. The reporting can then be accessed in pull mode through the web interface, easily accessible even to non-technical users. Additionally, we can push notifications like slack or email on campaign executions. Soon, we will also allow the sharing of reporting data with test management solutions like Xray

Figure 3: You can access the reporting on the UI or via notifications, Cerberus Testing Youtube.

The added value of reporting for our shift-right effort is in the measurement and associated decision-making. A first value is to have common objectives for different actors like Product Owner, Quality Engineer, Software Engineering, or an SRE. The next stage is to work collaboratively to implement the most valuable improvements based on our data.

In our case, we can start by acting on the various operational criteria of the experience.

Measure the user experience performance and reliability 

The selected use-cases to shift-right are oriented towards the user experience. On top of verifying the functional behavior, we added specific non-functional requirements control helpful to verify the user requirements are met. Then, the existing reporting allows us to look at the reliability aspects.

Cerberus Testing allows you to make non-functional requirements verifications like response time, browser log content, third-party API calls made. Concretely, we added control to ensure every page loads under 2 seconds and that the consentment partner API call is successful when accepting them on the homepage. Any degradation of a specific experience is then available via the notification of the reporting.

Figure 4: The reporting and analytics of Cerberus Testing, Capterra.

We can for example detect a response time problem only for one country on a specific browser. This is the power of continuous testing in shifting-right, localizing the problem fast. Using the reports available, we can verify reliability indicators like the flakiness ratio of our campaigns, stability of the execution, and the actual scheduling coverage.

The power of shifting right comes when taking actions based on quality attributes.

Close the feedback loop by improving the initial flow of work

We cover the way to reuse Continuous Testing assets for a shift-right approach. We rely on the test automation framework Cerberus Testing to accelerate our implementation. We saw that objective definition and its measurements are two key pillars.

Data is everything. It can drive much more factual discussion between various stakeholders. It helps to focus all the team on standard indicators, focused on the user experience. 

The Graal of Continuous Testing is to combine the most valuable practices to continuously improve the user experience. Shifting right is one piece of the puzzle we must master combining with other practices to achieve bottom-line results.

You can freely access Continuous Testing in the Cloud, start now.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.