The Frugal Way To Deliver Quality Gates in Production


User experience drives the bottom line of digital business. One failure, latency, or non-managed exception disrupting the user journey means a lost opportunity. Successful user experience is therefore a priority.

This pattern known as shift-right can be implemented in a variety of ways. We can select specific observability, testing, and alerting solutions to deploy in production. While this can do a part of the job, it creates another context hard to reconcile for the team.

In this article, we propose to rely on your test automation assets to build quality gates in production. We will use Cerberus Testing to demonstrate the added value of a shared and end-to-end collaboration.

In this previous article, we cover how to shift right with a continuous testing framework. We will detail the concrete steps to take in order to implement your quality gates in production:

  • Clarify your assets, assumptions and objectives
  • Test your key value and growth assumptions
  • Schedule a test automation campaign in production
  • Use analytics to improve the experience and delivery
  • Foster a context for continuous experience improvement

Let’s start by clarifying the key elements of your initiative.

Clarify your assets, assumptions and objectives

We first need to understand which digital assets are in scope. We have to start by identifying the software under test. From there, we can identify the related elements such as the requirement referential, automated tests, and production measurement.

The above elements give you a good picture of the user experience to address. You should know the various personas involved, their customer journeys, and which devices they mostly used. Your quality gates objective is to ensure the various contexts are working properly in production.

Figure 1: An example of traffic dilution to details per user journey, Medium.

That leads us to our value and growth assumptions, similarly to the Lean Startup methodology. Your value hypothesis represents the expected value you can get from improving the identified user experience. Your growth hypothesis represents your capacity to handle the execution of your quality gates in a variety of journeys, devices and with a frequent execution rate.

Your next step is to quickly validate or invalidate the hypotheses while still having a low commitment.

Test your key value and growth assumptions

You don’t need fancy processes or tools to test hypotheses. Be pragmatic and look for the most efficient ways. You can achieve that by asking good questions to the right persons, and by performing minimal testing on your tooling.

You derived from your initial analysis that your identified user journeys are worthwhile measuring and improving them; that’s your key-value hypothesis. Do your homework first and then involve the relevant stakeholders.

Start by looking at the application analytics dashboard, verifying that these user journeys are actually true paths taken by the customer, at a significant ratio. Then, verify this assumption by asking the product owner, manager, and more user-oriented team: “Do you care about improving these user journeys now and in the coming months?”.

Figure 2: The example of ecommerce customer journeys map, Venngage.

You can move to verify your implementation assumptions. You have to execute your tests on a variety of devices on a regular schedule in production, and access reporting. Check and ask your tooling team the following questions: “What are the available browsers and mobile devices? Can I run 50 to 100 tests every 5 minutes? Is there a native reporting and notification system?”.

Hoping you got these answers right you can continue. In our case with Cerberus Testing, these answers are positive, letting you proceed.

Schedule a test automation campaign in production

The definition of a test automation campaign requires various elements. First, we need to configure the scope of tests to execute. Then, define its execution parameters such as frequency, traceability. We end up by configuring the quality gates ratio and associated notifications.

The scope of tests to execute comes from your initial analysis. You can configure a fixed list or a dynamic one. We recommend the dynamic approach offering more decoupling and scalability. Instead of configuring a fixed list of tests, we will configure the criteria that dynamically select the matching ones.

Figure 3: Configure a campaign with dynamic filter on tag “monitoring”.

Your next step is to define the execution parameters. The first one is to define the frequency of execution. As you are implementing quality gates in production on the user experience, the tests should be quite regular. Even if you are not performing technical monitoring, the data you will collect will be valuable over time. 

Figure 4: Configure various campaign scheduling with a flexible crontab expression.

You can then define additional execution parameters such as systematic screenshots on error, increased robot traceability, or even automated rerun. The rerun ones are useful to avoid false positives but should be used wisely and not by default. Else, you end up hiding issues and also not having the exact performance measurement. 

Figure 5: Configure the notifications on defined triggers and channels.

The final step is to define the quality gates ratio and notifications. The ratio lets you define the number of tests and associated priority to consider the campaign as OK or KO. This is helpful when your test suite is larger. To start, we recommend you to leave it as it is. Lastly, configure your slack and email notifications to receive as soon as possible the update of your campaigns.

You are now ready to use your work to improve the user experience.

Use analytics to improve the experience and delivery

The value of analytics is to bring insight you cannot access another way. Most importantly, the bottom line results require taking decisions and actions on these insights. You can now leverage your quality gates in production.

The native analytics dashboard available allows you to identify improvements. Is the performance the same on all smartphone devices? Is there a specific country with an abnormal performance? Is there a time in the day where specific journeys are not working as expected? Are there unstable user experience paths, in which conditions? You can also drill down within the CI/CD pipeline to compare some data.

Figure 6: The native analytics dashboard available in Cerberus Testing for quality gates, Sourceforge.

These questions are clues to use as a detective. You need to analyze, interact with knowledgeable people, and ask more questions. You can rely on 5-why analysis and other types of problem-solving frameworks to accelerate your journey. Don’t be afraid to involve transversal stakeholders for specific topics, some problems are by nature of different causes and require different expertise to be solved.

Step by step, your work capitalizing on reusable assets can truly drive continuous improvement.

Foster a context for continuous experience improvement

Your quality gates in production are essential to keep the organization focused on the user experience. We covered the various prisms required to achieve them: UX, product management, engineering, and operations.

Driving continuous improvement is a necessity where the digital experience is the driver for growth. At the same time, the need for constant evolution and complexities to deliver represent a real challenge.

We shared an approach allowing you to capitalize on your existing assets, also minimizing the switching cost and mental overload on your teams. The result is an improved collaboration with minimal overhead of new processes or tooling.

Time flies, act now. Accelerate today with Cerberus Testing for Free.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service

Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.