The Top 5 Bottom Line Reasons for Continuous Testing


The rules have changed in the digital world in a globalized competition. The paradigm of “Too big to fail” is not valid anymore. Large companies can be disrupted by faster and more valuable actors in a matter of months.

Organizations have therefore the challenge of delivering valuable software changes at speed. This creates a requirement of accelerating the rate of changes while measuring their value creation. Continuous Testing can make a difference here.

In this article, we step back to cover the bottom-line benefits of Continuous Testing from a business perspective:

  • Drive continuous value with measurement
  • Accelerate leveraging automation and self-service
  • Reduce risk with continuous early feedback
  • Foster a culture of continuous improvement
  • Provide end-to-end visibility and transparency

For an implementation perspective, you can consult this article on how to make continuous testing.

Drive continuous value with measurement

The delivery of a successful user experience results from fast feedback loops adapting from what works and what does not work. Consequently, we need to deliver software changes continuously assessing “Are we creating value?”.

Figure 1: The 4 perspectives of value creation,

The measurement of value must happen as early as possible through the software delivery lifecycle (SDLC). The measurement technique depends on the stakeholder’s expectations, the technical implementation and testing capability. We can test the requirements, the first lines of code or the actual usability in production, all for different actors and timings.

Test automation supports continuous value measurement at various effects. The best example is user journey monitoring. It combines non-regression, functional and production testing to effectively measure valuable elements of the user experience. While it is hard to measure subjective criteria, we can continuously assess the conversion, availability and stability among others; the important factor is to focus on value for the stakeholders.

We now need the second element for Quality at Speed, that is Speed.

Accelerate leveraging automation and self-service

Delivering quality alone is not enough. We need to do it fast. Quality is Speed is our objective requiring to accelerate our cycle-time and, as a result, the lead-time of valuable user improvements.

The Accelerate report correlates the business performance with the IT capability of quicker, stable and reliable delivery. Test automation enables faster feedback loops of software changes, giving us the confidence to proceed with deliveries. In addition, automation opens the door to self-service with a repeatable and controlled process.

Figure 2: The performance of Elite group in the Accelerate Report, DORA 2019.

The common limiting factors in software delivery lies in the cross-roads of various actors’ collaboration, losing time in synchronization, waiting time or confusion. Automation of CI/CD gates is the primary pattern to accelerate streamlining the activities. With Quality Gates deployed in CI/CD, we can accelerate the design to operations value assessment. The other types of test automation activities can be provided directly to our stakeholders for fast feedback loops. This is what we do by providing observability reporting, shared dashboards and alerts.

But acceleration comes with a side-effect.

Reduce risk with continuous early feedback

Acceleration comes with trade-offs. One of them is to increase the level of risk taken. Like with a car, the faster we drive, the better ABS, pneumatic and control mechanisms we need. We have the same need for software delivery.

Continuous Testing provides the mechanisms to reduce the risk of accelerated software changes. We can assess early and continuously the changes performed against specific criteria. These criteria can be functional, oriented towards user features, or technical, testing non-functional requirements like performance, security, availability. The essential element of Continuous Testing is to select and align the more valuable testing methodologies for the particular risks in context.

Figure 3: An example of Continuous Testing to reduce risks, Synopsys.

Test automation is a subset of the testing techniques available to reduce risk. The business perspective is used in aligning a quality or test strategy. We can cascade the necessary automated tests at build time or runtime in the required environments with a clear risk reduction goal. Unit testing is for example very useful for reducing the risk with fast developers iterations. From a user experience perspective, performing gradual deployments with post-deployment and user monitoring helps us reduce the release risk.

Achieving this equilibrium is a journey, not a one-shot project. Our initiative requires the collaboration of various actors at scale. That’s where we need Culture.

Foster a culture of continuous improvement

As we progress in continuously measuring value, reducing the risk and leveraging automation, the behavior of our actors gradually change. For example, having automated tests becomes natural and even mandatory to be confident in deploying, whereas it was out of comfort in the past. That’s where the culture starts changing.

Continuous Testing is the materialization of the continuous improvement paradigm. The 3 P of Total Quality are present: People, Processes, Product. The People are continuously assessing the value creation from a user perspective, supported by Processes and Products. This perspective implies that our testing effort is not limited to the tester or engineering perimeter; we must involve the stakeholders. From there, we can work on actively creating efficient processes and products for their common mission.

Figure 4: The equilibrium of Quality for Continuous Testing, Source codes.

Test automation can easily remain closed to the outside world of testing teams. It is therefore essential to work in open and transversal collaboration to that effect. We cover in detail how to build a valuable test automation framework to respond to this challenge. Achieving continuous improvement in test automation is about combining valuable processes in time-boxed iterations. That way, we can iterate focusing on the impact we create supported by fast cycle-time, measurement and improvement rituals.

The measurement for Continuous Testing is both necessary and creating new opportunities for organizations.

Provide end-to-end visibility and transparency

A direct benefit from measurement and collaboration is to provide transversal visibility. The actors understand better the role of each party, their contribution and that they are on the same boat. Transparency is one foundation of trust.

The implementation of Continuous Testing requires more measurement at different stages of the software delivery chain. Combined with the need for acceleration, various actors must collaborate more closely to achieve common value creation. The results of their interactions on the end-to-end chain with factual reportings are truly beneficial to Quality at Speed in organizations. Measurement is the pre-requisite for improvement, remaining conscious about human biases to use them wisely as a guide from a user perspective. The objective is not to have 100% coverage or other types of vanity-oriented results.

Figure 5: The Cerberus Testing reporting available for shared visibility, TestingTools.

From a test automation perspective, we can leverage the traditional test reporting focusing on the test repository and its execution. We can measure the test suite execution time and flakiness ratio for instance. We must complement our reports on our key business goals of value delivery, faster and more reliable changes. Only after we can highlight the causation, correlation or other links of test automation benefits. Test automation must provide these 3 levels of zoom: individual test traceability, overall test reporting and test analytics.

Continuous testing constrains software delivery to quality

Creating bottom-line impact is the fuel of organizations aiming at the high-standard expected by the users. Companies have limited resources; their efforts must be concentrated on the most promising activities, one of them being Continuous Testing.

We cover the business benefits of Continuous Testing articulating a value proposition for the end-users, business and IT stakeholders of software delivery. Creating value through Continuous Testing is therefore a matter of implementation alignment in context.

There are surely must-have foundations, like minimal unit testing, CI/CD quality gates and post-deployment verification. But achieving the high standard is about the cherry on the cake, where we can create bottom-line benefits.

Continuous Testing is a journey full of challenges we will explore in the following article. Until then, stay well and happy testing!

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service

Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.