The Truth About Continuous Testing You Need To Know

Your users are expecting a customer experience meeting the high standards set by the leading actors. Delivering a high-standard user experience requires continuous improvement, explaining the need for faster software delivery. But acceleration comes with its side-effects of degraded quality without counter-forces.

The Quality at Speed paradigm is a good summary of what we are trying to achieve. We need to deliver software changes fast that meet the quality requirements. Testing in traditional ways lacks the essential element of speed, explaining the need for Continuous Testing within a Continuous Delivery environment.

The world of Testing has been evolving in various ways: shift-left, shift-right, test in production, among others. But what is changing with Continuous Testing? What are we trying to achieve? What are the impacts at the end of the day?

In this article, we cover the true changes coming with Continuous Testing:

• Continuous Testing requires a company-wide alignment
• The Graal of Continuous Shift-left and Shift-right
• SDLC and STLC merging into unified feedback loops
• Continuous feedback to all stakeholders, not only test or IT
• Focus on global usefulness more than local silo optimization
• Technology must be self-service, integrated and intelligent

You can consult The Top 5 Bottom Line Reasons for Continuous Testing for a better understanding of Continuous Testing motivations.

Continuous Testing requires a company-wide alignment

Continuous Testing is the natural consequence of evolving quality practices out of their natural silo of testing and QA teams. Valuable continuous testing will only come by providing value to the stakeholders, hence the need for a transversal alignment in the first place.

A prerequisite of aligning the priorities of Continuous Testing is to perform a Shift-Up & Spread exercise. This practice is shared by Iman Benlekehal, illustrating the need for a transversal alignment between the stakeholders on the quality expectations, meanings and attributes. Once this exercise has been performed, we can focus on implementing the Graal of Continuous Testing, providing the most valuable testing practice on the software value chain.

The core practices remain essential, like performing tests in non-production environments, implementing automated tests, managing the code quality. The key difference with Continuous Testing is to perform them systematically, supporting fast feedback loops. Taking the example of automated tests, we will not implement a full test suite for weeks. Instead, we need to align our flow of work in just-in-time to the small software changes that are being performed. That way, we can quickly provide feedback on the development done, adjusting directly if necessary. 

While the core practices are essential, we must act transversally, shifting left and right on the software lifecycle.

The Graal of Continuous Shift-left and Shift-right

The core practices remain more than valuable while their scalability is making the difference. The expansion of quality across the software lifecycle is identified with Shift-left – when acting upstream on the requirements for example – and Shift-right – more closely related to the actual user experience in production.

From a Shift-left perspective, Continuous Testing also requires leveraging automation while working in shorter time increments. A concrete example is to verify the use-cases and requirements early, assessing their pertinence, quality attributes, and description. We can leverage BDD or ATDD practices with automated checks for the first level of verifications, complementing afterward with a human review. Over time, we can build a better reusable library of components, enabling faster continuous testing loops.

We move to the Shift-Right context when focusing on the user experience, software operations and support. The SRE practices leveraging Monitoring is a simple example of Continuous Testing; we are performing every 5 minutes a verification of user journeys. The difference with Continuous Testing is to apply the paradigm of more frequent feedback loops with a broad range of practices. For example, load-test performed every 6 months before a big sales period is useful but remains risky. What happens if structural issues were added 4 months ago that is now too hard to change? That’s where Continuous Load-Test can help us detect earlier this type of problem.

Consequently, Continuous Testing requires profound changes of the existing lifecycle, namely software delivery and software testing.

SDLC and STLC merging into unified feedback loops

The software delivery lifecycle (SDLC) and software testing lifecycle (STLC) were traditionally two separate processes. The Software ones focused on specifying, implementing and deploying software, whereas the Testing ones on tests.  Continuous Testing is changing this paradigm to unified feedback loops.

Achieving Quality at Speed requires both delivering fast and with quality. The unified feedback loops including software and testing steps enable them to address quality requirements along the chain. CI/CD with Quality Gates is one concrete and valuable example: CI/CD coming from the software world are augmented with automated testing steps at the various stages. We can for example block the pipeline in a test environment for software changes not meeting the expected quality requirements.

A true change coming with Continuous Testing is the inclusion of systematic quality steps within the software development process. Our previous exercises of identifying the most valuable shift-left or shift-right practices are useful at this moment. We can prioritize the most valuable continuous testing increments on the identified software steps. For example, we can add security controls on the build or at the runtime of the applications. 

While implementing Continuous Testing, the actors will collaborate more closely on a shared mission they need to measure.

Continuous feedback to all stakeholders, not only test or IT

Traditional testing approaches tend to optimize the silo of QA with specific code or testing metrics. While a test coverage can be a useful supporting metric, it does not measure the value creation from a larger perspective. Continuous Testing makes a difference, giving continuous feedback to the various stakeholders.

Feedbacks are the prerequisite to a faster and more efficient adaptation. The continuous feedback we can leverage with Continuous Testing is therefore essential to reduce the risk of losing effort on non-valuable activities. A key aspect is to measure the right metric for the right purpose. Vanity metrics like the number of tests should be forgotten. We should again get back to the Shift-up process, where we identify the impact expected by our stakeholders.

Continuous measurement brings the additional question of timing. Like for blood pressure or losing weight, you don’t need to verify your indicators every minute or day. In Continuous Testing, that depends on the involved metric. For SRE on user journeys, we can measure the service indicators in a seconds interval while using time-series graphs for an analytics perspective. 

This shared and global perspective brought by Continuous Testing creates the habit of measuring the impact over local optimizations.

Focus on global usefulness more than local silo optimization

When you start Continuous Testing the first question is “Where to start?”. You are not anymore limited to a test environment to optimize automated tests. Therefore, Continuous Testing helps foster the actors to have a global picture, contributing to making an impact for the customer and the organization.

This global impact brings another challenge for the QA and testing actors. They must embrace a larger spectrum of interactions to deliver valuable quality elements. They have to engage with product management, engineering, infrastructure and platform. The list of buzzwords exemplifying this transformation to Continuous Testing is endless: DevTestOps, BusOps, etc. By sharing transversally, the actor’s priorities will be more aligned on the global objectives of the system.

Once a QA actor moves out of his comfort zone, he faces the challenges of implementing more broader practices. This challenge is valid for all actors with Continuous Testing; as the team works transversally to increase the impact of their activities, their practices evolved for more transversality. A concrete example is to evolve from SRE to Customer Reliability Engineering (CRE). We are no longer concerned only with site or service metrics; we look upstream from the user perspective.

Continuous Testing brings the last challenge to help the actors achieve this transversality, the one of Technology.

Technology must be self-service, integrated and intelligent

The unified feedback loops brought by Continuous Testing require speed and adaptation. From a technology perspective, we can start by leveraging self-service and automation to accelerate our cycle times. Additionally, the unified paradigm requires composing the different technologies from software and testing to work in harmony. Lastly, more intelligent systems are our hope for tomorrow’s tooling.

Self-service is achieved through matured automated processes. We were used to working with paper processes, which we later evolved to information systems. Right now, a large set of standard services are available in self-service. You can deal with a lot of government services from a portal for instance. We have the same stages of maturity to reach in Continuous Testing. For example, ensuring the scalable deployment of quality gates requires a robust library of templates and connectors to let the engineering team deploy them autonomously.

The interdependence of testing technologies with the ones part of the software delivery chain creates a pressure for composition. We must be able to quickly deploy a new testing activity within an existing process, in a matter of minutes. At the same time, we need a high degree of flexibility to adapt our processes; some tooling can become obsolete, other processes evolve, etc. The integration capability of testing products is therefore structuring. Our evolving ecosystem cannot work in isolation with lock-in, causing the system obsolescence in the mid-term.

The last Continuous Testing impact on Technology is to do more with less effort. We need to maximize the feedback loop’s value and frequency. From a system perspective, the human part is the limiting factor. We can automated tests, deployment and production observability, but the most complex decisions we take remain for the actors. That’s where data science can help us by automating complex decision matrices at scale. We are aiming for maximum automation of steps for faster loops.

In reality, Continuous Testing supports continuous learning loops for the customer value proposition.

Continuous Testing brings Continuous Customer Feedback Loops

The word Testing can induce in errors for the needs of Continuous Testing. We are not talking about impacts from or for testing. Our goal is to accelerate the rate of valuable adaptation to the user experience and value proposition.

We cover the implication of Continuous Testing from a broader perspective, processes, interactions. At the end of the day, we are helping the actors leverage modern processes and tools to meet the high standard.

Continuous Testing therefore brings Continuous Customer Feedbacks Loops to constantly improve the value delivery to the users. There is no end but only a journey of continuous improvement to meet and remain at the high standard.

For an implementation perspective, you can consult this article This Is How To Make Continuous Testing.

Looking for a Continuous Testing framework? Use Cerberus Testing for free.