Delivering a successful user experience is a real challenge in our current landscape. Digital consumers are expecting an intuitive yet reliable interaction with your products. Moreover, you have to constantly improve and evolve the offer to keep in line with the user expectations.

Consequently, the capability to perform rapid and robust changes to your digital platform is key. When you adapt a part of the system, you can test it quite easily. But then, how do you make sure you did not break something else?

This is where Regression Testing brings its value by retesting the unchanged parts of the application. It can be applied to different testing techniques (functional, security, etc) and on a particular scope (unit, full or partial).

In this article, we will share ordered practical steps you can take to implement robust Regression Tests. An e-commerce platform functionalities (search, login, buy) will be taken as an example through the next sections applying Business-process based testing (over other techniques such as boundary-value analysis or equivalence partitioning).

Clarify where Regression Test fits into Your Product Strategy

The first step is to step back on the overall problem you want to solve for your organisation. It is important to avoid rushing to implementation in this initial phase.

You probably looked at Regression Testing because of core functionalities impacted by recent releases. This is where you need to list the key functions you provide, clarifying their layers and level of changes.

Based on this scope you can formulate the contribution of your Regression Test into your Product Strategy: “Guarantee that my core functions are working anywhere and anytime”.

This is also the moment to clarify what Regression Tests are not for. They cannot fix a fragile software architecture. They are not exploratory tests and not necessarily functional or user interface tests. 

Define a clear scope for your Regression Test effort

Based on the work in the previous step, you will have an exhaustive list of regression tests you would like to perform. Certainly, this list will seem enormous!

Stay calm, we can rely on a useful technique: Test Prioritization. Rank first your most critical Business functions, usually balancing the business value and the frequency of usage. Then order them by frequency of changes, prioritizing the least changed parts.

From there, you can apply those guidelines to prioritize specific tests filtered out by the matrix  making sense in your context :

• Complex scenarios and test cases (likely to not be performed, manual and error-prone)
• Test cases that frequently fail (existing regression tests with a bad design, dataset, etc)
• System areas with recurring incidents (already impacting your product right now)

For each test prioritized in our example, we can clarify the scope of testing: Front, API or Back-end. Then we can define the most adequate test techniques to use: user interface, contract, integration or functional testing.

Be clear on your reporting, measurements and KPIs

Again, keep your temptation to rush for implementation. We still need clear goals and measurement in place to set you up for success.

Based on our prioritization matrix, we are able to compute our coverage objectives and set them at different levels over time. First, set your vision and the first milestones, as you will need to adapt them based on the implementation. 

The second types of KPIs you will need to set are related to your regression test performance. They can be linked with your digital product or with the tests themselves. In our example, Site Reliability Engineering (SRE) KPIs are well adapted to measure your platform performance.

We strongly recommend keeping those three categories between your goals. They are in fact linked to different profiles (Business, Product Owner, Development, QA, Operations) and  enable a better transversal understanding and collaboration.

Tune the value according to your context and stakes. In the same logic, define an ambitious goal you can iterate on and improve over-time.

Iterate on an incremental process in Shift-Left as possible

You are now ready to start your implementation with the proper foundations. Let’s start by sharing good practices in terms of organisation.

For the test implementation, we recommend that you stick to known solutions for now, for instance, Selenium Webdriver for performing web tests. It is more important to start to implement test cases in order to discover what needs to be addressed transversally.

For instance, you can start to implement front-end tests and discover that you are missing a fixed ID or that your framework is hiding elements by default. It will be then the most important task to fix prior to enriching your test suite.

A good practice is to perform a test of each typology you identify as a priority. That way, you will benefit from rapid feedback and a realistic vision of what needs to be done.

Once you have been able to demonstrate the various use-cases, you can think about the scalability of your process. This is where topics such as Automation, CI/CD and Test Design become a real priority.

Gradually automate while keeping your tests maintainable

For your first regression tests, we recommend that you automate them directly. First, because the functions you will be covering are core to your product and need to be verified at each release. Second, because it will force you to make them reliable to get value out of them.

In fact, the main problem you will tend to have while growing your regression test suite will be its maintainability. It is a key requirement for your tests to keep value over time. Not addressing this subject will make your test suite more problematic than the bugs you wanted to catch in the first place.

Test design is a skill as in architecture or software areas. Without it, your edifice will start to look like the “Tour de Pise”. You need robust, reliable and reusable elements within your tests, commonly referred to as a Test Library or modules.

In our e-commerce example, the login example is a classic one for all use-cases that will require this as a prerequisite, either on a Front, Back or API layer. This can also be applied to front-end customer journeys, ensuring that if a specific path changes, you just update the parameter in one place.

Regression Test is a Continuous Process from Design to Operations

Taking a step back from the process we shared, Regression Test requires a structured approach to set you up for success.

The initial steps are fundamental to articulate the problem to solve and the corresponding priorities you will define. From there, the next steps involve an iterative and incremental approach.

As a final recommendation, we recommend that your Regression Test also runs in production! Don’t you also need your core functions to be available in production firstly, in a complex operating environment? 😀