Delivering a successful user experience is a real challenge in our current landscape. Digital consumers are expecting an intuitive yet reliable interaction with your products. Moreover, you have to constantly improve and evolve the offer to keep in line with the user expectations.

Consequently, the capability to perform rapid and robust changes to your digital platform is key. When you adapt a part of the system, you can test it quite easily. But then, how do you make sure you did not break something else?

This is where Regression Testing brings its value by retesting the unchanged parts of the application. It can be applied to different testing techniques (functional, security, etc) and on a particular scope (unit, full or partial).

In this article, we will share ordered practical steps you can take to implement robust Regression Tests. An e-commerce platform functionalities (search, login, buy) will be taken as an example through the next sections applying Business-process based testing (over other techniques such as boundary-value analysis or equivalence partitioning).

Clarify where Regression Test fits into Your Product Strategy

The first step is to step back on the overall problem you want to solve for your organisation. It is important to avoid rushing to implementation in this initial phase.

You probably looked at Regression Testing because of core functionalities impacted by recent releases. This is where you need to list the key functions you provide, clarifying their layers and level of changes.

Figure 1: An example of functions analysis for your Regression Test strategy

Based on this scope you can formulate the contribution of your Regression Test into your Product Strategy: “Guarantee that my core functions are working anywhere and anytime”.

This is also the moment to clarify what Regression Tests are not for. They cannot fix a fragile software architecture. They are not exploratory tests and not necessarily functional or user interface tests. 

Define a clear scope for your Regression Test effort

Based on the work in the previous step, you will have an exhaustive list of regression tests you would like to perform. Certainly, this list will seem enormous!

Stay calm, we can rely on a useful technique: Test Prioritization. Rank first your most critical Business functions, usually balancing the business value and the frequency of usage. Then order them by frequency of changes, prioritizing the least changed parts.

Figure 2: Test Prioritization Matrix on a simple e-commerce example

From there, you can apply those guidelines to prioritize specific tests filtered out by the matrix  making sense in your context :

  • Complex scenarios and test cases (likely to not be performed, manual and error-prone)
  • Test cases that frequently fail (existing regression tests with a bad design, dataset, etc)
  • System areas with recurring incidents (already impacting your product right now)

For each test prioritized in our example, we can clarify the scope of testing: Front, API or Back-end. Then we can define the most adequate test techniques to use: user interface, contract, integration or functional testing.

Be clear on your reporting, measurements and KPIs

Again, keep your temptation to rush for implementation. We still need clear goals and measurement in place to set you up for success.

Based on our prioritization matrix, we are able to compute our coverage objectives and set them at different levels over time. First, set your vision and the first milestones, as you will need to adapt them based on the implementation. 

The second types of KPIs you will need to set are related to your regression test performance. They can be linked with your digital product or with the tests themselves. In our example, Site Reliability Engineering (SRE) KPIs are well adapted to measure your platform performance.

Figure 3: KPIs you can use to keep a big picture of your Regression Test

We strongly recommend keeping those three categories between your goals. They are in fact linked to different profiles (Business, Product Owner, Development, QA, Operations) and  enable a better transversal understanding and collaboration.

Tune the value according to your context and stakes. In the same logic, define an ambitious goal you can iterate on and improve over-time.

Iterate on an incremental process in Shift-Left as possible

You are now ready to start your implementation with the proper foundations. Let’s start by sharing good practices in terms of organisation.

For the test implementation, we recommend that you stick to known solutions for now, for instance, Selenium Webdriver for performing web tests. It is more important to start to implement test cases in order to discover what needs to be addressed transversally.

For instance, you can start to implement front-end tests and discover that you are missing a fixed ID or that your framework is hiding elements by default. It will be then the most important task to fix prior to enriching your test suite.

Figure 4: Step by step approach to update your plan early based on first iterations

A good practice is to perform a test of each typology you identify as a priority. That way, you will benefit from rapid feedback and a realistic vision of what needs to be done.

Once you have been able to demonstrate the various use-cases, you can think about the scalability of your process. This is where topics such as Automation, CI/CD and Test Design become a real priority.

Gradually automate while keeping your tests maintainable

For your first regression tests, we recommend that you automate them directly. First, because the functions you will be covering are core to your product and need to be verified at each release. Second, because it will force you to make them reliable to get value out of them.

Figure 5: The Regression Testing Manifesto we recommend to follow

In fact, the main problem you will tend to have while growing your regression test suite will be its maintainability. It is a key requirement for your tests to keep value over time. Not addressing this subject will make your test suite more problematic than the bugs you wanted to catch in the first place.

Test design is a skill as in architecture or software areas. Without it, your edifice will start to look like the “Tour de Pise”. You need robust, reliable and reusable elements within your tests, commonly referred to as a Test Library or modules.

In our e-commerce example, the login example is a classic one for all use-cases that will require this as a prerequisite, either on a Front, Back or API layer. This can also be applied to front-end customer journeys, ensuring that if a specific path changes, you just update the parameter in one place.

Regression Test is a Continuous Process from Design to Operations

Taking a step back from the process we shared, Regression Test requires a structured approach to set you up for success.

The initial steps are fundamental to articulate the problem to solve and the corresponding priorities you will define. From there, the next steps involve an iterative and incremental approach.

Figure 6: The Continuous Regression Testing Process

As a final recommendation, we recommend that your Regression Test also runs in production! Don’t you also need your core functions to be available in production firstly, in a complex operating environment? 😀

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://www.cerberus-testing.com, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.