5 New Metrics For Quality at Speed Test Automation


We are all tired of egos.

“I have greatly improved the number of tests, coverage, flakiness ratio”.

Great. But who’s using it? Which value did they create? At which speed?

Test automation is about accelerating the delivery of valuable software, bringing confidence to the team to deliver their changes.

Valuable metrics must therefore cascade from this goal, far from traditional KPIs even arguing formulas for computing an ROI.

Let’s see the Quality Engineering metrics you can use for test automation.

Follow Cerberus Testing for more open-source test automation.

Satisfaction and engagement of the product team

We can easily say to be customer-centric—what does that mean for test automation?

Valuable things are used by people on a regular basis and paid for.

If you still have your job, it’s already a good clue but that’s not sufficient.

The satisfaction and engagement of your product team with your automated tests is a good indicator of their usefulness.

They can be well implemented and non-technical; but people must notice at some point their value, even if it’s only 1 minute to give a go-nogo.

Use data on your automated tests such as the analytics of their usage, reaction to notifications, word of mouth.

You can even add surveys regularly and within test campaign reports.

Minimal waiting time per delivery cycle-time

Cross-functional teams have the objective to accelerate the delivery of valuable software with stability.

Their time is one the most valuable assets when they have to perform fast iterations, ideally within minutes.

They will not wait for a test campaign lasting more than a few minutes, even fewer hours.

Measuring the waiting-time induced by your automated tests is critical and must be contained over time, especially when adding tests.

Keep a focus on these metrics on your different campaigns to keep your efforts useful:

This metric can be more important than the tests in themselves executed: pass a threshold and your tests lose their value, becoming ignored. 

Lead-time to implement minimal quality gates

Software teams that accelerate have two common traits: they deliver change fasters and within new components.

Your reactivity is therefore key to keep helping them to iterate with Quality at Speed.

If you are not there, they will take the best alternative available at that time: manual test, unit test, or even nothing at the risk of discovering major issues later on.

The lead-time to implement the minimal quality gates is an important metric showing your capacity to adapt in an ever-changing environment.

If it’s complex to set up a minimal quality gate or have to reconsider too many things there is a problem, that is not necessarily duplication.

Most of the time issues in maintainability come from a lack of consideration in the design phases. You will have to invest more in decoupling and test design.

MTTA a bug from user-story definition

The Mean Time To Acknowledge (MTTA) is the average time it takes from when an alert is triggered to when work begins on the issue.

The feedback loop of acknowledgment starts from the user-story definition in test automation, hence the need for shift-left.

Users finding bugs in production do not come back.

Test automation goals to reduce risks must therefore be measured with these metrics, looking for issues in your process when discovering critical issues.

The goal begins to reach an acceptable level, not maximize it; remember it’s about risk reduction, not risk removal.

MTTR to fix flaky tests in non-regression campaign

We cannot trust systems or people that are unstable. It’s the same with automated tests.

Mean Time To Repair (MTTR) is the average time required to fix a failed component or device and return it to production status.

Unstable tests—also known as flaky tests—must be rapidly fixed to keep the team satisfied and iterating with confidence.

You can use the dashboards and alerting capabilities to remove tests from your campaigns changing their status or presence in the test suite.

A series of features are available within Cerberus Testing to reduce flakiness such as test data library, self-healing, or even rerun.

Test automation metrics to accelerate software delivery

Test automation objective is to bring confidence to deliver changes faster to software teams accelerating their rate of changes.

Metrics focusing on their success and what they find valuable is therefore what matters.

It’s not about optimizing a QA silo for the sake of it. Accelerate the delivery of valuable software is the mission of test automation for QA engineers.

There’s no time to lose. Start measuring and leveraging a ready-to-use platform to accelerate your test automation with Cerberus Testing.

Get your free plan here.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service

Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.