The Quality Framework You Need For Customer Journeys Monitoring

cerberus-cust-journey-featured-framework

Customer journeys are at the heart of a user experience leading to conversion, a critical area to act for organizations searching for digital performance. Companies’ transformations to deliver a successful UX do not happen overnight. Step-by-step, continuous improvements, test & learn is the path to success, with few shortcuts. 

Improvement starts with measurement. Customer journeys are no exception, hence the importance of their monitoring. Our quality initiative must incorporate this end-user experience perspective leveraging its capabilities, but how?

This article shares a step-by-step repeatable approach to iterate fast in your customer journey monitoring implementation. We go through a real e-commerce use case with actionable guidelines and recommended best practices.

As an introduction, you can read our previous articles to understand where quality meets customer journey monitoring and its 5 hidden powerful benefits.

The Customer Journeys Monitoring Context

The work of identifying the “right customer journeys” in your business context is the first step. You can perform this following the “Question Asker” model of QA in this article: 6 Quality Questions for Testing Customer Journeys. Now we can focus on building the “customer journeys right”.

In our case, we will work on an e-commerce retail and fashion website named Damart; that way, you can refer to this case for the traditional e-commerce journeys. We identify several journeys based upon use-cases ordinarily present in a functional non-regression campaign.

Figure 1: Test cases of the Conversion stage of customer journeys

Our customer journey monitoring will then focus on the Conversion stage of the e-commerce journey. We also included the main non-functional requirements of verifying the login capability is always fine. It is trivial, but forgetting to ensure this monitoring can lead to a drastic decrease in the business performance in case of disruption.

Using a framework is an excellent way to structure an efficient, repeatable process.

The Process to Monitor Customer Journeys

We will leverage the fast feedback loop cycles supported by Cerberus Testing: Define, Execute, Analyze. The framework areas supporting this cycle are respectively the Test Repository, Test Execution, and Test Reporting. Specific functionalities are leveraged to support the customer journey monitoring.

First things first, we start by defining and organizing the test case repository. The structure of the test cases is up to you; we recommend using the folder structure, step library, and application definition for expandable foundations. The key element to pivot between the test repository and execution is the use of labels we cover in the next section.

Figure 2: The steps and Cerberus Testing frameworks areas

The Execution area is the next one we will use. The process is similar to defining a standard test campaign; we can use the labels, test case selector, and choose the concerned environments. The main difference resides in the scheduling, usually much more frequent.

The last areas are Reporting and Analytics to support insights on our customer journey executions. We can switch our perspective from a single campaign to a more broad range of executions using the various dashboards.

Let’s start by tagging our tests for decoupled test and campaign management.

Organizing the Tests with Labels

Our test cases are ready for customer journey monitoring. A good practice is to leverage the non-functional regression test case to maintain alignment and early detection through the software delivery chain.

Customer journey campaigns can have various criteria: test case, application, system, environment, and labels. We prefer to use labels to decouple our test case repository from their execution. If we hard-code a specific test case, application, or environment, we can suddenly execute non-expected test cases. Explicit labels enable better visibility and explicit inclusion inside the monitoring campaign.

Figure 3: The use of labels for identifying the customer journeys cases

In our case, we tag our test cases with “Campaign_Scheduled” to identify them as part of the customer journey campaign. They still have other labels and requirements, useful for reporting purposes we will see later on.

The next step is to configure our test suite for monitoring, known as a Campaign.

Schedule a Monitoring Campaign

The pivot between our test case repository and execution is the label in our case. We recommend the campaign description to be clear on its goal and high cohesion for stakes of reporting and speed of execution.

The second tab, “TestCase List”, enables us to configure our test campaign content. Here we use the label previously defined only. As you can see below, we can scope the suite based on other criteria, such as “Test Case Criterias” or various labels.

Figure 4: The definition of our campaign using the labels

The next tab of “Environments, Countries & Robots List” must be tailored to the touchpoints identified in your customer journeys. Traditionally, the environment will be production, even staging for some cases. The countries and devices will depend upon your context and application deployments.

The execution settings are also flexible in their definition. For customer journeys, we recommend configuring a screenshot on error while limiting detailed tracing capabilities. Our goal is to react fast while measuring the actual performance of our tests; a screenshot is usually very useful, and limiting the data collection avoids biasing our test performance. Retry mechanisms should also be limited to don’t hide stability issues.

Our objective is to ensure the quality of our customer experience, not to have a green dashboard.

The priority of the scheduling is a critical point for customer journey monitoring with the limiting factor of parallel executions. The supervision campaign will usually be small and regularly executed, while non-regression ones tend to be larger and on-demand. We can guarantee that monitoring will pass first using the “Priority” field to 0, or at least lower than other campaigns defined; then, the queue management system will take care of the priorities.

The scheduler area will let you define one or more executions using a crontab format. The frequency of executions is usually a balance between the journeys’ value, criticality, and frequency. A minimum of an hourly execution is recommended. The last part is to define notifications by email and slack as required. It depends on your process; some teams integrate directly with the Cerberus Testing APIs from operational monitoring solutions.

Our customer journeys monitoring campaigns are all set to run regularly, starting to collect valuable data for the future. We can switch to the analytical part.

Report, Analyze and Take Actions

Various dashboards are available to support our visibility requirements. We differentiate Reporting from Analytics from the different perspectives and insights they can provide. Let’s start with the campaign reporting.

The first answer we are looking for is the execution status of a single campaign. We can access the native reporting in the Campaign Reporting area. Various dashboards are available to support our analysis: the global campaign status with a split by status, by test folder and labels, followed by a detailed table of all execution. Each test case execution report is then available with  traceability information.

Here’s an example of the status by labels, giving us both visibility on our customer journey monitoring with the “Campaign_Scheduled” tag. In contrast, the other tags provide us the stability perspective of each website area, namely Account, Cart, Connection, Homepage, Products.

Figure 5: The reporting by tag natively available in Cerberus Testing

Our next question is to understand the customer journey performance over time. We need to combine various campaign execution reports and statuses to give that visibility. The “Campaign Reporting Over Time” provides the Analytical dashboards we need with execution time split per devices and environment and the ratio of executions, ending with the stability dashboards.

Here’s an example, providing a visual representation of our flakiness ratio over time. We can then loop back to the other dashboards to understand if the instabilities are happening in a specific context. We can also zoom in on a particular campaign reporting and test case traces to narrow the problems.

Figure 6: The campaign stability dashboards giving a flakiness ratio

From here, we have a repeatable structure that can support our customer experience monitoring.

From Reactive to Proactive Customer Journeys Monitoring

The framework and process described here will support scalable foundations, starting from a good test case design, decoupling various requirements, and proper configuration. The expansion is then a balance to maintain.

 Our focus must remain on customer experience monitoring, supporting our quality initiative, and delivering value to our customers. The various elements defined support to evolve from a reactive to a proactive approach and from a siloed to a transversal one.

Sharing this customer perspective in operations is an excellent way to align the various teams, from product management to operations, on a common objective. Have a nice customer journey monitoring experience, starting and ending with a successful user experience.

Start now with Cerberus Testing.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.