How To Make Quality Gates in CI/CD with GitHub

Continuous Testing is a fundamental part of delivering valuable software optimizing our effort. While it is a simple term, making the concept a reality requires a deep dive into implementation details.

We covered in previous articles the bottom-line reasons, challenges, and the core test automation elements in Continuous Testing. We will now share in a series of articles more concrete implementation elements.

This first article focuses on setting up Quality Gates within a CI/CD context. As a result, you can verify the application requirements as soon as possible using test automation. We will cover the following steps :

  1. Clarify the architecture of the Quality Gates in CI/CD
  2. Define the Quality Gates content with a Test Campaign
  3. Implement the Quality Gates for the non-production deployment
  4. Configure sanity checks and post-deployment tests
  5. Continuously improve and shift-right afterward

You can follow this article by securing these prerequisites:

  • GitHub account, if you don’t have one go here
  • Cerberus Testing account, you can get a free cloud here
  • An application under test on GitHub that you can fork here
  • A standard application build and deployment pipeline

Let’s start with the end in mind by clarifying the architecture.

Clarify the architecture of the Quality Gates in CI/CD

Our Quality Gates acting in CI/CD aims to ensure the quality requirements we can verify with test automation. The best practice is to decouple the build, deployment and activation across various test environments. In our case, we rely on one test environment before going to production.

From an architectural perspective, our requirements are to ensure that the application goes through every test environment with automated test execution. If our quality criteria are not met, the application deployment must stop. As a preventive measure, we leave the possibility of a manual bypass with traceability and a manual validation for the production deployment.

Figure 1: The architecture of our CI/CD pipeline with Quality Gates.

Technically, we rely on the standard elements of GitHub and Cerberus Testing. We use the GitHub repository and the GitHub actions to perform the various steps using yml formats that are also versioned. We enable the test execution using the standard Cerberus Testing API that enables to trigger a campaign.

We can now define the content of our Quality Gates.

Define the Quality Gates content with a Test Campaign

The Quality Gates is the gatekeeper on which our software changes quality. It is the inspiration of Cerberus Testing with Cerberus being the dog guard of hell. In our concrete cases, we need to select the relevant test cases and their relative importance.

Our application provides a Web and API interface; our tests will focus on this area. We are not talking about unit tests that are part of the build. Cerberus Testing focuses on the non-regression and functional tests executed part of the pipeline. In our case, we can reuse the automated tests available on the application.

Figure 2: The Test Case Repository of Cerberus Testing to select our tests.

Our selection of test cases can then be grouped logically into a test campaign. In Cerberus Testing, a campaign enables us to regroup test cases based on different filters to then schedule or trigger them depending on various conditions. For the CI/CD pipeline, the trigger is the deployment of the application in a specific environment. Cerberus Testing also enables different notification mechanisms for the start and end of execution.

Figure 3: The Cerberus Testing Test Campaign to group tests and launch them together.

The configuration of the campaign is a step-by-step process you can follow using the user interface of the tutorial. Once defined properly, the campaigns are available through the APIs for an external integration.

We can now move to technical integration.

Implement the Quality Gates for the non-production deployment

The first step is to start deploying the Quality Gates as soon as possible in the test environment. Our test application has one test environment per country, FR and UK, where our tests will be executed.

The pipeline stages are described as code within GitHub using the GitHub actions. In this first step, we want to trigger the automated tests in the UAT environment after the application deployment. We can achieve this result by configuring the call to the Cerberus Testing campaign execution API.

Figure 4: The GitHub Pipeline and Actions triggering the Cerberus Testing campaign in UAT.

For our case, we rely on a simple bash script running after the deployment. The secrets and access management is handled by secrets, allowing us to remove that part from the configuration. The power of infrastructure as code is that this code is reusable for the web, API and any other application deployment we could have.

run_tests:
    needs: [deploy_UAT_FR,deploy_UAT_UK]
    name: Cerberus Testing on UAT
    runs-on: ubuntu-20.04
    steps:
    - uses: actions/checkout@v2
    - name: Run script file
      run: |
         chmod +x ./launchTest.sh
         ./launchTest.sh
      env:
        AUTHOR: ${{ github.event.pusher.name }}
        HOST: https://jftl2021.cerberus-testing.com
        CAMPAIGN: Website_NonRegression_UAT
        APIKEY: ${{ secrets.APIKEY }}
      shell: bash

We can repeat the same configuration for all environments we have for our application. In a traditional setup, that means executing the test in a QA, UAT, staging environment. The remaining environment is the one of production where the test to execute can be quite different.

Production deployment requires specific post-deployment checks also known as sanity checks.

Configure sanity checks and post-deployment tests

Sanity checks aim to verify the status of an application straight after its deployment in production. In the long run, monitoring and observability solutions take care of the application verification. The power of sanity checks is to do that after the deployment for a partial or complete deployment.

We will perform similar verification in our concrete case of an application exposing a Web and API interface. After the deployment in production, we will execute all the tests that are considered safe in a read-only mode. That way, we can guarantee the application status without impacting its operations.

Figure 5: The overall GitHub Actions pipeline up to the post-deployment sanity checks.

The configuration is similar to the previous one. The only difference is the campaign we will call in the API, being the post-production deployment one. The added value of relying on the campaign is the abstraction and scalability of the technique. We can add, update and remove tests without impacting the configuration of our pipelines.

Continuous Testing is a matter of performing regular checks at each stage of the deployment to reduce the risks.

Continuously improve and shift-right afterward

We cover in this article how you can deploy Quality Gates in CI/CD using GitHub actions. From now, you should be able to deploy various stages of test campaigns, being of non-regression, functional or post-deployment types.

The added value of these Quality Gates is not only for testers. The whole team can benefit from these more frequent feedback loops: the business, product, engineering, and operations teams. This is where the value of Continuous Testing is.

This article focuses on the middle area of Continuous Testing with the CI/CD part. But the overall Software Delivery Lifecycle (SDLC) is more transversal. Hence the importance of shifting left and right to cover the entire chain.

We will cover these other areas in the following articles. In the meanwhile, you can start using Cerberus Testing for free for your Continuous Testing initiative.
Start now.

Leave a comment

Your email address will not be published. Required fields are marked *

Cerberus-Testing Terms Of Service


Website Terms and Conditions of Use

1. Terms

By accessing this Website, accessible from http://vgeouyk.cluster030.hosting.ovh.net, you are agreeing to be bound by these Website Terms and Conditions of Use and agree that you are responsible for the agreement with any applicable local laws. If you disagree with any of these terms, you are prohibited from accessing this site. The materials contained in this Website are protected by copyright and trade mark law. These Terms of Service has been created with the help of the Terms of Service Generator and the Privacy Policy Template.

2. Use License

Permission is granted to temporarily download one copy of the materials on Cerberus Testing's Website for personal, non-commercial transitory viewing only. This is the grant of a license, not a transfer of title, and under this license you may not:

  • modify or copy the materials;
  • use the materials for any commercial purpose or for any public display;
  • attempt to reverse engineer any software contained on Cerberus Testing's Website;
  • remove any copyright or other proprietary notations from the materials; or
  • transferring the materials to another person or "mirror" the materials on any other server.

This will let Cerberus Testing to terminate upon violations of any of these restrictions. Upon termination, your viewing right will also be terminated and you should destroy any downloaded materials in your possession whether it is printed or electronic format.

3. Disclaimer

All the materials on Cerberus Testing’s Website are provided "as is". Cerberus Testing makes no warranties, may it be expressed or implied, therefore negates all other warranties. Furthermore, Cerberus Testing does not make any representations concerning the accuracy or reliability of the use of the materials on its Website or otherwise relating to such materials or any sites linked to this Website.

4. Limitations

Cerberus Testing or its suppliers will not be hold accountable for any damages that will arise with the use or inability to use the materials on Cerberus Testing’s Website, even if Cerberus Testing or an authorize representative of this Website has been notified, orally or written, of the possibility of such damage. Some jurisdiction does not allow limitations on implied warranties or limitations of liability for incidental damages, these limitations may not apply to you.

5. Revisions and Errata

The materials appearing on Cerberus Testing’s Website may include technical, typographical, or photographic errors. Cerberus Testing will not promise that any of the materials in this Website are accurate, complete, or current. Cerberus Testing may change the materials contained on its Website at any time without notice. Cerberus Testing does not make any commitment to update the materials.

6. Links

Cerberus Testing has not reviewed all of the sites linked to its Website and is not responsible for the contents of any such linked site. The presence of any link does not imply endorsement by Cerberus Testing of the site. The use of any linked website is at the user’s own risk.

7. Site Terms of Use Modifications

Cerberus Testing may revise these Terms of Use for its Website at any time without prior notice. By using this Website, you are agreeing to be bound by the current version of these Terms and Conditions of Use.

8. Your Privacy

Please read our Privacy Policy.

9. Governing Law

Any claim related to Cerberus Testing's Website shall be governed by the laws of fr without regards to its conflict of law provisions.

Thank you for your interest in cerberus,

In order to proceed with your SaaS we need some in order to confirm your instances accesses.

Please provide accurate contacts to secure the process.

You can cancel any time the subscription, please read Terms of Use.