How To Make Quality Gates in CI/CD with GitHub

Continuous Testing is a fundamental part of delivering valuable software optimizing our effort. While it is a simple term, making the concept a reality requires a deep dive into implementation details.

We covered in previous articles the bottom-line reasons, challenges, and the core test automation elements in Continuous Testing. We will now share in a series of articles more concrete implementation elements.

This first article focuses on setting up Quality Gates within a CI/CD context. As a result, you can verify the application requirements as soon as possible using test automation. We will cover the following steps :

Clarify the architecture of the Quality Gates in CI/CD
Define the Quality Gates content with a Test Campaign
Implement the Quality Gates for the non-production deployment
Configure sanity checks and post-deployment tests
Continuously improve and shift-right afterward

You can follow this article by securing these prerequisites:

GitHub account, if you don’t have one go here
Cerberus Testing account, you can get a free cloud here
An application under test on GitHub that you can fork here
A standard application build and deployment pipeline

Let’s start with the end in mind by clarifying the architecture.

Clarify the architecture of the Quality Gates in CI/CD

Our Quality Gates acting in CI/CD aims to ensure the quality requirements we can verify with test automation. The best practice is to decouple the build, deployment and activation across various test environments. In our case, we rely on one test environment before going to production.

From an architectural perspective, our requirements are to ensure that the application goes through every test environment with automated test execution. If our quality criteria are not met, the application deployment must stop. As a preventive measure, we leave the possibility of a manual bypass with traceability and a manual validation for the production deployment.

Figure 1: The architecture of our CI/CD pipeline with Quality Gates.

Technically, we rely on the standard elements of GitHub and Cerberus Testing. We use the GitHub repository and the GitHub actions to perform the various steps using yml formats that are also versioned. We enable the test execution using the standard Cerberus Testing API that enables to trigger a campaign.

We can now define the content of our Quality Gates.

Define the Quality Gates content with a Test Campaign

The Quality Gates is the gatekeeper on which our software changes quality. It is the inspiration of Cerberus Testing with Cerberus being the dog guard of hell. In our concrete cases, we need to select the relevant test cases and their relative importance.

Our application provides a Web and API interface; our tests will focus on this area. We are not talking about unit tests that are part of the build. Cerberus Testing focuses on the non-regression and functional tests executed part of the pipeline. In our case, we can reuse the automated tests available on the application.

Figure 2: The Test Case Repository of Cerberus Testing to select our tests.

Our selection of test cases can then be grouped logically into a test campaign. In Cerberus Testing, a campaign enables us to regroup test cases based on different filters to then schedule or trigger them depending on various conditions. For the CI/CD pipeline, the trigger is the deployment of the application in a specific environment. Cerberus Testing also enables different notification mechanisms for the start and end of execution.

Figure 3: The Cerberus Testing Test Campaign to group tests and launch them together.

The configuration of the campaign is a step-by-step process you can follow using the user interface of the tutorial. Once defined properly, the campaigns are available through the APIs for an external integration.

We can now move to technical integration.

Implement the Quality Gates for the non-production deployment

The first step is to start deploying the Quality Gates as soon as possible in the test environment. Our test application has one test environment per country, FR and UK, where our tests will be executed.

The pipeline stages are described as code within GitHub using the GitHub actions. In this first step, we want to trigger the automated tests in the UAT environment after the application deployment. We can achieve this result by configuring the call to the Cerberus Testing campaign execution API.

Figure 4: The GitHub Pipeline and Actions triggering the Cerberus Testing campaign in UAT.

For our case, we rely on a simple bash script running after the deployment. The secrets and access management is handled by secrets, allowing us to remove that part from the configuration. The power of infrastructure as code is that this code is reusable for the web, API and any other application deployment we could have.

run_tests:
    needs: [deploy_UAT_FR,deploy_UAT_UK]
    name: Cerberus Testing on UAT
    runs-on: ubuntu-20.04
    steps:
    - uses: actions/checkout@v2
    - name: Run script file
      run: |
         chmod +x ./launchTest.sh
         ./launchTest.sh
      env:
        AUTHOR: ${{ github.event.pusher.name }}
        HOST: https://jftl2021.cerberus-testing.com
        CAMPAIGN: Website_NonRegression_UAT
        APIKEY: ${{ secrets.APIKEY }}
      shell: bash

We can repeat the same configuration for all environments we have for our application. In a traditional setup, that means executing the test in a QA, UAT, staging environment. The remaining environment is the one of production where the test to execute can be quite different.

Production deployment requires specific post-deployment checks also known as sanity checks.

Configure sanity checks and post-deployment tests

Sanity checks aim to verify the status of an application straight after its deployment in production. In the long run, monitoring and observability solutions take care of the application verification. The power of sanity checks is to do that after the deployment for a partial or complete deployment.

We will perform similar verification in our concrete case of an application exposing a Web and API interface. After the deployment in production, we will execute all the tests that are considered safe in a read-only mode. That way, we can guarantee the application status without impacting its operations.

Figure 5: The overall GitHub Actions pipeline up to the post-deployment sanity checks.

The configuration is similar to the previous one. The only difference is the campaign we will call in the API, being the post-production deployment one. The added value of relying on the campaign is the abstraction and scalability of the technique. We can add, update and remove tests without impacting the configuration of our pipelines.

Continuous Testing is a matter of performing regular checks at each stage of the deployment to reduce the risks.

Continuously improve and shift-right afterward

We cover in this article how you can deploy Quality Gates in CI/CD using GitHub actions. From now, you should be able to deploy various stages of test campaigns, being of non-regression, functional or post-deployment types.

The added value of these Quality Gates is not only for testers. The whole team can benefit from these more frequent feedback loops: the business, product, engineering, and operations teams. This is where the value of Continuous Testing is.

This article focuses on the middle area of Continuous Testing with the CI/CD part. But the overall Software Delivery Lifecycle (SDLC) is more transversal. Hence the importance of shifting left and right to cover the entire chain.

We will cover these other areas in the following articles. In the meanwhile, you can start using Cerberus Testing for free for your Continuous Testing initiative.
Start now.

How To Make Quality Gates in CI/CD with GitHub