GitHub Actions

What you'll learn

• How to set up a GitHub Actions workflow to run your Artillery tests
• How to generate and view Artillery test reports from GitHub Actions
• How to schedule your Artillery tests to run at a specific time
• How to use Artillery Pro in a GitHub Actions workflow

Overview

Integrating Artillery with GitHub Actions allows you to track your service's performance while developing your applications in your existing GitHub repositories. The following guide will show you how to load-test your services using a GitHub Actions workflow.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
  target: "http://lab.artillery.io"
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 1
    max: 500

scenarios:
  - name: "Emit an event"
    engine: "socketio"
    flow:
      - emit:
          channel: "echo"
          data: "Hello from Artillery"
          response:
            channel: "echoResponse"
            data: "Hello from Artillery"

You can run an example of this test script and see it in action.

Setting up a GitHub Actions workflow

GitHub looks for configuration files placed inside your code repository in a directory called .github/workflows. The configuration files use YAML syntax to define the events that will trigger your workflow and the jobs and steps to execute.

For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the main branch of your GitHub repository. We'll also use the official Docker image to execute the load test without setting up Artillery as part of the workflow.

Create a GitHub Actions workflow configuration file placed in .github/workflows/load-test.yml with the following contents:

name: Artillery Socket.IO Load Test

on:
  push:
    branches:
      - main

jobs:
  artillery:
    runs-on: ubuntu-latest

    container: artilleryio/artillery:latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Execute load tests
        run: /home/node/artillery/bin/run run tests/performance/socket-io.yml

Commit this file to the main branch of your repository. Once you push the update to GitHub, GitHub Actions will execute the load test:

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.

First, your workflow needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command to generate the HTML report from the JSON file. Finally, you'll need to store the files as artifacts to access the reports upon completion of the workflow.

The configuration to accomplish these steps is the following:

name: Artillery Socket.IO Load Test

on:
  push:
    branches:
      - main

jobs:
  artillery:
    runs-on: ubuntu-latest

    container: artilleryio/artillery:latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Make reports directory
        run: mkdir reports

      - name: Execute load tests
        run: /home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml

      - name: Generate HTML report
        run: /home/node/artillery/bin/run report --output reports/report reports/report.json

      - name: Archive test report
        uses: actions/upload-artifact@v2
        with:
          name: artillery-test-report
          path: reports/*

After successful execution of the load test, GitHub Actions will store a zip file of the directory created as part of the workflow:

For more details on accessing the artifacts of a GitHub Actions workflow, read "Downloading workflow artifacts" section in GitHub's documentation.

Scheduling Artillery load tests

You can also schedule GitHub Actions to run a workflow on a recurring schedule using the on.schedule configuration setting, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

The on.schedule setting accepts a string using the POSIX cron syntax, with the times in UTC. The following configuration will automatically trigger the GitHub Actions workflow every day at midnight UTC:

name: Artillery Socket.IO Load Test

on:
  push:
    branches:
      - main
  schedule:
    - cron: '0 0 * * *'

jobs:
  artillery:
    runs-on: ubuntu-latest

    container: artilleryio/artillery:latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Make reports directory
        run: mkdir reports

      - name: Execute load tests
        run: /home/node/artillery/bin/run run --output reports/report.json tests/performance/socket-io.yml

      - name: Generate HTML report
        run: /home/node/artillery/bin/run report --output reports/report reports/report.json

      - name: Archive test report
        uses: actions/upload-artifact@v2
        with:
          name: artillery-test-report
          path: reports/*

Distributed load testing with Artillery Pro

You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.

To use Artillery Pro with GitHub Actions, you must have previously deployed the Artillery Pro backend to your AWS account. You also must set up the following environment variables as encrypted secrets:

• AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to access the Artillery Pro backend resources.
• AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user.

The following configuration will execute your Artillery test script on an Artillery Pro backend deployed in the us-east-1 region using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values set as encrypted secrets:

name: Artillery Socket.IO Load Test

on:
  push:
    branches:
      - main
  schedule:
    - cron: '0 0 * * *'

jobs:
  artillery:
    runs-on: ubuntu-latest

    container: artilleryio/artillery:latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Install Artillery Pro
        run: npm install artillery-pro@latest

      - name: Execute load tests on AWS (us-east-1 region)
        run: /home/node/artillery/bin/run artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.