Azure DevOps

What you'll learn

• How to set up a pipeline in Azure DevOps to run your Artillery tests
• How to generate and view Artillery test reports from Azure Pipelines
• How to schedule your Artillery tests to run at a specific time
• How to use Artillery Pro in Azure Pipelines

Overview

Integrating Artillery with Azure DevOps allows you to continuously track your service's performance while developing your applications. The following guide will show you how to load-test your services using the Azure Pipelines functionality that's a part of the family of Azure DevOps Services.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
  target: "http://lab.artillery.io"
  phases:
    - duration: 600
      arrivalRate: 50
  ensure:
    maxErrorRate: 1
    max: 500

scenarios:
  - name: "Emit an event"
    engine: "socketio"
    flow:
      - emit:
          channel: "echo"
          data: "Hello from Artillery"
          response:
            channel: "echoResponse"
            data: "Hello from Artillery"

You can run an example of this test script and see it in action.

Setting up Azure Pipelines

In an existing Azure DevOps project, go to the Pipelines section and create a new pipeline. This brings up the New Pipeline creation wizard. First, you need to point Azure to the code repository containing your Artillery test scripts to use for continuous integration. Azure DevOps supports the most popular code hosting services, but you can use any accessible Git or Subversion code repository:

For this guide, we'll use a GitHub code repository as an example. After granting permissions to GitHub, select the code repository containing your tests.

Next, Azure DevOps offers a few options to help configure your pipeline, depending on your code repository. In this guide, we'll select the Starter pipeline that provides the basics of an Azure Pipelines setup.

In the last step of the setup wizard, you can modify the starter configuration file to suit your needs. The configuration file for the pipeline is defined through a file usually named azure-pipelines.yml at the root of your code repository. The configuration files use YAML syntax to determine the events that will trigger your workflow and the jobs and steps to execute.

You can modify the configuration file directly in the setup wizard. For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the main branch of your repository. The pipeline will install the Node.js version 12 and install Artillery before executing the test script as part of the CI setup.

Update the configuration with the following contents:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js v12.x'

- script: npm install -g artillery@latest
  displayName: 'Install Artillery'

- script: artillery run tests/performance/socket-io.yml
  displayName: 'Execute load tests'

After updating the configuration file, click the Save and run button. You'll get a prompt to commit the azure-pipelines.yml file into your code repository. Once you commit the file into your repository's main branch, Azure DevOps will execute the load test:

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.

First, your workflow needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command to generate the HTML report from the JSON file. Finally, you'll need to store the files as artifacts to access the reports upon completion of the workflow.

The configuration to accomplish these steps is the following:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js v12.x'

- script: npm install -g artillery@latest
  displayName: 'Install Artillery'

- script: mkdir $(System.DefaultWorkingDirectory)/reports
  displayName: 'Make reports directory'

- script: artillery run --output reports/report.json tests/performance/socket-io.yml
  displayName: 'Execute load tests'

- script: artillery report --output reports/report reports/report.json
  displayName: 'Generate HTML report'

- publish: $(System.DefaultWorkingDirectory)/reports
  artifact: artillery-test-report

After successfully executing the load test, Azure DevOps will store the files inside the reports directory created as part of the workflow. You can find the published artifacts under the "Related" section of the test run results:

For more details on accessing the artifacts in Azure Pipelines, read the "Artifacts in Azure Pipelines overview" section in the Azure DevOps documentation.

Scheduling Artillery load tests

You can also trigger an Azure Pipelines run on a recurring schedule which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

The schedules trigger can be configured using the POSIX cron syntax, with the times in UTC. The following configuration will automatically trigger Azure Pipelines to run every day at midnight UTC:

schedules:
- cron: "0 0 * * *"
  displayName: 'Midnight (UTC) performance test'
  branches:
    include:
    - main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js v12.x'

- script: npm install -g artillery@latest
  displayName: 'Install Artillery'

- script: artillery run tests/performance/socket-io.yml
  displayName: 'Execute load tests'

For more details about scheduling test runs in Azure DevOps, read the "Configure schedules for pipelines" section in the Azure DevOps documentation.

Distributed load testing with Artillery Pro

You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.

To use Artillery Pro with Azure DevOps, you must have previously deployed the Artillery Pro backend to your AWS account. You also must set up the following environment variables as secret variables in your pipeline:

• AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to access the Artillery Pro backend resources.
• AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user.

The following configuration will execute your Artillery test script on an Artillery Pro backend deployed in the us-east-1 region using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values set as environment variables using the secret variables set above:

trigger:
- main

schedules:
- cron: "0 0 * * *"
  displayName: 'Midnight (UTC) performance test'
  branches:
    include:
    - main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  displayName: 'Install Node.js v12.x'

- script: npm install -g artillery@latest artillery-pro@latest
  displayName: 'Install Artillery and Artillery Pro'

- script: artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5
  displayName: 'Execute load tests on AWS (us-east-1 region)'
  env:
    AWS_ACCESS_KEY_ID: $(AWS_ACCESS_KEY_ID)
    AWS_SECRET_ACCESS_KEY: $(AWS_SECRET_ACCESS_KEY)

For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.