Skip to main content

Jenkins

What you'll learn
  • How to set up a Jenkins Pipeline to run your Artillery tests
  • How to generate and view Artillery test reports from Jenkins
  • How to schedule your Artillery tests to run at a specific time
  • How to use Artillery Pro in a Jenkins pipeline

Overview

Integrating Artillery with Jenkins allows you to track your service's performance while developing your applications using the popular open-source CI/CD tool. The following guide will show you how to load-test your services using a Jenkins Pipeline.

You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.

Artillery test script example

In this guide, we'll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
ensure:
maxErrorRate: 1
max: 500

scenarios:
- name: "Emit an event"
engine: "socketio"
flow:
- emit:
channel: "echo"
data: "Hello from Artillery"
response:
channel: "echoResponse"
data: "Hello from Artillery"

You can run an example of this test script and see it in action.

Setting up Jenkins Pipelines

This guide assumes you have an accessible Jenkins server running. The examples in this guide use Docker as a Jenkins agent for running Jenkins Pipelines, and build integration with GitHub to run the Artillery load test from a GitHub repository. Jenkins supports various build agents and integration with different source code management services, so you can easily modify the sample configuration files for your needs.

For this guide, the Artillery load test for the Socket.IO service will run after pushing new code to the main branch of an existing GitHub repository with our Artillery test. We'll also use the official Docker image to execute the load test without setting up Artillery as part of the Pipeline.

We'll use a Jenkins Pipeline to run our Artillery load tests. You can create a Jenkins Pipeline through the Jenkins UI or the newer Blue Ocean interface. For this guide, we'll initialize the Pipeline through the Jenkins interface and later create the Pipeline's configuration inside of the code repository.

Jenkins Pipeline Setup

The Jenkins Pipeline setup offers multiple options to set up a build and how to trigger it. The initial setup also asks how to define the Pipeline. The Pipeline can be set up directly through the Jenkins UI. However, for better control and traceability, a good choice is to create a Pipeline script in the code repository for your test, which we'll do in this example.

Defining Jenkins Pipeline in SCM

To have Jenkins look in our code repository, choose the "Pipeline script from SCM" Pipeline definition option and configure the repository based on your source code management service. Jenkins looks for a file called Jenkinsfile in the root of the code repository. The Jenkinsfile configuration supports two different ways to configure a Pipeline — Declarative and Scripted. Both formats work similarly, with some differences under the hood. For this guide, we'll follow the Declarative Pipeline syntax for the Jenkinsfile configuration.

After saving the Jenkins Pipeline, create a new file called Jenkinsfile in the root of your code repository with the following contents:

pipeline {
agent {
docker {
image 'artilleryio/artillery:latest'
args '-u root:root -i --entrypoint='
}
}

stages {
stage('Load Test') {
steps {
sh '/home/node/artillery/bin/artillery run tests/performance/socket-io.yml'
}
}
}
}

Note that when setting up the Artillery Docker image under agent, we're adding additional arguments that Jenkins will use to start the build:

  • -u root:root - By default, the Docker build agent will execute any actions inside the container using the jenkins user. This user does not have permission to run Artillery inside the container, so adding this argument will allow us to execute any actions as the root user inside the container.
  • -i - This argument sets the container to run in 'interactive' mode to allow Jenkins to execute any commands in the Pipeline stages.
  • --entrypoint= - Jenkins will use the Docker image's defined ENTRYPOINT when starting the container, which we don't need for continuous integration purposes. This argument overrides the ENTRYPOINT so we can execute commands in the container.

Commit this file to the main branch of your repository. You can manually trigger a build by clicking on the Build Now option inside of your Pipeline project on Jenkins:

Successful Artillery test run with Jenkins

Generating and viewing Artillery test reports

Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.

First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the report command to generate the HTML report from the JSON file. Finally, you'll need to archive the files as artifacts to access the reports upon completion of the job.

The following configuration accomplishes these steps:

pipeline {
agent {
docker {
image 'artilleryio/artillery:latest'
args '-u root:root -i --entrypoint='
}
}

stages {
stage('Load Test') {
steps {
sh 'mkdir reports'
sh '/home/node/artillery/bin/artillery run --output reports/report.json tests/performance/socket-io.yml'
sh '/home/node/artillery/bin/artillery report --output reports/report reports/report.json'
}
}
}

post {
success {
archiveArtifacts 'reports/*'
}
}
}

After successful execution of the load test, Jenkins will store the files from the reports directory created as part of the build, which you can download from the build details page on Jenkins:

Artillery report artifacts in Jenkins build

For more details on accessing the artifacts of a Jenkins build, read the "Recording tests and artifacts" section in the Jenkins documentation.

Scheduling Artillery load tests

You can also set up Jenkins to run a build on a recurring schedule by setting up a trigger in the Pipeline definition, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours.

You can include the triggers directive in the Jenkinsfile configuration file with the cron parameter using the POSIX cron syntax (with some minor differences). The cron trigger uses the time zone set in your Jenkins server. The following configuration will automatically trigger the Jenkins build every day at midnight, based on the server's time zone:

pipeline {
agent {
docker {
image 'artilleryio/artillery:latest'
args '-u root:root -i --entrypoint='
}
}

triggers {
cron('0 0 * * *')
}

stages {
stage('Load Test') {
steps {
sh 'mkdir reports'
sh '/home/node/artillery/bin/artillery run --output reports/report.json tests/performance/socket-io.yml'
sh '/home/node/artillery/bin/artillery report --output reports/report reports/report.json'
}
}
}

post {
success {
archiveArtifacts 'reports/*'
}
}
}

Distributed load testing with Artillery Pro

You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.

To use Artillery Pro with GitHub Actions, you must have previously deployed the Artillery Pro backend to your AWS account. You also must set up the following environment variables as secret text values in the Jenkins global credentials section:

  • AWS_ACCESS_KEY_ID - The access key ID of an IAM user with permissions to access the Artillery Pro backend resources. In this example, we'll set up this credential with the ID jenkins-aws-secret-key-id.
  • AWS_SECRET_ACCESS_KEY - The secret access key of the IAM user. In this example, we'll set up this credential with the ID jenkins-aws-secret-access-key.
Setting up AWS credentials in Jenkins for Artillery ProCompleted AWS credentials in Jenkins for Artillery Pro

The following configuration will execute your Artillery test script on an Artillery Pro backend deployed in the us-east-1 region using the AWS_ACCESS_KEY_ID (using the credential ID jenkins-aws-secret-key-id) and AWS_SECRET_ACCESS_KEY (using the credential ID jenkins-aws-secret-access-key) set as environment variables using the credentials specified in Jenkins:

pipeline {
agent {
docker {
image 'artilleryio/artillery:latest'
args '-u root:root -i --entrypoint='
}
}

triggers {
cron('0 0 * * *')
}

environment {
AWS_ACCESS_KEY_ID = credentials('jenkins-aws-secret-key-id')
AWS_SECRET_ACCESS_KEY = credentials('jenkins-aws-secret-access-key')
}

stages {
stage('Set up Artillery Pro') {
steps {
sh 'npm install -g artillery-pro@latest'
}
}
stage('Load Test on AWS') {
steps {
sh '/home/node/artillery/bin/artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5'
}
}
}
}

For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.