Running Artillery on AWS CodeBuild
What you'll learn
- How to set up an AWS CodeBuild project to run your Artillery tests
- How to generate Artillery test reports from AWS CodeBuild
- How to automatically upload Artillery test reports to Amazon S3
- How to schedule your Artillery tests to run at a specific time
- How to trigger distributed load tests from an AWS CodeBuild project
Integrating Artillery with AWS CodeBuild allows you to track your service's performance at scale while developing your applications on Amazon's AWS platform. The following guide will show you how to load-test your services using an AWS CodeBuild project.
You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo (opens in a new tab).
In this guide, we'll use the following Artillery test script to run a load test on a running Socket.IO (opens in a new tab) service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:
config: target: "<http://lab.artillery.io>" phases: - duration: 600 arrivalRate: 50 ensure: maxErrorRate: 1 max: 500 scenarios: - name: "Emit an event" engine: "socketio" flow: - emit: channel: "echo" data: "Hello from Artillery" response: channel: "echoResponse" data: "Hello from Artillery"
This guide assumes you have an existing AWS account with permissions to access the different resources used when setting up AWS CodeBuild projects, such as S3 (for storing and retrieving build artifacts) and CloudWatch (for generating and viewing build logs). For details on the required credentials and permissions, read the "Identity and access management in AWS CodeBuild" (opens in a new tab) section on the AWS CodeBuild User Guide.
We'll go through the process of setting up a new AWS CodeBuild project on the AWS console. After signing in to the console, go to the CodeBuild service to create a build project. The Create build project page provides many setup options, depending on your needs. The following are the main sections you'll need to set up for new CodeBuild projects:
AWS CodeBuild can pull in your source code from different providers, such as AWS CodeCommit, an S3 bucket, GitHub, and Bitbucket. Each source provider has different settings, both required and optional. As an example for this guide, we'll use GitHub as the source provider, using a repository from the connected account. This example also sets up a webhook that listens to any
PUSH events in the GitHub repository to trigger a new build.
You can choose different environments for your builds. AWS CodeBuild provides options to use a managed image (opens in a new tab) or a custom Docker image. We'll use a managed image based on Ubuntu 20.04 (
aws/codebuild/standard:5.0 image) for this guide, which contains Node.js version 14 as a runtime, which is needed to install Artillery.
This step also requires setting up a service role for your AWS CodeBuild project. This service role delegates permissions between the different AWS services used for builds, like generating logs in CloudWatch or sending artifacts to S3. You can create a new service role or use an existing one from your AWS account.
The buildspec is the configuration used by AWS CodeBuild to process the different settings and commands in each project build. The configuration file can be read from the source specified in the project setup or created directly in the AWS console. When using the source, CodeBuild looks for a file named
buildspec.yml in the source code root directory by default. If needed, you can change the file's name that AWS CodeBuild will look for in the repo. For this guide, we'll create a
buildspec.yml file in the GitHub repository.
Setting up build logs is optional for AWS CodeBuild. However, it's a good practice to set up logging to know what's happening inside your build environment. AWS CodeBuild can upload build logs to CloudWatch of an S3 bucket. For this guide, we'll use CloudWatch.
After creating the project on AWS CodeBuild, we can set up a buildspec in the source code repository to run our tests. For this guide, the buildspec will set up Artillery and run a load test for the Socket.IO service after pushing new code to the GitHub repository.
Create a buildspec configuration file placed inside the source code repository called
buildspec.yml with the following contents:
version: 0.2 phases: install: commands: - npm install -g artillery@latest build: commands: - artillery run tests/performance/socket-io.yml
Commit this file to the repository. Once you push the update to GitHub, AWS CodeBuild will execute the load test:
Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report (opens in a new tab).
First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the
report command (opens in a new tab) to generate the HTML report from the JSON file. Finally, AWS CodeBuild can send the generated files as artifacts to an S3 bucket upon completion of the job.
Before updating the buildspec to generate and save the reports, we need to ensure our AWS CodeBuild project is set up to publish artifacts to S3. In this guide, we'll set up our project to send artifacts to an S3 bucket that belongs to the AWS account, using the service role initially set up with the project.
Once the AWS CodeBuild project handles build artifacts, we can update the
buildspec.yml file with the following configuration to generate the Artillery load test report and send it to S3:
version: 0.2 phases: install: commands: - npm install -g artillery@latest pre_build: commands: - mkdir reports build: commands: - artillery run --output reports/report.json tests/performance/socket-io.yml post_build: commands: - artillery report --output reports/report reports/report.json artifacts: files: - "reports/*" name: artifacts/$CODEBUILD_BUILD_NUMBER
The configuration now uses the
post_build phases to handle the steps needed to create a new directory and generate the HTML report. These steps can also be placed in the
build phase instead if desired. The
artifacts section also uses the
$CODEBUILD_BUILD_NUMBER variable as a namespace to better associate the reports with a specific build when sending to the S3 bucket.
After running the build successfully, AWS CodeBuild will send all the generated files to the S3 bucket specified when setting up the artifacts for the project:
For more details on how to configure artifacts in a project build, read the artifacts section (opens in a new tab) in the AWS CodeBuild buildspec reference guide.
You can also automatically schedule builds to run on AWS CodeBuild with Amazon EventBridge, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours. You can create an Amazon EventBridge rule to target an AWS CodeBuild project and trigger a new build at a specific schedule.
When choosing a schedule, you can specify either a fixed rate (like every 12 hours) or use a cron expression (opens in a new tab) for more fine-grained control. The cron expression syntax is different from the standard POSIX cron syntax. For instance, the following cron expression sets up the schedule to trigger the build every day at midnight UTC:
You'll need to specify CodeBuild project as the target for the EventBridge rule and set the correct Project ARN (opens in a new tab) for your AWS CodeBuild project.
For more information, read the "Schedule automated builds using AWS CodeBuild" (opens in a new tab) tutorial on the Amazon EventBridge User Guide.
You can scale your load tests by using Artillery's AWS Lambda and AWS Fargate integrations to execute your tests.
Once your AWS CodeBuild project has the correct permissions, you can set up your builds to run distributed tests.
The following configuration will install the Artillery CLI and execute the test script in the
version: 0.2 phases: install: commands: - npm install -g artillery@latest build: commands: - artillery run-fargate --region us-east-1 --count 5 tests/performance/socket-io.yml