What you'll learn
- How to set up an AWS CodeBuild project to run your Artillery tests
- How to generate Artillery test reports from AWS CodeBuild
- How to automatically upload Artillery test reports to Amazon S3
- How to schedule your Artillery tests to run at a specific time
- How to use Artillery Pro in an AWS CodeBuild project
Integrating Artillery with AWS CodeBuild allows you to track your service's performance at scale while developing your applications on Amazon's AWS platform. The following guide will show you how to load-test your services using an AWS CodeBuild project.
You can find the complete source code used in this guide in the Artillery CI/CD Examples GitHub repo.
Artillery test script example
In this guide, we'll use the following Artillery test script to run a load test on a running Socket.IO service, sending 50 virtual users per second for 10 minutes and ensuring the aggregate maximum latency for the test is under 500 milliseconds:
- duration: 600
- name: "Emit an event"
data: "Hello from Artillery"
data: "Hello from Artillery"
You can run an example of this test script and see it in action.
Setting up AWS CodeBuild project
This guide assumes you have an existing AWS account with permissions to access the different resources used when setting up AWS CodeBuild projects, such as S3 (for storing and retrieving build artifacts) and CloudWatch (for generating and viewing build logs). For details on the required credentials and permissions, read the "Identity and access management in AWS CodeBuild" section on the AWS CodeBuild User Guide.
We'll go through the process of setting up a new AWS CodeBuild project on the AWS console. After signing in to the console, go to the CodeBuild service to create a build project. The Create build project page provides many setup options, depending on your needs. The following are the main sections you'll need to set up for new CodeBuild projects:
AWS CodeBuild can pull in your source code from different providers, such as AWS CodeCommit, an S3 bucket, GitHub, and Bitbucket. Each source provider has different settings, both required and optional. As an example for this guide, we'll use GitHub as the source provider, using a repository from the connected account. This example also sets up a webhook that listens to any
PUSH events in the GitHub repository to trigger a new build.
You can choose different environments for your builds. AWS CodeBuild provides options to use a managed image or a custom Docker image. We'll use a managed image based on Ubuntu 20.04 (
aws/codebuild/standard:5.0 image) for this guide, which contains Node.js version 14 as a runtime, which is needed to install Artillery.
This step also requires setting up a service role for your AWS CodeBuild project. This service role delegates permissions between the different AWS services used for builds, like generating logs in CloudWatch or sending artifacts to S3. You can create a new service role or use an existing one from your AWS account.
The buildspec is the configuration used by AWS CodeBuild to process the different settings and commands in each project build. The configuration file can be read from the source specified in the project setup or created directly in the AWS console. When using the source, CodeBuild looks for a file named
buildspec.yml in the source code root directory by default. If needed, you can change the file's name that AWS CodeBuild will look for in the repo. For this guide, we'll create a
buildspec.yml file in the GitHub repository.
Setting up build logs is optional for AWS CodeBuild. However, it's a good practice to set up logging to know what's happening inside your build environment. AWS CodeBuild can upload build logs to CloudWatch of an S3 bucket. For this guide, we'll use CloudWatch.
Creating the buildspec for AWS CodeBuild
After creating the project on AWS CodeBuild, we can set up a buildspec in the source code repository to run our tests. For this guide, the buildspec will set up Artillery and run a load test for the Socket.IO service after pushing new code to the GitHub repository.
Create a buildspec configuration file placed inside the source code repository called
buildspec.yml with the following contents:
- npm install -g artillery@latest
- artillery run tests/performance/socket-io.yml
Commit this file to the repository. Once you push the update to GitHub, AWS CodeBuild will execute the load test:
Generating and viewing Artillery test reports
Artillery can output a JSON file with additional details from the load test and use it to generate a self-contained HTML report.
First, the job needs to create a directory to place the test reports. Next, you can generate a JSON report when executing the Artillery load test. You can then use the
report command to generate the HTML report from the JSON file. Finally, AWS CodeBuild can send the generated files as artifacts to an S3 bucket upon completion of the job.
Before updating the buildspec to generate and save the reports, we need to ensure our AWS CodeBuild project is set up to publish artifacts to S3. In this guide, we'll set up our project to send artifacts to an S3 bucket that belongs to the AWS account, using the service role initially set up with the project.
Once the AWS CodeBuild project handles build artifacts, we can update the
buildspec.yml file with the following configuration to generate the Artillery load test report and send it to S3:
- npm install -g artillery@latest
- mkdir reports
- artillery run --output reports/report.json tests/performance/socket-io.yml
- artillery report --output reports/report reports/report.json
The configuration now uses the
post_build phases to handle the steps needed to create a new directory and generate the HTML report. These steps can also be placed in the
build phase instead if desired. The
artifacts section also uses the
$CODEBUILD_BUILD_NUMBER variable as a namespace to better associate the reports with a specific build when sending to the S3 bucket.
After running the build successfully, AWS CodeBuild will send all the generated files to the S3 bucket specified when setting up the artifacts for the project:
For more details on how to configure artifacts in a project build, read the artifacts section in the AWS CodeBuild buildspec reference guide.
Scheduling Artillery load tests
You can also automatically schedule builds to run on AWS CodeBuild with Amazon EventBridge, which is helpful if you want to execute your Artillery load tests at a specific time. For instance, you may wish to load-test your production applications outside of peak hours. You can create an Amazon EventBridge rule to target an AWS CodeBuild project and trigger a new build at a specific schedule.
When choosing a schedule, you can specify either a fixed rate (like every 12 hours) or use a cron expression for more fine-grained control. The cron expression syntax is different from the standard POSIX cron syntax. For instance, the following cron expression sets up the schedule to trigger the build every day at midnight UTC:
You'll need to specify CodeBuild project as the target for the EventBridge rule and set the correct Project ARN for your AWS CodeBuild project.
For more information, read the "Schedule automated builds using AWS CodeBuild" tutorial on the Amazon EventBridge User Guide.
Geographically distributed load testing with Artillery Pro
You can scale your load tests by using Artillery Pro to execute your tests. Artillery Pro allows you to run your Artillery tests from different geographic regions within your AWS infrastructure.
To use Artillery Pro on AWS CodeBuild, you must have previously deployed the Artillery Pro backend to your AWS account. You also must provide the correct permissions to allow your CodeBuild project to access the Artillery Pro backend. The simplest way to enable access to your Artillery Pro backend is to attach the role generated by Artillery Pro to the AWS CodeBuild role created with the project.
Once your AWS CodeBuild project has the correct permissions, you can set up your builds to use Artillery Pro. The following configuration will install the Artillery Pro CLI and execute the test script on an Artillery Pro backend deployed in the
- npm install -g artillery@latest artillery-pro@latest
- artillery run-test --cluster artillery-pro-cluster --region us-east-1 --count 5 tests/performance/socket-io.yml
For more options on executing your tests using Artillery Pro, check the Pro Commands section in the Command Line guide.