Skip to main content

· 6 min read
Hassy Veldstra
Load testing with real browsers

Load testing non-trivial dynamic web applications is no walk in the park.

If you've ever had to do it, you know. Creating test scripts is slow, error-prone, cumbersome, brittle, and just downright annoying. You have to deal with forms, track down which on-page actions kick off which request to the backend, and if any part of the flow depends on in-page Javascript... forget it.

Traditional load testing tools don't work well

The root cause of it all is simple: a mismatch in the level of abstraction. Load testing tools are designed to work with API endpoints, whereas a page is a better and more natural abstraction when testing web apps.

When testing an API, there's usually a spec available (e.g. an OpenAPI spec). Testing a web app? It's unlikely there's even a list of all API endpoints used on different pages. Different APIs may be called depending on in-page actions. Some calls may be made by in-page JS — and these have to be tracked down and emulated. Hours of fun in Chrome DevTools!

There hasn't been a good solution... up until now. We've been busy at Artillery HQ1 cooking something up.

Launching 10,000 browsers for fun and profit

What if you could run load tests with real browsers, at scale, and just reuse existing end-to-end testing scripts?

Artillery Playwright load testing with browsers demo

(the demo is also on YouTube)

A few things to highlight in the demo above:

  • We're re-using Playwright scripts, which we can also generate with playwright codegen, i.e. we can click around in a browser to create a test script
  • We get both backend and frontend performance metrics in the report, so we can see how load affects LCP or FCP over time
  • We're running thousands of browsers with zero infrastructure setup, and they run from your own AWS account - not a hosted cloud service. Did "ridiculous cost efficiency" cross your mind? -- so efficient you could run these tests in CICD multiple times a day. All sorts of nice security and data privacy-related thoughts too perhaps, since this would run in your own VPC. Why, yes, that's why we like cloud-native software that we can run ourselves too.

Are we also possibly talking 10x developer productivity for load testing non-trivial applications? Yes, yes, quite possibly.

How does that work?

How does that work? Playwright provides the browsers. Artillery provides the ability to launch a ridiculous number of them from your own AWS account, with no infra to manage.

Behind the scenes, we expose the Playwright API to Artillery via Artillery's extension interface, which is the same API that powers all of Artillery's many engines. This lets Artillery treat Playwright scripts as a description of what a virtual user scenario is. From that point onwards, as far as Artillery is concerned there is no difference between opening a TCP connection and exchanging Socket.IO messages (if we're testing Socket.IO) or launching a browser and running page-based actions (with the Playwright engine).

Artillery Pro can then pick up from there to launch swarms of Artillery workers running the Playwright engine on AWS ECS or Fargate for testing at scale.

OK, so we can but should we?

Just because we can do something doesn't mean we should, does it? We're running real browsers here, which is obviously going to be resource-intensive. Is it going to scale? Is it worth it?

Does it scale?

Yes, it certainly does. It's 2021, cloud compute is cheap & plentiful, and "just throw more containers at it" can be a perfectly legitimate tactic.

Again, we are launching whole browsers here, which is going to be resource-hungry. Let's look at Fargate with the beefiest configuration per-container we can get: 4 vCPUs and 12GB of memory. Running a container with that spec is us-east-2 for a whole hour is going to cost us a whopping 22 cents. If we run a 1,000 of those containers for an hour, we're looking at $220.

How much load can those 1,000 containers generate? In our unscientific tests running a simple but not-trivial Playwright flow, we could launch one Chromium instance every other second in each container, making around 20 HTTP requests per second each.

Your particular results will of course vary. Memory is going to be the main bottleneck, which will put a ceiling on how many concurrent browser instances we can run in a single container. The number of those will depend on the web app being tested and your Playwright scenario. The slower the app and the longer the test scirpt, the more concurrent Chrome instances you'll be running in each container.

The trade-off is developer productivity vs AWS costs. If you're testing a complex web app, especially on a tight schedule, you can be up and running much faster with a Playwright-based approach. Developer time is usually expensive, and cloud compute isn't - your call.

And hey, you don’t have to go full hog and load test exclusively with browsers, as fun as that may be. We can now run hybrid tests2 and smoke tests with browsers too.

Hybrid tests

In a hybrid test you’d generate most of the load on the backend with a traditional approach of sending HTTP requests to verious endpoints. You also run a Playwright-based test alongside as part of the same test (using scenario weights), to see how load on a key API endpoint can affect frontend metrics.

Smoke tests

Smoke testing is a common use-case for Artillery, and there's nothing stopping us from running our Playwright scripts with Artillery as smoke tests (also known as synthetic checks and E2E checks - think health checks on steroids). These are particularly useful for running in CICD to check that everything is still OK after every deployment.

Try it yourself!

The engine is open source with the code available on Github at artilleryio/artillery-engine-playwright.

You can try an end-to-end example locally by following along the README in the e2e example repo we put together - artillery-examples/browser-load-testing-playwright. The official repo includes a Dockerfile for running these tests in CICD as well.

Playwright support is available out-of-the-box to all Artillery Pro customers, including the free Dev plan.

We’d love to hear how you might use this. Hit us up on Twitter or on GitHub discussions. ✌️


  1. there isn't really an HQ, we're an all-remote distributed team
  2. You can’t do hybrid testing with anything else but Artillery so we’re going to say we’re coining a new term here. "Hybrid testing" - has a bit of a ring to it, doesn't it? Expect to see hybridtestingmanifesto.dev landing any day soon.

· 11 min read

Load testing GraphQL

See end-to-end example with Apollo and Prisma - artillery-examples/graphql-api-server.


The popularity of GraphQL for building and querying APIs has been on the rise for years, with no sign of slowing down. Developers can now build robust and flexible APIs that give consumers more choices when accessing their data. This flexibility, in turn, can help with application performance compared to traditional RESTful APIs. Providing API consumers the flexibility to fetch the data they need avoids the overhead of processing unnecessary information.

Despite all the advantages GraphQL provides to developers, it's also easy to introduce potential performance issues when constructing APIs:

  • Its flexibility can allow clients to craft inefficient queries when fetching server-side data, especially when using nested queries that need to get data from multiple database tables or sources. This problem is more prevalent when using ORMs with GraphQL since it can mask any inefficiencies in your database calls.
  • Since GraphQL allows consumers to build different queries as they need, it isn't easy to implement caching effectively. Since each request made to the API is potentially unique, developers can't use standard caching techniques. Some libraries provide built-in caching, but it may not work in all scenarios.

Most developers building GraphQL APIs eventually run into one or both of these issues. While these performance problems are simple to deal with, they're also easily overlooked. It's also not uncommon to find yourself with a sluggish API after deploying to production when dealing with actual data. That's why load testing your GraphQL APIs before and after deploying is more crucial than ever.

With Artillery, you can create test scripts that can test common workflows for GraphQL to smoke out any potential performance issues that can creep up when you least expect. Running regular load tests on your APIs both during development and against your production servers will keep your APIs performant and resilient for your customers. This article shows you how to use Artillery to keep your GraphQL services in check.

GraphQL API example

For this article, let's assume we have a GraphQL API with a data store, some queries to fetch this data, and mutations to manage data manipulation based on this diagram:

{% imgTag img/blog/graphql-example-schema.png "GraphQL API Example Schema" " " "border: 1px solid #ddd; margin: 0 auto;" %}

The data store consists of two tables to store user information and messages associated with that user for performing standard CRUD operations. There's a single query to fetch a specific user by their data store ID. The GraphQL API also has some mutations to create, update, and delete a user, along with creating messages associated with a single user.

Let's imagine that most consumers of the GraphQL API perform two primary flows after creating and logging in as the new user. The first flow will create new messages through the API and later fetch the user's account information along with their messages. The other flow handles updating a user. We want to make sure these flows remain performant under load for whoever accesses the API. Let's create a few load tests using Artillery.

Creating our first GraphQL load test

Most GraphQL APIs handle requests via standard HTTP GET and POST methods, meaning we can easily use Artillery's built-in HTTP engine to make requests for load testing. Both methods work similarly, with some differences related to how HTTP works. For instance, GET requests are easier to cache, while POST responses aren't cacheable unless they include specific headers. For simplicity in our load tests, we'll use the POST method to make our GraphQL API requests.

When served over HTTP, a GraphQL POST request accepts a JSON-encoded body with a query field containing the GraphQL query to execute on the server. The body can also include an optional variables field to set and use variables in the specified GraphQL query. We'll use both of these fields for our load test.

For our first test, we'll go through one of the flows described above:

  • First, we'll create a new user record through the GraphQL API and capture their email address from the GraphQL response. We'll generate randomized usernames and email addresses using Artillery's $randomString() method to ensure we don't run into uniqueness constraints when creating users during the test.
  • Next, we'll log in as the created user to test the login process and capture their ID from the data store.
  • With the user's ID in hand, we'll then use the loop construct to create 100 new messages associated with the user.
  • After creating those messages, we'll make a request to return the user data and their messages to verify how the API performs when fetching associated data between different database tables.
  • Finally, to clean up the database, we'll delete the user at the end of the flow, which will also delete the user's associated messages.

Let's see how this test script looks for executing this flow:

config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50

scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100

- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

For this example, we have our GraphQL API set up at https://graphql.test-app.dev. The Artillery test will run for ten minutes, generating 50 virtual users per second. In our scenario, we have our defined flow with a few POST requests made to the root URL (/) where the GraphQL API handles our requests.

Each step in our flow section works similarly. We'll have a JSON request to the GraphQL server with the query and variables fields (where necessary). The query field contains the GraphQL query or mutation sent to the server for processing. We'll use a pipe (|) when sending the data to make the body more readable in our test script.

Each query and mutation in our script is set up to handle variables since we'll use them for keeping track of the dynamic data generated throughout the test phase. That's where the variables field comes into play. This field can set up the variables we'll use through a standard JSON object. The GraphQL server will handle setting up the values where specified in a query or mutation.

For instance, in the step where we create a user, the createUser mutation expects an input argument, which is an object containing the username, email, and password fields. The variables field will construct the input object using fields required in the mutation argument and pass it to the query. It allows us to generate unique usernames and emails for each virtual user dynamically.

Assuming we save the test script in a file called graphql.yml, we can run it with Artillery using artillery run graphql.yml. After Artillery completes all the requests, you'll see how the GraphQL API performs under this load:

All VUs finished. Total time: 10 minutes, 39 seconds

--------------------------------
Summary report @ 15:31:12(+0900)
--------------------------------

vusers.created_by_name.Create and fetch messages flow: ...... 30000
vusers.created.total: ....................................... 30000
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 947.6
max: ...................................................... 20767.2
median: ................................................... 1465.6
p95: ...................................................... 20136.3
p99: ...................................................... 20136.3
http.request_rate: .......................................... 29/sec
http.requests: .............................................. 3120000
http.codes.200: ............................................. 3120000
http.responses: ............................................. 3120000
http.response_time:
min: ...................................................... 173
max: ...................................................... 4870
median: ................................................... 186.8
p95: ...................................................... 1312
p99: ...................................................... 3899

This example shows how straightforward it is to create a fully-fledged load test for GraphQL services. Artillery's HTTP engine has everything you need out of the box to check the most important actions of your GraphQL API. It's a simple yet powerful way to ensure your GraphQL services are resilient to any spikes in traffic and remain performant throughout. Although the performance of our GraphQL service isn't close to what we'd like, it gives us an excellent starting point for improvement.

Expanding the GraphQL load test

Let's add another scenario to validate the GraphQL API's performance further to expand our load test. For this flow, we want to validate that the service correctly handles password updates. For security purposes, the GraphQL service uses bcrypt hashing to ensure we don't store plain-text passwords in the database. Depending on how a service uses bcrypt, the hashing function can slow down password-related functionality, so it's an excellent candidate to check for performance purposes.

The second scenario for our test goes through the following steps:

  • We'll create a new user record and log in as that user, the same way we did in the first scenario.
  • Next, we'll log in as that user to test the login process and capture their ID from the data store.
  • With the user's ID in hand, we'll call the mutation for updating a user's database record to update their password. This step helps us verify that the request works as expected along with testing its performance.
  • Just like before, we'll delete the user at the end of the flow to keep the database clean of any test records.

The entire test script with both flows is as follows:

config:
target: "https://graphql.test-app.dev"
phases:
- duration: 600
arrivalRate: 50

scenarios:
- name: "Create and fetch messages flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- loop:
- post:
url: "/"
json:
query: |
mutation CreateMessageMutation($authorId: ID!, $body: String!) {
createMessage(authorId: $authorId, body: $body) {
id
body
}
}
variables:
authorId: "{{ userId }}"
body: "Message Body {{ $loopCount }}"
count: 100

- post:
url: "/"
json:
query: |
query Query($userId: ID!) {
user(id: $userId) {
username
email
messages {
id
body
}
}
}
variables:
userId: "{{ userId }}"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

- name: "Update password flow"
flow:
- post:
url: "/"
json:
query: |
mutation CreateUserMutation($input: CreateUserInput) {
createUser(input: $input) {
email
}
}
variables:
input:
username: "{{ $randomString() }}"
email: "user-{{ $randomString() }}@artillery.io"
password: "my-useful-password"
capture:
json: "$.data.createUser.email"
as: "userEmail"

- post:
url: "/"
json:
query: |
mutation LoginUserMutation($email: String!, $password: String!) {
loginUser(email: $email, password: $password) {
id
}
}
variables:
email: "{{ userEmail }}"
password: "my-useful-password"
capture:
json: "$.data.loginUser.id"
as: "userId"

- post:
url: "/"
json:
query: |
mutation UpdateUserMutation($updateUserId: ID!, $input: UpdateUserInput) {
updateUser(id: $updateUserId, input: $input) {
id
}
}
variables:
updateUserId: "{{ userId }}"
password: "my-new-password"

- post:
url: "/"
json:
query: |
mutation DeleteUserMutation($deleteUserId: ID!) {
deleteUser(id: $deleteUserId) {
id
}
}
variables:
deleteUserId: "{{ userId }}"

Running the test script with Artillery as we did previously (artillery run graphql.yml) shows the generated VUs going through both scenarios:

All VUs finished. Total time: 11 minutes, 20 seconds

--------------------------------
Summary report @ 15:43:19(+0900)
--------------------------------

vusers.created_by_name.Update password flow: ................ 18469
vusers.created.total: ....................................... 30000
vusers.created_by_name.Create and fetch messages flow: ...... 11531
vusers.completed: ........................................... 30000
vusers.session_length:
min: ...................................................... 899.1
max: ...................................................... 30451.4
median: ................................................... 1300.1
p95: ...................................................... 29445.4
p99: ...................................................... 29445.4
http.request_rate: .......................................... 28/sec
http.requests: .............................................. 1966900
http.codes.200: ............................................. 1966900
http.responses: ............................................. 1966900
http.response_time:
min: ...................................................... 172
max: ...................................................... 5975
median: ................................................... 186.8
p95: ...................................................... 1620
p99: ...................................................... 4867

As seen in our less-than-ideal performance results, GraphQL helps developers build flexible APIs with ease, but that flexibility comes at a cost if you're not careful. It's easy to accidentally introduce N+1 queries or implement ineffective caching that brings your GraphQL API to a grinding halt with only a handful of consumers. This example test script gives you an idea of how to use Artillery to run robust load tests against your GraphQL services to find any bottlenecks or other issues during periods of high traffic.

End-to-end example

An end-to-end example with Apollo and Prisma can be seen on artillery-examples/graphql-api-server.

· 2 min read

We are deligthed to announce that Artillery has been funded by YCombinator as part of the S21 batch.

Artillery + YCombinator

YCombinator have a long history of backing world-class companies such as Stripe, Dropbox, Airbnb, and Coinbase, and companies building developer tools specifically such as PagerDuty, Segment, Gitlab, Algolia, InfluxData, Armory, Heroku, and CoreOS. We're fired up to be part of the YC family.

What does this mean for Artillery?

Our mission remains the same it's been for over six years - help developers avoid outages & slowdowns, keep production fast & reliable, customers happy, and pagers silent. We are going to continue building the best solution for performance testing at scale, with help from our open source community, and our wonderful customers.

We have a big vision for what the future of performance testing should look like based on years of our own experience with DevOps and SRE, and hundreds of conversations with our users. Making that vision a reality will continue keeping us busy for a long time to come, and we're thrilled to have YC help us on this journey.

Join us

If you're passionate about SRE & DevOps, performance, open source and building tools for other developers you should consider joining us! Drop us a line on careers@artillery.io to say hi and learn more.

-- Team Artillery

· 12 min read

When performing a load test using Artillery, you can set a few conditions in your test script to ensure that the aggregate results meet different requirements using the ensure configuration setting. For instance, you can set a condition to verify that the aggregate median latency of your service is under 200 milliseconds, or that less than your scenarios have less than a 1% error rate. These checks are helpful in CI/CD environments since they can help confirm your service’s performance before deploying new updates or validate if your production services are meeting some of your SLOs.

However, you can use your existing Artillery test scripts to go beyond load testing and checking aggregate results of your HTTP services. With the artillery-plugin-expect plugin, you can easily extend Artillery’s built-in functionality to include functional testing. This plugin adds support for setting expectations and assertions using your current test scenarios, giving you a quick and straightforward solution to check the functionality of your services using your existing testing toolkit.

An example Artillery load test

Let’s take the following Artillery load testing script for an HTTP service as an example. The test script sends a consistent number of virtual users (50 VUs per second for 10 minutes), going through a single scenario. The scenario has three operations: the virtual user will log in to set a cookie with their authentication details, access a secure endpoint, and log out. The script also has a few checks to verify some aggregate results.

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
ensure:
p99: 300
maxErrorRate: 1

scenarios:
- flow:
- post:
url: "/login"
json:
username: "testuser"
password: "testpassword"
- get:
url: "/account"
- delete:
url: "/logout"

As part of the load test, we want to ensure that 99% of the scenarios should complete in under 300 milliseconds, with fewer than a 1% error rate. If the p99 latency is over 300 milliseconds, or more than 1% of the virtual users fail to complete their scenario entirely, the test script returns a non-zero exit code.

While this test is beneficial to confirm our service works well under load, we’d also like to validate that the HTTP service’s endpoints are working as expected. For instance, we want to check that the endpoints return the correct status code, content type, and properties. However, thanks to the artillery-plugin-expect plugin, you don't need a separate testing tool to add this support. Let's see how to set up this plugin and begin setting up these expectations in the same test script.

Using the artillery-plugin-expect plugin

Artillery plugins are created and distributed as standard npm packages and can be installed using the npm or Yarn package managers. Installing the latest version of the artillery-plugin-expect plugin depends on how you’ve set up Artillery in your test environment.

If Artillery is installed globally, you’ll need to install the plugin globally:

# Using npm.
npm install artillery-plugin-expect@latest --global

# Using Yarn (assuming Artillery is installed in /usr/local).
yarn global add artillery-plugin-expect@latest --prefix /usr/local

If you’re using Artillery as a local development dependency of a project (set up in a project’s package.json file), the plugin must be installed as a development dependency:

# Using npm.
npm install artillery-plugin-expect@latest --save-dev

# Using Yarn.
yarn add artillery-plugin-expect@latest --dev

Once the plugin is installed in the test environment, we’ll need to enable the plugin in the Artillery test script. Enabling plugins in Artillery is done with the config.plugins setting in the test script. In the case of artillery-plugin-expect, the configuration is the following:

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
ensure:
p99: 300
maxErrorRate: 1
plugins:
expect: {}

When configuring a plugin, Artillery will attempt to load an npm package using the naming convention of artillery-plugin-<plugin-name>, where <plugin-name> is the name used under config.plugins in the test script. Along with specifying the plugin’s name, we’ll also need to supply any configuration required for the plugin to work. In the case of artillery-plugin-expect, we don’t need to provide any additional configuration, so we’ll set an empty map.

To verify that the plugin is set up correctly, we can run the test script as we usually do. The load test will work the same as before, without any warnings messages if Artillery found the plugin. If the plugin is not set up correctly, the test will still execute, but we’ll see the following warning message appear at the start of the test run:

WARNING: Plugin expect specified but module artillery-plugin-expect could not be found (MODULE_NOT_FOUND)

Usually, this warning shows up because Artillery and the plugin are installed differently — for instance, Artillery is installed globally while the plugin is installed as a local dependency. If the test run shows any plugin warnings, double-check where both Artillery and artillery-plugin-expect are installed, along with the configuration in the test script.

Once we’ve confirmed the plugin is working, we can begin adding expectations and assertions to our test script. To start, we’ll define the following expectations that we want to validate for our functional tests:

  • POST /login - check that the service responds with a 200 OK status code.
  • GET /account - check that the service responds with a 200 OK status code, returns a JSON Content-Type header, and the body has a property named user.
  • DELETE /logout - check that the service responds with a 204 No Content status code.

Let’s add these expectations and assertions to the scenario operations in the Artillery test script:

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
ensure:
p99: 300
maxErrorRate: 1
plugins:
expect: {}

scenarios:
- flow:
- post:
url: "/login"
json:
username: "testuser"
password: "testpassword"
expect:
- statusCode: 200
- get:
url: "/account"
expect:
- statusCode: 200
- contentType: "json"
- hasProperty: "user"
- delete:
url: "/logout"
expect:
- statusCode: 204

Each request in the test scenarios we have now defines what we want to assert for each one. The plugin processes the expect keyword during the test run, checking each of the expectations set below, like statusCode, contentType, and hasProperty. These expectations and assertions cover what we wanted to check for our functional test.

When we run the test script using Artillery, the artillery-plugin-expect plugin will now print out a report on the state of the assertions for each request it makes:

* POST /login
ok statusCode 200
* GET /account
ok statusCode 200
ok contentType json
ok hasProperty user
* DELETE /logout
ok statusCode 204

With just a few additions to an existing Artillery test script, you can include functional testing alongside your load tests. It’s a simple yet powerful way to test an HTTP service without introducing new tools or libraries into your projects.

Separating load tests and functional tests in the same test script

With the existing test script, we’re also validating the assertions for every virtual user. This approach for functional testing is not the best one to take since there’s a good chance an HTTP service will have occasional errors while under heavy load. Ideally, we would run through our scenarios only once for functional tests.

Thankfully, we don’t have to create two separate test scripts to split up load and functional tests. Artillery allows us to set different configurations for distinct purposes using config.environments. This setting can let us reuse our load testing script while changing some configuration under different contexts. In our case, we want to have two environments:

  • A load testing environment, where we’ll send a large number of virtual users to the HTTP service for an extended amount of time and verify a few aggregate results after the test run.
  • A functional testing environment, where we’ll run our scenario with a single virtual user to validate expectations for each request once.

The main difference between each environment is the load phase and what we want to verify during or after a test run. We can use the config.environments setting to set these up separately, which we can then use to run the test script according to the testing type:

config:
target: "http://lab.artillery.io"
environments:
load:
phases:
- duration: 600
arrivalRate: 50
ensure:
p99: 300
maxErrorRate: 1
functional:
phases:
- duration: 1
arrivalCount: 1
plugins:
expect: {}

scenarios:
- flow:
- post:
url: "/login"
json:
username: "testuser"
password: "testpassword"
expect:
- statusCode: 200
- get:
url: "/account"
expect:
- statusCode: 200
- contentType: "json"
- hasProperty: "user"
- delete:
url: "/logout"
expect:
- statusCode: 204

Here, we’re defining two environments called load and functional. Each environment has different configuration settings suitable for the type of test we want to run under each. The rest of the configuration in the test script remains the same since it’ll target the same HTTP service and run the same scenarios. Although we won’t use the artillery-plugin-expect plugin during load testing, we can still keep it in the test script — it simply won’t run the expectations or assertions in that environment.

To run a specific environment, we’ll use the --environment flag in the command line when executing our tests, specifying the environment name we used in the test script:

# Runs load tests.
artillery run --environment load api-test.yml

# Runs functional tests.
artillery run --environment functional api-test.yml

When using the --environment load flag, Artillery will run the load test, sending 50 virtual users per second for 10 minutes and check the aggregate results at the end of the test run. With the --environment functional flag, Artillery will load the artillery-plugin-expect plugin and run through the scenario with a single virtual user, going through the expectations and verifying its assertions for each request only once.

Specifying different environments in an Artillery test script using the config.environments setting helps avoid duplication for your testing. It also allows us to reuse a single test script to handle various kinds of testing from one place.

Debugging expectations during functional testing

The artillery-plugin-expect plugin has a few methods to help you debug your functional tests if they’re not passing. When a functional test fails, the plugin will print out additional details about the expected and actual results for the expectations, like the request parameters, the response headers, and the body.

For example, if our POST /login request fails with a 500 Internal Server Error during a functional test, the plugin will print out the following details:

* POST /login
not ok statusCode 500
expected: 200
got: 500
Request params:
http://lab.artillery.io/login
{
"username": "testuser",
"password": "testpassword"
}
Headers:
content-type : application/json; charset=utf-8
content-length : 64
etag : W/"27-HSrEpTCl9tImXS685QUvpoLGKhI"
date : Mon, 02 Aug 2021 01:01:06 GMT
connection : keep-alive
keep-alive : timeout=5
User variables:
target : http://lab.artillery.io
$environment : functional
$uuid : 52f0cab7-32c8-4c24-a906-4d3703a37763

To get additional details about what the artillery-plugin-expect plugin is doing for a request, we can set the DEBUG=plugin:expect environment variable when executing the functional test:

DEBUG=plugin:expect artillery run --environment functional api-test.yml

Setting this environment variable will print out extra debugging information before each request to help you see the different types of checks that occur with the plugin. For example, the plugin’s debugging information for the GET /account request in our functional test prints out the following output:

plugin:expect Checking expectations +1s
plugin:expect checker: statusCode +0ms
plugin:expect check statusCode +0ms
plugin:expect checker: contentType +0ms
plugin:expect check contentType +0ms
plugin:expect expectation: { contentType: 'json' } +0ms
plugin:expect body: object +1ms
plugin:expect checker: hasProperty +1ms
plugin:expect check hasProperty +0ms

Although the plugin automatically prints out more details about the actual request and response with a failed assertion, using the DEBUG=plugin:expect environment variable is an additional way to help verify that we’re setting the correct expectation types and values in your test script.

Upcoming functional testing improvements for Artillery

Artillery 2.0 — currently a development release — includes the --solo flag that we can use to execute our tests with a single VU, regardless of the phase definition in the test script. This flag is ideal for functional testing since we wouldn't have to define two distinct phases for each environment. Using this flag, we can simplify the test script configuration for each environment:

config:
target: "http://lab.artillery.io"
phases:
- duration: 600
arrivalRate: 50
environments:
load:
ensure:
p99: 300
maxErrorRate: 1
functional:
plugins:
expect: {}

These configuration changes give us some extra flexibility for running our Artillery tests for different purposes:

# Install the development release of Artillery.
npm install -g artillery@dev

# Runs load tests without verifying aggregate results.
artillery run api-test.yml

# Runs load tests and verifies aggregate results after the test run.
artillery run --environment load api-test.yml

# Runs functional tests with a single VU.
artillery run --environment functional --solo api-test.yml

Help meet SLOs and validate your service’s functionality with a single Artillery test script

In this article, we covered a handful of the expectations provided by the plugin, like verifying a request’s status code or part of the response body. The plugin has additional expectations you can use to validate response headers, compare a captured value from a previous response, and more. You can learn more on the plugin’s documentation page.

Thanks to the artillery-plugin-expect plugin, you can use Artillery to easily cover both your performance and functional testing needs for your HTTP services with the same toolkit. You can reuse the same scenarios for both types of testing to help your applications meet SLOs and are functioning according to specifications.

You can try a live example of the artillery-plugin-expect plugin to see it in action using Artillery's SuperREPL tool.

· 20 min read

We've designed Artillery to be hackable and easy to extend. Plugins hook into Artillery’s internal APIs to allow you to include additional functionality like generating different kinds of data for your tests, integrating with third-party systems, or sending metrics to another platform.

Since Artillery uses Node.js, you can use tens of thousands of Node.js packages available on npm to take advantage of existing libraries you can integrate right away. You can find npm packages for almost anything, making it much easier to create and extend Artillery without much hassle.

Let’s see how easy it is to create a new plugin for Artillery. Imagine that we want to send our test run metrics to an AWS Lambda function to store in a DynamoDB database or send an event to an SNS topic. In this article, we’ll go through the process of creating a new plugin that extends Artillery’s functionality to do that. The plugin we’ll create will invoke a Lambda function after a test run using the AWS SDK for JavaScript library.

Setting up an NPM module

Plugins in Artillery are standard npm packages, and they can be either public or private. By default, Artillery will look in the npm package path for the plugin with artillery-plugin- prefixed to the name describing your plugin. In this example, we’ll name our plugin artillery-plugin-lambda since we will invoke Lambda functions in our code.

Creating the npm package is a straightforward process using the npm init command inside of a directory where you want to build your plugin. For our example, we’ll create a directory called artillery-plugin-lambda. When executing the npm init command without any arguments inside this directory, it will ask you a few questions for setting up the project, like the name of the package, a description, and other details. Most of the questions are optional, so you don’t have to answer all of them at this time. If you plan to publish the plugin later, it’s a good practice to add as much detail as possible to help other users find it.

> npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (artillery-plugin-lambda)
version: (1.0.0) 0.0.1
description: Send Artillery metrics to an AWS Lambda function
entry point: (index.js)
test command:
git repository:
keywords:
author: Dennis Martinez
license: (ISC) MIT
About to write to /tmp/artillery-plugin-lambda/package.json:

{
"name": "artillery-plugin-lambda",
"version": "0.0.1",
"description": "Send Artillery metrics to an AWS Lambda function",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "Dennis Martinez",
"license": "MIT"
}

Is this OK? (yes)

npm init will generate a package.json file for you based on your answers. This file serves as a starting point for your plugin. Later in the article, this file will get updated to include our dependencies like the AWS SDK.

Building the skeleton of the plugin

The package for a plugin is expected to export a Plugin class so Artillery can load it properly. The constructor for the class is a function that receives two arguments:

  • script - The full test script definition. The plugin code can modify the test script to extend the current load test definition, like attaching hooks to existing scenarios.
  • events - An EventEmitter which you can use to subscribe to events sent from Artillery during a test run. You can also use the argument to send custom metrics.

In addition to subscribing to Artillery events, plugins can also define a cleanup function with a done callback. This function gets called before Artillery exits upon the completion of a test run. It provides an opportunity to perform any cleanup that your plugin needs before wrapping up. We’ll see how to use this function later in the article.

To start constructing our plugin, let’s create a new file called index.js in our plugin directory. This file will serve as the entry point for our npm package. Inside the file, we can set up the Plugin class and a function to define the constructor of the class:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
// Your plugin code goes here.
}

As mentioned at the beginning of this section, Artillery plugins need to export a Plugin class. Here, we’re using the ArtilleryLambdaPlugin function to load the plugin’s code. The ArtilleryLambdaPlugin function receives the script and events arguments, and from there, you can create the custom functionality for your plugin. To see how these work, we’ll first demonstrate how to use the events argument and later see how to use the script argument to access our test script inside the plugin.

The events argument allows us to subscribe to the following events:

  • phaseStarted - Emitted when a new load phase starts.
  • phaseCompleted - Emitted when a load phase completes.
  • stats - Emitted after each reporting period during a test run (every 10 seconds or at the end of a load phase).
  • done - Emitted when all VUs complete their scenarios, and there are no more load phases to run.

Given that events is a Node.js EventEmitter object, we can set up a listener function that gets called when Artillery emits the event using .on. For example, let’s set up a listener for each of the events emitted by an Artillery test run to see how plugins work by printing a few messages to the console:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
// Prints out "🧑 New load phase started".
events.on('phaseStarted', function() {
console.log('🧑 New load phase started');
});

// Prints out "✅ Load phase complete".
events.on('phaseCompleted', function() {
console.log('✅ Load phase complete');
});

// Prints out "🖥️ Reporting".
events.on('stats', function() {
console.log('🖥️ Reporting');
});

// Prints out "🪄 All VUs finished".
events.on('done', function() {
console.log('🪄 All VUs finished');
});

return this;
}

Let’s verify the plugin is working correctly by loading it into an Artillery test script and putting it into action.

Loading the plugin into an Artillery test script

For purposes of this example, let’s assume we have the following Artillery test script called api-test.yml, which has three load phases and a scenario that hits two endpoints for an HTTP service:

config:
target: "https://test-app.dev/api"
phases:
- duration: 60
arrivalRate: 5
name: "Warm up"
- duration: 120
arrivalRate: 5
rampTo: 50
name: "Ramp up load"
- duration: 600
arrivalRate: 50
name: "Sustained load"

scenarios:
- flow:
- get:
url: "/products"
- post:
url: "/cart"
json:
productId: "123"

An existing test script can load an Artillery plugin by using the config.plugins setting. The setting will tell Artillery to load an npm package using the specified plugin name, prefixed with artillery-plugin- as mentioned at the beginning of this article. Our example plugin is named artillery-plugin-lambda, so we can configure our plugin using lambda:

config:
target: "https://test-app.dev/api"
phases:
- duration: 60
arrivalRate: 5
name: "Warm up"
- duration: 120
arrivalRate: 5
rampTo: 50
name: "Ramp up load"
- duration: 600
arrivalRate: 50
name: "Sustained load"
plugins:
lambda: {}

scenarios:
- flow:
- get:
url: "/products"
- post:
url: "/cart"
json:
productId: "123"

Under the config.plugins setting, we’re loading our plugin using lambda. Since we’re not providing any configuration to use in the plugin at this time, we’re setting an empty mapping for it. Later on, we’ll show how to set up some configuration that the plugin code can access and use internally.

Since plugins are npm packages, Artillery will look for them as any other package. That means if you installed Artillery globally in your system (like using npm install -g artillery@latest, for instance), it would try to find the artillery-plugin-lambda plugin in your global npm packages for this example. If you installed Artillery in your project (without the -g flag), it would search for the plugin in your project’s installed dependencies.

If we attempt to run the example test script with artillery run api-test.yml, the load test will run. However, it will print the following warning message in the console: WARNING: Plugin lambda specified but module artillery-plugin-lambda could not be found (MODULE_NOT_FOUND). The plugin is not installed either locally or globally because we don’t have an npm package yet to set up during development.

We don’t have to create an npm package for our test, though. Artillery allows us to set up the path of our plugin by using the ARTILLERY_PLUGIN_PATH environment variable. It’s a convenient way to tell Artillery where to look for any defined plugins. For this example, let’s assume our plugin is in the /home/artillery/plugins/artillery-plugin-lambda directory. We can load the plugin when running the example test script with the following command:

ARTILLERY_PLUGIN_PATH="/home/artillery/plugins" \
artillery run custom-metrics.yml

Artillery will look inside the /home/artillery/plugins directory for an npm package matching the name of the plugin it’s expecting. In our case, it’s expecting the artillery-plugin-lambda directory. It will find the directory and load the entry point, activating the plugin for the test run. If everything goes as expected, you will see the console messages set up in the plugin during the test run.

Allow users to configure the plugin

We’ve seen how to build the basics of an Artillery plugin, some of the events you can listen to, and how to load the plugin locally during development. Let’s begin setting up the main bits of functionality we want to extend in Artillery, which is invoking an AWS Lambda function when a test run completes.

Most Artillery plugins will have different settings to allow anyone using the plugin to add different configuration settings for their needs. For instance, you can set up access keys to third-party services that the plugin uses or define paths to generate various artifacts in the system running the test. For our example, we want to allow users of our plugin to let others set the name of the function they wish to invoke and the AWS region where the function exists. We’ll call these settings function and region.

For our example, we’ll have an AWS Lambda function called AggregateArtilleryResults located in the ap-northeast-1 region. We want our plugin’s configuration to look like this:

config:
target: "https://test-app.dev/api"
phases:
- duration: 60
arrivalRate: 5
name: "Warm up"
- duration: 120
arrivalRate: 5
rampTo: 50
name: "Ramp up load"
- duration: 600
arrivalRate: 50
name: "Sustained load"
plugins:
lambda:
function: "AggregateArtilleryResults"
region: "ap-northeast-1"

scenarios:
- flow:
- get:
url: "/products"
- post:
url: "/cart"
json:
productId: "123"

We have to tell our plugin to read these values from the test script to use them later when setting up our invocation functionality. The Plugin class constructor receives a script argument, which contains the entirety of the test script as an object. We can access the configuration for our plugin through this argument.

In the index.js file for our plugin, let’s delete the example code we used before and set up our custom functionality. First, we’ll access the configuration from the test script and set it up in a variable for future use:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
const self = this;

self.config = script.config.plugins['lambda'];

return this;
}

In this code, we first set up a reference to this by assigning it to the self variable to manage the original context when handling events, as we’ll show later. Next, we’re reading our test script through the script argument. We want to access our plugin’s configuration, so we’ll access script.config.plugins['lambda'] to get the configuration settings and assign them to self.config. Based on our test script above, the value of self.config will be a JavaScript object containing our settings:

{
function: "AggregateArtilleryResults",
region: "ap-northeast-1"
}

Because we’ll need these settings to configure the AWS SDK and invoke the correct Lambda function, a good practice is to alert any users loading the plugin if either of these settings is missing. A quick way to do this is by throwing an exception with a helpful message telling users that some configuration is missing:

'use strict';

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
const self = this;

self.config = script.config.plugins['lambda'];

if (!self.config.function) {
throw new Error('function is missing');
}

if (!self.config.region) {
throw new Error('region is missing');
}

return this;
}

With the added checks, if either the function or region setting is missing from the plugin configuration under lambda, the plugin will throw the exception and print out the respective message in the console at the beginning of the Artillery test run. Keep in mind that although your plugin won’t load after the error, the Artillery test script will continue running without using any of the plugin’s functionality.

Using the AWS SDK library after a test run

With the plugin configuration at hand, we can now set up the AWS SDK library to begin invoking the configured Lambda function.

As mentioned at the beginning of this article, you can use any Node.js package in your plugin. That means we can use the official AWS SDK for JavaScript created by Amazon. For this example, we’ll use version 3.x of the SDK. This version uses modularized packages to allow developers to load only the functionality they need. Since we only want to access AWS Lambda, we can set up the Lambda client of the SDK as a dependency using npm install:

npm install @aws-sdk/client-lambda

After installing the package, we can use it to set up the Lambda client in our plugin. We’ll need two functions from the SDK. The first function is LambdaClient, which initializes the client. The second function we need is the InvokeCommand to set up how we’ll invoke the function using the client. The SDK separates both client and commands, so we can import both functions and begin setting them up in the plugin:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
const self = this;

self.config = script.config.plugins['lambda'];

if (!self.config.function) {
throw new Error('function is missing');
}

if (!self.config.region) {
throw new Error('region is missing');
}

self.lambdaClient = function() {
return new LambdaClient({ region: self.config.region })
}

self.lambdaCommand = function(report) {
return new InvokeCommand({
FunctionName: self.config.function,
Payload: new TextEncoder().encode(JSON.stringify(report)),
});
}

return this;
}

The LambdaClient function is a class constructor where we can set different settings used for the SDK functionality. It accepts an object with various configuration settings for Lambda. In our case, we only care about selecting the region of the function, which we should already have from our test script configuration. We’ll set up a new LambdaClient instance and assign it to self.lambdaClient so we can use it to invoke the function.

The InvokeCommand function takes care of setting up the command used to invoke the Lambda function from the client instance. For our purposes, we’re configuring the command with two settings:

  • FunctionName - The name of the function to invoke, which we’ll get from the plugin’s configuration in the test script.
  • PayloadThe input to send to the Lambda function.

We’re assigning the InvokeCommand function to self.lambdaCommand, with an argument for our payload. The payload value we want to send to our Lambda function is our Artillery test run metrics when completed. The value of the payload needs to be a Uint8Array type, so we first have to encode it to the proper format using the TextEncoder class.

We have our AWS SDK setup wrapped up. All that’s left is to invoke the Lambda function after the Artillery test run. We can use the events argument to listen to the done event, which will contain the metrics we want to send to the Lambda function. However, we can’t invoke the Lambda function when listening to this event. Event handlers in Node.js are asynchronous, so the Artillery process will exit before giving us a chance to invoke the Lambda function.

To get around this issue, we can define the plugin’s cleanup function to invoke the Lambda function. The cleanup function can invoke the Lambda function asynchronously and wait for the response from AWS before the Artillery process exits. Our plugin will end up looking like this:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
const self = this;

self.config = script.config.plugins['lambda'];

if (!self.config.function) {
throw new Error('function is missing');
}

if (!self.config.region) {
throw new Error('region is missing');
}

self.lambdaClient = function() {
return new LambdaClient({ region: self.config.region })
}

self.lambdaCommand = function(report) {
return new InvokeCommand({
FunctionName: self.config.function,
Payload: new TextEncoder().encode(JSON.stringify(report)),
});
}

events.on('done', function(report) {
self.report = report;
});

return this;
}

ArtilleryLambdaPlugin.prototype.cleanup = async function(done) {
const client = this.lambdaClient();

try {
const result = await client.send(this.lambdaCommand(this.report));
console.log(`Sent Artillery report to Lambda function: ${this.config.function} (${this.config.region})`);
} catch(err) {
console.log("There was an error invoking the Lambda function");
} finally {
done(null);
}
};

First, we’ll listen to the done event to capture the test run metrics and assign it to self.report. Next, we’ll create the cleanup function for the Plugin class. We set up an async function since the AWS SDK makes its requests asynchronously, and we want to wait for the Lambda invocation response. Inside the function, we can access all the class properties we set in the constructor through this, like our LambdaClient and InvokeCommand functions for the AWS SDK, the plugin configuration settings, and the Artillery test run metrics.

With this information, we’ll invoke the function and await the response. If there aren’t any errors from the AWS SDK, we’ll print a message indicating a successful invocation of the function. If the AWS SDK throws any errors, we’ll catch them and let the plugin user know. Regardless of a successful or failed attempt to invoke the Lambda function, we’ll call the done callback function to tell Artillery that our plugin finished doing what it needs to do.

Our plugin is now feature-complete! We can test the plugin by ensuring we set the AWS SDK credentials and a valid Lambda function specified in the plugin’s configuration. With everything set up correctly, we can use the plugin to run any Artillery test and have an AWS Lambda function receive the complete test run metrics.

Add debugging info to help isolate issues

To help aid you or users of the plugin in debugging issues during test runs, it’s a good practice to have a simple way to print out debugging messages to get a glimpse at the inner workings of your plugin. Using console.log can work during development, but you don’t want to litter the console for users of the plugin.

A simple way to add unobtrusive debugging to your plugins is with a small debugging package called debug. This package helps you set up debugging messages that only appear in your output when using the DEBUG environment variable with specific values. Since debug is an npm package, that means you can easily integrate it into your plugin. Let’s set up the package and add a few debugging statements to see how it works.

First, we’ll set up the debug package as a dependency with npm install:

npm install debug

Inside the index.js entrypoint for the plugin, we can import the debug function and set it up to print out debug messages relevant to our plugin:

const debug = require('debug')('plugin:lambda');

The plugin:lambda setting when importing the package is the name we want to specify for our module. It allows us to print the debugging messages with the DEBUG=plugin:lambda environment variable. This code exposes a function — assigned to debug — that we can use. Here’s how our plugin would look with a few debugging messages when invoking the Lambda function so that we can see the AWS SDK responses:

'use strict';

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');
const debug = require('debug')('plugin:lambda');

module.exports.Plugin = ArtilleryLambdaPlugin;

function ArtilleryLambdaPlugin(script, events) {
const self = this;

self.config = script.config.plugins['lambda'];

if (!self.config.function) {
throw new Error('function is missing');
}

if (!self.config.region) {
throw new Error('region is missing');
}

self.lambdaClient = function() {
return new LambdaClient({ region: self.config.region })
}

self.lambdaCommand = function(report) {
return new InvokeCommand({
FunctionName: self.config.function,
Payload: new TextEncoder().encode(JSON.stringify(report)),
});
}

events.on('done', function(report) {
self.report = report;
});

return this;
}

ArtilleryLambdaPlugin.prototype.cleanup = async function(done) {
const client = this.lambdaClient();

try {
const result = await client.send(this.lambdaCommand(this.report));
console.log(`Sent Artillery report to Lambda function: ${this.config.function} (${this.config.region})`);
debug(result);
} catch(err) {
console.log("There was an error invoking the Lambda function");
debug(err);
} finally {
debug("Wrapping up");
done(null);
}
};

The last thing left to do is publish the plugin on npm so you can share your work with the world!

Publish the plugin on npm

Publishing an npm package is a straightforward process — in just three steps, you’ll have your Artillery plugin published in just a few minutes:

  1. First, you’ll need to sign up for a free npm account if you don’t have one already.
  2. Next, set up a registry user in your local system using the npm adduser command. This command will prompt you to log in using your npm account credentials.
  3. Finally, you can publish your package to npm using the npm publish command. This command picks up the information in the package.json file (generated by npm init) and makes your package available publicly.

If everything went well, you can visit npmjs.com and search for the plugin. It should appear available to everyone for immediate downloading. Now anyone can install your plugin and extend Artillery in a snap!

With your plugin available publicly, here are a few tips to help others get the most out of it:

  • When setting up your plugin, add as much information as possible to package.json, such as your name, the plugin’s homepage or code repository, and a brief but helpful description of how the plugin works. When others search for Artillery plugins on npmjs.com, this information will help them discover it.
  • Include a README file in your package with information about your plugin. The page for your plugin on npmjs.com will render the file to give more details on how it works. Adding installation notes, configuration settings, and debugging methods will considerably help others who want to use your plugin.
  • Do your best to follow semantic versioning to help your fellow developers depending on your plugin to keep track of any breaking changes. Make sure to increment your plugin’s version number and set up tags when updating the package.

Summary

This article shows just a small example of how you can extend Artillery’s functionality beyond what’s already included in the toolkit. Many developers have created convenient and practical plugins and made them available on npmjs.com. Here are a few plugins that can boost your load tests in different ways:

If you can’t find a plugin to do what you need, Artillery makes it dead-simple to create a plugin that listens to different events during your load tests and gives you the chance to perform any action you want to add your testing. And thanks to the npm ecosystem, you can find npm packages for just about anything you need. The only limit is your creativity!

· 11 min read

When running Artillery load tests, you'll want to go beyond testing locally or a single instance inside a continuous integration environment. It's essential to have the ability to scale your tests to generate as many requests as you need to ensure your services are stable and reliable. You'll also want to have a more realistic view of real-world usage for your applications by distributing your load test traffic geographically from different areas across the world.

While it's helpful to quickly run an Artillery load test locally, you need to be aware of a few caveats. Running your tests on a single instance won't scale well since the generated load depends on the underlying hardware. Unless you're running your tests from a high-performance system, it isn't easy to spin up the kind of load to match your testing needs. Also, running load tests on limited-performing hardware from only one place limits you from getting a clear picture of how well your services perform when someone accesses them from another region.

You can overcome all of these potential issues with Artillery Pro.

Artillery Pro to the rescue!

Artillery Pro is a layer built on top of the Artillery Core toolkit that allows you to run your load tests on the cloud using your own AWS account. At its core, Artillery Pro leverages the power and flexibility of Amazon AWS with the Amazon Elastic Container Service (ECS) to run your existing Artillery tests. It gives you the ability to scale and parallelize your tests from different geographic regions in a cost-effective manner, all within your AWS infrastructure.

Artillery Pro handles all the complex parts of running realistic and robust load tests while significantly reducing the overhead of time and money. Want to push the limits of your services? Artillery Pro can spin as many workers as you need to send more traffic on-demand. Need to check how your services perform with users on the opposite side of the world? Just run your tests from one of the 13 supported regions. Worried about managing infrastructure or the cost? Artillery Pro can use AWS Fargate, so you don't have to deal with servers while only paying for the resources used during testing.

In this article, you'll see how easy it is to get your existing Artillery load tests set up to use Artillery Pro. We'll show you how to set up your AWS environment and execute your tests from different regions around the world.

Setting up Artillery Pro

To get started with Artillery Pro, you need a license key provided by an active subscription or a trial. You can get started for free by signing up for the Dev plan on the Artillery Pro website. The Dev plan gives you access to essentially all of the Pro features with some limitations, such as running one test at a time, limited to one region and a limited number of workers. If you need to execute simultaneous test runs from different regions with more workers, you'll need a paid plan (Scale or Enterprise). For this article, we'll use Artillery Pro under the Scale plan.

As mentioned earlier, Artillery Pro builds on top of Artillery Core, so you'll need to install both Artillery and Artillery Pro. Provided you have Node.js version 12.0 and above installed, you can install both packages via npm:

npm install -g artillery@latest artillery-pro@latest

Before using Artillery Pro, you'll need to set up your environment with AWS credentials. You can handle this in one of the following ways:

  • Use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to set up the correct access and secret key associated with an IAM user in your AWS account.
  • Install the AWS CLI and configure your credentials using the command-line interface.

Either way will work, but we recommend using the AWS CLI tool to help you attach an AWS user policy to an IAM user to run your Artillery tests.

Once both packages are installed and your AWS credentials are set up, you'll need to set up the Artillery Pro backend in your AWS account. The backend is a fully serverless environment, using services such as DynamoDB, SQS, and S3 to handle the work of running your load tests on your AWS infrastructure without the burden of managing servers.

Artillery Pro handles this setup for you using the artillery deploy command. The command requires two settings: the AWS region where you want to deploy the backend and the license key for your Pro subscription. For instance, if you want to deploy the backend to the us-east-1 region of AWS, use the following command:

artillery deploy artillery deploy \
--region us-east-1 \
--license-key-file artillery-pro-license-key.txt

artillery deploy uses AWS CloudFormation to provision the different AWS services and create the necessary roles and policies required to run your tests on the cloud. You'll have the deployment progress shown in your terminal when running the command so you can view the created resources. This command will detect if you previously installed the Artillery Pro backend in your region, so you can safely run it more than once without accidentally deleting an existing setup.

After deploying the backend, you'll get prompted to attach a newly-created policy called artilleryio-cli-user to an existing IAM user in your AWS account. The policy gives an IAM user access to the resources created by Artillery Pro. If you installed the AWS CLI tool in your system, you can run the command shown in your terminal, changing the IAM user name to the one you want to use in your AWS account. You can also manually attach the policy from your browser by logging in to the AWS console and going to the IAM section.

With this initial setup completed, you're now ready to move on to the fun part - running your tests on AWS.

Running your tests from AWS

If you've run an Artillery load test in your local system, running your tests on AWS using Artillery Pro will look familiar. However, instead of creating virtual users and generating traffic from your computer, Artillery Pro will offload these tasks to an AWS ECS cluster with a defined number of workers using either EC2 instances or Fargate. Having your tests run on AWS allows you to run your tests from different regions in the world and scale your tests quickly and easily.

The first step for running Artillery tests on AWS is to create an ECS cluster in your AWS account. Since Artillery Pro supports both classic ECS clusters (with managed EC2 instances) and Fargate, you can create either type. To determine which type of cluster is best for your organization, read "Should you use ECS or Fargate?" in the Artillery documentation.

You can create the cluster from any of the AWS regions supported by Artillery Pro, but keep in mind that you can only run your tests in a single region with the Dev plan. You also need to ensure that the IAM user with the attached artilleryio-cli-user policy has the proper AWS permissions to use the ECS cluster.

Once your cluster is set up, you can now run your tests on AWS using Artillery Pro. For this article, we'll assume that you created an ECS cluster called artillery-pro-cluster in the us-east-1 region and have a test script called my-load-test.yml in your local directory.

To run the test script on AWS, you can use artillery run-test, specifying the cluster (using the --cluster flag), the AWS region (using the --region flag), and the number of workers (using the --count flag) as follows:

artillery run-test \
--cluster artillery-pro-cluster \
--region us-east-1 \
--count 5 \
my-load-test.yaml

By default, artillery run-test executes your tests in a classic ECS cluster using EC2 instances. If you want to use Fargate instead, you need to specify the cluster type with the --launch-type flag:

artillery run-test \
--cluster artillery-pro-cluster \
--region us-east-1 \
--launch-type ecs:fargate \
--count 5 \
my-load-test.yaml

These commands work similarly to executing your load tests with artillery run. However, artillery run-test will use the deployed backend to set up your test run using the defined cluster. Your test script and any dependencies used in the script, such as payload files or custom JavaScript code, get packaged up and sent to the cluster. Your tests will then run the same way as they do locally.

Adding the --count flag lets you leverage the power of AWS to scale up your load tests in an instance. By spinning up additional workers in your cluster, you can generate the load you need without worrying about server management or bandwidth capacity. And since you only pay for what you use on AWS, scaling up your tests with more workers won't cost you an arm and a leg.

Once the test run completes, Artillery Pro will clean up the workers and report your test results in the same way as if you executed your tests on your computer. However, you'll now have better insight into how your services perform from another section of the globe, which you can't observe when running your tests locally. This functionality is a must for today's globally accessible world.

Scaling your Artillery tests up and abroad

The example above shows how simple it is to run your tests from a different region. Seeing how your services perform from another country can give you a glimpse at how your application works for everyone.

With the Scale plan, you can use Artillery Pro to go even further with your load tests by generating traffic from multiple points of the globe simultaneously. Instead of checking how your service behaves with traffic in one place, you can quickly run up to ten tests from different regions concurrently with just a simple command-line change.

For instance, you may have your application hosted in a data center in Europe, and you want to see how your services work with your customers from the United States and Asia. Assuming you have ECS clusters set up in both the us-west-2 (Oregon) and ap-northeast-1 (Tokyo) regions, all you need to do is run your tests on those regions and let Artillery Pro handle the rest.

In one shell, run your test script in the us-west-2 region:

artillery run-test \
--cluster artillery-pro-cluster \
--region us-west-2 \
--launch-type ecs:fargate \
--count 5 \
my-load-test.yaml

Then, in another shell, execute the test in the ap-northeast-1 region:

artillery run-test \
--cluster artillery-pro-cluster \
--region ap-northeast-1 \
--launch-type ecs:fargate \
--count 5 \
my-load-test.yaml

With only two commands, you now have geographically distributed load testing from across the globe on AWS. It can't get any simpler than that. Running distributed tests like this is especially useful when done from a continuous integration service, ensuring that your services work well for everyone in the world after every change to your application.

One thing to keep in mind is that when you run your tests in a distributed fashion, each test runs independently of the other. While it's helpful to see the results for each region, you may want to aggregate and keep track of all the results. You can use the [artillery-publish-metrics plugin](https://artillery.io/docs/guides/plugins/plugin-publish-metrics.html) to send the results for each test run to an external monitoring and observability system like Datadog or Lightstep.

For even more resilient load testing, you can increase the number of workers for running your tests. By default, Artillery Pro runs your tests on a single worker on your cluster. If you need to increase this capacity, use the --count flag to increase the number of workers you want to provision for your test. For example, the following command executes a test using 10 cluster workers:

artillery run-test \
--cluster artillery-pro-cluster \
--region us-east-1 \
--count 10 \
my-load-test.yaml

It doesn't get easier to run load tests across the globe than with Artillery Pro

These days, making your applications and web services speedy and resilient isn't a nice-to-have feature - it's vital for any business that caters to users across the world. Your systems must work well for everyone, not just a subset of people nearest to your servers. Without knowing how your application behaves in other regions of the globe, you may be losing customers without realizing it.

Artillery Pro will help you ensure performance and reliability for your services by taking advantage of your AWS infrastructure to run globally distributed load tests. With just a few commands on top of the Artillery CLI, you can execute your existing Artillery test scripts on different ECS clusters located in distinct regions. You can also scale up your testing traffic as needed by provisioning additional workers.

The best part about it is that your team won't have to spend any time dealing with testing infrastructure or worrying about costs. Everything you need for testing will get set up on your own cloud environment. With Artillery Pro, you won't find an easier way to run load tests at scale worldwide.

· 24 min read

As web applications become more complex, more organizations have come to rely on splitting their applications into smaller, discrete services to help maintain them in the long run. Instead of building a huge, monolithic codebase that does everything, developers are beginning to use APIs and microservices to isolate specific functionality. This service-oriented architecture helps larger teams to work independently and make their systems easier to develop, test and deploy.

Since these separate applications now rely on each other for the entire system to function well, performance and reliability are crucial requirements for this architecture. If one application stops working for any reason, it'll likely affect other services that rely on it. Service-oriented architectures need to be able to withstand performance degradation and inevitable failures. Speed and resiliency are no longer optional.

To help ensure that your services can handle sudden changes in usage, such as a spike in traffic or an unforeseen event that rapidly increases the system's workload, you need to test for it ahead of time. As a result, performance and load testing are becoming increasingly critical to guarantee that each part of your system can deal with temporary or sustained traffic without breaking down.

Load testing will help you validate that your system's environment works as intended. For instance, you want to know that your logging and alerts get triggered if a service fails or if your auto-scaling system spins up additional servers due to the increased workload. You'll also want to keep track of performance during the software development lifecycle. If you deploy a slow application, you'll inevitably turn away potential customers.

While performing these tests occasionally to make sure things are working well, it's not enough to run them sporadically. Running load tests continuously becomes essential not only to keep your systems up and running but to ensure that you're providing the level of performance - often defined in service-level objectives or SLOs - expected by your team and customers.

In this article, we'll show you how to load-test an HTTP API service using Artillery. First, we'll set up a test script that will verify that the service can handle hundreds of requests at a time. Then, we'll add a few conditions to cover the SLOs defined for the service to let us know when our pre-defined thresholds aren't met. Finally, we'll integrate Artillery with CircleCI to ensure you always keep tabs on how your service performs over time.

What we'll test with Artillery

For this article, we'll use an example HTTP service of a JSON API running the backend of an e-commerce website. The API has a few endpoints for searching products by keyword, fetching product information, and adding items to a cart for purchase. In our fictional e-commerce site, let's say we've observed that the typical customer performs the following actions when they want to purchase a product:

  • They search for a product using a few keywords.
  • After receiving the search results, they click on a product - usually the first result.
  • They read the product information for a few seconds.
  • They decide this is the product they want and add the item to their cart.

This flow is vital for the company's business and needs to work at all times. If a customer can't find a product quickly or add it to their cart, they'll probably search for the product elsewhere and never return to our store. We want to test this flow to ensure the service will work properly at all times, especially during heavy-traffic periods such as Black Friday and Cyber Monday. Let's do that with Artillery.

Setting up an Artillery test script

Artillery test scripts are YAML files containing two main sections:

  • config: Allows you to set the target for performing the test, the load phases for defining how many virtual users you want to send to the target, and additional configuration settings.
  • scenarios: Allows you to describe which action you want a virtual user (VU) to take on your target service.

We'll create our test script for the e-commerce service inside the same code repository, under the tests/performance sub-directory. Inside this directory, we'll add a new file called search-and-add.yml. The name of the test script or sub-directory doesn't matter - you can place your tests anywhere.

Defining your target and load phases

First, let's write the configuration for our test script. The primary settings we need to set for Artillery in this section are the target URL where our service is located for testing and the load phases during the test. Deciding on the type of load you want to send for your service depends on your particular situation, such as the typical usage for your web application and your system architecture.

For this example, we've decided on the following load phases for our Artillery tests:

  • First, we'll warm up the HTTP service by sending a small amount of traffic. In this phase, Artillery will send five VUs to the service every second for one minute.
  • Next, we want to increase the load to the service gradually. After the warm-up phase, Artillery will go from 5 VUs to 50 VUs per second for two minutes, slowly sending more VUs every second.
  • Finally, we want to see how our service deals with a sustained load. We'll hammer the service with 50 new VUs every second for 10 minutes.

With our load phases defined, we can write it in our Artillery test script:

config:
target: "https://ecommerce.test-app.dev"
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
- duration: 600
arrivalRate: 50
name: Sustained load

Injecting data from an external file

Next, we want to set up some additional configuration to help with our test scenarios. As mentioned earlier, one of the things we want to test is when a user searches for a product. A good practice when running load tests is to check how your system behaves when accessing dynamic content by using different data during testing.

Artillery allows you to define a list of variables with different values, which you can then use in your test scenarios. You can do this in one of two ways - you can define the variables and data inside the test script using the config.variables setting, or you can load an external CSV file using the config.payload setting. For this example, we'll load data from a CSV file to use later when we define our test scenarios.

We'll place a file in the tests/performance sub-directory called keywords.csv in our code repository. This file will contain a few different keywords that customers typically use in the e-commerce website:

computer
video game
vacuum cleaner
toys
hair dryer

To load this data in the test script, we'll set up the config.payload setting in our test script:

config:
target: "https://ecommerce.test-app.dev"
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
- duration: 600
arrivalRate: 50
name: Sustained load
payload:
path: "keywords.csv"
fields:
- "keyword"

config.payload uses the path setting to point to the file containing the data, and the fields setting to define the variable name for use in our scenarios. The example CSV file only contains one value, so we only need to define one variable, called keyword. You'll see this variable in use later.

Configuring test scenario and steps

With our basic configuration prepared for the load test, it's now time to define the steps you want each VU to go through during the test.

In an Artillery test script, you can define one or more scenarios for each VU using the scenarios.flow setting. Each scenario contains one or more actions that the VU will go through. Since we're testing an HTTP service, you can define typical HTTP requests such as GET and POST like you would under regular use.

For this article, we'll define a single scenario which will go through the typical customer flow described earlier. Each VU that Artillery sends to the HTTP service will search using one of the keywords from the configured CSV payload, fetch the details, pause for a few seconds, and add it to their cart. Let's see how this is defined in an Artillery test script:

scenarios:
- name: "Search and add to cart"
flow:
- post:
url: "/search"
json:
kw: "{{ keyword }}"
capture:
- json: "$.results[0].id"
as: "productId"
- get:
url: "/products/{{ productId }}/details"
- think: 5
- post:
url: "/cart"
json:
productId: "{{ productId }}"

The scenario defined above is named "Search and add to cart", and it contains four steps under the flow setting.

The first step in the flow makes a POST request to our service's /search endpoint. The API endpoint expects a JSON object in the request body, with a key called kw containing the value of a keyword. We want to use one of the values from the CSV payload file, which we can access using " {{ keyword }}" . By default, Artillery will choose a random value from the CSV file. For instance, one virtual user will make a request with the body of {"kw": "computer"}, and the next virtual user makes a request with the body of {"kw": "video game"}.

The HTTP service returns a JSON response containing an array of different products found on the backend on a successful request. We want to grab the product ID of the first value returned to use in future steps in our scenario. Artillery allows you to do this by using the capture setting. This setting will enable us to parse the JSON response for the id property of the first object in the returned array (using JSONPath) and store the value into a variable called productId.

The second step of the flow uses the productId variable to make a GET request to the /products/:productId/details endpoint. This request uses the variable that we captured earlier and interpolates it as part of the URL to have each VU make different requests according to the keyword used in the first step.

Next, we want to simulate a customer spending time on the website after fetching the details of a product. In Artillery, we can easily accomplish this with the think setting. This setting pauses the test execution for the VU for a specified number of seconds before moving to the next step. Here, we're pausing the test for five seconds before we go to the final step.

Finally, the last step of this scenario is the action of adding the product to the cart. The HTTP service handles this action through the POST /cart endpoint. The endpoint also expects a JSON object as part of the request. The request body requires a productId key containing the ID of the product to add to the customer's cart. We already have this value stored in the productId variable, so we just need to include it in the JSON request.

With this, we've covered a full end-to-end flow for our load test using Artillery. Here's the complete test script with the configuration and scenarios defined:

config:
target: "https://ecommerce.test-app.dev"
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
- duration: 600
arrivalRate: 50
name: Sustained load
payload:
path: "keywords.csv"
fields:
- "keyword"

scenarios:
- name: "Search and add to cart"
flow:
- post:
url: "/search"
json:
kw: "{{ keyword }}"
capture:
- json: "$.results[0].id"
as: "productId"
- get:
url: "/products/{{ productId }}/details"
- think: 5
- post:
url: "/cart"
json:
productId: "{{ productId }}"

Running a load test

Let's make sure that the test script works as intended. If you have installed Artillery in your local system, you can run the test with the following command:

artillery run tests/performance/search-and-add.yml

This command will begin sending virtual users to your HTTP service, starting with the warm-up phase. Every 10 seconds, Artillery will print a report on the console for the number of scenarios executed during that period. At the end of the performance test, you'll receive a complete summary, including the number of scenarios launched and completed, the number of requests completed, response times, status codes, and any errors that may have occurred.

After your load test has finished, you'll receive a summary of the complete test defined in this section. Here's an example of what it will look like:

All virtual users finished
Summary report @ 08:28:52(+0000) 2021-06-05
Scenarios launched: 33689
Scenarios completed: 33689
Requests completed: 101067
Mean response/sec: 128.57
Response time (msec):
min: 9
max: 206
median: 13
p95: 99
p99: 111
Scenario counts:
Search and add to cart: 33689 (100%)
Codes:
200: 67378
201: 33689

Meeting SLOs in your load test

As part of our testing, we also want to check that our HTTP service maintains a certain level of performance and reliability while under load. We can place some assertions inside of our test script with Artillery to verify performance-related issues during load testing. These checks can help us meet some of our service-level objectives and detect potential problems quickly.

After discussing with the team and observing how the service behaves in normal circumstances, we decided upon validating the following in our test:

  • 99% of the requests made in the scenario (99th percentile) should have an aggregate response time of 200 milliseconds or less.
  • Less than 0.5% of all requests are allowed to fail.

We can define these SLOs as part of the test using the ensure setting as part of the test configuration as part of the config section:

config:
target: "https://ecommerce.test-app.dev"
phases:
- duration: 60
arrivalRate: 5
name: Warm up
- duration: 120
arrivalRate: 5
rampTo: 50
name: Ramp up load
- duration: 600
arrivalRate: 50
name: Sustained load
payload:
path: "keywords.csv"
fields:
- "keyword"
ensure:
p99: 200
maxErrorRate: 0.5

scenarios:
- name: "Search and add to cart"
flow:
- post:
url: "/search"
json:
kw: "{{ keyword }}"
capture:
- json: "$.results[0].id"
as: "productId"
- get:
url: "/products/{{ productId }}/details"
- think: 5
- post:
url: "/cart"
json:
productId: "{{ productId }}"

With this setting, the tests run the same way as before. However, Artillery will check if the aggregate p99 latency and the number of errors at the end of the test run. If the p99 latency is over 200ms or over 0.5% of requests failed to complete, Artillery with exit with a non-zero exit code. In a continuous integration pipeline, it will cause the test run to fail.

Setting up CircleCI

Now that we have a working load test for our HTTP service, the next step is to set up a continuous integration service to run these scenarios frequently. We can find plenty of excellent choices for CI/CD services out there. For this article, we'll use CircleCI as the place to continuously run our Artillery tests.

CircleCI has a generous free plan to help you get started. To sign up for their service and begin using their free tier, you'll need a GitHub or BitBucket account. CircleCI currently only supports these two services. Once you authenticate through GitHub or BitBucket and grant the necessary permissions to CircleCI, we can see our existing code repositories.

In this example, we have a code repository on GitHub called ecommerce-backend, containing the codebase for the HTTP service and the Artillery test script we created in this article. Click on the "Set Up Project" button for this project, and CircleCI will help you choose a programming language to get a pre-populated configuration file. We'll create the CircleCI configuration file from scratch, so we can skip this step.

CircleCI - Set up project

CircleCI works by connecting to your code repository and looking for a configuration file called .circleci/config.yml. This file contains all the steps required to build an environment and execute your workflow. After setting up the project on CircleCI, it will check the configuration file after each commit. Let's start by building a basic CircleCI configuration that will load an environment with Artillery ready to run our tests.

The contents of the .circleci/config.yml file will be the following:

version: 2.1

jobs:
build:
docker:
- image: artilleryio/artillery:1.7.2
steps:
- checkout
- run:
name: Execute load tests
command: /home/node/artillery/bin/artillery run tests/performance/search-and-add.yml

All CircleCI configuration files require a version setting to tell CircleCI which version of their platform to use for our builds. The latest available version at this time is version 2.1, so we'll use it for this example.

Next, we define the jobs setting to define the environment we want for our builds and the steps to perform during each build. We can define one or more jobs for each build and set it up as a workflow, but we'll create a single job called build for this article. Inside our build job, we'll do two things - set up the environment where we'll run our load test, and define which steps to take to perform the test.

CircleCI provides multiple options for starting an environment to run any workflow with their service, like spinning up a Windows or macOS environment. One of the easiest and most common ways to create an environment for our builds on CircleCI is to use Docker. Artillery provides pre-defined images with all we need to run our test script. Using the docker setting in the configuration file, we can define the image name using image, and CircleCI will pull the image and run it as a container.

Next, we use the steps setting to define what actions to take inside the Docker container. This section allows us to set a series of steps. Here, we only need to do two steps. First, we'll have to place the code from our repository inside the container, so Artillery has access to the test script. CircleCI has a special step called checkout, which does this for us.

With the test script inside the Docker container, we can now run the load test since Artillery is already pre-installed. The run step allows us to execute any command inside the container. Currently, Artillery is not in the environment's PATH (which will be corrected soon), so we'll have to point to the location of the binary (/home/node/artillery/bin/artillery). The command will run inside of the directory containing our codebase, so we can point to the test script directly there.

Now that the CircleCI configuration is complete, we can commit it to the code repository and push it to GitHub. CircleCI will detect the commit, find the configuration file, and trigger a new build with the defined steps. If everything is set up correctly, CircleCI will pull the Docker image, start it up, place the code inside, and execute the Artillery load test. After a few minutes, Artillery will finalize the test, and if the defined SLOs pass, we'll have our first passing build.

CircleCI - Successful Build

Generate a report

While it's nice to see the aggregate report of our load test, it doesn't give us too much of an indicator at first glance on how our HTTP backend handled the different phases. We can scroll through the output provided by Artillery, but it's tedious to go through the entire list. Instead, we can use Artillery's support to generate an HTML report of the complete test, which gives us excellent oversight on the totality of the test run.

To generate an HTML report of an Artillery test run, we need to perform two steps. First, we need to create a JSON report of the test run. We can do this by modifying the step that runs our tests and include the --output flag:

- run:
name: Execute load tests
command: /home/node/artillery/bin/artillery run --output reports/report.json tests/performance/search-and-add.yml

The second step in the process is to take the generated JSON report and convert it to an HTML report using the artillery report command. This command will take a JSON report created by Artillery and create a nicely formatted HTML report. We can add this as an additional step in our CircleCI configuration file:

- run:
name: Generate HTML report
command: /home/node/artillery/bin/artillery report --output reports/report.html reports/report.json

To view these reports after your test run completes on CircleCI, you need to store them as artifacts. CircleCI has another special step called store_artifacts that allows you to upload any artifacts created during a build and store them for up to 30 days. You can then access any uploaded files through the CircleCI build results page. We'll add this step after generating the HTML report:

- store_artifacts:
path: reports/

If you notice, we're generating both the JSON and HTML reports in the reports directory, which doesn't exist in our code repository. Without this directory, Artillery will raise an error, and our tests will fail, so we'll have to create the directory before running our test and generating the reports. Our updated CircleCI configuration file looks like this:

version: 2.1

jobs:
build:
docker:
- image: artilleryio/artillery:1.7.2
steps:
- checkout

- run:
name: Make reports directory
command: mkdir reports

- run:
name: Execute load tests
command: /home/node/artillery/bin/artillery run --output reports/report.json tests/performance/search-and-add.yml

- run:
name: Generate HTML report
command: /home/node/artillery/bin/artillery report --output reports/report.html reports/report.json

- store_artifacts:
path: reports/

When we commit the file to GitHub, our tests will run again. After the tests complete, you will see both the JSON and HTML reports under the "Artifacts" tab of the CircleCI build.

CircleCI - Artifacts
CircleCI - HTML Report

Run a nightly build

We've been triggering a build on CircleCI after each commit to the code repository, but this isn't ideal. The load test takes about 13 minutes to complete, which can cause delays to your entire team's development workflow. If you have multiple team members committing code all day, you'll quickly find your CircleCI builds backed up, especially on the free tier, which only runs one build at a time.

Instead of running our tests after every commit, let's say we only want to run them once a day during off-peak hours. This way, we can keep track of our service's performance every day while preventing slow build times for the development team while they're working.

CircleCI uses workflows to give you the choice to run jobs after committing to the codebase or by scheduling them at specific times. A workflow is a way for CircleCI to orchestrate defined jobs in different ways. For instance, you may have three jobs you want to run sequentially, or you only want to trigger a build if someone makes a code commit to a specific branch in your repository.

Our existing CircleCI configuration doesn't have a workflow defined, so CircleCI triggers a build every time we commit new code to the repository. Let's change that to only run our load test every day at 12:00 AM UTC. To schedule a recurring job on CircleCI, we have to define a workflows section in our configuration outside of the jobs section. Our example will have the following workflow definition:

workflows:
version: 2
nightly:
jobs:
- build
triggers:
- schedule:
cron: "0 0 * * *"
filters:
branches:
only:
- main

Like the CircleCI configuration itself, workflows also require a version. At this time, CircleCI only has version 2 available. After that, we define a workflow by giving it a name. In this example, our workflow is called nightly.

Under our job name, we need to define which jobs we want to run in this workflow using the jobs setting. We can define one or more jobs, but we currently only have one job, build, so we'll include it here.

The other setting, triggers, is where we'll specify how we want to execute this workflow. The only option available for the triggers setting is the schedule key, which tells CircleCI we want to run this on a recurring schedule. The schedule can be set using the cron key, which is required and uses standard POSIX crontab syntax. The configuration above sets the schedule to midnight UTC every single day.

The schedule key also requires the filters key, a map that limits which branches you want to run the workflow on. We're only interested in running the load test from the primary main branch, so we'll define it using the branches.only setting under filters. If we don't specify a filter, CircleCI will run the job on all branches in your repository, so it's important to control where we want to trigger any jobs.

The finalized CircleCI configuration will now look like this:

version: 2.1

jobs:
build:
docker:
- image: artilleryio/artillery:1.7.2
steps:
- checkout

- run:
name: Make reports directory
command: mkdir reports

- run:
name: Execute load tests
command: /home/node/artillery/bin/artillery run --output reports/report.json tests/performance/search-and-add.yml

- run:
name: Generate HTML report
command: /home/node/artillery/bin/artillery report --output reports/report.html reports/report.json

- store_artifacts:
path: reports/

workflows:
version: 2
nightly:
jobs:
- build
triggers:
- schedule:
cron: "0 0 * * *"
filters:
branches:
only:
- main

When committing this file, it won't trigger a build as it did previously. The load test will now only execute at around 12:00 AM UTC, giving you a chance to check your service every day without blocking the rest of your team.

Caveats of load testing in CI

While the tests in this article validate that the service performs as intended when under load, it comes with a few potential pitfalls you need to be aware of.

Generally speaking, continuous integration environments are not the ideal place for executing load-generating tests. These systems often aren't under our control, so we won't know how the build environment can handle testing. The environments provided by these services usually are lower-powered virtualized servers, making it challenging to produce realistic load in a performant way. It also limits our ability to scale, preventing us from generating additional load or running tests in parallel. Another issue is that the CI service runs in a single location, which doesn't guarantee your service will perform well across the globe.

Artillery Pro solves these issues. Instead of directly executing the load tests from CI, you can use your continuous integration service to trigger the load tests from an environment outside of the continuous integration service. It will give you more control over the build environment and the load you want to send to your service. Artillery Pro also resolves scalability issues by allowing you to spin up additional workers distributed in different regions of the world to give you a better sense of how your services perform, especially in production.

Where to go from here?

In this article, we've set up a continuous load test that will keep our HTTP service in check and send automatic alerts if the service doesn't meet our SLOs. This setup helps with maintaining reliability and performance at the levels we expect at all times. Still, there's more we can do with our load testing besides daily testing.

We can use our continuous integration environment to run other tests, like unit and end-to-end tests, alongside our Artillery load test when the primary branch in the repo receives an update. With a robust testing pipeline in place, we can increase our confidence in our application, allowing us to set up continuous delivery and automatically deploy new changes to our customers.

We can also improve upon our reporting and observability. While Artillery has basic checks for SLOs, publishing load testing metrics to an external monitoring/observability system like Datadog or Lightstep will help us better define and evaluate our SLOs. The artillery-plugin-publish-metrics plugin integrates easily with our existing test scripts to automatically send Artillery test results to take our monitoring to the next level.

It's essential not only to verify that your application's services work as expected but that they perform as expected regardless of the workload. Using a robust load testing tool like Artillery alongside a continuous integration service such as CircleCI will help you regularly check that your team meets SLOs, maintain fast and reliable production systems, and keep your team and customers happy.

· 14 min read

When performance matters...

Imagine the following scenario: you're part of a sizeable project using a microservice-based architecture, with multiple teams (who aren't necessarily in the same geographical location) working on a number of microservices.

All teams may be deploying to a single target if you're fortunate (or if the project is still in relatively early stages), such as Kubernetes or ECS, but it's likely that there's at least some AWS Lambda/Serverless in the mix, as well as a more traditional EC2-based deployment somewhere. There's an API gateway which exposes public APIs that are backed by some of the microservices, and a variety of purely internal microservices that provide APIs which are only ever used by other microservices.

Most of your microservices probably expose HTTP-based APIs (whether RESTful or GraphQL), but it's very likely that there's a variety of other interfaces and transports used in the system: there's probably a number of services which use SQS, Kinesis or another queue-like mechanism to receive work or output results, there may be gRPC somewhere in the mix, or perhaps a real-time component using WebSockets or Socket.io.

A typical microservice architecture diagram

An imaginary polyglot microservice-based system.
Note: any resemblance to a real-world architecture is accidental

More importantly for the purposes of this article: performance & reliability are a key requirement for this system. Perhaps you're in e-commerce, where every 100ms you can shave off response time affects the bottom line. Or it's an IoT system that will need to be able to receive and process hundreds of thousands of messages every minute from a fleet of real-world sensors. Maybe it's some kind of a ticketing or event system, with regular spikes in traffic that must be dealt with or your company's call centers get DDoS'd by irate customers. Whatever it is, performance and reliability are important to this project, and you know you will want to invest time and effort into it.

... how do we make sure our performance goals are met?

This is the situation development teams find themselves in all the time (and one we are very familiar with and have helped with many times in the course of our consulting work). As a team lead, developer, SDET or QA engineer who's tasked with implementing a performance testing pipeline there's likely to be many questions that you feel need to be answered, e.g.:

  1. Where does performance testing fit into the development, testing, and delivery process?
  2. How do we run performance tests in our CI/CD pipeline?
  3. How do we define performance goals & SLOs for our services? Can we have those checked automatically in CI?
  4. What types of performance tests are there? Which ones do we need for our services?
  5. How do we organize our test suites? What are the best practices around structuring those tests, avoiding duplicating effort and wasting time, and sharing common solutions across teams?
  6. How do we pick which services to test? Do we need to test all of our microservices? What about testing different environments and configurations easily?
  7. How do we encourage collaboration on performance testing and involve everyone: developers, testers, SREs and product managers? (performance is everyone's responsibility!)
  8. How do we make performance testing results available to all interested parties? (developers, testers, product managers, SREs)
  9. How do we make sure that the costs of running load tests frequently at scale do not become astronomical?
  10. What's the best way to get started with all of this?

That's a lot of questions, but fortunately we have the answers! The goal of this series of articles is to give you a practical methodology to follow with your team, and make it so clear that you could read this guide and start tomorrow.

By the end of this series, you will end up with:

  • A scaffold for building out a comprehensive test suite for all of your APIs -- internal and external, regardless of where they may be deployed, what protocol they use to communicate, or which techology stack they're built with. Whether it's a serverless Node.js service, a Go microservice deployed on Kubernetes that consumes Kinesis, or a SQS-based worker service.
  • An understanding of how to integrate performance tests into your team's CI/CD pipeline.
  • The ability to run both load tests and acceptance/smoke/functional tests using the same test definitions, e.g. to first verify a new deployment by running a few happy path scenarios, and then blasting it with lots of traffic to test how it responds under load.
  • Making the metrics generated by your tests (such as latency and RPS) available in an external monitoring system such as Datadog.

... and more!

Let's get started!

No surprises here, we'll be using Artillery

This article presumes that you're familiar with Artillery. If you haven't used it before, I'd suggest having a look through the docs, especially the Core Concepts guide, and an overview of what Artillery tests look like.

Where do we get started?

While we could start with any service, if we want to make progress quickly (e.g. to have something interesting to present to the rest of the team) and demonstrate the value of what we are doing, it's best if we start with a service which satisfies the following criteria:

  • It has a small API surface. Ideally only a couple of API endpoints.
  • It would obviously benefit from load tests, so:
    • It has experienced issues under load in the past or you have good reasons to suspect that it may
    • It's on the critical path for some important functionality of the system, and if it experiences hiccups, parts of the system may go down, become unavailable and affect end users
    • It has high performance requirements (which may be expressed as TPS, desirable p99 response time, number of concurrent users, or a combination of several objectives)

An example of such a service might be an authentication service which issues JWTs allowing users to access other API endpoints.

How do we organize our test code?

Now that we've picked a service we want to test, the next question is how to organize our code.

Monorepo!

Keeping extensibility of our test suite in mind (we will be adding more scenarios, hopefully for many other services and APIs), and with cross-team and cross-functional collaboration also firmly in our thoughts, we usually recommend using a monorepo approach, that is a single source code repository that will contain test definitions for all (or many) services and APIs:

  • A single repo makes the test suite easier to extend and maintain. There's only one repo to clone. All of the test code is there, and existing tests for other services can serve as good examples for someone just starting to write tests for their API.
  • Access control is simple. You only need to share one repo with other teams rather than 10 or 20 (which would be the case if performance tests are kept alongside the microservices' codebase).
  • A monorepo encourages and simplifies code reuse in tests, and helps keep things more DRY.
  • A monorepo is also easier to work with when you're building CI/CD jobs and pipelines. The CI server needs to be able to clone only one known repo to run an automated test against any existing microservice.

(Aside: here's a nice read over on Dan Luu's blog on reasons monorepos can work well for some projects.)

Setting up our project structure

Acme Corp Logo

Let's pretend we're working for Acme Corp for the moment

We're going set up the following directory structure for our tests:

acme-corp-api-tests/
- services/
- README.md
- package.json
  • acme-corp-api-tests -- This is the name of the repo that will hold our test code. Naming is one of the two hardest things in computing, and org-or-project-name followed by api-tests tends to work well as a name. (Not to give too much away, but in the follow up to this article we'll cover how to use the same test definitions to run either load tests or acceptance/smoke/functional tests, so a more generic "api-tests" makes more sense than something specific like "load-tests").
  • services -- this is where our test definitions will reside
  • README.md -- the README.md will cover the basics of how to use and extend the test suite
  • package.json -- this will contain dependency information for our tests and include Artillery / Artillery Pro, any third party Artillery plugins being used, or npm modules that are used in custom JS code in our test definitions.

(You can go ahead and clone the template acme-corp-api-tests repo we have set up and use that as your base. )

Service-specific code

Next, we'll create a directory under services to hold the test code for the microservice we are starting with, e.g. services/auth-service. We will follow a convention for each of the services that we write tests for as well, as follows:

acme-corp-api-tests/
- services/
- auth-service/
- scenarios/
- config.yaml
- functions.js
- overrides.slo-response-time.json
  • scenarios will contain a number of YAML files, each defining one virtual user scenario
  • config.yaml will contain service-specific information, such as a list of environments that the service is deployed to (e.g. dev, test, uat, staging etc), configuration of any service-specific HTTP headers, any variable data etc.
  • We will have a number of overrides.something.json files, which we will use in conjunction with the --overrides flag in artillery run command to be able to set the amount of generated load dynamically. For example we may have two overrides files, one setting the level of load to baseline that our service is expected to handle, and another one which emulates peak traffic spikes that we need to be able to deal with.
  • functions.js will hold any custom JS code that we may need to write for our scenarios.

Next, we'll look at each one of these in more detail.

Service scenarios in scenarios/

The number and nature of these will depend on the service being tested. A simple scenario for an imaginary auth service that tests the happy path may look like this:

{% raw %}

scenarios:
- name: Authenticate with valid credentials
flow:
- post:
url: "/auth"
json:
username: "{{ username }}"
password: "{{ password }}"
expect:
- statusCode: 200
- contentType: json

{% endraw %}

One thing to note here is the expect attributes we are setting (even though we're getting slightly ahead of ourselves here, it's worth doing this early). Those are for the expectations plugin for Artillery, which we will employ to reuse our scenario definitions for acceptance/smoke/functional testing. Until the plugin is enabled, those annotations don't have any effect -- we'll look at how we can use the plugin in a follow up article.

When writing tests for a service, you'd typically start with covering the happy path or the endpoints that are more likely to be performance-sensitive, and then increase coverage over time.

Service-specific configuration in config.yaml

Adding this configuration will allow us to test different versions of our service, e.g. our service deployed in a dev and a staging environment seamlessly. Since we also want to be able to run these tests against a local instance of our service (e.g. when writing new code or when learning about how a service works by reading and running the scenarios), we will define a "local" environment as well.

config:
target: "" # we don't set a target by default
environments:
# Describe an environment that we'll refer to as "dev". This is the name
# we will use with the --environment flag in artillery run command
# See https://artillery.io/docs/cli-reference/#run for details
dev:
# There's a dev deployment of this service on the following URL:
target: "https://auth-service-dev.acme-corp.internal"
# If we want to set service and environment-specific HTTP headers,
# we can do that here as well, e.g. to set an API key that must
# be used to access the service.
defaults:
headers:
x-api-key: "0xcoffee"
# We will almost certainly want to run these tests against a copy of the
# service running locally, so we define a "local" environment:
local:
target: "http://localhost:8080"
# Tell Artillery to load service-specific custom code
processor: "./functions.js"
# For our imaginary auth-service, we may use a CSV file with a set of known
# valid credentials to test with. This file may or may not actually be stored
# in the Git repo (and it may be encrypted at rest if it is) depending on the
# security requirements.
payload:
- path: "./username-password.csv"
fields:
- username
- password
#
# Any other service-specific configuration goes here, e.g. if there's an
# Artillery plugin that we use only for this service's tests, we'd load it
# here with the "plugins" attribute.
#

Service-specific custom JS code in functions.js

If we need to write a bit of custom JS (maybe using some npm modules) to customize the behavior of virtual users for this service (e.g. to generate some random data in the format the the service expects), we can write and export our functions from this module to make them available to this service's scenarios.

The key thing to remember here is that if you use an npm module, make sure it's declared in the top-level `package.json'.

Overrides in overrides.something.json

Let's say our auth service has the following SLOs:

  1. The service will handle 500 authentications per second at peak
  2. 99% of calls will complete in 200ms or less
  3. No more than 0.1% of calls will fail (defined as a non 2xx response code)

We could encode these requirements with the following Artillery configuration:

{
"config": {
"phases": [
{
"duration": 120,
"arrivalRate": 10,
"rampTo": 20,
"name": "Warm up the service"
},
{
"duration": 240,
"arrivalRate": 20,
"rampTo": 100,
"name": "Ramp to high load"
},
{
"duration": 600,
"arrivalRate": 100,
"name": "Sustained high load"
}
],
"ensure": {
"maxErrorRate": 0.1,
"p99": 200
}
}
}

Let's look at these overrides in more detail:

Arrival phases

Artillery does not do constant RPS (as it would not be possible to do meaningfully with scenarios composed of more than one request), but let's say we arrived at the arrivalRate values above through some experimentation to achieve >1000 RPS consistently in the "Sustained high load" phase.

Automatic SLO checking with ensure

Artillery supports automatic checking of response time and error rate conditions after a test run is completed. Here we're encoding our SLOs for the service of p99 response time being <= 200ms, and a max 0.1% acceptable error rate. Once the test run is over, Artillery will compare the actual metrics with the objectives, and if they're higher (e.g. p99 latency is over 200ms), the CLI will exit with a non-zero exit code, which will in turn fail the CI/CD pipeline job.

Note: that Artillery can only measure latency which includes both the application's response time and the time for the request and response to traverse the network. Depending on where the Artillery is run from, those values may differ very slightly (e.g. when Artillery is run from an ECS cluster in the same VPC as the service under test) or quite substantially (e.g. when an Artillery test is run from us-east-1 on a service deployed in eu-central-1). Use your judgement when setting those ensure values.

It is possible to make Artillery measure just the application's response time, but that requires that your services make that information available with a custom HTTP header such as the X-Response-Time header, and that some custom JS code is written.

Over to you!

Now that you've set up a project structure that's extensible and re-usable, go ahead and write a scenario or two for your service. To run your tests, you'd use a command like this:

artillery run --config ./services/auth-service/config.yaml --overrides "$(cat ./services/auth-service/overrides.slos.json)" -e dev ./services/auth-service/login.yaml

· 6 min read

If you're building a real-time backend in Node.js such as a chat server or the backend for a collaborative app, chances are you're using Socket.io (and you're in good company - Microsoft, Zendesk, Trello and countless others use it too).

We won't go into the details of what Socket.io does - if you're reading this, you probably know it very well already. What we'll look into is performance testing of Socket.io-based applications with Artillery.

The Situation

You have a Socket.io-based app. Maybe it's still in development, or maybe it's already running in production. You'd like to test your app under heavy load.

Perhaps you're interested in how much traffic it can handle before falling over. Maybe you want to test if you can scale horizontally by adding more app servers behind the load balancer. Or perhaps you're interested in doing some CPU profiling with perf or dtrace to improve runtime performance.

Whatever the reason may be, step one is to generate some load on your application, and that's exactly what this article covers. Read on to find out how to use Artillery to load test a Socket.io app.

Artillery 101

In case you haven't used Artillery before and need a quick intro to what it is: Artillery is a modern load testing toolkit that focuses on ease of use and developer happiness. It's designed for testing complex apps with multi-step scenarios (such as e-commerce backends, chat servers and all kinds of transactional APIs), and comes with batteries included for testing a variety of protocols and integrating with a variety of monitoring tools. It's very lightweight and very easy to get started with.

Artillery <3 Socket.io

Artillery offers first-class native support for Socket.io out of the box. The Socket.io engine in Artillery lets you send data to the server and optionally wait for and verify responses that come back. You can also mix HTTP and Socket.io actions in the same scenario.

Let's look at an example

# simple-socketio-load-test.yaml
config:
target: "http://my-backend.local"
phases:
- duration: 600
arrivalRate: 5
scenarios:
- name: "Connect and send a bunch of messages"
flow:
- loop:
- emit:
channel: "send message"
data: "hello world!"
- think: 1
count: 50

(Note that we are connecting to the Socket.io endpoint, not an underlying transport endpoint, e.g. the WebSocket URL exposed by Socket.io)

What will happen when we run this script with artillery run simple-socketio-load-test.yaml?

The target config value tells us that Artillery will connect to the application running on http://my-backend.local.

Looking at the phases definition, we see that Artillery will simulate 5 new users arriving to use the application every second for 600 seconds (resulting in 3000 users arriving in the space of 10 minutes).

What will those users do?

Each user spawned by Artillery will pick and run through one of the scenarios defined in the test script. In our case, there's just one scenario, which will have each user send 50 messages with a second's pause in between and disconnect from the server.

Testing the demo chat app

Let's see how we would test a real Socket.io application. We'll use the chatroom demo bundled with Socket.io.

The app is a simple chatroom where users can choose a nickname, join the room, and publish and receive messages.

The source code for the chatroom is available on Github: https://github.com/socketio/socket.io/tree/master/examples/chat.

Writing the test

Our test isn't going to be complex, but we will try to make it realistic by modeling three different kinds of users:

  1. Lurkers, who typically make up the majority of users in any given chatroom. They may send the occasional message, but are mostly receiving messages from others.
  2. Mostly-quiet users, that will engage now and then, but stay quiet most of the time.
  3. Chatty users, who will send a lot of messages.

The scenario for a lurker is very simple:

{% raw %}

 # A lurker - join, listen for a while, and leave.
- name: "A user that just lurks"
weight: 90
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "lurker-{{ $randomString() }}"
- think: 60

{% endraw %}

The nickname for the user is set with a template which generates a random alpanumeric string ($randomString() is a built-in function in Artillery templates).

The scenario for a chatty makes use of the ability to run custom logic written in Javascript as part of a scenario:

{% raw %}

  - name: "A chatty user"
weight: 10
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "chatty-{{ $randomString() }}"
- emit:
channel: "new message"
data: "{{ greeting }}"
- loop:
- function: "setMessage"
- emit:
channel: "new message"
data: "{{ message }}"
- think: 10
count: 10
- think: 60

{% endraw %}

After joining the chatroom and greeting other users (the greeting message is picked from a number of predefined strings at random), the user will send 10 messages with a 10 second pause between each message.

The setMessage function is pretty straightforward:

const MESSAGES = [
'what a nice day',
'how\'s everybody?',
'how\'s it going?',
'what a lovely socket.io chatroom',
'to be or not to be, that is the question',
'Romeo, Romeo! wherefore art thou Romeo?',
'now is the winter of our discontent.',
'get thee to a nunnery',
'a horse! a horse! my kingdom for a horse!'
];

function setMessage(context, events, done) {
// pick one of the messages
const index = Math.floor(Math.random() * MESSAGES.length);
// make it available to templates as "message"
context.vars.message = MESSAGES[index];
return done();
}

All scenarios in the test script also make use of the weighing feature in Artillery, which allows us to "weigh" the scenarios relative to each other making some more likely to be picked by spawned users than others. For this test script, we have assigned a weight of 75 to lurkers, 15 to mostly-quiet users, and 10 to chatty users, meaning that on average 75% of all users created during a test run will be of the lurker type for example.

To see how everything fits together, check out the code Github under hassy/socketio-load-test-artillery-example.

Running the tests

Running the tests is easy:

artillery run socketio-chat-load-test.yaml

Or if you cloned the repository above:

npm install # install dependencies
npm run load # run the load test

Questions or comments?

If you have any questions or commments, feel free to reach out on Github Discussions or on Discord: https://discord.com/invite/37vGhH3NMB

Happy load testing!