Artillery Best Practices

We designed Artillery with production-grade load testing in mind:

Load testing frequently, e.g. on automated schedules or as part of CI/CD pipelines
Load testing at high-scale

Artillery comes with support for best practices baked-in. This guide outlines some of the best practices that we see most of our users adopt to make the most out of load testing.

Fundamentals

Set up automated checks

An Artillery load test can generate many pages worth of metric reports. Whilst all that data is invaluable when you want to investigate the root cause of a performance issue, most of the time we just want to see if everything was OK or not at a glance.

Artillery gives you multiple ways to summarize the results of a load test:

Use the Apdex plugin for automated scoring of HTTP API performance
Use the ensure plugin to create flexible success/fail conditions for all load tests
Advanced: use Artillery's JSON reports to create custom checks

Setting up automated checks frees up your time from inspecting and summarizing load test results manually, and makes it clear when a load test may have identified an issue that needs to be investigated.

Connect monitoring & observability

Artillery integrates with multiple monitoring and observability systems out of the box.

Connecting Artillery to your monitoring system makes it possible to correlate metrics from backend services and APIs with metrics reported by Artillery load tests, which is useful for debugging issues uncovered by load tests.

Put load testing in your CI/CD pipeline

Load testing from CI/CD pipelines helps detect performance regressions and downward trends early on.

There's a couple of ways to run load tests from a CI/CD pipeline:

As part of a release workflow, where a load test is triggered by deployment events, such as deployment to a pre-production environment
As part of a release workflow, where a load test serves as a pre-promotion quality gate, such as running a load test on the staging environment before a limited canary roll-out into production
Running load tests on a schedule, such as running a nightly load test on the staging environment

Learn more about integrating Artillery with your CI/CD system in Artillery in CI/CD guides.

Run distributed tests from the cloud

A realistic load test is a distributed load test that runs from the cloud.

Realistic load test that produce meaningful results do not run from developer machines, or directly from CI/CD nodes. This is a common anti-pattern seen on teams who are early in their load testing adoption journey.

A developer's local machine is great for great for developing test scripts themselves, but for running realistic and meaningful load tests always scale them out horizontally. Artillery comes with distributed load testing at scale built in.

Spike or soak, avoid middle ground

As a rule of thumb, your load tests should fall into one of two groups:

Spike tests (also know as flash tests) - these are tests that run for short periods of time (<30m) and ramp up traffic quickly to create a large burst of traffic quickly.

This type of load testing is especially good for uncovering issues with:
- autoscaling configurations
- service startup time issues
- misconfigurations of service liveness/readiness checks
- CPU bottlenecks in application code
Soak tests - these are tests that run over extended periods of time, typically between 6-12 hours at a time (although longer soak tests of 24 hours or longer are not uncommon either). These tests usually run at no more than 10-20% above baseline load.

This type of load testing is employed to find lifecycle issues, such as:
- misconfigured resource pools
- memory leaks and other types of resource leaks
- resource saturation issues

Make your application perform an all-out sprint or run a marathon.

Organizing test code

Use `config.environments`

Use config.environments to create different configuration profiles for different environments or types of load tests.

Separate configuration and scenario definitions

Separate scenarios into individual files and use the --config flag to combine test configs with individual scenarios. This makes reusing scenario definitions in multiple different load test configurations easier.

Keeping scenario definitions in isolated files also makes it easier to re-use those scenarios for end-to-end testing and synthetic monitoring with Skytrace (opens in a new tab).

Use a single repo for all performance tests

We find that it's best to place all load test specs and configurations into the same repo, even though the tests may target different services or sub-systems.

This simplifies both code reuse between different tests, and dependency management (such as Artillery itself, plugins and extensions, and custom code used by the tests).

Advanced

Load test in production

Load testing in production is a practice with a very high payoff when it comes to increasing performance and reliability of production systems. Contrary to popular belief, load testing in production does not have to be risky and can be done safely.

We outline the theory and reasoning behind production load testing, and provide a hands-on guide for going from zero to a production load test in a two-part guide on our blog:

Create tagging conventions

Having a consistent strategy for tagging your tests will allow you to filter and search for tests more easily, and to aggregate and analyze test results across multiple runs.

For Artillery Cloud, use the --tags flag to pass a comma-separated list of tags to your test script.
When using publish-metrics to send data to an observability platform, use the tags key in your test script to pass a list of tags.

Here are some recommendations we have seen work well:

env

Track performance across different environments (to account for differences in application or infrastructure). For example, you may use your staging environment to gather baselines, and your production environment to run synthetic tests.

e.g. env:staging, env:prod, env:qa

version

Trace performance issues back to the specific version of the application under test (such as a service version or commit SHA). For example, you could use git rev-parse --short HEAD to get a short Git SHA to tag your test with.

e.g. version:a231b3e, version:1.2.3

service

Build more robust service dashboards by using service-specific tests. For example, you may want to track the performance of your system by running a production smoke test for each service in your architecture.

e.g. service:auth, service:billing

test_id

Filter to a specific test run. If using Artillery Cloud, you will be able to do this by default without passing this tag.

e.g. test_id:{{ $testId }}, testId:twjdf_dm43kdchd4ymtyb7zzney3rkkxqgk_trbe

type

Differentiate between the same test being run for different purposes. For example, you may run nightly tests to gather baselines on the environment, but also run a pre-release soak test in the same environment every two weeks.

e.g. type:spike, type:baseline, type:soak, type:acceptance

ci

Distinguish local runs from CI runs, so you can gather data over time under specific circumstances in CI.

e.g. ci:true, ci:jenkins

team

Enable teams to build team-specific dashboards where they can query on specific tests they ran.

e.g. team:capybara, team:billing

actor

Enable users to filter tests they ran. For example, in Github Actions, you could achieve this with the github.actor variable.

e.g. actor:bernardobridge, actor:hassy@yourdomain.com

Artillery at enterprise scale

Use Artillery Dashboard

We designed the Artillery Dashboard for teams that run load tests continuously and in large volumes to provide a central location for visualizing, reviewing, analyzing and collaborating on performance reports.

→ More information on https://www.artillery.io/cloud (opens in a new tab)

Consider creating a convenience wrapper

We find that teams that offer Artillery as part of an internal developer platform often create a company-specific CLI wrapper around the Artillery CLI (or integrate it into an existing internal CLI) to help with things like:

Making common team or company-specific tests easier to run (e.g. running a test by a simple short name rather than by specifying paths to config and scenario files).
Adding a set of tags automatically to every test, such as the environment and the author.
Integrating Artillery tests with other systems. For example by injecting appropriate secrets from Vault (opens in a new tab) into Artillery tests.

Why Artillery?Artillery Cloud