Getting started with Artillery Cloud

Artillery Cloud is a hosted service that works alongside the Artillery CLI and lets you:

See and track load test runs in real-time
View & analyze performance metrics generated by load test runs (response times, errors, custom ensure checks, and custom metrics)
See a green/red overview of all load testing activity on your team
Create custom charts and dashboards to help make sense of application-specific metrics
Organize load test reports across projects and teams
Annotate and comment on individual test reports to include debugging information or results of performance investigations
Share tests with external stakeholders and other teams

How it works

All load tests still run completely within your own environment (i.e. locally, on AWS Lambda, AWS Fargate or Azure Container Instances). The use of the cloud dashboard is opt-in per test run. The CLI will send test reports to the cloud dashboard only if the --record flag is used.

Get started

Create an Artillery Cloud account

Create an Artillery Cloud account by signing in with your GitHub account on https://app.artillery.io (opens in a new tab)

Upgrade Artillery CLI to latest version

Artillery CLI v2.0.0 or later is required to use Artillery Cloud. We recommend always running the latest release of the Artillery CLI.

Upgrade with:

npm install -g artillery@latest

Create an API key

To enable the Artillery CLI to send data to Artillery Cloud you will need an API key.

Configure the CLI to send data to Artillery Cloud

The use of Artillery Cloud is opt-in. You have to tell the CLI when you want test results to be sent to the cloud dashboard.

To record a test run to the dashboard, use the --record and --key flags with the artillery run command. For example, if you're currently running load tests with:

artillery run my-service-test.yml

To enable cloud dashboard reporting, add --record and set the API key with --key:

artillery run my-service-test.yml --record --key 0x12345_my_api_key

The API key may also be set via the ARTILLERY_CLOUD_API_KEY environment variable. When that variable is set, the --key flag may be omitted.

Best practices

Naming tests

By default, Artillery Cloud will use the name of the test script file as the name of the test.

You can override this by using the --name flag.

artillery run service-foo.yml --name "service-foo-spike-test" --record --key $MY_ARTILLERY_CLOUD_KEY

The name of the test is used in several places in the dashboard:

The name shown in the list of test runs
The name of the group in group view
When showing other runs of the same test in Comparison View
As the basis for calculating and showing performance trends across multiple runs

Using tags

Having a consistent strategy for tagging your tests will allow you to filter and search for tests more easily, and to aggregate and analyze test results across multiple runs.

Use the --tags flag to pass a comma-separated list of tags to your test script, for example:

artillery run my-service-test.yml --tags "env:staging,service:auth,version:1.2.3" --record --key $MY_ARTILLERY_CLOUD_KEY

Here are some recommendations we have seen work well:

env — different environments where the application is deployed

Track performance across different environments to account for differences in application or infrastructure.

For example, you may use your staging environment to gather baselines under high load, and your production environment to run synthetic tests at lower loads.

Examples: env:staging, env:prod, env:qa

service — name of the application or service

Build more robust service dashboards by using service-specific tests. For example, you may want to track the performance of your system by running a production smoke test for each service in your architecture.

Examples: service:auth, service:billing

version — version ID of the application or service

Trace performance issues back to the specific version of the application under test, such as a service version or commit SHA.

For example, you could use git rev-parse --short HEAD to get a short Git SHA to tag your test with.

Examples: version:a231b3e, version:1.2.3

type — type of load test

Differentiate between the same test being run for different purposes.

For example, you may run nightly tests to gather baselines on the environment, but also run a pre-release soak test in the same environment every two weeks.

Examples: type:spike, type:baseline, type:soak, type:acceptance

ci — whether a run was triggered from CI or not

Distinguish ad-hoc test runs from CI runs, so you can gather data over time under specific circumstances in CI.

Examples: ci:true, ci:jenkins

team — name of the team running the test

Enable teams to build team-specific dashboards where they can query on specific tests they ran.

Examples: team:billing, team:capybara

actor — ID of the person or workflow that triggered the test

Useful for filtering by ID of the person or workflow.

For example, in GitHub Actions, you can use the github.actor or github.job variables.

Examples: actor:github-post-deployment-check, actor:deepak@example.com

Troubleshooting

`Error: unable to reach Artillery Cloud API`

If you see this error message, it means that the CLI is unable to reach the Artillery Cloud API.

The most common cause of this error is a corporate proxy or firewall blocking POST requests to the Artillery Cloud API (https://app.artillery.io (opens in a new tab)).

To check if requests are blocked run the following command:

curl -X POST https://app.artillery.io/api/ping

You should see:

{"message":"pong"}

If you see an error message, such as a network timeout, or a message indicating that the request was blocked, you will need to work with your network administrators to allow requests to the Artillery Cloud API.

In the meantime, you can try running Artillery from a machine on a different network (e.g. from your home network, or from a cloud instance).

Artillery Best Practices Examples