Artillery Pro CLI
Artillery Pro has entered maintenance mode and will be officially sunset in June 2024. New releases will be made only to address critical bugs or security vulnerabilities.
The distributed load testing functionality of Artillery Pro is now available in the main open source distribution of Artillery.
Requirements
Artillery Pro needs to be installed. Once Artillery Pro is installed, several new commands become available in the artillery
CLI. This page documents those commands.
Artillery Pro is on a different release cycle from Artillery and has its own versioning.
All Artillery Pro versions are compatible with both Artillery v1 and Artillery v2. If an Artillery Pro release requires a specific version of Artillery, it will be indicated in the Changelog and checked by Artillery Pro itself at runtime.
run-test
- run a test from your cloud environment
artillery run-test [options] <test>
This command is similar to the Core run
command, but instead of running a test from a single machine, the run-test
command runs it from an Amazon Elastic Container Service (opens in a new tab) or Amazon Fargate (opens in a new tab) cluster. This command lets you scale up and run your tests in distributed mode across multiple workers, and from a number of geographical regions.
For example, if you have a test script called my-script.yaml
, you can run it in a single, local instance with Artillery Core using the run
command:
artillery run my-script.yaml
The run-test
command makes it easy to scale up and run the same script from the cloud. For example, you can scale up to 20 workers (instead of single worker) running in the us-east-1
region on Fargate with the following Artillery Pro command:
artillery run-test \
--region us-east-1 \
--cluster mycluster \
--count 20 \
--launch-type ecs:fargate \
my-script.yaml
See Running tests from AWS for a detailed guide on running Artillery tests from AWS.
Options
The run-test
command shares several flags with run
, which allows for a seamless transition from running tests locally to running them in the cloud.
Option | Description |
---|---|
--output , -o | Write a JSON report to a file |
--environment , -e | Run the test using the specified environment |
--config , -c | Load config section from another file |
--overrides | Override values in the test script dynamically |
--target , -t | Set or override target URL for the test |
--insecure , -k | Turn off TLS verification. Should not be used in production |
These flags work in the exact same way in run-test
as in run
.
The run-test
command has several other flags specifically for configuring how a test should be executed from a cloud environment.
Option | Description |
---|---|
--cluster | The name of the cluster to run the test from |
--launch-type | Set the platform to run tests from. Possible values: ecs:fargate and ecs:ec2 (default) |
--count | Set the number of workers for running the test |
--secret | Make a secret created with set-secret command available in the test as an environment variable |
--region , -r | The region where the test will run from. Artillery Pro supports 13 different regions |
--launch-config | A JSON object describing launch configuration for ECS tasks |
--subnet-ids | [Fargate-only] Comma-separated list of subnet IDs to place tasks into |
--security-group-ids | [Fargate-only] Comma-separated list of security group IDs to attach to tasks |
--max-duration <time> | Set maximum duration of a test run. Duration is set with a string like 10m for 10 minutes or 1h for one hour. Only integer values may be specified. Supported units are ms , s , m and h . The test run will be stopped if the running time of a test exceeds the limit |
--dotenv <path> | Set environment variables in workers with a .env file |
--tags <tags> | Comma-separated list of tags in key:value format to tag the test run, for example: --tags "team:sre,service:foo" |
--note <msg> | Add a note to the test run |
Examples
Run a test by launching 10 workers on Amazon Fargate in the US West (Oregon) region:
artillery run-test \
--region us-west-2 \
--cluster mycluster \
--count 10 \
--launch-type ecs:fargate \
my-script.yaml
Run a test by launching 20 workers on Amazon ECS in the Europe (Ireland) region:
artillery run-test \
--region eu-west-1 \
--cluster mycluster \
--count 20 \
--launch-type ecs:ec2 \
my-script.yaml
stop-test
- stop a running test
The stop-test
command stops a running test gracefully, by stopping worker tasks and performing other clean up tasks as necessary.
artillery stop-test <id>
The run-test
command prints test IDs at the beginning of each test run. IDs of currently running tests may also be seen by running artillery get-tests
.
Example
Stop a running test with id = e84a482c-cf34-404c-a95d-22446b92a2c1.
artillery stop-test e84a482c-cf34-404c-a95d-22446b92a2c1
describe-test-run
v3
Description
Return the details of a test that was run or is still running, as JSON.
Synopsis
artillery describe-test-run <id>
id
is the test run id, which is printed by test-run
or returned by list-test-runs
command.
Examples
# Get the details for a test run with id = test-e7107e1
artillery describe-test-run test-e7107e1
list-test-runs
v3
Description
List ids of past test runs, as JSON.
Synopsis
artillery list-test-runs
delete-test-run
v3
Description
Delete a test run by its id.
Examples
artillery delete-test-run aad11369-9c3e-4f18-b517-62aa2a376705
get-tests
v2 • list-tests
v3
List currently running tests.
# Artillery Pro v2
artillery get-tests
# Artillery Pro v3
artillery list-tests
Options
Option | Description |
---|---|
--json | Format output as JSON |
create-test
v2 • create-test-bundle
v3
Description
The create-test-bundle
command resolves a test script's dependencies (such as external CSV files, custom JS code and its dependencies), and uploads everything needed to run the script to a location in Amazon S3.
The name given to the test may then be used with the run-test
command to reduce the amount of time it takes to set up a test environment before the running the test script.
You don't need to create a test bundle before running a test script with
run-test
. This command provides a way to optimize the running time of a
script with many dependencies, but its usage is optional.
Synopsis
artillery create-test-bundle [options] <script>
Options
Option | Description |
---|---|
--name , -n | The name for the test (to be used in run-test ) |
--config | Use common test configuration for this named test |
Examples
Create a named test from a test script in my-script.yaml
called my-named-test
:
artillery create-test-bundle --name my-named-test my-script.yaml
If the script makes use of CSV files for variable data, plugins, or custom code (which may make use of external npm
dependencies), all of those references will be resolved automatically.
After creating the test, you can run the test using run-test
with the --bundle
option:
artillery run-test \
--region ap-northeast-1 \
--count 25 \
--bundle my-named-test
delete-test
v2 • delete-test-bundle
v3
Description
Delete a named test bundle created with create-test-bundle
.
Synopsis
artillery delete-test-bundle <name>
Examples
Delete an existing named test called my-named-test
:
artillery delete-test-bundle my-named-test
list-tests
v2 • list-test-bundles
v3
Description
List all named tests created with create-test-bundle
.
Synopsis
artillery list-test-bundles [options]
Options
Option | Description |
---|---|
--json | Format output as JSON |
deploy
- deploy Artillery Pro backend
Description
Set up the Artillery Pro backend components in your AWS account. A CloudFormation stack is created (or updated) with the resources needed to run Artillery Pro tests. See Installing Artillery Pro for more information on deploying the backend.
Synopsis
artillery deploy [options]
Options
Option | Description |
---|---|
--region , -r | The AWS region to deploy the backend into |
--upgrade | Upgrade an existing Artillery Pro installation in an AWS account |
--license-key-file | Specify the path to file containing an Artillery Pro license key |
--license-key-text | Specify the contents of an Artillery Pro license key |
--set <key=value> | Customize names of resources in the CloudFormation stack |
Upgrading an existing installation
To upgrade an existing installation of Artillery Pro, install the latest version of the CLI, and run the deploy
command in the region in which the backend is already deployed, e.g.:
artillery deploy --upgrade --region us-east-1
Customizing CloudFormation resource names
You can customize the names of some resources created by the Artillery Pro CloudFormation stack (to follow naming standards in your organization for example). The --set
flag currently supports the following options:
Key | Description |
---|---|
cli-user-role-name | The name of IAM role for CLI users |
cli-user-policy-name | The name of IAM managed policy assumed by CLI user role |
Example: set a custom name for the IAM role for CLI users who run tests
This examples presumes an existing installation in us-east-1
that we're wishing to upgrade, and set the name of the IAM role generated for Artillery Pro CLI users to tools-sre-artillery
.
artillery deploy --set cli-user-role-name=tools-sre-artillery --region us-east-1
set-secret
artillery set-secret [options]
The set-secret
command creates an encrypted secret value which can be used in test runs. You can then access these values in your test scenarios via the $env
variable.
How are the secrets stored?
The secrets are encrypted and stored securely in AWS Parameter Store (opens in a new tab) in your AWS account (under /artilleryio/
scope).
Options
Option | Description |
---|---|
--name , -n | Name of the secret |
--value , -v | Value of the secret |
--backend | Specify the secret storage backend (currently only ssm ) |
--region , -r | AWS region where the secret should be created |
Examples
Create a secret called API_KEY
with a value of 012345
in the US East (N. Virginia) region:
artillery create-secret --name API_KEY --value 012345 --region us-east-1
You can then make this secret available in a test run using the --secret
flag in the run-test
command:
artillery run-test --secret API_KEY --launch-type ecs:fargate my-test.yml
The value for the secret will be available as an API_KEY
environment variable accessible via the special $env
variable:
- post:
url: "/pets"
headers:
x-api-key: "{{ $env.API_KEY }}"
json:
name: Tiki
species: pony
list-secrets
v3
Description
List secrets created with set-secret
.
Synopsis
artillery list-secrets
list-tags
v3
Description
List all tags that have been used to tag test runs, as JSON.
Synopsis
artillery list-tags
post-note
v3
Description
Add a note to a test run. The notes are available in the output from describe-test-run
.
A common use-case is using post-note
to annotate test runs with links to descriptions of services tested, monitoring dashboards, project management boards, or runbooks.
Synopsis
artillery post-note --test-run-id <id> --text <text>
Examples
artillery post-note \
--test-run-id test-e7107e1 \
--text "See monitoring dashboard on https://app.datadoghq.com/dashboards/foo-service "
set-config-value
Description
Set runtime configuration values which will be available to workers running on ECS/Fargate in a secure way. These configuration values can be used for installing private packages from npm (opens in a new tab) and Artifactory (opens in a new tab), and for customizing your npm configuration.
See Private npm
packages for more information on using private packages in your tests.
Synopsis
artillery set-config-value [options]
Options
Option | Description |
---|---|
--name , -n | Name of the configuration parameter (see available parameters below) |
--value , -v | Value of the configuration parameter |
--region , -r | AWS region where Artillery Pro is deployed |
Configuration parameters
You can set values for the following configuration parameters:
Parameter | Description |
---|---|
NPM_TOKEN | Auth token to use with npm |
NPM_REGISTRY | URL of private npm registry to use |
ARTIFACTORY_AUTH | Auth token to use with Artifactory |
ARTIFACTORY_EMAIL | Email to use with Artifactory |
NPMRC | Customize .npmrc with arbitrary config |
GITHUB_CLIENT_ID , GITHUB_CLIENT_SECRET | Client ID and secret. Set these to enable GitHub-based login for Artillery dashboard |
GITHUB_ALLOWED_USERS , GITHUB_ALLOWED_ORGS | A JSON-encoded list of usernames or organization names which are allowed to access the dashboard, for example: ["artilleryio"] |
Examples
Set the npm authentication token for an Artillery Pro deployment in the US East (N. Virginia) region:
artillery set-config-value \
--name NPM_TOKEN \
--value 01234567 \
--region us-east-1
Set the loglevel
setting for the .npmrc
configuration to debug
for the workers in an Artillery Pro deployment in the Europe (Ireland) region:
artillery set-config-value \
--name NPMRC \
--value "loglevel=debug" \
--region eu-west-1
subscription
artillery subscription [options]
The subscription
command shows the status of your team's Artillery Pro subscription. You can also use this command to request an Artillery Pro license key to be sent to an email address associated with the subscription.
Options
Option | Description |
---|---|
--email | Request license key to be sent to an email address |