Artillery CLI

What you'll learn

• How to run Artillery tests from the command line
• How to run tests from a single machine with artillery run
• How to accomplish other tasks with the Artillery CLI, such as generating HTML reports

Overview

Requirements

This guide assumes you've already read our Installing Artillery guide, and installed Artillery on your machine.

For a reference of commands added by Artillery Pro please refer to Artillery Pro CLI.

Using the CLI

Listing all commands

Run artillery --help to see all of the available commands.

To see more information about a specific command, such as the full list of flags it supports, use the --help flag with that command. For instance, to list all flags supported by the run command:

artillery run --help

Getting version info

Run artillery version to print the version of Artillery (and Artillery Pro if installed), as well as information about the environment, such as Node.js version and the OS.

$ artillery version

        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/


VERSION INFO:

Artillery Core: 2.0.0-10
Artillery Pro:  3.0.0

Node.js: v17.3.0
OS:      darwin

run - run a test script

artillery run [options] <script>

The run command runs a test script from the local machine. The basic way to run a test is with:

artillery run my-script.yaml

Options

Option	Description
--output, -o	Write a JSON report to a file
--environment, -e	Run the test using the specified environment
--config, -c	Load config section from another file
--overrides	Override values in the test script dynamically
--target, -t	Set or override target URL for the test
--variables, -v	Set scenario variables dynamically
--dotenv <path>	Set environment variables with a .env file
--insecure, -k	Turn off TLS verification. Should not be used in production
--quiet, -q	Run in "quiet" mode
--platform	Set platform. Options: local, aws:lambda. Default: local
--platform-opt	Set platform-specific options

Runtime Platforms

By default Artillery will run the test from the local machine. Since v2.0.0-21 Artillery can also run tests on arbitrary platforms. Currently, AWS Lambda support is available in preview mode.

Platform: local machine (local)

This is the default platform if a platform is not set explicitly.

Platform: AWS Lambda (aws:lambda)

To run an Artillery test on AWS Lambda, using a test script in hello-artillery.yml, run the following command. This presumes that an AWS profile has already been configured locally with sufficient IAM permissions.

# Run a distributed test on AWS Lambda in us-east-1 region using 25 workers:

artillery run \
  --platform aws:lambda \
  --platform-opt region=us-east-1 \
  --count 25 \
  hello-artillery.yml

Please see Running tests on AWS Lambda for more information.

AWS Lambda-specific options

Option	Description
region	AWS region to execute the test from, e.g. --platform-opt=us-east-1
memory-size	Amount of memory to make available to each Lambda in MB. A number between 128 and 10240. Default: 4096
security-group-ids and subnet-ids	Comma-separated lists of subnet IDs and security group IDs to configure Lambdas to run in a VPC. Default: none, Lambdas will run in the default secure AWS-owned VPC
architecture	Set CPU architecture to use. Default is arm64 (Graviton). Legacy x86 may be set with x86_64

Create a JSON report

You can tell Artillery to write a JSON report of a test run into a file. The JSON report can be used to generate an HTML report with the report command, or to run queries on with a tool like jq.

artillery run --output report.json my-script.yaml

Test different deployment environments

A service is usually deployed in more than one place. For example, your team may use pull request or preview environments, which create a separate deployment of a service for each pull request or feature branch. Most teams often have one or more staging deployment and at least one production deployment. Artillery lets you describe those environments in your test script and switch between to run tests.

Environment-specific configuration is set in the config.environments settings in the test script:

config:
  target: "https://service-foo.acmecorp.digital" # default target
  phases:
    - arrivalRate: 50
      duration: 600
  environments:
    local-dev:
      target: "http://localhost:8080"
      phases:
      - arrivalRate: 10
        duration: 60
    preprod:
      target: "https://service-foo.preprod.acmecorp.digital"
scenarios:
    # Scenario definitions would go here.

The above example script defines two environments: local-dev and preprod. Environment-specific definitions such as target or phases will override the top-level ones defined in config.

You may then run the same test against the service running locally on http://localhost:8080 with:

artillery run --environment local-dev my-script.yaml

The same script could be used in a post-deployment CI job to run tests against the preprod version of the service on https://service-foo.preprod.acmecorp.digital with:

artillery run --environment preprod my-script.yaml

Extract and reuse configuration

While Artillery test scripts allow for any number of scenarios to be defined in the same test script, it's useful to keep individual scenarios in their own files for reusability and to prevent duplicating the configuration settings.

Consider the following example layout of a repository containing two Artillery test scripts, edit-records.yaml and search-records.yaml:

acmecorp-backend-tests/
  └── services/
      └── service-foo/
          ├── config.yaml
          └── scenarios/
              ├── edit-records.yaml
              └── search-records.yaml

For both tests, you can define the same config settings to run each test script individually. However, if you need to modify the test configuration (like changing the target URL, for instance), you would need to make the same change on all scripts, which has the risk of forgetting to update all test scripts where needed.

To keep your test script settings identical, you can reuse the same config settings by defining a separate YAML file. In this example, the config.yaml contains common configuration used for both test scripts:

config:
  target: "https://service-foo.acmecorp.digital"
  phases:
    - arrivalRate: 50
      duration: 600

With a shared configuration, each test script can only contain a scenarios section to define the scenarios for executing the test. You can use the shared configuration with both test scripts using the --config flag:

# Run edit-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/edit-records.yaml

# Run search-records scenarios with the shared configuration:
artillery run --config config.yaml scenarios/search-records.yaml

When using --config, you can merge different configuration settings between the shared configuration file and the test scripts. However, the settings from the shared configuration will take precedence over the same settings inside the test scenario.

Override parts of the test script on the fly

If you want to override parts of a test script from the command line, you can use the --overrides flag. The flag accepts a JSON string with parts of the test script you wish to override.

For example, you have the following test script, containing a phase generating 50 virtual users per second for 10 minutes:

config:
  target: "https://service-foo.acmecorp.digital"
  phases:
    - arrivalRate: 50
      duration: 600
scenarios:
  # Scenario definitions would go here.

You can use the --overrides flag to override the configuration and run a phase generating 1 virtual user per second for 10 seconds:

artillery run \
    --overrides '{"config": { "phases": [{ "duration": 10, "arrivalRate": 1 }] } }' \
    my-script.yaml

If you're running Artillery on Windows using the Command Prompt, you'll need to wrap the --overrides flag values using double-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --overrides "{\"config\": { \"phases\": [{ \"duration\": 10, \"arrivalRate\": 1 }] } }" \
    my-script.yaml

If you're running Artillery on Windows using PowerShell, you'll need to wrap the --overrides flag values using single-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --overrides '{\"config\": { \"phases\": [{ \"duration\": 10, \"arrivalRate\": 1 }] } }' \
    my-script.yaml

Set scenario variables on the fly

To define variables on the fly to use in your test scenarios, the --variables flag allows you to set a JSON string containing one or more variable names with multiple values.

You can also load environment variables from a dotenv file with the --dotenv flag.

The following example makes two variables - color and size - available in scenario definitions:

artillery run \
    --variables '{ "color": ["red", "yellow", "blue"], "size": [120, 150, 200] }' \
    my-script.yaml

The variables may be used as normal in scenario definitions:

scenario:
  - name: Create record
    flow:
      - post:
          url: "/items"
          json:
            itemColor: "{{ color }}"
            itemSize: "{{ size }}"

As with setting variables in the configuration, each virtual user will get one of the available values for each variable.

If you're running Artillery on Windows using the Command Prompt, you'll need to wrap the --variables flag values using double-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --variables "{ \"color\": [\"red\", \"yellow\", \"blue\"], \"size\": [120, 150, 200] }" \
    my-script.yaml

If you're running Artillery on Windows using PowerShell, you'll need to wrap the --variables flag values using single-quotes and escape the inner quotes of the JSON string using backslashes:

artillery run \
    --variables '{ \"color\": [\"red\", \"yellow\", \"blue\"], \"size\": [120, 150, 200] }' \
    my-script.yaml

Override the target URL

The --target flag lets you override the specified target URL of a test script:

artillery run --target https://service-bar.acmecorp.digital my-script.yaml

Turn off TLS verification

By default, Artillery will reject SSL certificates that it's unable to validate. You can disable certificate validation with the --insecure flag:

artillery run --insecure my-script.yaml

This flag is useful for testing a service in a development or staging environment that doesn't have a SSL certificate that can be validated, like a self-signed certificate.

caution

You should never use the --insecure flag on a production environment. Ignoring certificate errors in a production system can lead to potential security vulnerabilities, like man-in-the-middle attacks.

Suppress output when running tests

When running a test, Artillery prints a report on the console every 10 seconds for the number of scenarios executed during that period. At the end of the performance test, it prints a complete summary.

You can suppress this output if you don't need it (like when running tests on a continuous integration environment) using the --quiet flag:

artillery run --quiet my-script.yaml

---

quick - test a single HTTP endpoint

artillery quick [options] <target>

This command allows you to send some requests to a single endpoint, without writing a test script.

The quick command only works to test HTTP services. You cannot run quick tests for other services, such as Socket.IO or WebSockets.

Options

Option	Description
--count, -c	Specify a fixed number of arrivals
--num, -n	Specify the number of GET requests each new arrival will send
--content-type, -t	Set the Content-Type for the request (defaults to application/json)
--output, -o	Write a JSON report to a file
--insecure, -k	Allow insecure TLS connections
--quiet, -q	Run in "quiet" mode
--rate, -r	Specify the number of new arrivals per second v1 only

Examples

Run a quick test which generates 20 virtual users, each sending 100 GET requests to the specified target:

artillery quick \
    --count 20 \
    --num 100 \
    https://service-foo.preprod.acmecorp.digital/items

---

report - create an HTML report

artillery report [options] <file>

The report command converts a JSON report generated by the run command into a self-contained HTML report.

Options

Option	Description
--output, -o	File name of the HTML report (optional, defaults to adding .html extension to the name of the JSON file)

Generating an HTML report

First, run a test and create a JSON report with the --output flag.

artillery run --output test-run-report.json my-script.yaml

You can then use the generated JSON report to create an HTML report:

artillery report test-run-report.json

This will create a test-run-report.json.html file in the current directory which you can open in a browser.