Core Concepts

What you'll learn

• Key high-level concepts for using Artillery
• Configuration and scenario sections in Artillery test scripts
• What load phases and virtual users are

High-level view

Artillery runs performance tests. You write test definitions, and run them with Artillery.

As Artillery runs a test, it will track and record performance metrics, such as API response times, throughput, and errors. As the test runs, Artillery will report those metrics so that performance of the system can be observed in real-time. A summary report is also provided at the end of the test run.

Artillery is centered around scenarios which describe how virtual users (VUs)

Test definitions

A test definition is written as a YAML file (with the option for near-infinite customization with Node.js code, which may use any public or private npm packages). The artillery CLI is then used to run that test definition.

A test definition is composed of one or more scenarios (in the scenarios section) and test config (in the config section).

A "hello world" Artillery test definition looks like this:

config:
  target: https://www.artillery.io
  phases:
    - duration: 60
      arrivalRate: 1
    - duration: 300
      arrivalRate: 10
scenarios:
  - flow:
      - get:
          url: "/docs"
      - get:
          url: "/docs/resources"

Virtual users (VUs) and scenarios

Artillery is centered around the concept of virtual users.

A "virtual user" is exactly what the name suggests - a simulation of a real user (or a client) using a service or an API. Each virtual user is completely independent of other virtual users - just like in the real world. For example, when testing an HTTP-based service, each virtual user will open & maintain its own TCP connection(s), and maintain its own cookies and any other data.

For example, if you're testing a RESTful ecommerce backend, a scenario may be composed of:

1. Making a request to the search API endpoint to search for a product.
2. Making a request to add one of the products returned in the results to basket.
3. Loading the basket.
4. Initiating the checkout flow.

Config

Test config sets the target service to be tested, and the workload (the shape of traffic). It is also used to load data from external CSV files, load & configure plugins, and load custom Javascript code.

Workload modeling

Load phases

A load phase tells Artillery how many virtual users to create over a period of time. Load phases is how you model load on a service for the duration of a test run. A load test will usually have multiple load phases, such as a warm up phase, followed by a gentle ramp up phase, which is then followed by one or more heavy load phases. Let's look at an example:

phases:
  - duration: 300
    arrivalRate: 10
    name: Warm up phase

This code describes a load phase which lasts 300 seconds (or 5 minutes), where 10 new virtual users will be created every second.

Each virtual user will pick and run one of the scenarios in the test definition and run it to completion. A test run ends when all virtual users finish running their scenarios. This is an important distinction to understand: the duration of a test is not the combined duration of all of its load phases, but the amount of time it takes for all virtual users created in those load phases to finish their scenarios.

Example - How long will a test run for?

Let's look at an example. Imagine we define a test with a single load phase which lasts five seconds and creates one virtual user every second (so a total of fives virtual users will be created). The test definition contains a single scenario, which sends two HTTP requests one after another.

How long will that test take to run?

The answer is "it depends", with a couple of factors that need to be considered:

1. How quickly the target service can respond to each request, which will vary for different kinds of requests (e.g. a GET request that returns a cached value vs a POST that creates a new record in a database)
2. How quickly the service will actually respond to requests. The response time from the service will likely change as more load is placed on the it, i.e. it will probably get slower at some point
3. Network latency and performance which can add some overhead to how long a request takes to be completed, before the virtual user moves on to the next one
4. The number of steps in the scenario

Imagine that each HTTP request takes exactly 60 seconds. The total running time of the test is going to be:

• 5 seconds during which virtual users will be created
• each virtual user will make two HTTP requests, and the total duration of the session for each VU will be 60 * 2 = 120 seconds
• the last VU is going be created at second 5 of the test run, and will take 120 seconds to complete its scenario
• hence the total running time of the test run is going to be 5 + 120 = 125 seconds

Load tests vs smoke tests

Artillery can be used for both load testing and smoke testing (also known as synthetic checks, or synthetic tests). The only difference between a load test and a smoke test is the number of VUs created:

• A load test creates many VUs
• A smoke test creates just one VU

The rest of it is the same: the scenario definition can be re-used, as well as configuration (such as using a plugin to send metrics to an external monitoring system).