Skip to main content

Writing Your First Test

What you'll learn
  • What an Artillery test definition looks like
  • How to build your first Artillery test definition
  • How to run an Artillery test script

Let's say you want to test the performance of a JSON API that runs the backend of an e-commerce website. Many e-commerce sites experience traffic spikes on occasion (like on Black Friday / Cyber Monday, for instance). An excellent way to verify that your backend will handle a sudden influx of visitors during these times is to simulate the expected flows that your customers do when they visit your site.

On an e-commerce website, the typical customer will perform the following actions when they want to purchase a product:

  • Search for a keyword for the product they want to purchase.
  • Click on the first result returned by the site.
  • Read the product information.
  • Add the item to the cart.

You'll want to check that this flow will work well when hundreds of users perform these actions during the same period. Let's see how we can use Artillery to load-test these actions by writing a test definition.

Configuration​

Target URL​

First, you need to define the base URL for the application you want to test. You can do this by setting config.target:

config:
  target: "https://example.com/api"

Load Phases​

Next, you'll need to determine the load phases for the test by setting config.phases. For each load phase, you'll define how many virtual users you want to generate and how frequently you want this traffic to send requests to your backend. You can set up one or more load phases, each with a different number of users and duration.

For this performance test, we'll start define three load phases:

config:
  target: "https://example.com/api"
  phases:
    - duration: 60
      arrivalRate: 5
      name: Warm up
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: Ramp up load
    - duration: 600
      arrivalRate: 50
      name: Sustained load

The first phase is a slow ramp-up phase to warm up the backend. This phase will send five virtual users to your backend every second for 60 seconds.

The second phase that follows will start with five virtual users and gradually send more users every second for the next two minutes, peaking at 50 virtual users at the end of the phase.

The final phase simulates a sustained spike of 50 virtual users every second for the next ten minutes. This phase is meant to stress test your backend to check the system's sustainability over a more extended period.

Injecting data from an external file​

One of the sections to verify as part of the performance test involves searching for a product. While you can make a request to a URL that will search for a specific product, it's a good practice to check how your system behaves when accessing dynamic content, such as searching for a product.

Artillery allows you to load an external CSV file and inject its contents into your scenarios as a variable. For example, you can have a CSV file called keywords.csv in your test directory with the following contents:

computer
video game
vacuum cleaner
toys
hair dryer

You can load this CSV file in Artillery with the config.payload setting:

config:
  target: "https://example.com/api"
  phases:
    - duration: 60
      arrivalRate: 5
      name: Warm up
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: Ramp up load
    - duration: 600
      arrivalRate: 50
      name: Sustained load
  payload:
    path: "keywords.csv"
    fields:
      - "keyword"

config.payload requires the following settings:

  • path: The name of the CSV file you want to load into your test script. The file must use a path is relative to the location of your test definition file.
  • fields: The variable name you want to use in your test scenarios. In this example, the CSV file contains only one value per row, but you can also define multiple variables if each row contains more than one value.

Scenarios​

The test definition built so far sets up the configuration for your performance test. Now it's time to define the steps you want the virtual users to go through during your test.

In an Artillery test definition, you can define multiple scenarios for each virtual user (defined by scenarios.flow), and each scenario will contain one or more operations. This example test definition is for an HTTP-based API, so you can define typical HTTP requests such as GET and POST. Let's write a scenario covering the flow described at the beginning of this section.

Adding a scenario and flow​

To begin defining scenarios, you can add the scenarios setting in your test script. scenarios is an array where you can define one or more scenario definitions. Let's start adding the first definition with an optional name:

scenarios:
  - name: "Search and buy"

The first request in the flow is to tell our virtual users to search for a keyword. The e-commerce API handles this request through the POST /search endpoint. The API endpoint expects a JSON object in the request body. The object should have a key called kw with a value of the keyword searched by the user coming from the external CSV file you defined earlier in the config.payload setting.

Here's how you can define this step in your Artillery test script:

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"

Artillery will generate a POST /search request for the base URL for each generated virtual user when running this performance test. The request's body will contain a JSON object containing a key called kw and sets the value of one of the keywords inside the payload. For instance, one virtual user will make a request with the body of {"kw": "computer"}, and the next virtual user makes a request with the body of {"kw": "video game"}.

Using values from a response​

For our next step in the scenario, we want to make a request to fetch the details of the first result returned by the backend. However, you'll need to look into the response for the ID you need before making that request. Artillery allows you to capture and parse the response of any request in a scenario using the capture setting inside your step definition:

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"
          capture:
            - json: "$.results[0].id"
              as: "productId"

The e-commerce backend under test returns a JSON response containing an array of objects with the search results for the specified keyword. We're only interested in the ID for the first result, so we're parsing the JSON response and storing the id property into a variable called productId, which we can use later in the test.

With the product ID in hand, you can make a subsequent request to fetch the product details. The endpoint for this request in the e-commerce backend is GET /product/:productId/details, where :productId is the product's ID. You already have this information, so you can add the next step in your scenario:

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"
          capture:
            - json: "$.results[0].id"
              as: "productId"
      - get:
          url: "/product/{{ productId }}/details"

This request uses the productId variable that you captured earlier and interpolates it as part of the URL.

Pausing test execution​

Next, let's take a brief pause in the test execution before proceeding. This action will simulate a user spending time on the website after making the previous request to fetch a product's details. This action is straightforward in Artillery with the think setting inside the test definition:

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"
          capture:
            - json: "$.results[0].id"
              as: "productId"
      - get:
          url: "/product/{{ productId }}/details"
      - think: 5

In this example, your test will pause for five seconds after the virtual user makes the GET /product/:productId/details request.

Finally, let's cover the step where the user decides to put the product in their shopping cart. The API handles this action through the POST /cart endpoint. This endpoint requires a productId as part of the body. The request can be made similar to the search step done earlier:

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"
          capture:
            - json: "$.results[0].id"
              as: "productId"
      - get:
          url: "/product/{{ productId }}/details"
      - think: 5
      - post:
          url: "/cart"
          json:
            productId: "{{ productId }}"

In this step, you'll use the same productId captured when performing a search and sending it to the POST /cart endpoint to simulate the "Add to Cart" functionality for the e-commerce site.

Running a performance test​

You've covered a full end-to-end flow for your performance test. Let's put both the config and scenarios together for the complete Artillery test definition:

config:
  target: "https://example.com/api"
  phases:
    - duration: 60
      arrivalRate: 5
      name: Warm up
    - duration: 120
      arrivalRate: 5
      rampTo: 50
      name: Ramp up load
    - duration: 600
      arrivalRate: 50
      name: Sustained load
  payload:
    path: "keywords.csv"
    fields:
      - "keyword"

scenarios:
  - name: "Search and buy"
    flow:
      - post:
          url: "/search"
          json:
            kw: "{{ keyword }}"
          capture:
            - json: "$.results[0].id"
              as: "productId"
      - get:
          url: "/product/{{ productId }}/details"
      - think: 5
      - post:
          url: "/cart"
          json:
            productId: "{{ productId }}"

Save this script in a file called search-and-add-to-cart.yml. If you have installed Artillery, you could run the performance test with the following command:

artillery run search-and-add-to-cart.yml

This command will begin launching virtual users, starting with the first phase in your test definition.

Artillery will print a report on the console for every 10 second time window as the test runs, which includes performance metrics collected in that time window. At the end of the test run, you'll receive a complete summary, including the number of VUs launched and completed, the number of requests completed, response times, status codes, and any errors that may have occurred.

Table of contentsOn this page