Expectations & Assertions
What you'll learn
- How to add assertions and expectations to HTTP scenarios with
artillery-plugin-expect
plugin - How to use the plugin to run functional tests for your API with Artillery
Overview
The expect plugin turns Artillery into a smoke / functional testing tool for HTTP APIs.
Features
- Use expectations and assertions in your HTTP scenarios
- Use the same
artillery
command to run functional / acceptance tests on your APIs - See details of any failed assertions (request headers & body, response etc)
- Run post-deployment smoke tests from CI/CD pipelines
Usage
Enable the plugin
To enable the plugin, add it to the config
section of a test script.
config:
target: "http://lab.artillery.io"
plugins:
expect: {}
Add expectations to HTTP requests
scenarios:
- name: Get a movie
flow:
- get:
url: "/movies/5"
capture:
- json: "$.title"
as: title
expect:
- statusCode: 200
- contentType: json
- hasProperty: title
- equals:
- "From Dusk Till Dawn"
- "{{ title }}"
Run the test
Run your script that uses expectations with:
artillery run --quiet my-script.yaml
(The --quiet
option is to stop Artillery from printing its default reports to the console.)
Run load + functional tests
This plugin allows for the same scenario to be re-used for either load testing or functional testing of an API, with the help of config.environments
definition in your test script.
(The only real difference between the two, of course, is how many virtual users you run -- only one for functional tests, and lots of virtual users for a load test.)
config:
target: "https://my.api.internal"
environments:
#
# This is our load testing profile, where we create a lot of virtual users.
# Note that we don't load the plugin here, so that we don't see the output
# from the plugin.
#
load:
phases:
- duration: 600
arrivalRate: 10
#
# This is our functional testing profile, with a single virtual user, and
# the plugin enabled.
#
functional:
phases:
- duration: 1
arrivalCount: 1
plugins:
expect: {}
scenarios:
# Your scenario definitions go here.
You could run the script above in load testing mode with:
artillery run --environment load script.yaml
and in functional testing mode with:
artillery run --environment functional script.yaml
Configuration
Formatters
The plugin supports a number of different output formats. The formatter may be set with the outputFormat
config option.
The following formatters are available:
pretty
(default) - prints HTTP method, URL for each request, along with each expectationprettyError
- only prints details of failed expectationsjson
- provides newline-delimited JSON outputsilent
- suppresses all output from the plugin (NOTE: this formatter is only available in v2 of the plugin)
Metrics
Metrics support is available in v2 or later of the plugin.
The plugin tracks successful and failed expectations with custom metrics.
expect.ok
- count of all expectations that have been successfulexpect.failed
- count of all expectations that failedexpect.ok.{type}
- count of successful expectations of{type}
, for example:expect.ok.statusCode
forstatusCode
expectationsexpect.failed.{type}
- count of failed expectations of{type}
, for example:expect.failed.statusCode
forstatusCode
expectations
Expectations
statusCode
Check that the status code of the response.
expect:
- statusCode: 201
The value may also be a list to indicate that any of the codes in the list is OK:
expect:
- statusCode:
- 200
- 301
notStatusCode
This check succeeds if the status code is anything but the list of codes specified.
For example, this expectation succeeds when the response status code is anything but 403
or 500
:
expect:
- notStatusCode:
- 403
- 500
contentType
Check the value of Content-Type
header.
hasProperty
and notHasProperty
When the response is JSON, check that the response object has a property (or does not in case of notHasProperty
). Same behavior as lodash#has
.
expect:
- hasProperty: 'data[0].id'
equals
Check that two or more values are the same. NOTE only primitive values (e.g. booleans, strings and numbers) are currently supported.
- get:
url: "/pets/f037ed9a"
capture:
- json: "$.species"
as: species
expect:
- equals:
- "{{ species }}"
- "dog"
hasHeader
Check that the response contains a header.
- get:
url: "/pets/f037ed9a"
expect:
- hasHeader: "object-version"
headerEquals
Check that the response contains a header and its value matches some string.
- get:
url: "/pets/f037ed9a"
expect:
- headerEquals:
- "object-version"
- "v3"
Multiple headers with the same name can also be matched. For example, if we expect a request to return two Set-Cookie
headers with some known values we can check that they match like this:
- get:
url: "/protected/url"
expect:
- headerEquals:
# Expect two Set-Cookie headers returned by this request,
# in a particular order:
- set-cookie
-
- platform=astmh;version=1
- status=draft;version=1
matchesRegexp
Check that response body matches a regular expression. The regular expression provided must be a string which is a valid argument to the RegExp constructor.
- get:
url: "/pets/f037ed9a"
expect:
- matchesRegexp: .+ # body is not empty
cdnHit
This expectation checks for presence of a cache hit/miss header from a CDN. The following CDNs are supported:
- Cloudflare
- CloudFront
- Fastly
- Vercel
The check will succeed if:
- A CDN cache hit/miss header is present in the response (one of
cf-cache-status
,x-cache
,x-vercel-cache
) - The value of the header is either
hit
orstale
- get:
url: "/pets/f037ed9a"
expect:
cdnHit: true
Debugging
If you're having issues with the plugin, you can print out helpful debugging messages using the DEBUG
environment variable.
Print expectations and values
Set DEBUG=plugin:expect
when running your tests to view when using an expectation and any values to check.
DEBUG=plugin:expect artillery run my-script.yaml
More Information
The source code for artillery-plugin-expect
is available on Github at https://github.com/artilleryio/artillery-plugin-expect. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.