Publishing Metrics / Monitoring
What you'll learn
- How to publish metrics from your Artillery tests to external monitoring and observability systems
- How to install the official
publish-metrics
plugin - How to configure metrics and traces for Datadog, Influx, Honeycomb, Lightstep, and Statsd
Overview
The official artillery-publish-metrics
plugin lets you send metrics and events/traces from Artillery to external monitoring and observability systems.
Supported targets
- Datadog metrics via agent or HTTP API
- Prometheus metrics via Pushgateway
- Honeycomb events
- Lightstep spans
- Mixpanel events
- InfluxDB metrics with Telegraf + StatsD plugin
- StatsD metrics
We are working on adding more targets (such as CloudWatch, Splunk, ELK and Prometheus). You can follow the plugin's development on Github.
Published metrics
All metrics generated by Artillery during a test run will be sent to target monitoring system (Datadog, Prometheus, InfluxDB and StatsD).
That includes both built-in and custom metrics.
Published traces
Honeycomb and Lightstep integrations send events/traces for all HTTP requests, with the following tags:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Usage
Install the plugin <=v2.0.0-12
tip
The plugin has been included into Artillery itself starting with version 2.0.0-13 and does not need to be installed separately.
To check you version of Artillery run artillery version
The plugin needs to be installed in the same scope (globally or as a project-specific dependency) as Artillery.
If you installed Artillery globally (i.e. with npm install -g artillery
) then the plugin needs to be installed globally too:
npm install -g artillery-plugin-publish-metrics
If artillery
is installed as a project-specific dependency (i.e. in a directory with package.json
in it), install it with:
t
npm install artillery-plugin-publish-metrics
Configure a target for metrics/traces
Datadog
The plugin supports sending metrics to an already running Datadog agent or directly to Datadog API.
If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).
- To send metrics to Datadog, set
type
todatadog
- Set
apiKey
to an API key to send metrics directly to Datadog via its HTTP API, or: - If
apiKey
is not set, metrics will be sent to a Datadog agent:- Set
host
to the hostname/IP of the agent (defaults to127.0.0.1
) - Set
port
to the port that the agent is listening on (defaults to8125
)
- Set
prefix
-- use a prefix for metric names created by Artillery; defaults toartillery.
tags
-- a list ofname:value
strings to use as tags for all metrics sent during a testevent
-- send a Datadog event when the test starts/finishestitle
-- set to customize the event title; defaults toArtillery.io Test
+ timestamptext
-- set to customize the event textpriority
--normal
orlow
; defaults tolow
tags
-- a list of event specific tags in thevalue:name
formatalertType
--error
,warning
,info
orsuccess
; defaults toinfo
send
-- set tofalse
to turn off the event. By default, if anevent
is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
Debugging
Set DEBUG=plugin:publish-metrics:datadog-statsd
when running your tests to print out helpful debugging messages when sending metrics to Datadog.
DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml
Example: Datadog with an agent
config:
plugins:
publish-metrics:
# apiKey not set, so the plugin will assume that the agent is running:
- type: datadog
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:datadog-agent"
event:
priority: low
tags:
- "testId:mytest123"
Example: Datadog with an API key
config:
plugins:
publish-metrics:
- type: datadog
# DD_API_KEY is an environment variable containing the API key
apiKey: "{{ $processEnvironment.DD_API_KEY }}"
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:datadog-api"
event:
title: "Plugin integration test"
priority: high
tags:
- "testId:mytest123"
Prometheus (Pushgateway)
Artillery metrics can be sent to Prometheus via Pushgateway.
- Set
type
toprometheus
and setpushgateway
to the URL of the Pushgateway instance - Set an optional
prefix
(defaults toartillery
) andtags
Metric names in Prometheus
Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.
This integration creates three metric names (when using the default artillery
prefix):
artillery_counters
artillery_summaries
artillery_rates
Individual Artillery metrics are then made available via the metric
tag. For example:
- The built-in Artillery metric
http.response_time
is of type "summary", i.e. it has the following fields:min
/max
/p95
/p99
- Those values can be looked up in Prometheus through the
artillery_summaries
metric with tagmetric:http_response_time_p95
forp95
Example: Prometheus
config:
plugins:
publish-metrics:
- type: prometheus
pushgateway: "http://kubernetes.docker.internal:9091"
tags:
- "testId:mytest123"
- "type:loadtest"
Honeycomb
Honeycomb integration sends an event for every HTTP response (rather than pre-aggregated metrics).
- To send events to Honeycomb, set
type
tohoneycomb
- Set
apiKey
to API/write key - Set
dataset
to the name of a dataset you want to send events to - Optional: set
enabled
tofalse
to disable the integration - Optional: set
sampleRate
to sample rate (default:1
i.e. send all events) (Honeycomb docs)
The following properties are set on every event:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:honeycomb
when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.
DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml
Example: Honeycomb
config:
plugins:
publish-metrics:
- type: honeycomb
# HONEYCOMB_API_KEY is an environment variable which contains the API key
apiKey: "{{ $processEnvironment.HONEYCOMB_API_KEY }}"
dataset: "artillery-test"
sampleRate: 1
Lightstep
Lightstep integration sends a span for every HTTP response (rather than pre-aggregated metrics).
- To send events to Lightstep, set
type
tolightstep
- Set
accessToken
to an access token - Set
componentName
to the name of a component for which the spans will be sent - Default tags for every span may be added by setting key-value pairs under
tags
- Optional: set
enabled
tofalse
to disable the integration
The following tags are set on every span:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:lightstep
when running your tests to print out helpful debugging messages when sending metrics to Lightstep.
DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml
Example: Lightstep
config:
plugins:
publish-metrics:
- type: lightstep
# LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
accessToken: "{{ $processEnvironment.LIGHTSTEP_ACCESS_TOKEN }}"
component: artillery-tests
tags:
team: sre
purpose: peak-load
Mixpanel
Set type
to mixpanel
, and set the project token with projectToken
.
plugins:
publish-metrics:
- type: mixpanel
projectToken: "my-mixpanel-project-token"
Debugging
Set DEBUG=plugin:publish-metrics:mixpanel
when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.
DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml
StatsD
- To send metrics to StatsD, set
type
tostatsd
- Set
host
andport
to hostname/IP and port of the agent (if different from the default127.0.0.1:8125
) - Set
prefix
to use a custom prefix for metric names created by Artillery; defaults toartillery.
Example: StatsD
config:
plugins:
publish-metrics:
- type: statsd
prefix: 'artillery.publish_metrics_plugin.'
InfluxDB/Telegraf
- To send metrics to Telegraf (with Telegraf's statsd Service Plugin), set
type
toinfluxdb-statsd
- All other options are the same as for Datadog (other than
apiKey
which is not relevant).
Example: InfluxDB/Telegraf
config:
plugins:
publish-metrics:
- type: influxdb-statsd
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:influxdb-statsd"
event:
priority: low
tags:
- "testId:mytest123"
More Information
The source code for artillery-plugin-expect
is available on Github at https://github.com/artilleryio/artillery-plugin-publish-metrics. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.