Publishing Metrics / Monitoring
What you'll learn
- How to publish metrics from your Artillery tests to external monitoring and observability systems
- How to configure metrics and traces for CloudWatch, Datadog, Influx, Honeycomb, Lightstep, and Statsd
Overview
The official artillery-publish-metrics
plugin lets you send metrics and events/traces from Artillery to external monitoring and observability systems.
Supported targets
- AWS CloudWatch
- Datadog - agent or HTTP API
- New Relic
- Prometheus - via Pushgateway
- Honeycomb events
- Lightstep spans
- Mixpanel events
- InfluxDB - via Telegraf + StatsD plugin
- StatsD or any StatsD-compatible system
We are planning to add more targets (such as Splunk, ELK, New Relic) in future.
Published metrics
All metrics generated by Artillery during a test run will be sent to target monitoring system (Datadog, Prometheus, InfluxDB and StatsD) by default. That includes both built-in and custom metrics.
CloudWatch and Datadog integrations allow you to customize the list of metrics that will be sent through includeOnly
and excluded
options.
Published traces
Honeycomb and Lightstep integrations send events/traces for all HTTP requests, with the following tags:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Usage
CloudWatch
Enable CloudWatch by adding a configuration with type: cloudwatch
and setting the AWS region that you want metrics to be sent to:
config:
plugins:
publish-metrics:
- type: cloudwatch
region: eu-west-1
By default, all Artillery metrics will be sent to CloudWatch. Each Artillery metric will create a
custom CloudWatch metric, which incur extra AWS fee for each metric.
You can configure a specific list of metrics to send with the includeOnly
setting (see Configuration section below).
You can learn more about AWS CloudWatch custom metric pricing in this blog post from Vantage.sh: https://www.vantage.sh/blog/cloudwatch-metrics-pricing-explained-in-plain-english
Important: AWS credentials have to be present in the environment and have sufficient IAM permissions to publish CloudWatch metrics.
Configuration options
region
- CloudWatch region where metrics will be sentnamespace
- Metric namespace. Defaults to "artillery".name
- The value of the defaultName
dimension attached to every metric. Defaults to "loadtest".excluded
- A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.includeOnly
- A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.dimensions
- A list of extra dimensions to attache to every metric asname
/value
pairs. Defaults to an empty list.
Example: CloudWatch configuration
The following configuration enables CloudWatch reporting. All metrics generated by the test will be sent to us-east-1
region, and tagged with extra dimensions to specify the team running the tests (SQA), and the name of the service under test (checkout-svc
).
config:
plugins:
publish-metrics:
- type: cloudwatch
region: us-east-1
namespace: continous-testing
dimensions:
- name: Team
value: SQA
- name: Service
value: checkout-svc
Datadog
The plugin supports sending metrics to an already running Datadog agent or directly to Datadog API.
If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).
By default, all Artillery metrics will be sent to Datadog. Each Artillery metric will create a custom Datadog metric, which will have an associated charge.
You can configure a specific list of metrics to send with the includeOnly
setting (see Configuration section below).
- To send metrics to Datadog, set
type
todatadog
- Set
apiKey
to an API key to send metrics directly to Datadog via its HTTP API, or: - If
apiKey
is not set, metrics will be sent to a Datadog agent:- Set
host
to the hostname/IP of the agent (defaults to127.0.0.1
) - Set
port
to the port that the agent is listening on (defaults to8125
)
- Set
apiHost
-- use this to override the default Datadog endpoint, e.g. to use Datadog EU set the value toapp.datadoghq.eu
(default isapp.datadoghq.com
) Note: this only works whenapiKey
is set, and not when Datadog agent is usedprefix
-- use a prefix for metric names created by Artillery; defaults toartillery.
tags
-- a list ofname:value
strings to use as tags for all metrics sent during a testexcluded
- A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.includeOnly
- A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.event
-- send a Datadog event when the test starts/finishestitle
-- set to customize the event title; defaults toArtillery.io Test
+ timestamptext
-- set to customize the event textpriority
--normal
orlow
; defaults tolow
tags
-- a list of event specific tags in thevalue:name
formatalertType
--error
,warning
,info
orsuccess
; defaults toinfo
send
-- set tofalse
to turn off the event. By default, if anevent
is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
Debugging
Set DEBUG=plugin:publish-metrics:datadog-statsd
when running your tests to print out helpful debugging messages when sending metrics to Datadog.
DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml
Example: Datadog with an agent
config:
plugins:
publish-metrics:
# apiKey not set, so the plugin will assume that the agent is running:
- type: datadog
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:datadog-agent"
event:
priority: low
tags:
- "testId:mytest123"
Example: Datadog with an API key
config:
plugins:
publish-metrics:
- type: datadog
# DD_API_KEY is an environment variable containing the API key
apiKey: "{{ $processEnvironment.DD_API_KEY }}"
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:datadog-api"
event:
title: "Plugin integration test"
priority: high
tags:
- "testId:mytest123"
New Relic
To send metrics to New Relic, set type
to newrelic
:
config:
plugins:
publish-metrics:
- type: newrelic
apiKey: "{{ $processEnvironment.NEW_RELIC_LICENSE_KEY }}"
prefix: 'artillery.'
attributes:
- "type:soak-test"
- "service:my-service"
By default, all Artillery metrics will be sent to New Relic. Each Artillery metric will create a custom New Relic metric, which may have an associated cost.
Configuration options
- To send metrics to New Relic, set
type
tonewrelic
. - Set
licenseKey
to the license key for the account you want to send the metrics to region
--us
(default) oreu
. Thes sets default New Relic endpoint. If your account hosts data in the EU data center set the region toeu
.prefix
-- set a prefix for metric names created by Artillery; defaults toartillery.
attributes
-- a list of name:value strings to use as tags for all metrics sent during a testexcluded
-- a list of metric names which should not be sent to New Relic. Defaults to an empty list, i.e. all metrics are sent to New Relic.includeOnly
-- a list of specific metrics to send to New Relic. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to New Relic.
Prometheus (Pushgateway)
Artillery metrics can be sent to Prometheus via Pushgateway.
- Set
type
toprometheus
and setpushgateway
to the URL of the Pushgateway instance - Set an optional
prefix
(defaults toartillery
) andtags
Metric names in Prometheus
Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.
This integration creates three metric names (when using the default artillery
prefix):
artillery_counters
artillery_summaries
artillery_rates
Individual Artillery metrics are then made available via the metric
tag. For example:
- The built-in Artillery metric
http.response_time
is of type "summary", i.e. it has the following fields:min
/max
/p95
/p99
- Those values can be looked up in Prometheus through the
artillery_summaries
metric with tagmetric:http_response_time_p95
forp95
Example: Prometheus
config:
plugins:
publish-metrics:
- type: prometheus
pushgateway: "http://kubernetes.docker.internal:9091"
tags:
- "testId:mytest123"
- "type:loadtest"
Honeycomb
Honeycomb integration sends an event for every HTTP response (rather than pre-aggregated metrics).
- To send events to Honeycomb, set
type
tohoneycomb
- Set
apiKey
to API/write key - Set
dataset
to the name of a dataset you want to send events to - Optional: set
enabled
tofalse
to disable the integration - Optional: set
sampleRate
to sample rate (default:1
i.e. send all events) (Honeycomb docs)
The following properties are set on every event:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:honeycomb
when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.
DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml
Example: Honeycomb
config:
plugins:
publish-metrics:
- type: honeycomb
# HONEYCOMB_API_KEY is an environment variable which contains the API key
apiKey: "{{ $processEnvironment.HONEYCOMB_API_KEY }}"
dataset: "artillery-test"
sampleRate: 1
Lightstep
Lightstep integration sends a span for every HTTP response (rather than pre-aggregated metrics).
- To send events to Lightstep, set
type
tolightstep
- Set
accessToken
to an access token - Set
componentName
to the name of a component for which the spans will be sent - Default tags for every span may be added by setting key-value pairs under
tags
- Optional: set
enabled
tofalse
to disable the integration
The following tags are set on every span:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:lightstep
when running your tests to print out helpful debugging messages when sending metrics to Lightstep.
DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml
Example: Lightstep
config:
plugins:
publish-metrics:
- type: lightstep
# LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
accessToken: "{{ $processEnvironment.LIGHTSTEP_ACCESS_TOKEN }}"
component: artillery-tests
tags:
team: sre
purpose: peak-load
Mixpanel
Set type
to mixpanel
, and set the project token with projectToken
.
plugins:
publish-metrics:
- type: mixpanel
projectToken: "my-mixpanel-project-token"
Debugging
Set DEBUG=plugin:publish-metrics:mixpanel
when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.
DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml
StatsD
- To send metrics to StatsD, set
type
tostatsd
- Set
host
andport
to hostname/IP and port of the agent (if different from the default127.0.0.1:8125
) - Set
prefix
to use a custom prefix for metric names created by Artillery; defaults toartillery.
Example: StatsD
config:
plugins:
publish-metrics:
- type: statsd
prefix: 'artillery.publish_metrics_plugin.'
InfluxDB/Telegraf
- To send metrics to Telegraf (with Telegraf's statsd Service Plugin), set
type
toinfluxdb-statsd
- All other options are the same as for Datadog (other than
apiKey
which is not relevant).
Example: InfluxDB/Telegraf
config:
plugins:
publish-metrics:
- type: influxdb-statsd
prefix: 'artillery.publish_metrics_plugin.'
tags:
- "testId:mytest123"
- "reporterType:influxdb-statsd"
event:
priority: low
tags:
- "testId:mytest123"
More Information
The source code for artillery-plugin-expect
is available on Github at https://github.com/artilleryio/artillery-plugin-publish-metrics. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.