publish-metrics
- Monitoring & Observability Integrations
Overview
The artillery-publish-metrics
plugin enables you to send metrics, events, and/or traces from Artillery to a variety of external monitoring and observability systems. This includes native support for popular platforms as well as a versatile OpenTelemetry reporter, which allows data to be sent to any monitoring and observability system that supports OpenTelemetry/OTLP ingestion.
Supported Targets
Native Integrations
- AWS CloudWatch
- Datadog - agent (opens in a new tab) or HTTP API (opens in a new tab)
- New Relic
- Prometheus - via Pushgateway (opens in a new tab)
- Honeycomb events
- Lightstep spans
- Mixpanel events
- InfluxDB - via Telegraf + StatsD plugin (opens in a new tab)
- StatsD or any StatsD-compatible system
OpenTelemetry Support
OpenTelemetry support enables you to send metrics and/or traces to any monitoring and observability platform that supports OTLP (opens in a new tab) by using OTLP exporters (HTTP/JSON
, HTTP/protobuf
, gRPC
) and sending traces to Zipkin (opens in a new tab) by using the Zipkin exporter.
Published Metrics
All metrics generated by Artillery during a test run are sent to the selected monitoring system by default, encompassing both built-in and custom metrics. CloudWatch, Datadog, New Relic, Splunk and OpenTelemetry integrations allow you to customize the list of metrics that will be sent through includeOnly
and excluded
options.
Published Traces and Events
Direct integrations like Honeycomb and Lightstep, as well as the OpenTelemetry reporter, capture traces and events for all HTTP requests.
Published Events
Direct integrations for Datadog, New Relic, Splunk, Dynatrace and InfluxDB/Telegraf allow you to configure sending events that are sent at the start and end of the test.
Usage
CloudWatch
Enable CloudWatch (opens in a new tab) by adding a configuration with type: cloudwatch
and setting the AWS region that you want metrics to be sent to:
config:
plugins:
publish-metrics:
- type: cloudwatch
region: eu-west-1
By default, all Artillery metrics will be sent to CloudWatch. Each Artillery metric will create a
custom CloudWatch metric (opens in a new tab), which incur extra AWS fee for each metric.
You can configure a specific list of metrics to send with the includeOnly
setting (see Configuration section below).
You can learn more about AWS CloudWatch custom metric pricing in this blog post from Vantage.sh: https://www.vantage.sh/blog/cloudwatch-metrics-pricing-explained-in-plain-english (opens in a new tab)
Important: AWS credentials have to be present in the environment and have sufficient IAM permissions to publish CloudWatch metrics.
Configuration options
region
- CloudWatch region where metrics will be sentnamespace
- Metric namespace. Defaults to "artillery".name
- The value of the defaultName
dimension attached to every metric. Defaults to "loadtest".excluded
- A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.includeOnly
- A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.dimensions
- A list of extra dimensions to attache to every metric asname
/value
pairs. Defaults to an empty list.
Example: CloudWatch configuration
The following configuration enables CloudWatch reporting. All metrics generated by the test will be sent to us-east-1
region, and tagged with extra dimensions to specify the team running the tests (SQA), and the name of the service under test (checkout-svc
).
config:
plugins:
publish-metrics:
- type: cloudwatch
region: us-east-1
namespace: continous-testing
dimensions:
- name: Team
value: SQA
- name: Service
value: checkout-svc
Datadog
The plugin supports sending metrics to an already running Datadog agent (opens in a new tab) or directly to Datadog API (opens in a new tab).
If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).
By default, all Artillery metrics will be sent to Datadog. Each Artillery metric will create a custom Datadog metric (opens in a new tab), which will have an associated charge.
You can configure a specific list of metrics to send with the includeOnly
setting (see Configuration section below).
- To send metrics to Datadog, set
type
todatadog
- Set
apiKey
to an API key to send metrics directly to Datadog via its HTTP API, or: - If
apiKey
is not set, metrics will be sent to a Datadog agent:- Set
host
to the hostname/IP of the agent (defaults to127.0.0.1
) - Set
port
to the port that the agent is listening on (defaults to8125
)
- Set
apiHost
-- use this to override the default Datadog endpoint, e.g. to use Datadog EU set the value toapp.datadoghq.eu
(default isapp.datadoghq.com
) Note: this only works whenapiKey
is set, and not when Datadog agent is usedprefix
-- use a prefix for metric names created by Artillery; defaults toartillery.
tags
-- a list ofname:value
strings to use as tags for all metrics sent during a testexcluded
- A list of metric names which should not be sent to Datadog. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.includeOnly
- A list of specific metrics to send to Datadog. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.event
-- send a Datadog event when the test starts/finishestitle
-- set to customize the event title; defaults toArtillery.io Test
+ timestamptext
-- set to customize the event textpriority
--normal
orlow
; defaults tolow
tags
-- a list of event specific tags in thename:value
formatalertType
--error
,warning
,info
orsuccess
; defaults toinfo
send
-- set tofalse
to turn off the event. By default, if anevent
is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
Debugging
Set DEBUG=plugin:publish-metrics:datadog-statsd
when running your tests to print out helpful debugging messages when sending metrics to Datadog.
DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml
Example: Datadog with an agent
config:
plugins:
publish-metrics:
# apiKey not set, so the plugin will assume that the agent is running:
- type: datadog
prefix: "artillery.publish_metrics_plugin."
tags:
- "testId:mytest123"
- "reporterType:datadog-agent"
event:
priority: low
tags:
- "testId:mytest123"
Example: Datadog with an API key
config:
plugins:
publish-metrics:
- type: datadog
# DD_API_KEY is an environment variable containing the API key
apiKey: "{{ $env.DD_API_KEY }}"
prefix: "artillery.publish_metrics_plugin."
tags:
- "testId:mytest123"
- "reporterType:datadog-api"
event:
title: "Plugin integration test"
priority: high
tags:
- "testId:mytest123"
New Relic
To send metrics to New Relic (opens in a new tab), set type
to newrelic
:
By default, all Artillery metrics will be sent to New Relic. Each Artillery metric will create a custom New Relic metric (opens in a new tab), which may have an associated cost.
Configuration options
- To send metrics to New Relic, set
type
tonewrelic
. licenseKey
-- required - The license key for the account you want to send the metrics toregion
--us
(default) oreu
. Thes sets default New Relic endpoint. If your account hosts data in the EU data center set the region toeu
.prefix
-- set a prefix for metric names created by Artillery; defaults toartillery.
attributes
-- a list ofname:value
strings to use as tags for all metrics sent during a testexcluded
-- a list of metric names which should not be sent to New Relic. Defaults to an empty list, i.e. all metrics are sent to New Relic.includeOnly
-- a list of specific metrics to send to New Relic. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to New Relic.event
-- set to send a New Relic event when the test starts/finishes.accountId
-- required - Your New Relic account ID (opens in a new tab).eventType
-- set to customize the event's name, defaults toArtillery_io_Test
. Must be a string that is a combination of alphanumeric characters, underscores, and colons.send
-- set tofalse
to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)attributes
-- optional list ofname:value
strings to use as attributes/tags for events sent during a test. By default Artillery sends thetarget: <target set in the script config>
,timestamp: <timestamp of start/end of test>
andphase: 'Test Started' / 'Test Finished'
attributes. Anyattributes
set will be sent in addition to the default ones. Check character [restrictions] for attributes here (opens in a new tab)
Example: New Relic
config:
plugins:
publish-metrics:
- type: newrelic
licenseKey: "{{ $processEnvironment.NEW_RELIC_LICENSE_KEY }}"
prefix: "artillery."
attributes:
- "type:soak-test"
- "service:my-service"
event:
accountId: "{{ $processEnvironment.NEW_RELIC_ACCOUNT_ID }}"
eventType: "Artillery_load_test"
attributes:
- "alertType:info"
- "priority:low"
- "testId:myTest123"
Debugging
Set DEBUG=plugin:publish-metrics:newrelic
when running your tests to print out helpful debugging messages when sending metrics to New Relic.
DEBUG=plugin:publish-metrics:newrelic artillery run my-script.yaml
Splunk
The plugin supports sending metrics (opens in a new tab) and events (opens in a new tab) to Splunk Observability Cloud (opens in a new tab) via Ingest API.
By default, all Artillery metrics will be sent to Splunk. Each Artillery metric will create a custom Splunk metric, which will have an associated charge.
To send events set and configure event
setting (see Configuration section below)
Configuration options
- To send metrics and/or events to Splunk, set
type
tosplunk
. - Set
accessToken
to your organisation'sINGEST
access token (opens in a new tab) realm
-- use this to override the default Splunk endpoint which is set to theus0
realm. A realm is a self-contained deployment that hosts organizations. You can find your realm name on your profile page in the user interface.prefix
-- use a prefix for metric names created by Artillery; defaults to artillery.dimensions
-- a list ofname:value
strings to use as dimensions for all metrics sent during a test. Dimensions (opens in a new tab) are metadata sent in along with the metrics in the form of key-value pairs. They provide additional information about the metric, such as the name of the host that sent the metric. Check out the type of information (opens in a new tab) suitable for dimensions, and dimensions name requirements (opens in a new tab).excluded
-- A list of metric names which should not be sent to Splunk. Defaults to an empty list, i.e. all metrics are sent to Splunk.includeOnly
-- A list of specific metrics to send to Splunk. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to Splunk.event
-- set to send an event to Splunk when the test starts/finishes.eventType
-- event name. Can not contain any blank splaces(" "). Defaults toArtillery_io_Test
.send
-- set tofalse
to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)dimensions
-- a list ofname:value
strings to use as dimensions for events sent to Splunk. By default Artillery sends thetarget: <target set in the script config>
,timestamp: <timestamp of start/end of test>
andphase: 'Test-Started' / 'Test-Finished'
dimensions. Anydimensions
set in script will be sent in addition to the default ones.properties
-- a list ofname:value
strings to use as properties for events sent to Splunk. See the difference between dimensions and properties here (opens in a new tab)
For information on how to manage data ingested through the Splunk API consult Splunk docs (opens in a new tab).
Example: Splunk
config:
plugins:
publish-metrics:
- type: splunk
realm: eu0
# SP_ACCESS_TOKEN is an environment variable containing the API key
accessToken: "{{ $processEnvironment.SP_ACCESS_TOKEN }}"
prefix: "artillery.publish_metrics_plugin."
dimensions:
- "host:server_1"
- "host_id:1.2.3.4"
event:
eventType: "Artillery_load_test"
dimensions:
- "environment:production"
- "testId:myTest123"
properties:
- "use:QA"
Debugging
Set DEBUG=plugin:publish-metrics:splunk when running your tests to print out helpful debugging messages when sending metrics to Splunk
DEBUG=plugin:publish-metrics:splunk artillery run my-script.yaml
Prometheus (Pushgateway)
Artillery metrics can be sent to Prometheus via Pushgateway (opens in a new tab).
- Set
type
toprometheus
and setpushgateway
to the URL of the Pushgateway instance - Set an optional
prefix
(defaults toartillery
) andtags
Metric names in Prometheus
Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.
This integration creates three metric names (when using the default artillery
prefix):
artillery_counters
artillery_summaries
artillery_rates
Individual Artillery metrics are then made available via the metric
tag. For example:
- The built-in Artillery metric
http.response_time
is of type "summary", i.e. it has the following fields:min
/max
/p95
/p99
- Those values can be looked up in Prometheus through the
artillery_summaries
metric with tagmetric:http_response_time_p95
forp95
Example: Prometheus
config:
plugins:
publish-metrics:
- type: prometheus
pushgateway: "http://kubernetes.docker.internal:9091"
tags:
- "testId:mytest123"
- "type:loadtest"
Dynatrace
The plugin supports sending metrics and events to Dynatrace through Dynatrace Metric and Event API's (v2).
By default, all Artillery metrics will be sent to Dynatrace. Each Artillery metric will create a custom Dynatrace metric, which will have an associated charge.
To send events set and configure event
setting (see Configuration section below)
Configuration options
- To send metrics to Dynatrace, set
type
todynatrace
. apiToken
-- required - The access token (opens in a new tab) must have themetrics.ingest
and/orevents.ingest
scope assigned to it depending on wether you are sending metrics, events or both to Dynatrace.envUrl
-- required - Set to your Dynatrace environment URL. For Managed, Dynatrace for Government or Environment ActiveGate this URL ishttps://{your-domain}/e/{your-environment-id}
( In case of Environment ActiveGate{your-domain}
represents your Activegate domain). ForSaaS
the URL ishttps://{your-environment-id}.live.dynatrace.com
prefix
-- use a prefix for metric names created by Artillery; defaults toartillery.
dimensions
-- a list ofname:value
strings to use as dimensions for all metrics sent during a test.excluded
-- A list of metric names which should not be sent to Dynatrace. Defaults to an empty list, i.e. all metrics are sent to Dynatrace.includeOnly
-- A list of specific metrics to send to Dynatrace. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to Dynatrace.event
-- set to send an event to Dynatrace when the test starts/finishes.eventType
-- the type of the event. It can be one of the values listed here (opens in a new tab). Defaults toCUSTOM_INFO
title
-- the title of the event. Defaults toArtillery_io_Test
.send
-- set tofalse
to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)entitySelector
-- string. The entity selector (opens in a new tab), defining a set of Dynatrace entities to be associated with the event. Only entities that have been active within the last 24 hours can be selected.properties
-- a list ofname:value
strings to use as properties for events sent to Dynatrace. By default Artillery sends theTarget: <target set in the script config>
andPhase: 'Test-Started' / 'Test-Finished'
properties. Anyproperties
set in script will be sent in addition to the default ones.
For information on how to manage data ingested through the Dynatrace Metric API consult Dynatrace docs (opens in a new tab).
Example: Dynatrace
config:
plugins:
publish-metrics:
- type: dynatrace
# DY_API_TOKEN is an environment variable containing the API key
apiToken: "{{ $processEnvironment.DY_API_TOKEN }}"
envUrl: "{{ $processEnvironment.DY_ENVIRONMENT_URL }}"
prefix: "artillery."
dimensions:
- "service:my-service"
- "host_id:1.2.3.4"
event:
title: "Loadtest"
entitySelector: "type(SERVICE),entityName.equals(MyService)"
properties:
- "Tool:Artillery"
- "Load per minute:100"
- "Load pattern:production"
Debugging
Set DEBUG=plugin:publish-metrics:dynatrace when running your tests to print out helpful debugging messages when sending metrics to Dynatrace
DEBUG=plugin:publish-metrics:dynatrace artillery run my-script.yaml
Honeycomb
Honeycomb (opens in a new tab) integration sends an event for every HTTP response (rather than pre-aggregated metrics).
- To send events to Honeycomb, set
type
tohoneycomb
- Set
apiKey
to API/write key - Set
dataset
to the name of a dataset you want to send events to - Optional: set
enabled
tofalse
to disable the integration - Optional: set
sampleRate
to sample rate (default:1
i.e. send all events) (Honeycomb docs (opens in a new tab))
The following properties are set on every event:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:honeycomb
when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.
DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml
Example: Honeycomb
config:
plugins:
publish-metrics:
- type: honeycomb
# HONEYCOMB_API_KEY is an environment variable which contains the API key
apiKey: "{{ $env.HONEYCOMB_API_KEY }}"
dataset: "artillery-test"
sampleRate: 1
Lightstep
Lightstep (opens in a new tab) integration sends a span for every HTTP response (rather than pre-aggregated metrics).
- To send events to Lightstep, set
type
tolightstep
- Set
accessToken
to an access token (opens in a new tab) - Set
componentName
to the name of a component for which the spans will be sent - Default tags for every span may be added by setting key-value pairs under
tags
- Optional: set
enabled
tofalse
to disable the integration
The following tags are set on every span:
url
- full URL of the requesthost
- hostname + portmethod
- HTTP method, e.g.GET
statusCode
- status code, e.g.200
responseTimeMs
- time-to-first-byte of the response in milliseconds
Debugging
Set DEBUG=plugin:publish-metrics:lightstep
when running your tests to print out helpful debugging messages when sending metrics to Lightstep.
DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml
Example: Lightstep
config:
plugins:
publish-metrics:
- type: lightstep
# LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
accessToken: "{{ $env.LIGHTSTEP_ACCESS_TOKEN }}"
component: artillery-tests
tags:
team: sre
purpose: peak-load
Mixpanel
To send data to Mixpanel (opens in a new tab) set type
to mixpanel
, and set the project token (opens in a new tab) with projectToken
.
plugins:
publish-metrics:
- type: mixpanel
projectToken: "my-mixpanel-project-token"
Debugging
Set DEBUG=plugin:publish-metrics:mixpanel
when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.
DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml
StatsD
- To send metrics to StatsD, set
type
tostatsd
- Set
host
andport
to hostname/IP and port of the agent (if different from the default127.0.0.1:8125
) - Set
prefix
to use a custom prefix for metric names created by Artillery; defaults toartillery.
Example: StatsD
config:
plugins:
publish-metrics:
- type: statsd
prefix: "artillery.publish_metrics_plugin."
InfluxDB/Telegraf
- To send metrics to Telegraf (with Telegraf's statsd Service Plugin (opens in a new tab)), set
type
toinfluxdb-statsd
- All other options are the same as for Datadog (other than
apiKey
which is not relevant).
Example: InfluxDB/Telegraf
config:
plugins:
publish-metrics:
- type: influxdb-statsd
prefix: "artillery.publish_metrics_plugin."
tags:
- "testId:mytest123"
- "reporterType:influxdb-statsd"
event:
priority: low
tags:
- "testId:mytest123"
OpenTelemetry
The plugin supports OpenTelemetry (opens in a new tab), and uses its API and SDK's to generate, record and export Artillery's metrics and traces in a standardized way. This way you can send that data directly to any observability and monitoring backend that supports OpenTelemetry, or to your own OpenTelemetry Collector (opens in a new tab).
Currently it supports sending:
- Metrics via OTLP (OpenTelemetry Protocol) exporter with HTTP/JSON, HTTP/protobuf and gRPC;
- Traces via OTLP exporter with HTTP/JSON and HTTP/protobuf, as well as via Zipkin exporter.
Configuration options
To send data using OpenTelemetry, set type
to open-telemetry
.
serviceName
-- resource attribute - logical name of the service. Defaults toArtillery-test
. Used for both metrics and traces;metrics
-- set this option to send metrics:exporter
-- exporter used to send metrics. Defaults tootlp-http
- OTLP exporter with HTTP/JSON. Set tootlp-proto
to use OTLP exporter withHTTP/protobuf
, or tootlp-grpc
to use OTLP exporter withgRPC
;endpoint
* -- set to your chosen monitoring and observability systems' endpoint for ingesting OpenTelemetry metrics. Look for this information in the documentation for your chosen observability system;headers
--key: value
pairs that exporter will add as headers to the outgoing requests. Look for this information in the documentation for your chosen observability system;excluded
-- A list of metric names which should not be sent. Defaults to an empty list, i.e. all metrics are sent;includeOnly
-- A list of specific metrics to send. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent;attributes
--key: value
pairs to be added to all metrics. By default Artillery addsstartTime
andendTime
for each recorded metric marking the time period that the particular metric value represents;
traces
-- set this option to send traces:exporter
-- exporter used to send traces. Defaults tootlp-http
- OTLP exporter with HTTP/JSON. Set tootlp-proto
to use OTLP exporter withHTTP/protobuf
, or tozipkin
to use the Zipkin exporter;endpoint
* -- set to your chosen monitoring and observability systems' endpoint for ingesting OpenTelemetry traces. Look for this information in the documentation for your chosen observability system;headers
--key: value
pairs that exporter will add as headers to the outgoing requests. Look for this information in the documentation for your chosen observability system;sampleRate
-- sample rate (defaults to 1 i.e. send all spans);useRequestNames
-- set totrue
to set span name to requestname
provided in test script, if one is provided (by OTel convention span name is initially set to the{http.request.method}
);attributes
-- attributeskey: value
pairs to add to each span before export in addition to the default ones.
NOTE: The endpoint
is equivalent to the OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
(opens in a new tab) or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
(opens in a new tab) environment variables, so it does not automatically append the /v1/metrics
or /v1/metrics
to the endpoints.
Defaults to http://localhost:4318/v1/metrics
(or /v1/traces
) for HTTP/JSON
and HTTP/protobuf
and to http://localhost:4318
for gRPC
.
When sending tracing data, by default Artillery sets the following on each span:
- the
http_request_started
andhttp_request_ended
events on each span; - the following attributes:
url.full
-- full URL of the request;server.address
-- hostname;server.port
-- port;http.request.method
-- HTTP method e.g.GET
;http.response.status_code
-- HTTP status code e.g.200
;responseTimeMs
-- time-to-first-byte of the response in milliseconds.
Example
publish-metrics:
- type: "open-telemetry"
serviceName: "{{ $processEnvironment.SERVICE_NAME }}"
metrics:
exporter: "otlp-http"
endpoint: "{{ $processEnvironment.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }}"
headers:
Api-Key: "{{ $processEnvironment.API_KEY }}"
attributes:
environment: "test"
traces:
exporter: "otlp-proto"
endpoint: "{{ $processEnvironment.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }}"
headers:
Authorization: "{{ $processEnvironment.API_KEY }}"
useRequestNames: true
attributes:
environment: "test"
tool: "Artillery"
Debugging
Set DEBUG=plugin:publish-metrics:open-telemetry
when running your tests to print out helpful debugging messages when sending metrics and/or traces with OpenTelemetry.
DEBUG=plugin:publish-metrics:open-telemetry artillery run my-script.yaml