`publish-metrics` - Monitoring & Observability Integrations

Overview

The artillery-publish-metrics plugin enables you to send metrics, events, and/or traces from Artillery to a variety of external monitoring and observability systems. This includes native support for popular platforms as well as a versatile OpenTelemetry reporter, which allows data to be sent to any monitoring and observability system that supports OpenTelemetry/OTLP ingestion.

Supported Targets

Native Integrations

AWS CloudWatch
Datadog - agent (opens in a new tab) or HTTP API (opens in a new tab)
New Relic
Prometheus - via Pushgateway (opens in a new tab)
Honeycomb events
Lightstep spans
Mixpanel events
InfluxDB - via Telegraf + StatsD plugin (opens in a new tab)
StatsD or any StatsD-compatible system

OpenTelemetry Support

OpenTelemetry support enables you to send metrics and/or traces to any monitoring and observability platform that supports OTLP (opens in a new tab) by using OTLP exporters (HTTP/JSON, HTTP/protobuf, gRPC) and sending traces to Zipkin (opens in a new tab) by using the Zipkin exporter.

Published Metrics

All metrics generated by Artillery during a test run are sent to the selected monitoring system by default, encompassing both built-in and custom metrics. CloudWatch, Datadog, New Relic, Splunk and OpenTelemetry integrations allow you to customize the list of metrics that will be sent through includeOnly and excluded options.

Published Traces and Events

Direct integrations like Honeycomb and Lightstep, as well as the OpenTelemetry reporter, capture traces and events for all HTTP requests.

Published Events

Direct integrations for Datadog, New Relic, Splunk, Dynatrace and InfluxDB/Telegraf allow you to configure sending events that are sent at the start and end of the test.

Usage

CloudWatch

Enable CloudWatch (opens in a new tab) by adding a configuration with type: cloudwatch and setting the AWS region that you want metrics to be sent to:

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: eu-west-1

By default, all Artillery metrics will be sent to CloudWatch. Each Artillery metric will create a custom CloudWatch metric (opens in a new tab), which incur extra AWS fee for each metric. You can configure a specific list of metrics to send with the includeOnly setting (see Configuration section below).

You can learn more about AWS CloudWatch custom metric pricing in this blog post from Vantage.sh: https://www.vantage.sh/blog/cloudwatch-metrics-pricing-explained-in-plain-english (opens in a new tab)

Important: AWS credentials have to be present in the environment and have sufficient IAM permissions to publish CloudWatch metrics.

Configuration options

region - CloudWatch region where metrics will be sent
namespace - Metric namespace. Defaults to "artillery".
name - The value of the default Name dimension attached to every metric. Defaults to "loadtest".
excluded - A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
includeOnly - A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
dimensions - A list of extra dimensions to attache to every metric as name/value pairs. Defaults to an empty list.

Example: CloudWatch configuration

The following configuration enables CloudWatch reporting. All metrics generated by the test will be sent to us-east-1 region, and tagged with extra dimensions to specify the team running the tests (SQA), and the name of the service under test (checkout-svc).

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: us-east-1
        namespace: continous-testing
        dimensions:
          - name: Team
            value: SQA
          - name: Service
            value: checkout-svc

Datadog

The plugin supports sending metrics to an already running Datadog agent (opens in a new tab) or directly to Datadog API (opens in a new tab).

If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).

By default, all Artillery metrics will be sent to Datadog. Each Artillery metric will create a custom Datadog metric (opens in a new tab), which will have an associated charge.

You can configure a specific list of metrics to send with the includeOnly setting (see Configuration section below).

To send metrics to Datadog, set type to datadog
Set apiKey to an API key to send metrics directly to Datadog via its HTTP API, or:
If apiKey is not set, metrics will be sent to a Datadog agent:
- Set host to the hostname/IP of the agent (defaults to 127.0.0.1)
- Set port to the port that the agent is listening on (defaults to 8125)
apiHost -- use this to override the default Datadog endpoint, e.g. to use Datadog EU set the value to app.datadoghq.eu (default is app.datadoghq.com) Note: this only works when apiKey is set, and not when Datadog agent is used
prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
tags -- a list of name:value strings to use as tags for all metrics sent during a test
excluded - A list of metric names which should not be sent to Datadog. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
includeOnly - A list of specific metrics to send to Datadog. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
event -- send a Datadog event when the test starts/finishes
- title -- set to customize the event title; defaults to Artillery.io Test + timestamp
- text -- set to customize the event text
- priority -- normal or low; defaults to low
- tags -- a list of event specific tags in the name:value format
- alertType -- error, warning, info or success; defaults to info
- send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)

Debugging

Set DEBUG=plugin:publish-metrics:datadog-statsd when running your tests to print out helpful debugging messages when sending metrics to Datadog.

DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml

Example: Datadog with an agent

config:
  plugins:
    publish-metrics:
      # apiKey not set, so the plugin will assume that the agent is running:
      - type: datadog
        prefix: "artillery.publish_metrics_plugin."
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-agent"
        event:
          priority: low
          tags:
            - "testId:mytest123"

Example: Datadog with an API key

config:
  plugins:
    publish-metrics:
      - type: datadog
        # DD_API_KEY is an environment variable containing the API key
        apiKey: "{{ $env.DD_API_KEY }}"
        prefix: "artillery.publish_metrics_plugin."
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-api"
        event:
          title: "Plugin integration test"
          priority: high
          tags:
            - "testId:mytest123"

New Relic

To send metrics to New Relic (opens in a new tab), set type to newrelic:

By default, all Artillery metrics will be sent to New Relic. Each Artillery metric will create a custom New Relic metric (opens in a new tab), which may have an associated cost.

Configuration options

To send metrics to New Relic, set type to newrelic.
licenseKey -- required - The license key for the account you want to send the metrics to
region -- us (default) or eu. Thes sets default New Relic endpoint. If your account hosts data in the EU data center set the region to eu.
prefix -- set a prefix for metric names created by Artillery; defaults to artillery.
attributes -- a list of name:value strings to use as tags for all metrics sent during a test
excluded -- a list of metric names which should not be sent to New Relic. Defaults to an empty list, i.e. all metrics are sent to New Relic.
includeOnly -- a list of specific metrics to send to New Relic. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to New Relic.
event -- set to send a New Relic event when the test starts/finishes.
- accountId -- required - Your New Relic account ID (opens in a new tab).
- eventType -- set to customize the event's name, defaults to Artillery_io_Test. Must be a string that is a combination of alphanumeric characters, underscores, and colons.
- send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
- attributes -- optional list of name:value strings to use as attributes/tags for events sent during a test. By default Artillery sends the target: <target set in the script config>, timestamp: <timestamp of start/end of test> and phase: 'Test Started' / 'Test Finished' attributes. Any attributes set will be sent in addition to the default ones. Check character [restrictions] for attributes here (opens in a new tab)

Example: New Relic

config:
  plugins:
    publish-metrics:
      - type: newrelic
        licenseKey: "{{ $processEnvironment.NEW_RELIC_LICENSE_KEY }}"
        prefix: "artillery."
        attributes:
          - "type:soak-test"
          - "service:my-service"
        event:
          accountId: "{{ $processEnvironment.NEW_RELIC_ACCOUNT_ID }}"
          eventType: "Artillery_load_test"
          attributes:
            - "alertType:info"
            - "priority:low"
            - "testId:myTest123"

Debugging

Set DEBUG=plugin:publish-metrics:newrelic when running your tests to print out helpful debugging messages when sending metrics to New Relic.

DEBUG=plugin:publish-metrics:newrelic artillery run my-script.yaml

Splunk

The plugin supports sending metrics (opens in a new tab) and events (opens in a new tab) to Splunk Observability Cloud (opens in a new tab) via Ingest API.

By default, all Artillery metrics will be sent to Splunk. Each Artillery metric will create a custom Splunk metric, which will have an associated charge.

To send events set and configure event setting (see Configuration section below)

Configuration options

To send metrics and/or events to Splunk, set type to splunk.
Set accessToken to your organisation's INGEST access token (opens in a new tab)
realm -- use this to override the default Splunk endpoint which is set to the us0 realm. A realm is a self-contained deployment that hosts organizations. You can find your realm name on your profile page in the user interface.
prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
dimensions -- a list of name:value strings to use as dimensions for all metrics sent during a test. Dimensions (opens in a new tab) are metadata sent in along with the metrics in the form of key-value pairs. They provide additional information about the metric, such as the name of the host that sent the metric. Check out the type of information (opens in a new tab) suitable for dimensions, and dimensions name requirements (opens in a new tab).
excluded -- A list of metric names which should not be sent to Splunk. Defaults to an empty list, i.e. all metrics are sent to Splunk.
includeOnly -- A list of specific metrics to send to Splunk. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to Splunk.
event -- set to send an event to Splunk when the test starts/finishes.
- eventType -- event name. Can not contain any blank splaces(" "). Defaults to Artillery_io_Test.
- send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
- dimensions -- a list of name:value strings to use as dimensions for events sent to Splunk. By default Artillery sends the target: <target set in the script config>, timestamp: <timestamp of start/end of test> and phase: 'Test-Started' / 'Test-Finished' dimensions. Any dimensions set in script will be sent in addition to the default ones.
- properties -- a list of name:value strings to use as properties for events sent to Splunk. See the difference between dimensions and properties here (opens in a new tab)

For information on how to manage data ingested through the Splunk API consult Splunk docs (opens in a new tab).

Example: Splunk

config:
  plugins:
    publish-metrics:
      - type: splunk
        realm: eu0
        # SP_ACCESS_TOKEN is an environment variable containing the API key
        accessToken: "{{ $processEnvironment.SP_ACCESS_TOKEN }}"
        prefix: "artillery.publish_metrics_plugin."
        dimensions:
          - "host:server_1"
          - "host_id:1.2.3.4"
        event:
          eventType: "Artillery_load_test"
          dimensions:
            - "environment:production"
            - "testId:myTest123"
          properties:
            - "use:QA"

Debugging

Set DEBUG=plugin:publish-metrics:splunk when running your tests to print out helpful debugging messages when sending metrics to Splunk

DEBUG=plugin:publish-metrics:splunk artillery run my-script.yaml

Prometheus (Pushgateway)

Artillery metrics can be sent to Prometheus via Pushgateway (opens in a new tab).

Set type to prometheus and set pushgateway to the URL of the Pushgateway instance
Set an optional prefix (defaults to artillery) and tags

Metric names in Prometheus

Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.

This integration creates three metric names (when using the default artillery prefix):

artillery_counters
artillery_summaries
artillery_rates

Individual Artillery metrics are then made available via the metric tag. For example:

The built-in Artillery metric http.response_time is of type "summary", i.e. it has the following fields: min/max/p95/p99
Those values can be looked up in Prometheus through the artillery_summaries metric with tag metric:http_response_time_p95 for p95

Example: Prometheus

config:
  plugins:
    publish-metrics:
      - type: prometheus
        pushgateway: "http://kubernetes.docker.internal:9091"
        tags:
          - "testId:mytest123"
          - "type:loadtest"

Dynatrace

The plugin supports sending metrics and events to Dynatrace through Dynatrace Metric and Event API's (v2).

By default, all Artillery metrics will be sent to Dynatrace. Each Artillery metric will create a custom Dynatrace metric, which will have an associated charge.

To send events set and configure event setting (see Configuration section below)

Configuration options

To send metrics to Dynatrace, set type to dynatrace.
apiToken -- required - The access token (opens in a new tab) must have the metrics.ingest and/or events.ingest scope assigned to it depending on wether you are sending metrics, events or both to Dynatrace.
envUrl -- required - Set to your Dynatrace environment URL. For Managed, Dynatrace for Government or Environment ActiveGate this URL is https://{your-domain}/e/{your-environment-id} ( In case of Environment ActiveGate {your-domain} represents your Activegate domain). For SaaS the URL is https://{your-environment-id}.live.dynatrace.com
prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
dimensions -- a list of name:value strings to use as dimensions for all metrics sent during a test.
excluded -- A list of metric names which should not be sent to Dynatrace. Defaults to an empty list, i.e. all metrics are sent to Dynatrace.
includeOnly -- A list of specific metrics to send to Dynatrace. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to Dynatrace.
event -- set to send an event to Dynatrace when the test starts/finishes.
- eventType -- the type of the event. It can be one of the values listed here (opens in a new tab). Defaults to CUSTOM_INFO
- title -- the title of the event. Defaults to Artillery_io_Test.
- send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)
- entitySelector -- string. The entity selector (opens in a new tab), defining a set of Dynatrace entities to be associated with the event. Only entities that have been active within the last 24 hours can be selected.
- properties -- a list of name:value strings to use as properties for events sent to Dynatrace. By default Artillery sends the Target: <target set in the script config> and Phase: 'Test-Started' / 'Test-Finished' properties. Any properties set in script will be sent in addition to the default ones.

For information on how to manage data ingested through the Dynatrace Metric API consult Dynatrace docs (opens in a new tab).

Example: Dynatrace

config:
  plugins:
    publish-metrics:
      - type: dynatrace
        # DY_API_TOKEN is an environment variable containing the API key
        apiToken: "{{ $processEnvironment.DY_API_TOKEN }}"
        envUrl: "{{ $processEnvironment.DY_ENVIRONMENT_URL }}"
        prefix: "artillery."
        dimensions:
          - "service:my-service"
          - "host_id:1.2.3.4"
        event:
          title: "Loadtest"
          entitySelector: "type(SERVICE),entityName.equals(MyService)"
          properties:
            - "Tool:Artillery"
            - "Load per minute:100"
            - "Load pattern:production"

Debugging

Set DEBUG=plugin:publish-metrics:dynatrace when running your tests to print out helpful debugging messages when sending metrics to Dynatrace

DEBUG=plugin:publish-metrics:dynatrace artillery run my-script.yaml

Honeycomb

Honeycomb (opens in a new tab) integration sends an event for every HTTP response (rather than pre-aggregated metrics).

To send events to Honeycomb, set type to honeycomb
Set apiKey to API/write key
Set dataset to the name of a dataset you want to send events to
Optional: set enabled to false to disable the integration
Optional: set sampleRate to sample rate (default: 1 i.e. send all events) (Honeycomb docs (opens in a new tab))

The following properties are set on every event:

url - full URL of the request
host - hostname + port
method - HTTP method, e.g. GET
statusCode - status code, e.g. 200
responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging

Set DEBUG=plugin:publish-metrics:honeycomb when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.

DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml

Example: Honeycomb

config:
  plugins:
    publish-metrics:
      - type: honeycomb
        # HONEYCOMB_API_KEY is an environment variable which contains the API key
        apiKey: "{{ $env.HONEYCOMB_API_KEY }}"
        dataset: "artillery-test"
        sampleRate: 1

Lightstep

Lightstep (opens in a new tab) integration sends a span for every HTTP response (rather than pre-aggregated metrics).

To send events to Lightstep, set type to lightstep
Set accessToken to an access token (opens in a new tab)
Set componentName to the name of a component for which the spans will be sent
Default tags for every span may be added by setting key-value pairs under tags
Optional: set enabled to false to disable the integration

The following tags are set on every span:

url - full URL of the request
host - hostname + port
method - HTTP method, e.g. GET
statusCode - status code, e.g. 200
responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging

Set DEBUG=plugin:publish-metrics:lightstep when running your tests to print out helpful debugging messages when sending metrics to Lightstep.

DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml

Example: Lightstep

config:
  plugins:
    publish-metrics:
      - type: lightstep
        # LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
        accessToken: "{{ $env.LIGHTSTEP_ACCESS_TOKEN }}"
        component: artillery-tests
        tags:
          team: sre
          purpose: peak-load

Mixpanel

To send data to Mixpanel (opens in a new tab) set type to mixpanel, and set the project token (opens in a new tab) with projectToken.

plugins:
  publish-metrics:
    - type: mixpanel
      projectToken: "my-mixpanel-project-token"

Debugging

Set DEBUG=plugin:publish-metrics:mixpanel when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.

DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml

StatsD

To send metrics to StatsD, set type to statsd
Set host and port to hostname/IP and port of the agent (if different from the default 127.0.0.1:8125)
Set prefix to use a custom prefix for metric names created by Artillery; defaults to artillery.

Example: StatsD

config:
  plugins:
    publish-metrics:
      - type: statsd
        prefix: "artillery.publish_metrics_plugin."

InfluxDB/Telegraf

To send metrics to Telegraf (with Telegraf's statsd Service Plugin (opens in a new tab)), set type to influxdb-statsd
All other options are the same as for Datadog (other than apiKey which is not relevant).

Example: InfluxDB/Telegraf

config:
  plugins:
    publish-metrics:
      - type: influxdb-statsd
        prefix: "artillery.publish_metrics_plugin."
        tags:
          - "testId:mytest123"
          - "reporterType:influxdb-statsd"
        event:
          priority: low
          tags:
            - "testId:mytest123"

OpenTelemetry

The plugin supports OpenTelemetry (opens in a new tab), and uses its API and SDK's to generate, record and export Artillery's metrics and traces in a standardized way. This way you can send that data directly to any observability and monitoring backend that supports OpenTelemetry, or to your own OpenTelemetry Collector (opens in a new tab).

Currently it supports sending:

Metrics via OTLP (OpenTelemetry Protocol) exporter with HTTP/JSON, HTTP/protobuf and gRPC;
Traces via OTLP exporter with HTTP/JSON and HTTP/protobuf, as well as via Zipkin exporter.

Configuration options

To send data using OpenTelemetry, set type to open-telemetry.

serviceName -- resource attribute - logical name of the service. Defaults to Artillery-test. Used for both metrics and traces;
metrics -- set this option to send metrics:
- exporter -- exporter used to send metrics. Defaults to otlp-http - OTLP exporter with HTTP/JSON. Set to otlp-proto to use OTLP exporter with HTTP/protobuf, or to otlp-grpc to use OTLP exporter with gRPC;
- endpoint* -- set to your chosen monitoring and observability systems' endpoint for ingesting OpenTelemetry metrics. Look for this information in the documentation for your chosen observability system;
- headers -- key: value pairs that exporter will add as headers to the outgoing requests. Look for this information in the documentation for your chosen observability system;
- excluded -- A list of metric names which should not be sent. Defaults to an empty list, i.e. all metrics are sent;
- includeOnly -- A list of specific metrics to send. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent;
- attributes -- key: value pairs to be added to all metrics. By default Artillery adds startTime and endTime for each recorded metric marking the time period that the particular metric value represents;
traces -- set this option to send traces:
- exporter -- exporter used to send traces. Defaults to otlp-http - OTLP exporter with HTTP/JSON. Set to otlp-proto to use OTLP exporter with HTTP/protobuf, or to zipkin to use the Zipkin exporter;
- endpoint* -- set to your chosen monitoring and observability systems' endpoint for ingesting OpenTelemetry traces. Look for this information in the documentation for your chosen observability system;
- headers -- key: value pairs that exporter will add as headers to the outgoing requests. Look for this information in the documentation for your chosen observability system;
- sampleRate -- sample rate (defaults to 1 i.e. send all spans);
- useRequestNames -- set to true to set span name to request name provided in test script, if one is provided (by OTel convention span name is initially set to the {http.request.method});
- attributes -- attributes key: value pairs to add to each span before export in addition to the default ones.

NOTE: The endpoint is equivalent to the OTEL_EXPORTER_OTLP_METRICS_ENDPOINT (opens in a new tab) or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (opens in a new tab) environment variables, so it does not automatically append the /v1/metrics or /v1/metrics to the endpoints.

Defaults to http://localhost:4318/v1/metrics (or /v1/traces) for HTTP/JSON and HTTP/protobuf and to http://localhost:4318 for gRPC.

When sending tracing data, by default Artillery sets the following on each span:

the http_request_started and http_request_ended events on each span;
the following attributes:
- url.full -- full URL of the request;
- server.address -- hostname;
- server.port -- port;
- http.request.method -- HTTP method e.g. GET;
- http.response.status_code -- HTTP status code e.g. 200;
- responseTimeMs -- time-to-first-byte of the response in milliseconds.

Example

publish-metrics:
  - type: "open-telemetry"
    serviceName: "{{ $processEnvironment.SERVICE_NAME }}"
    metrics:
      exporter: "otlp-http"
      endpoint: "{{ $processEnvironment.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }}"
      headers:
        Api-Key: "{{ $processEnvironment.API_KEY }}"
      attributes:
        environment: "test"
    traces:
      exporter: "otlp-proto"
      endpoint: "{{ $processEnvironment.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }}"
      headers:
        Authorization: "{{ $processEnvironment.API_KEY }}"
      useRequestNames: true
      attributes:
        environment: "test"
        tool: "Artillery"

Debugging

Set DEBUG=plugin:publish-metrics:open-telemetry when running your tests to print out helpful debugging messages when sending metrics and/or traces with OpenTelemetry.

DEBUG=plugin:publish-metrics:open-telemetry artillery run my-script.yaml

Extensions & Plugins expect

publish-metrics - Monitoring & Observability Integrations

Overview

Supported Targets

Native Integrations

OpenTelemetry Support

Published Metrics

Published Traces and Events

Published Events

Usage

CloudWatch

Configuration options

Example: CloudWatch configuration

Datadog

Debugging

Example: Datadog with an agent

Example: Datadog with an API key

New Relic

Configuration options

Example: New Relic

Debugging

Splunk

Configuration options

Example: Splunk

Debugging

Prometheus (Pushgateway)

Metric names in Prometheus

Example: Prometheus

Dynatrace

Configuration options

Example: Dynatrace

Debugging

Honeycomb

Debugging

Example: Honeycomb

Lightstep

Debugging

Example: Lightstep

Mixpanel

Debugging

StatsD

Example: StatsD

InfluxDB/Telegraf

Example: InfluxDB/Telegraf

OpenTelemetry

Configuration options

Example

Debugging

`publish-metrics` - Monitoring & Observability Integrations