Skip to main content

Publishing Metrics / Monitoring

What you'll learn
  • How to publish metrics from your Artillery tests to external monitoring and observability systems
  • How to install the official publish-metrics plugin
  • How to configure metrics and traces for Datadog, Influx, Honeycomb, Lightstep, and Statsd

Overview​

The official artillery-publish-metrics plugin lets you send metrics and events/traces from Artillery to external monitoring and observability systems.

Supported targets​

We are working on adding more targets (such as CloudWatch, Splunk, ELK and Prometheus). You can follow the plugin's development on Github.

Published metrics​

All metrics generated by Artillery during a test run will be sent to target monitoring system (Datadog, Prometheus, InfluxDB and StatsD).

That includes both built-in and custom metrics.

Published traces​

Honeycomb and Lightstep integrations send events/traces for all HTTP requests, with the following tags:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Usage​

Install the plugin <=v2.0.0-12​

tip

The plugin has been included into Artillery itself starting with version 2.0.0-13 and does not need to be installed separately.

To check you version of Artillery run artillery version

The plugin needs to be installed in the same scope (globally or as a project-specific dependency) as Artillery.

If you installed Artillery globally (i.e. with npm install -g artillery) then the plugin needs to be installed globally too:

npm install -g artillery-plugin-publish-metrics

If artillery is installed as a project-specific dependency (i.e. in a directory with package.json in it), install it with: t

npm install artillery-plugin-publish-metrics

Configure a target for metrics/traces​

Datadog​

The plugin supports sending metrics to an already running Datadog agent or directly to Datadog API.

If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).

  • To send metrics to Datadog, set type to datadog
  • Set apiKey to an API key to send metrics directly to Datadog via its HTTP API, or:
  • If apiKey is not set, metrics will be sent to a Datadog agent:
    • Set host to the hostname/IP of the agent (defaults to 127.0.0.1)
    • Set port to the port that the agent is listening on (defaults to 8125)
  • prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
  • tags -- a list of name:value strings to use as tags for all metrics sent during a test
  • event -- send a Datadog event when the test starts/finishes
    • title -- set to customize the event title; defaults to Artillery.io Test + timestamp
    • text -- set to customize the event text
    • priority -- normal or low; defaults to low
    • tags -- a list of event specific tags in the value:name format
    • alertType -- error, warning, info or success; defaults to info
    • send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)

Debugging​

Set DEBUG=plugin:publish-metrics:datadog-statsd when running your tests to print out helpful debugging messages when sending metrics to Datadog.

DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml

Example: Datadog with an agent​

config:
  plugins:
    publish-metrics:
      # apiKey not set, so the plugin will assume that the agent is running:
      - type: datadog
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-agent"
        event:
          priority: low
          tags:
            - "testId:mytest123"

Example: Datadog with an API key​

config:
  plugins:
    publish-metrics:
      - type: datadog
        # DD_API_KEY is an environment variable containing the API key
        apiKey: "{{ $processEnvironment.DD_API_KEY }}"
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-api"
        event:
          title: "Plugin integration test"
          priority: high
          tags:
            - "testId:mytest123"

Prometheus (Pushgateway)​

Artillery metrics can be sent to Prometheus via Pushgateway.

  • Set type to prometheus and set pushgateway to the URL of the Pushgateway instance
  • Set an optional prefix (defaults to artillery) and tags

Metric names in Prometheus​

Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.

This integration creates three metric names (when using the default artillery prefix):

  • artillery_counters
  • artillery_summaries
  • artillery_rates

Individual Artillery metrics are then made available via the metric tag. For example:

  • The built-in Artillery metric http.response_time is of type "summary", i.e. it has the following fields: min/max/p95/p99
  • Those values can be looked up in Prometheus through the artillery_summaries metric with tag metric:http_response_time_p95 for p95

Example: Prometheus​

config:
  plugins:
    publish-metrics:
      - type: prometheus
        pushgateway: "http://kubernetes.docker.internal:9091"
        tags:
          - "testId:mytest123"
          - "type:loadtest"

Honeycomb​

Honeycomb integration sends an event for every HTTP response (rather than pre-aggregated metrics).

  • To send events to Honeycomb, set type to honeycomb
  • Set apiKey to API/write key
  • Set dataset to the name of a dataset you want to send events to
  • Optional: set enabled to false to disable the integration
  • Optional: set sampleRate to sample rate (default: 1 i.e. send all events) (Honeycomb docs)

The following properties are set on every event:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging​

Set DEBUG=plugin:publish-metrics:honeycomb when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.

DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml

Example: Honeycomb​

config:
  plugins:
    publish-metrics:
      - type: honeycomb
        # HONEYCOMB_API_KEY is an environment variable which contains the API key
        apiKey: "{{ $processEnvironment.HONEYCOMB_API_KEY }}"
        dataset: "artillery-test"
        sampleRate: 1

Lightstep​

Lightstep integration sends a span for every HTTP response (rather than pre-aggregated metrics).

  • To send events to Lightstep, set type to lightstep
  • Set accessToken to an access token
  • Set componentName to the name of a component for which the spans will be sent
  • Default tags for every span may be added by setting key-value pairs under tags
  • Optional: set enabled to false to disable the integration

The following tags are set on every span:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging​

Set DEBUG=plugin:publish-metrics:lightstep when running your tests to print out helpful debugging messages when sending metrics to Lightstep.

DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml

Example: Lightstep​

config:
  plugins:
    publish-metrics:
      - type: lightstep
        # LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
        accessToken: "{{ $processEnvironment.LIGHTSTEP_ACCESS_TOKEN }}"
        component: artillery-tests
        tags:
          team: sre
          purpose: peak-load

Mixpanel​

Set type to mixpanel, and set the project token with projectToken.

plugins:
    publish-metrics:
      - type: mixpanel
        projectToken: "my-mixpanel-project-token"

Debugging​

Set DEBUG=plugin:publish-metrics:mixpanel when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.

DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml

StatsD​

  • To send metrics to StatsD, set type to statsd
  • Set host and port to hostname/IP and port of the agent (if different from the default 127.0.0.1:8125)
  • Set prefix to use a custom prefix for metric names created by Artillery; defaults to artillery.

Example: StatsD​

config:
  plugins:
    publish-metrics:
      - type: statsd
        prefix: 'artillery.publish_metrics_plugin.'

InfluxDB/Telegraf​

  • To send metrics to Telegraf (with Telegraf's statsd Service Plugin), set type to influxdb-statsd
  • All other options are the same as for Datadog (other than apiKey which is not relevant).

Example: InfluxDB/Telegraf​

config:
  plugins:
    publish-metrics:
      - type: influxdb-statsd
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:influxdb-statsd"
        event:
          priority: low
          tags:
            - "testId:mytest123"

More Information​

The source code for artillery-plugin-expect is available on Github at https://github.com/artilleryio/artillery-plugin-publish-metrics. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.

Table of contentsOn this page