Publishing Metrics / Monitoring

What you'll learn

• How to publish metrics from your Artillery tests to external monitoring and observability systems
• How to install the official publish-metrics plugin
• How to configure metrics and traces for Datadog, Influx, Honeycomb, Lightstep, and Statsd

Overview

The official artillery-publish-metrics plugin lets you send metrics and events/traces from Artillery to external monitoring and observability systems.

Supported targets

• Datadog metrics (via agent or HTTP API)
• Honeycomb events
• Lightstep spans
• Mixpanel events
• InfluxDB metrics with Telegraf + StatsD plugin
• StatsD metrics

We are working on adding more targets (such as CloudWatch, Splunk, ELK and Prometheus). You can follow the plugin's development on Github.

Published metrics

Datadog, Mixpanel, InfluxDB and StatsD integration send the following aggregated metrics:

• Virtual user metrics: scenarios.created, scenarios.completed - number of sessions created and successfully completed
• HTTP-specific metrics:
  • requests.completed -- number of requests completed successfully
  • latency.min / max / median / p95 / p99 -- response time latency distribution
  • codes.2xx / 3xx / 4xx / 5xx -- response code counts
  • rps.mean -- mean per/second rate of successful responses
• Errors: errors.ETIMEOUT / ENOTFOUND / EMFILE - any errors encountered

Published traces

Honeycomb and Lightstep integrations send events/traces for all HTTP requests, with the following tags:

• url - full URL of the request
• host - hostname + port
• method - HTTP method, e.g. GET
• statusCode - status code, e.g. 200
• responseTimeMs - time-to-first-byte of the response in milliseconds

Usage

Install the plugin

The plugin needs to be installed in the same scope (globally or as a project-specific dependency) as Artillery.

If you installed Artillery globally (i.e. with npm install -g artillery) then the plugin needs to be installed globally too:

npm install -g artillery-plugin-publish-metrics

If artillery is installed as a project-specific dependency (i.e. in a directory with package.json in it), install it with: t

npm install artillery-plugin-publish-metrics

Configure a target for metrics/traces

Datadog

The plugin supports sending metrics to an already running Datadog agent or directly to Datadog API.

If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).

• To send metrics to Datadog, set type to datadog
• Set apiKey to an API key to send metrics directly to Datadog via its HTTP API, or:
• If apiKey is not set, metrics will be sent to a Datadog agent:
  • Set host to the hostname/IP of the agent (defaults to 127.0.0.1)
  • Set port to the port that the agent is listening on (defaults to 8125)
• prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
• tags -- a list of name:value strings to use as tags for all metrics sent during a test
• event -- send a Datadog event when the test starts/finishes
  • title -- set to customize the event title; defaults to Artillery.io Test + timestamp
  • text -- set to customize the event text
  • priority -- normal or low; defaults to low
  • tags -- a list of event specific tags in the value:name format
  • alertType -- error, warning, info or success; defaults to info
  • send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)

Debugging

Set DEBUG=plugin:publish-metrics:datadog-statsd when running your tests to print out helpful debugging messages when sending metrics to Datadog.

DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml

Example: Datadog with an agent

config:
  plugins:
    publish-metrics:
      # apiKey not set, so the plugin will assume that the agent is running:
      - type: datadog
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-agent"
        event:
          priority: low
          tags:
            - "testId:mytest123"

Example: Datadog with an API key

config:
  plugins:
    publish-metrics:
      - type: datadog
        # DD_API_KEY is an environment variable containing the API key
        apiKey: "{{ $processEnvironment.DD_API_KEY }}"
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-api"
        event:
          title: "Plugin integration test"
          priority: high
          tags:
            - "testId:mytest123"

Honeycomb

Honeycomb integration sends an event for every HTTP response (rather than pre-aggregated metrics).

• To send events to Honeycomb, set type to honeycomb
• Set apiKey to API/write key
• Set dataset to the name of a dataset you want to send events to
• Optional: set enabled to false to disable the integration
• Optional: set sampleRate to sample rate (default: 1 i.e. send all events) (Honeycomb docs)

The following properties are set on every event:

• url - full URL of the request
• host - hostname + port
• method - HTTP method, e.g. GET
• statusCode - status code, e.g. 200
• responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging

Set DEBUG=plugin:publish-metrics:honeycomb when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.

DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml

Example: Honeycomb

config:
  plugins:
    publish-metrics:
      - type: honeycomb
        # HONEYCOMB_API_KEY is an environment variable which contains the API key
        apiKey: "{{ $processEnvironment.HONEYCOMB_API_KEY }}"
        dataset: "artillery-test"
        sampleRate: 1

Lightstep

Lightstep integration sends a span for every HTTP response (rather than pre-aggregated metrics).

• To send events to Lightstep, set type to lightstep
• Set accessToken to an access token
• Set componentName to the name of a component for which the spans will be sent
• Default tags for every span may be added by setting key-value pairs under tags
• Optional: set enabled to false to disable the integration

The following tags are set on every span:

• url - full URL of the request
• host - hostname + port
• method - HTTP method, e.g. GET
• statusCode - status code, e.g. 200
• responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging

Set DEBUG=plugin:publish-metrics:lightstep when running your tests to print out helpful debugging messages when sending metrics to Lightstep.

DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml

Example: Lightstep

config:
  plugins:
    publish-metrics:
      - type: lightstep
        # LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
        accessToken: "{{ $processEnvironment.LIGHTSTEP_ACCESS_TOKEN }}"
        component: artillery-tests
        tags:
          team: sre
          purpose: peak-load

Mixpanel

Set type to mixpanel, and set the project token with projectToken.

plugins:
    publish-metrics:
      - type: mixpanel
        projectToken: "my-mixpanel-project-token"

Debugging

Set DEBUG=plugin:publish-metrics:mixpanel when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.

DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml

StatsD

• To send metrics to StatsD, set type to statsd
• Set host and port to hostname/IP and port of the agent (if different from the default 127.0.0.1:8125)
• Set prefix to use a custom prefix for metric names created by Artillery; defaults to artillery.

Example: StatsD

config:
  plugins:
    publish-metrics:
      - type: statsd
        prefix: 'artillery.publish_metrics_plugin.'

InfluxDB/Telegraf

• To send metrics to Telegraf (with Telegraf's statsd Service Plugin), set type to influxdb-statsd
• All other options are the same as for Datadog (other than apiKey which is not relevant).

Example: InfluxDB/Telegraf

config:
  plugins:
    publish-metrics:
      - type: influxdb-statsd
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:influxdb-statsd"
        event:
          priority: low
          tags:
            - "testId:mytest123"

More Information

The source code for artillery-plugin-expect is available on Github at https://github.com/artilleryio/artillery-plugin-publish-metrics. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.