Skip to main content

Publishing Metrics / Monitoring

What you'll learn
  • How to publish metrics from your Artillery tests to external monitoring and observability systems
  • How to configure metrics and traces for CloudWatch, Datadog, Influx, Honeycomb, Lightstep, and Statsd

Overview​

The official artillery-publish-metrics plugin lets you send metrics and events/traces from Artillery to external monitoring and observability systems.

Supported targets​

We are planning to add more targets (such as Splunk, ELK, New Relic) in future.

Published metrics​

All metrics generated by Artillery during a test run will be sent to target monitoring system (Datadog, Prometheus, InfluxDB and StatsD) by default. That includes both built-in and custom metrics.

CloudWatch and Datadog integrations allow you to customize the list of metrics that will be sent through includeOnly and excluded options.

Published traces​

Honeycomb and Lightstep integrations send events/traces for all HTTP requests, with the following tags:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Usage​

CloudWatch​

Enable CloudWatch by adding a configuration with type: cloudwatch and setting the AWS region that you want metrics to be sent to:

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: eu-west-1

By default, all Artillery metrics will be sent to CloudWatch. Each Artillery metric will create a custom CloudWatch metric, which incur extra AWS fee for each metric. You can configure a specific list of metrics to send with the includeOnly setting (see Configuration section below).

You can learn more about AWS CloudWatch custom metric pricing in this blog post from Vantage.sh: https://www.vantage.sh/blog/cloudwatch-metrics-pricing-explained-in-plain-english

Important: AWS credentials have to be present in the environment and have sufficient IAM permissions to publish CloudWatch metrics.

Configuration options​

  • region - CloudWatch region where metrics will be sent
  • namespace - Metric namespace. Defaults to "artillery".
  • name - The value of the default Name dimension attached to every metric. Defaults to "loadtest".
  • excluded - A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
  • includeOnly - A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
  • dimensions - A list of extra dimensions to attache to every metric as name/value pairs. Defaults to an empty list.

Example: CloudWatch configuration​

The following configuration enables CloudWatch reporting. All metrics generated by the test will be sent to us-east-1 region, and tagged with extra dimensions to specify the team running the tests (SQA), and the name of the service under test (checkout-svc).

config:
  plugins:
    publish-metrics:
      - type: cloudwatch
        region: us-east-1
        namespace: continous-testing
        dimensions:
          - name: Team
            value: SQA
          - name: Service
            value: checkout-svc

Datadog​

The plugin supports sending metrics to an already running Datadog agent or directly to Datadog API.

If Datadog agents have already been set up on your infrastructure, then publishing via the agent is probably preferable. Publishing via the HTTP API is useful when running in environments which do not have the agent (e.g. when running Artillery on AWS Lambda or AWS Fargate).

By default, all Artillery metrics will be sent to Datadog. Each Artillery metric will create a custom Datadog metric, which will have an associated charge.

You can configure a specific list of metrics to send with the includeOnly setting (see Configuration section below).

  • To send metrics to Datadog, set type to datadog
  • Set apiKey to an API key to send metrics directly to Datadog via its HTTP API, or:
  • If apiKey is not set, metrics will be sent to a Datadog agent:
    • Set host to the hostname/IP of the agent (defaults to 127.0.0.1)
    • Set port to the port that the agent is listening on (defaults to 8125)
  • apiHost -- use this to override the default Datadog endpoint, e.g. to use Datadog EU set the value to app.datadoghq.eu (default is app.datadoghq.com) Note: this only works when apiKey is set, and not when Datadog agent is used
  • prefix -- use a prefix for metric names created by Artillery; defaults to artillery.
  • tags -- a list of name:value strings to use as tags for all metrics sent during a test
  • excluded - A list of metric names which should not be sent to CloudWatch. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
  • includeOnly - A list of specific metrics to send to CloudWatch. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to CloudWatch.
  • event -- send a Datadog event when the test starts/finishes
    • title -- set to customize the event title; defaults to Artillery.io Test + timestamp
    • text -- set to customize the event text
    • priority -- normal or low; defaults to low
    • tags -- a list of event specific tags in the value:name format
    • alertType -- error, warning, info or success; defaults to info
    • send -- set to false to turn off the event. By default, if an event is configured, it will be sent. This option makes it possible to turn event creation on/off on the fly (e.g. via an environment variable)

Debugging​

Set DEBUG=plugin:publish-metrics:datadog-statsd when running your tests to print out helpful debugging messages when sending metrics to Datadog.

DEBUG=plugin:publish-metrics:datadog-statsd artillery run my-script.yaml

Example: Datadog with an agent​

config:
  plugins:
    publish-metrics:
      # apiKey not set, so the plugin will assume that the agent is running:
      - type: datadog
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-agent"
        event:
          priority: low
          tags:
            - "testId:mytest123"

Example: Datadog with an API key​

config:
  plugins:
    publish-metrics:
      - type: datadog
        # DD_API_KEY is an environment variable containing the API key
        apiKey: "{{ $processEnvironment.DD_API_KEY }}"
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:datadog-api"
        event:
          title: "Plugin integration test"
          priority: high
          tags:
            - "testId:mytest123"

New Relic​

To send metrics to New Relic, set type to newrelic:

config:
  plugins:
    publish-metrics:
      - type: newrelic
        apiKey: "{{ $processEnvironment.NEW_RELIC_LICENSE_KEY }}"
        prefix: 'artillery.'
        attributes:
          - "type:soak-test"
          - "service:my-service"

By default, all Artillery metrics will be sent to New Relic. Each Artillery metric will create a custom New Relic metric, which may have an associated cost.

Configuration options​

  • To send metrics to New Relic, set type to newrelic.
  • Set licenseKey to the license key for the account you want to send the metrics to
  • region -- us (default) or eu. Thes sets default New Relic endpoint. If your account hosts data in the EU data center set the region to eu.
  • prefix -- set a prefix for metric names created by Artillery; defaults to artillery.
  • attributes -- a list of name:value strings to use as tags for all metrics sent during a test
  • excluded -- a list of metric names which should not be sent to New Relic. Defaults to an empty list, i.e. all metrics are sent to New Relic.
  • includeOnly -- a list of specific metrics to send to New Relic. No other metrics will be sent. Defaults to an empty list, i.e. all metrics are sent to New Relic.

Prometheus (Pushgateway)​

Artillery metrics can be sent to Prometheus via Pushgateway.

  • Set type to prometheus and set pushgateway to the URL of the Pushgateway instance
  • Set an optional prefix (defaults to artillery) and tags

Metric names in Prometheus​

Prometheus requires that every metric name is registered ahead of time, whereas metrics can be generated on the fly in Artillery, i.e. there is no way to know all metric names that may be generated by Artillery during a test run ahead of time.

This integration creates three metric names (when using the default artillery prefix):

  • artillery_counters
  • artillery_summaries
  • artillery_rates

Individual Artillery metrics are then made available via the metric tag. For example:

  • The built-in Artillery metric http.response_time is of type "summary", i.e. it has the following fields: min/max/p95/p99
  • Those values can be looked up in Prometheus through the artillery_summaries metric with tag metric:http_response_time_p95 for p95

Example: Prometheus​

config:
  plugins:
    publish-metrics:
      - type: prometheus
        pushgateway: "http://kubernetes.docker.internal:9091"
        tags:
          - "testId:mytest123"
          - "type:loadtest"

Honeycomb​

Honeycomb integration sends an event for every HTTP response (rather than pre-aggregated metrics).

  • To send events to Honeycomb, set type to honeycomb
  • Set apiKey to API/write key
  • Set dataset to the name of a dataset you want to send events to
  • Optional: set enabled to false to disable the integration
  • Optional: set sampleRate to sample rate (default: 1 i.e. send all events) (Honeycomb docs)

The following properties are set on every event:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging​

Set DEBUG=plugin:publish-metrics:honeycomb when running your tests to print out helpful debugging messages when sending metrics to Honeycomb.

DEBUG=plugin:publish-metrics:honeycomb artillery run my-script.yaml

Example: Honeycomb​

config:
  plugins:
    publish-metrics:
      - type: honeycomb
        # HONEYCOMB_API_KEY is an environment variable which contains the API key
        apiKey: "{{ $processEnvironment.HONEYCOMB_API_KEY }}"
        dataset: "artillery-test"
        sampleRate: 1

Lightstep​

Lightstep integration sends a span for every HTTP response (rather than pre-aggregated metrics).

  • To send events to Lightstep, set type to lightstep
  • Set accessToken to an access token
  • Set componentName to the name of a component for which the spans will be sent
  • Default tags for every span may be added by setting key-value pairs under tags
  • Optional: set enabled to false to disable the integration

The following tags are set on every span:

  • url - full URL of the request
  • host - hostname + port
  • method - HTTP method, e.g. GET
  • statusCode - status code, e.g. 200
  • responseTimeMs - time-to-first-byte of the response in milliseconds

Debugging​

Set DEBUG=plugin:publish-metrics:lightstep when running your tests to print out helpful debugging messages when sending metrics to Lightstep.

DEBUG=plugin:publish-metrics:lightstep artillery run my-script.yaml

Example: Lightstep​

config:
  plugins:
    publish-metrics:
      - type: lightstep
        # LIGHTSTEP_ACCESS_TOKEN is an environment variable which contains the token
        accessToken: "{{ $processEnvironment.LIGHTSTEP_ACCESS_TOKEN }}"
        component: artillery-tests
        tags:
          team: sre
          purpose: peak-load

Mixpanel​

Set type to mixpanel, and set the project token with projectToken.

plugins:
    publish-metrics:
      - type: mixpanel
        projectToken: "my-mixpanel-project-token"

Debugging​

Set DEBUG=plugin:publish-metrics:mixpanel when running your tests to print out helpful debugging messages when sending metrics to Mixpanel.

DEBUG=plugin:publish-metrics:mixpanel artillery run my-script.yaml

StatsD​

  • To send metrics to StatsD, set type to statsd
  • Set host and port to hostname/IP and port of the agent (if different from the default 127.0.0.1:8125)
  • Set prefix to use a custom prefix for metric names created by Artillery; defaults to artillery.

Example: StatsD​

config:
  plugins:
    publish-metrics:
      - type: statsd
        prefix: 'artillery.publish_metrics_plugin.'

InfluxDB/Telegraf​

  • To send metrics to Telegraf (with Telegraf's statsd Service Plugin), set type to influxdb-statsd
  • All other options are the same as for Datadog (other than apiKey which is not relevant).

Example: InfluxDB/Telegraf​

config:
  plugins:
    publish-metrics:
      - type: influxdb-statsd
        prefix: 'artillery.publish_metrics_plugin.'
        tags:
          - "testId:mytest123"
          - "reporterType:influxdb-statsd"
        event:
          priority: low
          tags:
            - "testId:mytest123"

More Information​

The source code for artillery-plugin-expect is available on Github at https://github.com/artilleryio/artillery-plugin-publish-metrics. It's an officially supported plugin and we'd love to hear your feedback and ideas for improvement.

Table of contentsOn this page