OpenTelemetry

Added in: v2.0.0-37

⚠️

Experimental

This feature is under active development. We aim to keep it stable, but it may need to introduce breaking changes. Please open an issue (opens in a new tab) or discussion (opens in a new tab) for any feedback.

This reporter uses OpenTelemetry (opens in a new tab) (also referred to as OTel) to generate, record and export Artillery's metrics and traces in a standardized way, allowing you to send that data directly to virtually any observability and monitoring backend that supports OpenTelemetry (you can consult a list of vendors who natively support OpenTelemetry (opens in a new tab)), as well as to your own OTel Collector (opens in a new tab).

To use the OpenTelemetry reporter set type to open-telemetry in the publish-metrics plugin configuration. You can use Artillery’s VS Code Extension (opens in a new tab) to help you navigate through the needed fields in the plugin.

Support

Artillery Engines

Available tracing data per engine

HTTP engine

When sending tracing data, Artillery creates a trace per each scenario created, holding all requests that are executed as a part of that scenario. Each request is then broken down further into the following child spans (representing the request timings as they are recorded in the got response.timings.phases object):

  • dns_lookup
  • tcp_handshake
  • tls_negotiation
  • request
  • first_byte
  • download

The following attributes are set by default on all request spans:

  • test_id: unique id for the test run

    Added in: v2.0.6

  • vu.uuid: unique id for the virtual user
  • url.full: full URL of the request
  • server.address: hostname
  • server.port: port
  • http.request.method: HTTP method e.g. GET
  • http.response.status_code: HTTP status code e.g. 200
  • response.time.ms: time-to-first-byte of the response in milliseconds

Playwright engine

Added in: v2.0.1

A trace is created for each scenario/virtual user execution and it holds a span for each:

  • test.step helper call: remains active for the duration of the test.step function.
  • Page navigated: This span is active for the duration of time consumed on the given page.

Information added to spans:

  • test_id: unique id for the test run attribute is set on all spans.

    Added in: v2.0.6

  • vu.uuid: unique id for the virtual user attribute is set on all spans.
  • browser.memory_used_mb: represents usedJSHeapSize. This attribute is set on page spans when extendedMetrics setting is enabled in the Playwright engine config.
  • dominteractive: time it takes for DOM to become interactive. This attribute is set on page spans when extendedMetrics setting is enabled in the Playwright engine config.
  • Attributes set with attributes in the tracing config are added to all spans.
  • Web Vitals (opens in a new tab) (when available) are recorded as events, as well as set as attributes on page spans. Events are named by the given Web Vital abbreviation (e.g.TTFB for Time To First Byte), and attribute names are in the following format:
    • web_vitals.<web_vital_metric_name>.value: the value of given web vital metric. This attribute is set on page spans, and only when available.
    • web_vitals.<web_vital_metric_name>.rating: the rating of given web vital metric. This attribute is set on page spans, and only when available.
  • Errors are recorded as exceptions and as span status set to ERROR.

OTeL protocols per signal type

  • Metrics via OTLP (OpenTelemetry Protocol) exporter with HTTP/JSON, HTTP/protobuf and gRPC;
  • Traces via OTLP exporter with HTTP/JSON, HTTP/protobuf, gRPC as well as via Zipkin exporter.

Configuration

TIP: If you want to send data to Honeycomb, New Relic, Datadog, Dynatrace, Grafana or Lightstep, check out the Guides for popular Observability Platforms for quick start configuration guides.

Requirements

In most cases basic configuration will require:

  • OTLP endpoint for the metrics and/or traces
  • Header for authentication

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: <your-service-name>  
        metrics:
          endpoint: "{{ $env.OTLP_METRICS_ENDPOINT }}" 
          headers:
            <auth-header-name>: "{{ $env.API_KEY }}"
        traces:
          endpoint: "{{ $env.OTLP_TRACES_ENDPOINT }}"
          headers:
            <auth-header-name>: "{{ $env.API_KEY }}"

Steps

  • Set serviceName to the name of your service. It will apply to both metrics and traces
  • Specify signal type/s you want to send and configure them with:
    • endpoint for the signal type in question
    • headers in key: value format. (The authentication (e.g. API key) is usually provided via header of platform specific format)

This information can be found in your chosen observability backends documentation.

Obtaining the right configuration

Look into your chosen observability platforms documentation for the following information:

  1. Signal type supported - make sure ingesting the signal type that you want to send (traces and/or metrics) is supported.
  2. Underlying transport protocol for OTLP supported (HTTP/JSON, HTTP/protobuf or gRPC) for each signal type (metrics/traces). You will need this information to understand if you need to override the reporters default exporter setting. Artillery defaults to OTLP HTTP/JSON but has exporters for all 3 OTLP options.
  3. OTLP API endpoints to send data to. Keep in mind that the endpoints will be different for metrics and traces. If the same endpoint is provided for both signal types, you will need to append the /v1/metrics or /v1/traces to it.
  4. Headers needed. Usually at least one header is needed as that is how authentication is provided. Header format (name and value) as well as information you need for authentication (e.g. API key which will be set in the header value) will be provided in your chosen observability platforms documentation
  5. Any additional information that the platform might ask you to set (e.g. specific attributes).

TIP: Sometimes, in your chosen observability platforms documentation, the information mentioned above will be located inside the part on how to instrument your service/app with OpenTelemetry to export data to the chosen platform.

Configuration options

Example configuration
publish-metrics:
  - type: "open-telemetry"
    serviceName: "my service"
    metrics:
      exporter: "otlp-http"
      endpoint: "{{ $env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }}"
      headers:
        Api-Key: "{{ $env.API_KEY }}"
      attributes:
        environment: "test"
    traces:
      exporter: "otlp-proto"
      endpoint: "{{ $env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }}"
      headers:
        Authorization: "{{ $env.API_KEY }}"
      useRequestNames: true
      attributes:
        environment: "test"
        tool: "Artillery"
      smartSampling:
        thresholds:
          firstByte: 500

Main

NameValid optionsDescription
serviceNamestring (defaults to Artillery.test)Logical name of the service. Represents service.name resource attribute. Used for both metrics and traces.
metricskey:value settingsSet this option to send and configure metrics
traceskey:value settingsSet this option to send and configure traces

Metrics

NameValid optionsDescription
exporter
  • - otlp-http (OTLP via HTTP/JSON)(default)
  • - otlp-proto (OTLP via HTTP/protobuf)
  • - otlp-grpc (OTLP via gRPC)
Exporter used to send metrics.
endpoint
  • - http://localhost:4318/v1/metrics for HTTP
  • - http://localhost:4317/v1/metrics for gRPC
OTLP endpoint for metric ingestion provided by your chosen observability platform.
headerskey: value pairsPlatform specific headers. Tipically used for providing authentication.
excludedlist of stringsA list of metric names which should not be sent.
includeOnlylist of stringsA list of specific metrics to send. No other metrics will be sent.
attributeskey: value pairsCustom attributes to be added to all metrics.

Traces

NameValid optionsDescription
exporter
  • - otlp-http (OTLP via HTTP/JSON)(default)
  • - otlp-proto (OTLP via HTTP/protobuf)
  • - otlp-grpc (OTLP via gRPC)
  • - zipkin (OTLP via gRPC)
Exporter used to send traces.
endpoint
  • - Valid endpoint
  • Defaults to:
  • - http://localhost:4318/v1/traces for HTTP
  • - http://localhost:4317/v1/traces for gRPC
Endpoint (OTLP or zipkin) for trace ingestion provided by your chosen observability platform.
headerskey:value pairsPlatform specific headers. Tipically used for providing authentication.
sampleRatevalue between 0 and 1 (default 1)Sample rate. Percentage of traces to send.
useRequestNames
  • - true
  • - false (default)
If true the request names provided in test script will be used as span names
attributeskey:value pairsCustom attributes to be added to each span.
smartSamplingconfiguration settingsSet to export only traces that have errors, HTTP status code over 400 or exceed thresholds set (if any thresholds are set)
Smart sampling (HTTP engine)

Added in: v2.0.4

When smartSampling is set for traces, only traces that are considered outliers will be exported. Traces that pass the following criteria are considered outliers:

  • Errors - any span with error status (e.g. request that failed due to an error)
  • HTTP Status Code 400 or more
  • Exceeded time to first byte (TTFB (opens in a new tab)) threshold (if set with firstByte)
  • Exceeded total request time threshold (if set with total)
traces:
  smartSampling:
    thresholds:
      firstByte: 500 
      total: 2000

Thresholds:

NameValueDescription
firstByteNumber (milliseconds)threshold for time to first byte (TTFB (opens in a new tab))
totalNumber (miliseconds)threshold for total request time
Example configuration for smart sampling

With the configuration shown below the reporter will export traces with:

  • Errors
  • HTTP Status 400 or more
  • TTFB over 500 miliseconds
  • Total time of request over 2000 miliseconds
  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: <your-service-name>  
        traces:
          endpoint: "{{ $env.OTLP_TRACES_ENDPOINT }}"
          headers:
            <auth-header-name>: "{{ $env.API_KEY }}"
          smartSampling:
            thresholds:
              firstByte: 500
              total: 2000

Environment variables config options

Some additional configuration options can be set through environment variables:

Environment VariableSignal specificDescription
OTEL_EXPORTER_OTLP_HEADERSNOEquivalent to headers in test script, set to the headers required by your chosen platform.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINTYESOTLP API endpoint for metrics ingestion
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTYESOTLP API endpoint for traces ingestion
OTEL_EXPORTER_OTLP_CERTIFICATENOThe path to the file containing trusted root certificate to use when verifying an OTLP trace, metric, or log server's TLS credentials. By default the host platform's trusted root certificate is used.
OTEL_EXPORTER_OTLP_TRACES_CERTIFICATEYESThe path to the file containing trusted root certificate to use when verifying an OTLP trace server's TLS credentials. By default the host platform's trusted root certificate is used.
OTEL_EXPORTER_OTLP_METRICS_CERTIFICATEYESThe path to the file containing trusted root certificate to use when verifying an OTLP metrics server's TLS credentials. By default the host platform's trusted root certificate is used.
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATENOThe path to the file containing trusted client certificate/chain for clients private key to use when verifying an OTLP trace, metric and log server's TLS credentials. Must provide a private client key when providing a certificate/chain. By default no chain file is used.
OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATENOThe path to the file containing trusted client certificate/chain for clients private key to use when verifying an OTLP trace server's TLS credentials. Must provide a private client key when providing a certificate/chain. By default no chain file is used.
OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATENOThe path to the file containing trusted client certificate/chain for clients private key to use when verifying an OTLP metrics server's TLS credentials. Must provide a private client key when providing a certificate/chain. By default no chain file is used.
OTEL_EXPORTER_OTLP_CLIENT_KEYNOThe path to the file containing private client key to use when verifying an OTLP trace, metric or log client's TLS credentials. Must provide a client certificate/chain when providing a private client key. By default no client key file is used.
OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEYYESThe path to the file containing private client key to use when verifying an OTLP trace client's TLS credentials. Must provide a client certificate/chain when providing a private client key. By default no client key file is used.
OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEYYESThe path to the file containing private client key to use when verifying an OTLP metric client's TLS credentials. Must provide a client certificate/chain when providing a private client key. By default no client key file is used.

Note: When using environment variables keep in mind that you still need to configure the reporter in the script. The minimum reporter configuration in the script will require specifying the type of signal to send.

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        metrics: {}
        traces: {}

NOTE: Less often, platforms will require you to install and configure the OTel Collector, or the platforms own Agent. In those cases the information on how to install and configure the Collector/Agent will be provided in your chosen platforms documentation.

On Artillery's side configuration will be the same or very similar - you will need to provide an endpoint and authentication through either the script or environment variable/s.

Guides for popular Observability Platforms

The following guides demonstrate only basic reporter configuration, however additional configuration options are available.

Datadog

Datadog

Requirements

  • Datadog API Key
  • Datadog requires a Datadog Agent to ingest OTLP data. The Agent setup is different depending which platform you're using (check below).

Note: Lambda is not currently supported.

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: "my_service" 
        metrics:
          attributes:
            environment: "test"
            type: "Load test"
        traces: {}

Fargate - additional setup

Added in: v2.0.4

Artillery sets up the Datadog Agent for you as a sidecar container. For it to start, all you need to do is make sure a DD_API_KEY is set in a .env file by using the --dotenv flag. Example:

artillery run-fargate scenario.yaml --dotenv ./.env

Local runs - additional setup

You'll need to setup a Datadog Agent in your machine. You can follow this guide (opens in a new tab) to set it up using Docker.

Reference

Here is a small demo (opens in a new tab) on seeing your Playwright engine traces on Datadog.

Honeycomb

Honeycomb

Requirements

Configuration Template

  plugins:
    publish-metrics:
      - type: open-telemetry
        serviceName: <your_service_name>
        traces:
          endpoint: https://api.honeycomb.io/v1/traces
          headers:
            x-honeycomb-team: "{{ $env.HONEYCOMB_API_KEY }}"
        metrics:
          endpoint: https://api.honeycomb.io/v1/metrics
          headers:
            x-honeycomb-team: "{{ $env.HONEYCOMB_API_KEY }}"
            x-honeycomb-dataset: <your_dataset_name>

NOTE: x-honeycomb-dataset needs to be set for metrics. We advise setting it to the same as your serviceName so that metrics and traces are grouped under the same dataset (namespace).

Reference

For additional information about OpenTelemetry data in Honeycomb consult Honeycomb's documentation (opens in a new tab).

New Relic

New Relic

Requirements

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: <your-service-name>
        metrics:
          endpoint: "https://{{ $env.NEW_RELIC_OTLP_ENDPOINT }}/v1/metrics"
          headers:
            api-key: "{{ $env.NR_LICENSE_KEY }}"
        traces:
          endpoint: "https://{{ $env.NEW_RELIC_OTLP_ENDPOINT }}/v1/traces"
          headers:
            api-key: "{{ $env.NEW_RELIC_LICENSE_KEY }}"

Endpoint

OrganisationBase URL
US OTLPhttps://otlp.nr-data.net
EU OTLPhttps://otlp.eu01.nr-data.net
US FedRAMP OTLPhttps://gov-otlp.nr-data.net

Reference

For more information, consult the New Relic documentation (opens in a new tab).

Dynatrace

Dynatrace

Requirements

Note: this guide is only for sending traces to ActiveGate (opens in a new tab). In order to send traces to OneAgent consult the Dynatrace documentation (opens in a new tab)

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: <your-service-name>
        metrics:
          exporter: otlp-proto
          endpoint: "{{ $env.DYNATRACE_OTLP_ENDPOINT }}/v1/metrics"
          headers:
            Authorization: "Api-Token {{ $env.API_ACCESS_TOKEN }}"
 
        traces:
          exporter: otlp-proto
          endpoint: "{{ $env.DYNATRACE_OTLP_ENDPOINT }}/v1/traces" 
          headers:
            Authorization: "Api-Token {{ $env.API_ACCESS_TOKEN }}" 

NOTE: Dynatrace only supports HTTP/protobuf, so we change the exporter value from the default to otlp-proto in this case.

Endpoints

ActiveGate TypeBase URL
Dynatrace SaaShttps://{your-environment-id}.live.dynatrace.com/api/v2/otlp
Local Environment ActiveGate(1)https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/otlp
Tip: Environment ActiveGates listen by default on port 9999. If you changed that port, adjust the port in the URL accordingly.

API Access Token scopes

To obtain an access token, in the Dynatrace web UI, go to Access tokens. Use the following access scopes for the signals you are exporting:

Signal TypeAccess Scope
metricsmetrics.ingest
tracesopenTelemetryTrace.ingest

Reference

For any aditional information on OpenTelemetry data in Dynatrace, consult the Dynatrace documentation (opens in a new tab).

Lightstep (ServiceNow Observability Cloud)

Lightstep (ServiceNow Observability Cloud)

Requirements

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: "my_service" 
        metrics:
          endpoint: "{{ $env.LIGHTSTEP_OTLP_METRICS_ENDPOINT }}"
          headers:
            Lightstep-Access-Token: "{{ $env.LIGHTSTEP_ACCESS_TOKEN }}"
        traces:
          endpoint: "{{ $env.LIGHTSTEP_OTLP_TRACES_ENDPOINT }}"
          headers:
            Lightstep-Access-Token: "{{ $env.LIGHTSTEP_ACCESS_TOKEN }}"

Endpoints

Choose the endpoint that matches the type of signal you wish to send and to your Lightstep/Service Now service type, and replace the placeholders with your data if needed (e.g. replace {microsatellite_ip} with your microsatellite_ip).

Service TypeSignal TypeEndpoint
Public Satellitesmetricshttps://ingest.lightstep.com:443/metrics/otlp/v0.9
traceshttps://ingest.lightstep.com:443/v1/traces
On-Premise Satellitesmetricshttps://<microsatellite_ip>:<microsatellite_port>/metrics/otlp/v0.9
traceshttps://<microsatellite_ip>:<microsatellite_port>/v1/traces
Tip: Replace <microsatellite_ip> and <microsatellite_port> with the correct ones.

Reference

For additional information about sending OpenTelemetry data to Lightstep (ServiceNow Observability Cloud), check Lightstep documentation (opens in a new tab).

Grafana

Grafana

Requirements

  • Access Policy Token with correct scopes
  • Account Information obtained in Cloud Portal (opens in a new tab) -> OpenTelemetry card -> Configure:
    • Instance ID (e.g. 130834)
    • OTLP Endpoint (e.g. https://otlp-gateway-prod-us-central-0.grafana.net/otlp)

Configuration Template

  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: <your-service-name> 
        metrics:
          exporter: otel-proto
          endpoint: "{{ $env.GRAFANA_OTLP_ENDPOINT }}"
          headers:
            Authorization: "Basic: {{ $env.GRAFANA_AUTH_CODE }}" 
        traces:
          exporter: otel-proto
          endpoint: "{{ $env.GRAFANA_OTLP_ENDPOINT }}"
          headers:
            Authorization: "Basic: {{ $env.GRAFANA_AUTH_CODE }}" 

Generating Auth Code

  1. Create an Access Policy Token (opens in a new tab) with the metrics:write and/or traces:write scopes depending on the data you will be sending.
  2. Make a base64 encoding of the <your-instance-id>:<your-token> string and store it as an environment variable (e.g. GRAFANA_AUTH_CODE).

Reference

For additional information about sending OpenTelemetry data to Grafana, check their documentation (opens in a new tab)

Debugging

Set DEBUG=plugin:publish-metrics:open-telemetry when running your tests to print out helpful debugging messages when sending metrics and/or traces with OpenTelemetry.

DEBUG=plugin:publish-metrics:open-telemetry artillery run my-script.yaml