Artillery

CLI & Core

• Add parallel action for making parallel requests in HTTP tests

Fargate

• Add ability to suppress metrics output with --quiet flag - docs
• Add eu-north-1 to supported regions

Lambda

• Add ability to suppress metrics output with --quiet flag - docs

Playwright

• Fix issue preventing stack traces for errors from being displayed
• Update Playwright version to 1.43.1 - docs

Publish Metrics

• Make reporters using OpenTelemetry compatible with parallel requests

Publish Metrics

• Sending traces to AWS X-Ray is now available through the CloudWatch reporter
• Allow replacing of all tracing span names with replaceSpanNameRegex config option - Docs
• Fix bug in Datadog reporter causing last batches of traces to occasionally drop
• Fix errors happening when sampleRate tracing option set
• Fix errors in tracing happening when using HTTP streaming
• Display number of spans exported in the Artillery report (plugins.publish-metrics.spans.exported ) when tracing is enabled

CLI & Core

• Fix incorrectly declared dependencies in Artillery CLI

Artillery Cloud

• Fix issue preventing ARTILLERY_CLOUD_API_KEY environment variable from being picked up by Fargate when using --dotenv

Metrics by Endpoint

• Report request URLs for requests that resulted in a network error such as ETIMEOUT or ECONNRESET - docs

CLI and Core

• Fix issue causing some intermediate reports to not be reported
• Fix issue with incorrect formatting of time units in “phase started” messages

CLI and Core

• Add support for loading custom code from ES modules - docs
• Add support for using async functions as hooks - docs

Playwright

• Update Playwright version to 1.42.1
• Fix bug that prevented traces from Playwright tests from being sent to Datadog and Honeycomb

AWS Fargate

• Tests running in Fargate will now use AWS Distro for OpenTelemetry instead of Datadog Agent for sending traces to Datadog - docs
• Fix bug that prevented setting CPU and memory allocation with the --launch-config flag

Playwright

• Make expect from @playwright/test work natively with Typescript support - Documentation
• Use target as Playwright’s baseURL by default, allowing test scripts to include only relative paths in page navigations (e.g. page.goto('/docs') ) - Documentation
• Update Playwright version to 1.41.2 - Documentation

Artillery Cloud

• Fix missing phase events in Artillery Cloud when running Fargate tests

CLI & Core

• Improve Typescript support to work with more packages, by exposing config.bundling.external to set packages that can’t be bundled as external - Documentation
• Make test run id available to test runs via {{ $testId }} in scenarios/config and vuContext.vars.$testId in custom code
• Routine dependency updates

Publish Metrics

• Automatically send all traces and metrics sent via the opentelemetry reporter with a test_id attribute, making it easier to search for all relevant data from a particular test run

Fargate

• Improve error message and handling for putting CloudWatch retention policy

Artillery will now aim to release a new CLI version every first and third Monday of each month, when possible.

Artillery Cloud

• Fix bug causing only one of the links to worker logs to be displayed in Artillery Dashboard

Playwright

• Update Playwright version used by Artillery to v1.41.0 - Documentation
• Add support for configuring Playwright’s testIdAttribute setting - Documentation
• Add INP to core web vital metrics reported by Artillery - Documentation
• Enable headless mode by default when running in Fargate, as headed mode is not compatible with Fargate

Fargate

• Improve Fargate startup logic to prevent bug where workers would hang
• Fix bug where Fargate tests running on Windows would crash trying to resolve paths to dependencies in folders
• Set default bucket lifecycle policy for S3 bucket containing artifacts from test runs - Documentation
• Set default retention policy (180 days) on Cloudwatch Logs generated during runs, with override possible - Documentation
• Add support for running using Fargate Spot by using the --spot flag - Documentation

Lambda

• Fix Lambda bundle size issue preventing all Lambda tests from running
• Add default bucket lifecycle policy (2-7 days retention) for S3 bucket containing artifacts from test runs - Documentation

CLI & Core

• Fix $randomString(length) utility to return correct number of characters and allow length larger than 10

Publish Metrics

• Fix bug in Fargate tests where traces sent with OpenTelemetry reporter would not arrive or arrive inconsistently formatted to observability platforms
• Add tracing support for the Datadog, New Relic and Dynatrace reporters using OpenTelemetry
• Improve traces sent via Honeycomb reporter to be proper traces with full metadata, exceptions, events, etc

Fargate

• Enable support for 12 more AWS regions (eu-west-2, eu-west-3, us-east-2, us-west-2, ca-central-1, ap-east-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, me-south-1, sa-east-1 )
• Add support for sending Playwright and HTTP traces to Datadog APM - Documentation
• Add support for --variables flag in run-fargate command - Documentation
• Show supported regions when using the --region flag

CLI & Core

• Add TypeScript support for custom code for tests running locally or on AWS Fargate. This is an experimental feature. - Documentation

Playwright

• VUs will now run in isolated browsers contexts rather than new browser instances by default for better performance - Documentation
• HTTP status codes are now tracked and reported through the browser.page.codes.<code> metric - Documentation
• Fix bug in before/ after hooks that prevented the Playwright engine from working - Documentation
• Add experimental TypeScript support. See the docs for current limitations - Documentation

OpenTelemetry

• Fix issue with virtual users not failing when using test.step() with OpenTelemetry Playwright tracing enabled
• OpenTelemetry tracing can now be used to send HTTP and Playwright engine traces to Datadog in Fargate tests - Documentation
• Add support for smart sampling when using HTTP tracing with the OpenTelemetry reporter

CLI & Core

• Add time unit support for think steps in scenarios
• metrics-by-endpoint plugin is now enabled by default to capture per-URL metrics in HTTP tests. The metrics will be available only in the JSON report and not output to the console, unless the plugin is loaded explicitly by the user.
• Configure CPU and memory for Fargate tests with --cpu and --memory flags
• Fix issue with $env not resolving correctly in the config section
• Fix an issue with --scenario-name not picking up the correct scenario
• Add a new fake-data plugin to allow for realistic test data to be generated in your tests - docs
• Fix handling of metrics with non-alphanumeric characters in ensure checks
• Improve error handling in the expect plugin

OpenTelemetry

• Add experimental tracing support for Playwright tests, with tracing of test.step() calls, page navigation events, and Core Web Vitals metrics
• Improve conformance to OTel semantic conventions:
  • Include error messages for spans with HTTP response codes of 400 or higher
  • Include HTTP_REQUEST_CONTENT_LENGTH, HTTP_USER_AGENT and HTTP_FLAVOR attributes
• Fix issue with virtual user ID attribute (vu.uuid) not always being set correctly
• Fix inconsistencies in error handling

Playwright

• Add experimental tracing support via OpenTelemetry with publish-metrics plugin
• Fix FID Core Web Vital metric not being emitted as expected

ECS/Fargate

• Configure CPU and memory for Fargate tests with --cpu and --memory flags

Socket.IO

• Allow validation of any number of arguments in response/acknowledge with new args/on options
• More flexible validations with ability to wait for multiple responses
• Fix issue with displaying match errors in the report

New features & improvements

Engines

• playwright: Update Playwright version to latest (1.39.0).
• playwright: Add a convenience test.step() helper, allowing you to break your tests into steps that emit custom timing metrics - Docs.

CLI/Core

• New CLI flag --scenario-name, allowing you to specify which scenario to run by its name in the test script - Docs.
• Expose mean as a metric.
• Include scenario object in all engine hooks - you can now access this under context.scenario - Docs.

Plugins

publish-metrics - OpenTelemetry

• Traces for HTTP requests are now nested under individual scenarios.
• Include all request metrics in traces: dns_lookup, tcp_handshake, tls_negotiation, request, first_byte, download.
• Include VU errors in traces.

Fixes

Artillery Dashboard

• Ensure all Artillery console logs are sent correctly to Artillery Cloud.
• Fix issue with phase information not being sent correctly.

Engines

• socketio: Allow nested options (e.g. namespace) in emit without throwing validation error - Docs.
• socketio: Allow emit action to be an array of strings - Docs.

CLI/Core

• fargate: Fix issue with --packages flag.
• fargate: Include apdex plugin and playwright engine by default in image, preventing additional installation in workers.
• Allow unnamed capture groups to be used in capture.
• Display meaningful error when engine doesn't match the engine provided in config.engine.

Plugins

• apdex - Allow frustrated, satisfied and tolerated metrics to be checked with ensure plugin.
• apdex - Prevent plugin from crashing when using processor functions.
• publish-metrics - Set headers as metadata for otel-grpc exporter in OpenTelemetry reporter - Docs.
• publish-metrics - Create endpoint path correctly so it works with managed instances of Dynatrace.
• publish-metrics: Fix OpenTelemetry reporter debug logging.

New features

• Dynatrace support - the official publish-metrics plugin can now be configured to send Artillery metrics and events to Dynatrace. Read more in the docs
• config.http.defaults - default headers and cookies for all requests in a test may now be set via config.http.defaults

Fixes & improvements

Artillery Dashboard

• Send load phase start/end timings for visualization
• Send CLI exit code to display in test run results
• Fix test run status not always getting reported correctly
• Fix bug which caused test runs on AWS Lambda to not be tagged correctly

Core

• Allow variables set with --variables to be used anywhere in config
• Fix for encoding of array values in query strings
• Make sure CLI exit handlers are always run
• Fix bug affecting match groups in regexp captures

Plugins

• publish-metrics - Exit early if required config is missing for publish-metrics
• publish-metrics - Reduce memory usage of the plugin
• expect - add useRequestNamesOnly option
• ensure - fix bug where ensure config under config.plugins.ensure would not get picked up

Apdex scores

The load test summary view will now calculate and display Apdex scores for load tests that make use the built-in apdex plugin.

Apdex (Application Performance Index) is an open standard for measuring performance of HTTP services. The Apdex score converts response times into a single score that reflects user satisfaction with the service.

Fixes & Improvements

App-wide

• Update colors to improve legibility and consistency
• Add exit animations for all dropdown-like elements
• Improve loading states on all buttons
• Improve consistency of dialogs everywhere
• Improve consistency of spacing between elements across the app

Main view

• Tweak the display of Artillery’s logo in the navbar
• Fix responsiveness on Load Tests view
• Improve UX for Saved Views and the Filter Bar with long input strings
• Match test run ID icon in filter dropdown to one used elsewhere

Load test view

• Make font sizes consistent in metadata panel
• Improve styling of text log view, JSON log view, and table components
• Fix issue with loading default CLI avatar images on notes
• Fix issue with scroll gradient covering first note in the notes list
• Improve spacing on Edit Note dialog

Charts

• Add intelligent X axis scaling depending on duration of the test
• Improve Y axis layout on Load Summary chart

AWS Fargate support

This release adds support for running tests on AWS Fargate. Scale out your load tests horizontally from any of the six supported regions.

artillery run-fargate --count 10 --region us-east-1 my-script.yml

Supported regions:

• us-east-1
• us-west-1
• eu-west-1
• eu-central-1
• ap-south-1
• ap-northeast-1

AWS Fargate is a serverless solution that requires no infrastructure to be managed, and unlike AWS Lambda does not impose a restriction on the duration of your load tests.

Artillery's Playwright integration is fully supported on Fargate, to let you run browser-based load tests at scale with ease.

Playwright support

Playwright support is now built into Artillery. The integration in artillery-engine-playwright is now bundled into the Artillery CLI.

• The version of Playwright used by the integration has been updated to v1.3.4
• Web Vitals tracking has been improved to provide more accurate measurements
• Playwright scenario file may now be loaded via testFunction attribute rather than flowFunction
• flow attribute may now be omitted in Playwright engine scenarios

Other fixes & improvements

• Pass lambdaRoleArn to AWS Lambda engine correctly
• Fix issue with returning matched groups from regexp capture
• Track & report downloaded data in HTTP tests via http.downloaded_bytes metric
• Environment variables may now be accessed through the special $env variable (the current way of accessing those via $processEnvironment is still supported but will be deprecated in future)
• Dependency updates to address deprecation and security notices

New features

• AWS Lambda - when running tests on AWS Lambda, Artillery will automatically bundle any plugins loaded via config.plugins and engines loaded via config.engines
• Checks & assertions: new notStatusCode check in the expect plugin
• Datadog integration: includeOnly and excluded options may be used to configure the list of Artillery metrics which will be sent to Datadog

Fixes & improvements

• When artillery run is used with the --output option, Artillery will check if destination exists before running the test
• Miscellaneous dependency updates to address deprecation warnings and security advisories

We have released Artillery v2.0.0-28, which adds support for AWS CloudWatch, and features a number of performance and stability improvements.

(GitHub release page: https://github.com/artilleryio/artillery/releases/tag/artillery-2.0.0-28)

Breaking changes

• AWS Lambda: the default IAM role and policy are now versioned. If you attached any custom permissions to the artilleryio-default-lambda-role role, you will need to re-attach it to the new artilleryio-default-lambda-role-20230116 role
• CLI: The probe command has been removed from the CLI. The functionality has been moved to the new Skytrace CLI

New features

• Add support for publishing test metrics to AWS CloudWatch
• HTTP - compressed responses (e.g. gzip) will be decompressed by default now. (Set gzip: false to disable automatic decompression if required to improve performance.)
• CLI - suggest closest match if a command name is mistyped

AWS Lambda

• Print custom role ARN if provided
• Add versioning to default Lambda role and policy created for running tests

Fixes & improvements

• Fix arrival phases with arrivalRate and rampTo set to 0
• Improve rampTo logic to result in more predictable and deterministic ramps
• Fix issue with maxVusers setting that would lead to unexpected number of VUs to be generated
• Fix wildcard message handling support in Socket.io
• Fix issue with setting some properties on the global artillery object

AWS Lambda

• Improve test startup logic: workers will wait for each other to be ready before kicking off the test run
• Fix some intermediate metrics not being flushed & reported
• Fix handling of memory-size and security-group-ids options

Artillery CLI

New HTML reports

We have improved the layout and styling of reports generated with artillery report.

--dotenv flag

You can now load environment variables from a file with the --dotenv flag. Those env vars are available in Artillery scenarios through the $env variable.

Dashboard

Visual polish

• Improved homepage responsiveness
• Updated load test table with new context tags and new flexible tags
• Updated context menus
• Updated header styling in load test view.
• New and improved selected tags filter bar
• Show loading skeletons while data is being fetched

CDK installer

We released an official AWS CDK-based installer for Artillery dashboard.

https://github.com/artilleryio/installer-aws-cdk

Flexible deployment modes

The dashboard can now be deployed in more than one way to fit the exact requirements of your organization:

• As an internal-facing, inside a VPC. This is the default deployment mode, and it will require an existing VPN setup to access the app. TLS is optional with this setup.
• As a publig public-facing, which requires TLS with a pre-provisioned ACM certificate.
• Custom domain may be configured on top of the provisioned ALB DNS name.

GitHub Authentication

We added support for authentication through GitHub.

Aurora Serverless v1 Support

An Aurora Serverless database cluster may now be used as the backing store for Artillery Pro.

This requires Data API to be enabled on the cluster. Only Aurora Serverless v1 is supported right now.

Dashboard

Fixes & Improvements

• Improve loading states on the Settings screen
• Fix bug with context menu getting cut off
• Fix double-trigger on context menu buttons

Artillery Pro

Artillery Pro v3.0.0-10 has been released (full changelog)

New

• Add support for loading enviroment variables into workers from a .env file with --dotenv flag
• Include extra information in JSON reports: test metadata, test run ID, and ensure check results
• Include launch platform and subnet information in environment summary output
• Test runs are automatically tagged with test_run_id tag, which is set to the test run ID
• Print test run ID at the end of the test run

Fixes & Improvements

• Exclude raw histogram data from JSON reports to reduce file size
• Fix an issue which could prevent ensure checks from running
• Fix an issue where only one --secret flag would be taken into account
• Show auto-guessed subnet IDs when subnets aren't set explicitly with --subnet-ids in run-test

Artillery Probe

We have launched the first version of Artillery Probe, a smart and user-friendly HTTP client built into Artillery

• Send HTTP requests from the command line
• Send JSON, arbitrary request bodies, set custom headers, querystrings, forms, and more
• See request performance waterfalls
• Syntax highlighting for responses
• Query JSON responses with JMESPath
• Set expectations and checks on the command-line for quick acceptance testing and smoke testing

More details on our blog: https://www.artillery.io/blog/swiss-army-knife-for-http-testing

Artillery Probe is available in the most recent release of Artillery:

npm install -g artillery@latest

Artillery CLI

• Improve validation of test scripts
• Add wait action to WebSocket scenarios
• Fix an issue with aggregating summary metrics under certain conditions

Detailed changelog for the CLI is available on https://github.com/artilleryio/artillery/releases

Dashboard

• Improve test status indicators
• Fix issue with tooltip positioning on metric charts
• Fix keyboard navigation in tag search dropdown

Website and Docs

• Typography & readability improvements

Docs

• Documentation search is now available
• A whole lot of polish to improve readability: layout, typography, colors, layout & spacing
• Need to get to the docs quickly? Try docs.art

Dashboard

Polish & fixes:

• Add horizontal scrolling to the main load test view when the number of column exceeds display size
• Load test view: clicking on tag in the sidebar will filter for that tag
• Order of charts in the sidebar matches that on the page
• Fix Y-axis overflow in load test list view
• Fix bug with allowing duplicate tag selection
• Fix "Copy JSON" button on load test report view
• Fix for an exception shown when some metrics from a test run are missing
• Fix spacing between tags in load test view
• Fix filter icon colors
• Fix styling of "clear" button in tag filter component

Artillery CLI

• Fixed issue with saving large test run results produced by long-running (8+ hours) load tests
• Fixed issue with aggregating summary metrics at the end of a test run under certain conditions
• Add --max-duration flag to run-test command to set a limit on running time of a test
• Handle Fargate throttling when launching multiple large load tests in parallel

Artillery Operator

New kubectl plugin! Use `kubectl artillery scaffold' to scaffold tests for existing Kube services. Available from GitHub releases on https://github.com/artilleryio/artillery-operator/releases/tag/v0.1.1

Dashboard

Delete load test results

Sometimes you don't need to keep a test result around - maybe the run did not complete and you don't need the report, or perhaps you ran the test in the wrong environment and the results are meaningless. Whatever the reason, you can now delete test results you don't need.

In case you missed it in the previous changelog, this can be done from the CLI too with the - Add delete-test-run command.

Artillery CLI

Artillery v2.0.0-13 has been released:

• Fix issue with phase names sometimes not being printed
• Add support for matching multiple headers with the same name in the expectations plugin (see an example in the docs)
• The expectation plugin, endpoint metrics plugin, and external metric publisher plugin are now bundled with Artillery

Dashboard v0.5.0

(Artillery Pro Dashboard is in early beta with a small number of Artillery Pro customers. If you'd like to join the beta, please let us know via team@artillery.io)

Last week we made a few changes focused on improving querying & filtering, providing more context around test run data, and adding a bit of extra polish.

More powerful filtering

You can now search & filter by:

• tags
• date range
• status
• or a combination of the above

We also improved the tag filtering input and made tag filter query easier to read.

Customize main load test view

You can now customize the columns shown in the main load test view to be able to find what you're looking for faster.

More test metadata

We've exposed more metadata for test runs in the sidebar. You can now see more information for each test run now, such as:

• Full artillery command used to trigger the test run
• CPU and memory configuration for the workers
• Version information for Artillery CLI used to trigger the test
• Names of secrets used by the test run

Progress indicators

Sometimes filtering and searching can take a bit of time, especially for more complex queries. We've added loading indicators in a few places to make it clear that the UI is busy doing the thing you asked it to do.

Fixes

• Fixed bug which would cause the view to scroll to the top when viewing Notes
• Fixed "Finished at" time being shown incorrectly for tests which are in progress

Artillery Pro v3.0.0-6

• Add delete-test-run command
• describe-test-run returns extra metadata about a test run:
  • cliArgs - full CLI args that a test was run with
  • secrets - names of secrets passed to the test run
  • platformConfig - platform-specific configuration, e.g. CPU and memory config on Fargate/ECS