Increase maxConcurrentRecordings from 3 to 5 to increase the probability of capturing traces for failed VUs (#3533 )
Increase upload timeout for traces to help make sure large trace recordings are uploaded to Artillery Cloud (#3533 )
Azure ACI
Client ID and client secret must be provided via AZURE_CLIENT_ID and AZURE_CLIENT_SECRET environment variables rather than CLI flags. This brings Artillery CLI in line with Azure SDK’s DefaultAzureCredential credential chain. (#3525 )
Add support for overriding worker startup timeout via WORKER_WAIT_TIMEOUT_SEC environment variable (#3527 )
Fix issue that caused tests comprised of a single TypeScript file to fail to run (#3528 )
The report command has ben removed. As an alternative consider setting up Artillery Cloud for visualizing test metrics, or setting up an OpenTelemetry integration with an external monitoring system with the publish-metrics plugin. (#3431 )
Node.js v22.13.0 (current active LTS) is the recommended version of Node.js for running Artillery now
Add --env-file flag as an alternative for --dotenv flag. This makes it consistent with the Node.js --env-file flag. The --dotenv flag will be deprecated in a future release (#3376 )
Add tracking of response times by HTTP status code. A new set of metrics (e.g. http.response_time.2xx or http.response_time.5xx) is now reported to provide more granular view of response times in a test (#3326 )
Fix an issue that caused incorrect “multiple batches of metics” warnings when running tests with pause phases (#3331 )
Playwright
Upgrade Playwright to v1.48.0
Azure
Fix: Make values loaded from an env file with --dotenv / --env-file flag available to workers containers (rather than just the Artillery process running inside the worker) (#3376 )
Stagger startup of containers in large load tests to prevent rate limit errors from Azure services (#3371 )
OpenTelemetry
Add support for setting resource-level attributes (#3335 )
Artillery Cloud
Send CI related information to Artillery Cloud. This makes the following information available in Artillery Cloud:
Whether a test run was triggered in CI or not, and which CI service was used
For tests triggered in GitHub Actions - provide a link back to the job run on GitHub Actions
Dependencies
Upgrade json-plus to address a critical security vulnerability (#3369 )
Artillery will now always use Lambda Container Images to run tests on AWS Lambda. The --container flag introduced in v2.0.12 is no longer required and will be removed in the next release. Creating Lambda functions from .zip archives is no longer supported.
All existing tests should continue running normally without needing any changes. If you run into any issues please open an issue .
This change brings improved startup times for tests running on AWS Lambda and brings AWS Lambda support to parity with AWS Fargate.
Fixes
Increase startup timeout time for Lambdas running in a VPC
Exit with non-zero code if a Lambda worker fails
Fix issue that could lead tests to crash when triggered from a Windows machine
Expose current worker ID to Lambda workers as WORKED_ID environment variable
Playwright
Accept 0 values for defaultNavigationTimeout and defaultTimeout settings
A warning will be printed if a Playwright testFunction cannot be found
Artillery can now use Lambda Container Images to run tests on AWS Lambda. Pass the --container flag to run-lambda command to use a container instead of a zip file. No further configuration is required.
This change brings two main benefits:
Much faster startup times for tests
Ability to use certain features that were previously only available in tests running on AWS Fargate:
Run test scripts that TypeScript hooks
Use third-party dependencies
Use the built-in publish-metrics plugin
Container-based Lambda functions will become the default option in a future release of Artillery.
Playwright
Update Playwright to v1.44.0
Fix tests failing on Fargate with ENOSPC error with Playwright tracing enabled
Improve reliability of trace collection:
Reduce trace collection cooldown period to 1-5m
Increase the number of concurrent trace recordings to 3 up from 1
Track the number of collected/discarded recordings with browser.traces_collected and browser.traces_discarded metrics
CLI & Core
Make path to the config directory available to test scripts via the special $dirname variable - docs
The --scenario-name flag will now default to using an exact string match, and fall back to using a regex
Clarify the meaning of --tags flag
Remove an unnecessary dependency
Slack integration
Artillery now includes an official Slack integration, which can be used to post test results to a Slack channel automatically
Make expect from @playwright/test work natively with Typescript support - Documentation
Use target as Playwright’s baseURL by default, allowing test scripts to include only relative paths in page navigations (e.g. page.goto('/docs') ) - Documentation
Fix missing phase events in Artillery Cloud when running Fargate tests
CLI & Core
Improve Typescript support to work with more packages, by exposing config.bundling.external to set packages that can’t be bundled as external - Documentation
Make test run id available to test runs via {{ $testId }} in scenarios/config and vuContext.vars.$testId in custom code
Routine dependency updates
Publish Metrics
Automatically send all traces and metrics sent via the opentelemetry reporter with a test_id attribute, making it easier to search for all relevant data from a particular test run
Fargate
Improve error message and handling for putting CloudWatch retention policy
Enable support for 12 more AWS regions (eu-west-2, eu-west-3, us-east-2, us-west-2, ca-central-1, ap-east-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, me-south-1, sa-east-1 )
Add support for sending Playwright and HTTP traces to Datadog APM - Documentation
Add support for --variables flag in run-fargate command - Documentation
Show supported regions when using the --region flag
CLI & Core
Add TypeScript support for custom code for tests running locally or on AWS Fargate. This is an experimental feature. - Documentation
Playwright
VUs will now run in isolated browsers contexts rather than new browser instances by default for better performance - Documentation
HTTP status codes are now tracked and reported through the browser.page.codes.<code> metric - Documentation
Fix bug in before/ after hooks that prevented the Playwright engine from working - Documentation
Add experimental TypeScript support. See the docs for current limitations - Documentation
OpenTelemetry
Fix issue with virtual users not failing when using test.step() with OpenTelemetry Playwright tracing enabled
OpenTelemetry tracing can now be used to send HTTP and Playwright engine traces to Datadog in Fargate tests - Documentation
Add support for smart sampling when using HTTP tracing with the OpenTelemetry reporter
Add time unit support for think steps in scenarios
metrics-by-endpoint plugin is now enabled by default to capture per-URL metrics in HTTP tests. The metrics will be available only in the JSON report and not output to the console, unless the plugin is loaded explicitly by the user.
Configure CPU and memory for Fargate tests with --cpu and --memory flags
Fix issue with $env not resolving correctly in the config section
Fix an issue with --scenario-name not picking up the correct scenario
Add a new fake-data plugin to allow for realistic test data to be generated in your tests - docs
Fix handling of metrics with non-alphanumeric characters in ensure checks
Improve error handling in the expect plugin
OpenTelemetry
Add experimental tracing support for Playwright tests, with tracing of test.step() calls, page navigation events, and Core Web Vitals metrics
The load test summary view will now calculate and display Apdex scores for load tests that make use the built-in apdex plugin.
Apdex (Application Performance Index) is an open standard for measuring performance of HTTP services. The Apdex score converts response times into a single score that reflects user satisfaction with the service.
Fixes & Improvements
App-wide
Update colors to improve legibility and consistency
Add exit animations for all dropdown-like elements
Improve loading states on all buttons
Improve consistency of dialogs everywhere
Improve consistency of spacing between elements across the app
Main view
Tweak the display of Artillery’s logo in the navbar
Fix responsiveness on Load Tests view
Improve UX for Saved Views and the Filter Bar with long input strings
Match test run ID icon in filter dropdown to one used elsewhere
Load test view
Make font sizes consistent in metadata panel
Improve styling of text log view, JSON log view, and table components
Fix issue with loading default CLI avatar images on notes
Fix issue with scroll gradient covering first note in the notes list
Improve spacing on Edit Note dialog
Charts
Add intelligent X axis scaling depending on duration of the test
AWS Fargate is a serverless solution that requires no infrastructure to be managed, and unlike AWS Lambda does not impose a restriction on the duration of your load tests.
Artillery’s Playwright integration is fully supported on Fargate, to let you run browser-based load tests at scale with ease.
Playwright support
Playwright support is now built into Artillery. The integration in artillery-engine-playwright is now bundled into the Artillery CLI.
The version of Playwright used by the integration has been updated to v1.3.4
Web Vitals tracking has been improved to provide more accurate measurements
Playwright scenario file may now be loaded via testFunction attribute rather than flowFunction
flow attribute may now be omitted in Playwright engine scenarios
Other fixes & improvements
Pass lambdaRoleArn to AWS Lambda engine correctly
Fix issue with returning matched groups from regexp capture
Track & report downloaded data in HTTP tests via http.downloaded_bytes metric
Environment variables may now be accessed through the special $env variable (the current way of accessing those via $processEnvironment is still supported but will be deprecated in future)
Dependency updates to address deprecation and security notices
AWS Lambda - when running tests on AWS Lambda, Artillery will automatically bundle any plugins loaded via config.plugins and engines loaded via config.engines
Checks & assertions: new notStatusCode check in the expect plugin
Datadog integration: includeOnly and excluded options may be used to configure the list of Artillery metrics which will be sent to Datadog
Fixes & improvements
When artillery run is used with the --output option, Artillery will check if destination exists before running the test
Miscellaneous dependency updates to address deprecation warnings and security advisories
AWS Lambda: the default IAM role and policy are now versioned. If you attached any custom permissions to the artilleryio-default-lambda-role role, you will need to re-attach it to the new artilleryio-default-lambda-role-20230116 role
CLI: The probe command has been removed from the CLI. The functionality has been moved to the new Skytrace CLI
New features
Add support for publishing test metrics to AWS CloudWatch
HTTP - compressed responses (e.g. gzip) will be decompressed by default now. (Set gzip: false to disable automatic decompression if required to improve performance.)
CLI - suggest closest match if a command name is mistyped
AWS Lambda
Print custom role ARN if provided
Add versioning to default Lambda role and policy created for running tests
Fixes & improvements
Fix arrival phases with arrivalRate and rampTo set to 0
Improve rampTo logic to result in more predictable and deterministic ramps
Fix issue with maxVusers setting that would lead to unexpected number of VUs to be generated
Fix wildcard message handling support in Socket.io
Fix issue with setting some properties on the global artillery object
AWS Lambda
Improve test startup logic: workers will wait for each other to be ready before kicking off the test run
Fix some intermediate metrics not being flushed & reported
Fix handling of memory-size and security-group-ids options
We have improved the layout and styling of reports generated with artillery report.
—dotenv flag
You can now load environment variables from a file with the --dotenv flag. Those env vars are available in Artillery scenarios through the $env variable.
Dashboard
Visual polish
Improved homepage responsiveness
Updated load test table with new context tags and new flexible tags
Updated context menus
Updated header styling in load test view.
New and improved selected tags filter bar
Show loading skeletons while data is being fetched
CDK installer
We released an official AWS CDK -based installer for Artillery dashboard.
The dashboard can now be deployed in more than one way to fit the exact requirements of your organization:
As an internal-facing, inside a VPC. This is the default deployment mode, and it will require an existing VPN setup to access the app. TLS is optional with this setup.
As a publig public-facing, which requires TLS with a pre-provisioned ACM certificate.
Custom domain may be configured on top of the provisioned ALB DNS name.
GitHub Authentication
We added support for authentication through GitHub.
Hassy Veldstra
Bernardo Guerreiro
