Skip to main content

Changelog

v3.0.0-21​

Fixes & Improvements​

  • Update Artillery version used in cloud workers to v2.0.0-26
  • Improve handling of arguments to the --tags flag in run-test command

v3.0.0-20​

Fixes & Improvements​

  • Improve test startup time by skipping bundling of built-in plugins
  • Performance improvements for data operations on Aurora Serverless

v3.0.0-19​

Fixes & Improvements​

  • [list-tags] Improve handling of a large number of tags
  • Update Playwright to v1.25.1

v3.0.0-18​

Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.

v3.0.0-17​

Fixes & Improvements​

  • [list-tags] Improve handling of a large number of distinct tags
  • Routine dependency updates

v3.0.0-16​

Fixes & Improvements​

  • Upgrade worker images to use Artillery v2.0.0-23

v3.0.0-15​

New features​

Fixes & Improvements​

  • [list-tests] Fix an issue which could lead to test run details not being displayed
  • Routine dependency updates

v3.0.0-14​

Fixes & Improvements​

  • [run-test] Fix issue where some worker ECS task ARNs would not get stored
  • [run-test] Fix issue which could lead to tests stopped because of an error to not get marked as such
  • [stop-test-run] Fix issue which could lead to stopped tests not getting marked as such
  • [list-test-runs] Add --max-items and --offset flags for pagination
  • [describe-test-run] Fix issue which could lead to intermediate metrics objects not being returned in sorted order
  • Update worker images to Artillery v2.0.0-21
  • General performance improvements

v3.0.0-13​

  • Add support for setting Artillery dashboard-specific configuration with set-config-value

v3.0.0-12​

Fixes & Improvements​

  • Fix issue which could prevent secondary deployments from being upgraded

v3.0.0-11​

Fixes & Improvements​

  • Fix issue with admin:run-db-migrations that would not mark migrations as done despite them completing successfully
  • Various minor fixes for the new Aurora Serverless v1 support

v3.0.0-10​

New features​

run-test​

  • add support for loading enviroment variables into workers from a .env file with --dotenv flag
  • include extra information in JSON reports: test metadata, test run ID, and ensure check results
  • include launch platform and subnet information in environment summary output
  • test runs are automatically tagged with test_run_id tag, which is set to the test run ID
  • print test run ID at the end of the test run

Improvements​

run-test​

  • exclude raw histogram data from JSON reports to reduce file size
  • fix an issue which could prevent ensure checks from running
  • fix an issue where only one --secret flag would be taken into account
  • show auto-guessed subnet IDs when subnets aren't set explicitly with --subnet-ids

describe-test-run​

  • pretty-print JSON output by default

Beta features​

v3.0.0-9​

Date: Apr 6, 2022

New features​

  • [run-test] Add --max-duration flag to set maximum running time of a test

Fixes & Improvements​

  • Improve handling of ThrottlingException errors when launching multiple tests with hundreds of workers on Fargate

v3.0.0-8​

Date: Apr 4, 2022

Fixes & Improvements​

  • Fix issue in describe-test-run command with reading test data created by earlier version of Artillery Pro

v3.0.0-7​

Date: Mar 29, 2022

Fixes & Improvements​

  • Improve handling of reports and logs generated by long-running tests

v3.0.0-6​

Date: Mar 8, 2022

New features​

  • Add delete-test-run command
  • describe-test-run returns extra metadata about a test run:
    • cliArgs - full CLI args that a test was run with
    • secrets - names of secrets passed to the test run
    • platformConfig - platform-specific configuration, e.g. CPU and memory config on Fargate/ECS

v3.0.0-5​

Date: Feb 8, 2022

Fixes & Improvements​

  • Allow secondary deployments of Artillery Pro in another region in the same AWS account
  • Performance improvements in list-test-runs

v3.0.0-4​

Date: Jan 10, 2022

Fixes & Improvements​

  • [list-test-runs] Add ability to filter results by tags
  • All of Artillery Pro commands are now exposed to Artillery v2
  • Routine dependency updates

v3.0.0-3​

Date: Dec 8, 2021

Fixes & Improvements​

  • Improve AWS SSO support. Artillery will detect expired SSO sessions rather than trying to fallback on another authentication method.
    • Note: AWS SSO session duration needs to exceed the duration of tests run with Artillery as credentials derived from SSO won't be refreshed automatically
  • Fix issue where default credentials refresh behavior of AWS SDK could stop working as expected
  • [run-test] Only create Artillery Fargate cluster when --cluster option is not set.

v3.0.0-2​

Date: Dec 3, 2021

Fixes & Improvements​

  • [run-test] Fix issue which would launch tests from the default cluster rather than one specified with the --cluster flag

v3.0.0-1​

Date: Dec 2, 2021

This is another preview release of the new v3 line of Artillery Pro.

tip
  • A backend upgrade is required to use new features in this release. It is not backwards-compatible with Artillery Pro v2.
  • Preview releases of Artillery Pro v3 support Artillery Core v1.7.x but we recommend starting the upgrade process to Artillery Core v2 as soon as possible

New Features​

  • Support for credentials received from AWS SSO
  • Support for running Playwright tests (requires Artillery Core v2)
  • Fargate workers will be launched with 4 vCPUs by default
  • When running tests with Artillery v2 multithreading is now on by default for better performance

Fixes & Improvements​

  • Some commands have been renamed for better consistency:
    • create-test is now create-test-bundle
    • list-tests is now list-test-bundles
    • delete-test is now delete-test-bundle
    • get-tests is now list-tests (to align with run-test and stop-test)
  • Fixed an issue where tests with more than 25 workers could fail on startup
  • Artillery workers can now use up to 12GB of memory
  • Speed improvements launching tests with a large number of workers on Fargate
  • Various reliability and stability improvements
  • There is now a limit of 50MB for captured console logs
  • Improved error messages

v3.0.0-0​

Date: Oct 19, 2021

First preview release of the new v3 line of Artillery Pro.

tip

A backend upgrade is required to use new features in this release. It is not backwards-compatible with Artillery Pro v2.

New Features​

  • Support for using Artillery Core v2
  • Add describe-test-run, list-test-runs, list-tags, list-secrets, and post-note commands
  • Artillery Pro can now be deployed to ap-east-1 (Bahrain) and me-south-1 (Hong Kong) regions
    • Note that Artillery Pro has to be deployed to one of those regions to be able to run tests from that region.
  • Add support for running tests in eu-west-3 (Paris)
  • [run-test]: test run IDs may be set with --test-run-id flag
  • [run-test]: a test run may be annotated with --note flag
  • [run-test]: --task-role-name may be used to provide additional IAM permissions to Artillery workers
  • [run-test]; --tags flag may be used to tag the test run
    • Note: a name tag is set automatically for every test run if not set explicitly
  • Node.js 14.17.6 (LTS) or higher is now the recommended version of Node.js for running Artillery Pro

Fixes & Improvements​

  • Add --json option to subscription command

2.3.0​

Date: Dec 2, 2021

Production release of v2.3 line. Please see changelogs for v2.3.0-0, v2.3.0-1 and v2.3.0-2 for details.

v2.3.0-2​

Date: July 27, 2021

  • Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
  • Important: Backend upgrade is required to use new features in this release

New Features​

  • Add get-tests command to list running tests
  • Add stop-test command to stop a running test gracefully
  • [deploy] Add ability to override the name of some CloudFormation resources with --set flag

Fixes & Improvements​

  • Fix an issue that could stop long test runs (tests running for >1 hour)
  • subscription command will now display plan name and backend version
  • Fix an issue that would prevent clean shutdown when trying to launch a test on ECS EC2 cluster with insufficient capacity
  • CloudWatch log group name will now include the name of the ECS cluster

v2.3.0-1​

Date: June 16, 2021

  • Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
  • Backend upgrade not required

New Features​

  • [run-test] ecs:fargate is now the default launch type
  • [run-test] Ignore SIGINT (Ctrl+C in the terminal). Test runs may be stopped cleanly by sending SIGTERM to the Artillery process

Fixes & Improvements​

  • [run-test] Artillery workers will use Artillery Core v1.7.3
  • [run-test] Fix issue where successful test runs would not exit immediately and the CLI would hang for some time
  • [run-test] Improve clean up of ECS tasks when a test run fails to start properly
  • Address security advisories in dependencies (as reported by npm audit)

v2.3.0-0​

Date: April 21, 2021

  • Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
  • Backend upgrade not required

New Features​

  • [run-test] Startup time improvements for tests with large npm dependency trees
  • [run-test][`ensure`](https://artillery.io/docs/guides/guides/test-script-reference#Setting-success-conditions-with-ensure) conditions are now fully supported
  • [run-test] Unique worker IDs are now available via the WORKER_ID environment variable
  • [run-test] Worker wait timeout may now be customized via ECS_WAIT_TIMEOUT environment variable
  • [run-test] Artillery workers will use Artillery Core v1.7.2 with its upgraded HTTP engine, which improves throughput for HTTP-based tests

Fixes & Improvements​

  • Node.js >= 12.13.0 (LTS) is the recommended version for running Artillery Pro

v2.2.3​

Date: March 19, 2021

Note: This release requires a backend upgrade with artillery deploy

New Features​

  • New set-secret command can be used to create encrypted secrets (such as API keys or passwords) to use in test definitions
  • [run-test] Add --secret <name> flag to make a secret available in a test run
  • [run-test] Add --launch-type flag to specify where tests should run from
  • Dev account support

Changes​

  • [run-test] The --fargate is deprecated in favor of --launch-type ecs:fargate

Fixes & Improvements​

  • [AWS] Handle cases when there's no InternetGateway attached to the default VPC
  • Miscellaneous internal improvements

v2.2.2​

Date: November 17, 2020

Fixes & Improvements​

  • Fix an issue which could prevent more than 3 concurrent test runs under some conditions

v2.2.1​

Date: October 27, 2020

Fixes & Improvements​

  • Address third-party dependency deprecation notices and other issues
  • Minor performance improvements
  • Fix issues when running tests with >100 workers under some conditions

v2.2.0​

Date: March 31, 2020

Please make sure update the [Artillery Pro backend](https://artillery.io/docs/pro/setting-up/#deploy-the-backend-with-artillery-deploy) after upgrading the CLI before running any tests.

New Features​

  • Artillery Pro tests may now be run from private subnets with no public internet access with AWS PrivateLink to enable running Artillery Pro tests in highly sensitive "air gapped" environments.
  • [run-test] The --region option is now optional. The region that the Artillery Pro backend is deployed to will be used when running tests.
  • [run-test] The --public-subnet-ids option is now optional when running tests on Fargate. If not set, public subnets in the default VPC will be used.
  • [run-test] The --public-subnet-ids option has been replaced by --subnet-ids and is scheduled for deprecation in a future release.
  • [run-test] ulimit settings passed to containers may now be customized via --launch-config option. The default nofile limit is now set to 8192 to support high-concurrency tests.

Fixes & Improvements​

  • Startup time improvements for tests running in EU and AP regions

v2.2.0-3​

Date: December 19, 2019

This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.

New Features​

Fixes & Improvements​

  • Fixed an issue with could sometimes stop tests that make use of a large number of npm dependencies from starting up correctly.

v2.2.0-2​

Date: December 13, 2019

Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.

v2.2.0-1​

Date: October 14, 2019

This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.

Fixes & Improvements​

  • Fixed an issue with setting NPM_REGISTRY to a custom URL
  • Fixed RPS values reported by run-test command being lower than actual values under some circumstances

v2.2.0-0​

Date: October 9, 2019

This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.

New Features​

  • Private NPM registries are now supported - configure access to private npm modules
  • run-cluster command has been renamed to run-test. Either may still be used to run tests, but run-cluster will be deprecated at some point in the future in favor of run-test.
  • Add set-config-value command which allows for various Artillery Pro configuration values to be set

Fixes & Improvements​

  • CSV payload paths may now contain variables
  • Various performance improvements in the CLI and workers
  • An issue with create-test which could result in incorrect test names to be set under some conditions

v2.1.3​

Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.

v2.1.2​

Fixes​

  • Various fixes for auto-detection of npm modules and other assets required by test scripts.

v2.1.1​

Fixes​

  • Fix for artillery deploy where --license-key-file option would not work as expected under some conditions

v2.1.0​

New Features​

  • Yarn is now supported. If a yarn.lock file can be found, yarn will be used to install test dependencies by the workers instead of npm
  • run-cluster and create-test commands will now list the files which will get packaged up and made available to workers

Fixes​

  • Fix a bug with handling of @scoped dependencies in custom JS code

v2.0.0​

New Features​

  • 9 more AWS regions are now supported. Artillery Pro can now be deployed to any of the following 13 regions: us-east-1, us-east-2, us-west-2, eu-west-1, us-west-1, ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, eu-central-1, eu-west-2, or ca-central-1.
  • activate-pro command has been deprecated to make Artillery Pro CLI feel more like the official AWS CLI. End-users of the CLI such as developers or testers using run-cluster as well as machine users such as CI/CD jobs will no longer need to activate their installation of Artillery Pro CLI, as long as they have adequate IAM permissions to interact with the Artillery Pro backend deployed to their team's AWS account.
  • setup-pro command has been renamed to deploy. Any existing scripts using setup-pro will continue working, but setup-pro but will be deprecated in a future release.
  • config.includeFiles can be used to provide a list of files to package up along the test scripts and any custom JS functions
  • New subscription command to check subscription status and to request license keys

Optimisations And Fixes​

  • Better progress and error messages in run-cluster
  • Worker Docker image size reduced by 25% - faster startup time on Fargate
  • Default memory allocation reduced on ECS
  • Workers now run Node.js 12.2.0 which offers better performance
  • Improved error handling and signaling in workers
  • Improved CloudWatch logging from workers
  • Fix for --overrides in run-cluster which could lead to workers crashing under certain conditions
  • Fix for plugins not always being detected correctly
  • Fix for deploy failing under certain condition with IAM
  • Support HTTP responses where headers exceed 8kb in size
  • Fix for a race condition in run-cluster which could lead to CLI not detecting that a test run has completed successfully
  • Better detection of ECS/Fargate scheduling errors
  • Various other minor performance improvements in the CLI and the backend

v1.3.3​

  • Artillery workers will now use 1024 CPU units (equivalent to 1 vCPU) and 2048 memory units by default on both Fargate and classic ECS to improve performance
  • Fixes an issue where the CLI would sometimes hang indefinitely due to getting into an inconsistent internal state because of Fargate taking too long to provision required containers
  • The CLI will now provide progress updates as Fargate provisions the required resources (number of tasks ready / being provisioned / pending)
  • Improvements to the accuracy of intermediate reports
  • Improves Node v11 support

v1.3.2​

  • Fix for cpu/memory overrides with --launch-config sometimes not being set correctly when running on Fargate
  • Fix for subscription limits sometimes not being applied correctly
  • Activation with a Github account now requires the use of the --github flag
  • Artillery Pro will now print a notification when an update is available
  • Upgrade worker containers to use Node.js 10
  • Various improvements to error handling & providing friendlier error messages

v1.3.1​

  • Stability improvements for the installation/upgrade process of backend components
  • Speed improvements in metric aggregation & reporting

v1.3.0​

  • Serverless load testing with Fargate is here! To use Fargate, add a --fargate flag, and tell Fargate which public subnets to use with --public-subnet-ids, and your test will run from serverless Fargate containers rather than managed EC2 instances. Any existing Artillery test script, any scale, without having to manage any EC2 VMs.
  • Fixed a bug which could cause run-cluster to hang and never exit despite the test completing successfully.
  • Various internal stability and performance improvements.

Important - Artillery Pro backend components need to be upgraded with artillery setup-pro before using v1.3.0.

v1.2.7​

  • ECS worker containers can be customized now (with ECS container overrides for cpu, memory, and environment) with --launch-config
  • Custom DEBUG variable may be set to customize debug output in CloudWatch logs for a test run
  • Alpha AWS Fargate support (get in touch on team@artillery.io if you'd like to give it a go)
  • Various internal improvements

v1.2.6​

  • Autodetect plugins used in a test script and make them available in worker containers running on ECS
  • Performance improvements

v1.2.5​

  • File uploads: setContentLengthHeader option may now be set
  • Non-flat custom JS code dependencies are resolved properly now (for test scripts that load a custom JS module which loads other local modules in turn)
Table of contentsOn this page