Changelog
v3.0.0-21
Fixes & Improvements
- Update Artillery version used in cloud workers to v2.0.0-26
- Improve handling of arguments to the
--tags
flag inrun-test
command
v3.0.0-20
Fixes & Improvements
- Improve test startup time by skipping bundling of built-in plugins
- Performance improvements for data operations on Aurora Serverless
v3.0.0-19
Fixes & Improvements
- [
list-tags
] Improve handling of a large number of tags - Update Playwright to v1.25.1
v3.0.0-18
Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.
v3.0.0-17
Fixes & Improvements
- [
list-tags
] Improve handling of a large number of distinct tags - Routine dependency updates
v3.0.0-16
Fixes & Improvements
- Upgrade worker images to use Artillery v2.0.0-23
v3.0.0-15
New features
- Access to the dashboard may now be set by GitHub org via the
GITHUB_ALLOWED_ORGS
config value. Note that this requires that the user's membership is set to public.
Fixes & Improvements
- [
list-tests
] Fix an issue which could lead to test run details not being displayed - Routine dependency updates
v3.0.0-14
Fixes & Improvements
- [
run-test
] Fix issue where some worker ECS task ARNs would not get stored - [
run-test
] Fix issue which could lead to tests stopped because of an error to not get marked as such - [
stop-test-run
] Fix issue which could lead to stopped tests not getting marked as such - [
list-test-runs
] Add--max-items
and--offset
flags for pagination - [
describe-test-run
] Fix issue which could lead to intermediate metrics objects not being returned in sorted order - Update worker images to Artillery v2.0.0-21
- General performance improvements
v3.0.0-13
- Add support for setting Artillery dashboard-specific configuration with
set-config-value
v3.0.0-12
Fixes & Improvements
- Fix issue which could prevent secondary deployments from being upgraded
v3.0.0-11
Fixes & Improvements
- Fix issue with
admin:run-db-migrations
that would not mark migrations as done despite them completing successfully - Various minor fixes for the new Aurora Serverless v1 support
v3.0.0-10
New features
run-test
- add support for loading enviroment variables into workers from a .env file with
--dotenv
flag - include extra information in JSON reports: test metadata, test run ID, and
ensure
check results - include launch platform and subnet information in environment summary output
- test runs are automatically tagged with
test_run_id
tag, which is set to the test run ID - print test run ID at the end of the test run
Improvements
run-test
- exclude raw histogram data from JSON reports to reduce file size
- fix an issue which could prevent
ensure
checks from running - fix an issue where only one
--secret
flag would be taken into account - show auto-guessed subnet IDs when subnets aren't set explicitly with
--subnet-ids
describe-test-run
- pretty-print JSON output by default
Beta features
- Add beta support for using Aurora Serverless v1 clusters as backing store
v3.0.0-9
Date: Apr 6, 2022
New features
- [
run-test
] Add--max-duration
flag to set maximum running time of a test
Fixes & Improvements
- Improve handling of ThrottlingException errors when launching multiple tests with hundreds of workers on Fargate
v3.0.0-8
Date: Apr 4, 2022
Fixes & Improvements
- Fix issue in
describe-test-run
command with reading test data created by earlier version of Artillery Pro
v3.0.0-7
Date: Mar 29, 2022
Fixes & Improvements
- Improve handling of reports and logs generated by long-running tests
v3.0.0-6
Date: Mar 8, 2022
New features
- Add
delete-test-run
command describe-test-run
returns extra metadata about a test run:- cliArgs - full CLI args that a test was run with
- secrets - names of secrets passed to the test run
- platformConfig - platform-specific configuration, e.g. CPU and memory config on Fargate/ECS
v3.0.0-5
Date: Feb 8, 2022
Fixes & Improvements
- Allow secondary deployments of Artillery Pro in another region in the same AWS account
- Performance improvements in
list-test-runs
v3.0.0-4
Date: Jan 10, 2022
Fixes & Improvements
- [
list-test-runs
] Add ability to filter results by tags - All of Artillery Pro commands are now exposed to Artillery v2
- Routine dependency updates
v3.0.0-3
Date: Dec 8, 2021
Fixes & Improvements
- Improve AWS SSO support. Artillery will detect expired SSO sessions rather than trying to fallback on another authentication method.
- Note: AWS SSO session duration needs to exceed the duration of tests run with Artillery as credentials derived from SSO won't be refreshed automatically
- Fix issue where default credentials refresh behavior of AWS SDK could stop working as expected
- [
run-test
] Only create Artillery Fargate cluster when--cluster
option is not set.
v3.0.0-2
Date: Dec 3, 2021
Fixes & Improvements
- [
run-test
] Fix issue which would launch tests from the default cluster rather than one specified with the--cluster
flag
v3.0.0-1
Date: Dec 2, 2021
This is another preview release of the new v3 line of Artillery Pro.
tip
- A backend upgrade is required to use new features in this release. It is not backwards-compatible with Artillery Pro v2.
- Preview releases of Artillery Pro v3 support Artillery Core v1.7.x but we recommend starting the upgrade process to Artillery Core v2 as soon as possible
New Features
- Support for credentials received from AWS SSO
- Support for running Playwright tests (requires Artillery Core v2)
- Fargate workers will be launched with 4 vCPUs by default
- When running tests with Artillery v2 multithreading is now on by default for better performance
Fixes & Improvements
- Some commands have been renamed for better consistency:
create-test
is nowcreate-test-bundle
list-tests
is nowlist-test-bundles
delete-test
is nowdelete-test-bundle
get-tests
is nowlist-tests
(to align withrun-test
andstop-test
)
- Fixed an issue where tests with more than 25 workers could fail on startup
- Artillery workers can now use up to 12GB of memory
- Speed improvements launching tests with a large number of workers on Fargate
- Various reliability and stability improvements
- There is now a limit of 50MB for captured console logs
- Improved error messages
v3.0.0-0
Date: Oct 19, 2021
First preview release of the new v3 line of Artillery Pro.
tip
A backend upgrade is required to use new features in this release. It is not backwards-compatible with Artillery Pro v2.
New Features
- Support for using Artillery Core v2
- Add
describe-test-run
,list-test-runs
,list-tags
,list-secrets
, andpost-note
commands - Artillery Pro can now be deployed to
ap-east-1
(Bahrain) andme-south-1
(Hong Kong) regions- Note that Artillery Pro has to be deployed to one of those regions to be able to run tests from that region.
- Add support for running tests in
eu-west-3
(Paris) - [
run-test
]: test run IDs may be set with--test-run-id
flag - [
run-test
]: a test run may be annotated with--note
flag - [
run-test
]:--task-role-name
may be used to provide additional IAM permissions to Artillery workers - [
run-test
];--tags
flag may be used to tag the test run- Note: a
name
tag is set automatically for every test run if not set explicitly
- Note: a
- Node.js 14.17.6 (LTS) or higher is now the recommended version of Node.js for running Artillery Pro
Fixes & Improvements
- Add
--json
option tosubscription
command
2.3.0
Date: Dec 2, 2021
Production release of v2.3 line. Please see changelogs for v2.3.0-0, v2.3.0-1 and v2.3.0-2 for details.
v2.3.0-2
Date: July 27, 2021
- Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
- Important: Backend upgrade is required to use new features in this release
New Features
- Add
get-tests
command to list running tests - Add
stop-test
command to stop a running test gracefully - [deploy] Add ability to override the name of some CloudFormation resources with
--set
flag
Fixes & Improvements
- Fix an issue that could stop long test runs (tests running for >1 hour)
subscription
command will now display plan name and backend version- Fix an issue that would prevent clean shutdown when trying to launch a test on ECS EC2 cluster with insufficient capacity
- CloudWatch log group name will now include the name of the ECS cluster
v2.3.0-1
Date: June 16, 2021
- Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
- Backend upgrade not required
New Features
- [run-test]
ecs:fargate
is now the default launch type - [run-test] Ignore
SIGINT
(Ctrl+C in the terminal). Test runs may be stopped cleanly by sendingSIGTERM
to the Artillery process
Fixes & Improvements
- [run-test] Artillery workers will use Artillery Core v1.7.3
- [run-test] Fix issue where successful test runs would not exit immediately and the CLI would hang for some time
- [run-test] Improve clean up of ECS tasks when a test run fails to start properly
- Address security advisories in dependencies (as reported by
npm audit
)
v2.3.0-0
Date: April 21, 2021
- Note: This is a preview release. If you run into issues, you can downgrade back to v2.2.3 safely
- Backend upgrade not required
New Features
- [run-test] Startup time improvements for tests with large
npm
dependency trees - [run-test][`ensure`](https://artillery.io/docs/guides/guides/test-script-reference#Setting-success-conditions-with-ensure) conditions are now fully supported
- [run-test] Unique worker IDs are now available via the
WORKER_ID
environment variable - [run-test] Worker wait timeout may now be customized via
ECS_WAIT_TIMEOUT
environment variable - [run-test] Artillery workers will use Artillery Core v1.7.2 with its upgraded HTTP engine, which improves throughput for HTTP-based tests
Fixes & Improvements
- Node.js >= 12.13.0 (LTS) is the recommended version for running Artillery Pro
v2.2.3
Date: March 19, 2021
Note: This release requires a backend upgrade with artillery deploy
New Features
- New
set-secret
command can be used to create encrypted secrets (such as API keys or passwords) to use in test definitions - [run-test] Add
--secret <name>
flag to make a secret available in a test run - [run-test] Add
--launch-type
flag to specify where tests should run from Dev
account support
Changes
- [run-test] The
--fargate
is deprecated in favor of--launch-type ecs:fargate
Fixes & Improvements
- [AWS] Handle cases when there's no InternetGateway attached to the default VPC
- Miscellaneous internal improvements
v2.2.2
Date: November 17, 2020
Fixes & Improvements
- Fix an issue which could prevent more than 3 concurrent test runs under some conditions
v2.2.1
Date: October 27, 2020
Fixes & Improvements
- Address third-party dependency deprecation notices and other issues
- Minor performance improvements
- Fix issues when running tests with >100 workers under some conditions
v2.2.0
Date: March 31, 2020
Please make sure update the [Artillery Pro backend](https://artillery.io/docs/pro/setting-up/#deploy-the-backend-with-artillery-deploy) after upgrading the CLI before running any tests.New Features
- Artillery Pro tests may now be run from private subnets with no public internet access with AWS PrivateLink to enable running Artillery Pro tests in highly sensitive "air gapped" environments.
- [
run-test
] The--region
option is now optional. The region that the Artillery Pro backend is deployed to will be used when running tests. - [
run-test
] The--public-subnet-ids
option is now optional when running tests on Fargate. If not set, public subnets in the default VPC will be used. - [
run-test
] The--public-subnet-ids
option has been replaced by--subnet-ids
and is scheduled for deprecation in a future release. - [
run-test
]ulimit
settings passed to containers may now be customized via--launch-config
option. The defaultnofile
limit is now set to 8192 to support high-concurrency tests.
Fixes & Improvements
- Startup time improvements for tests running in EU and AP regions
v2.2.0-3
Date: December 19, 2019
This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.New Features
- Private npm registries powered by Artifactory are now supported. Please refer to Configuring Access To Private npm Modules in the docs for the details.
Fixes & Improvements
- Fixed an issue with could sometimes stop tests that make use of a large number of npm dependencies from starting up correctly.
v2.2.0-2
Date: December 13, 2019
Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.
v2.2.0-1
Date: October 14, 2019
This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.Fixes & Improvements
- Fixed an issue with setting
NPM_REGISTRY
to a custom URL - Fixed RPS values reported by
run-test
command being lower than actual values under some circumstances
v2.2.0-0
Date: October 9, 2019
This is an early-access release of Artillery Pro for users who would like early access to most recent features. If you run into any issues, please get in touch on support@artillery.io.New Features
- Private NPM registries are now supported - configure access to private npm modules
run-cluster
command has been renamed torun-test
. Either may still be used to run tests, butrun-cluster
will be deprecated at some point in the future in favor ofrun-test
.- Add
set-config-value
command which allows for various Artillery Pro configuration values to be set
Fixes & Improvements
- CSV payload paths may now contain variables
- Various performance improvements in the CLI and workers
- An issue with
create-test
which could result in incorrect test names to be set under some conditions
v2.1.3
Maintenance release updating third-party dependencies to address deprecation notices and CVE advisories.
v2.1.2
Fixes
- Various fixes for auto-detection of npm modules and other assets required by test scripts.
v2.1.1
Fixes
- Fix for
artillery deploy
where--license-key-file
option would not work as expected under some conditions
v2.1.0
New Features
- Yarn is now supported. If a
yarn.lock
file can be found,yarn
will be used to install test dependencies by the workers instead ofnpm
run-cluster
andcreate-test
commands will now list the files which will get packaged up and made available to workers
Fixes
- Fix a bug with handling of @scoped dependencies in custom JS code
v2.0.0
New Features
- 9 more AWS regions are now supported. Artillery Pro can now be deployed to any of the following 13 regions: us-east-1, us-east-2, us-west-2, eu-west-1, us-west-1, ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1, eu-central-1, eu-west-2, or ca-central-1.
activate-pro
command has been deprecated to make Artillery Pro CLI feel more like the official AWS CLI. End-users of the CLI such as developers or testers usingrun-cluster
as well as machine users such as CI/CD jobs will no longer need to activate their installation of Artillery Pro CLI, as long as they have adequate IAM permissions to interact with the Artillery Pro backend deployed to their team's AWS account.setup-pro
command has been renamed todeploy
. Any existing scripts usingsetup-pro
will continue working, butsetup-pro
but will be deprecated in a future release.config.includeFiles
can be used to provide a list of files to package up along the test scripts and any custom JS functions- New
subscription
command to check subscription status and to request license keys
Optimisations And Fixes
- Better progress and error messages in
run-cluster
- Worker Docker image size reduced by 25% - faster startup time on Fargate
- Default memory allocation reduced on ECS
- Workers now run Node.js 12.2.0 which offers better performance
- Improved error handling and signaling in workers
- Improved CloudWatch logging from workers
- Fix for
--overrides
inrun-cluster
which could lead to workers crashing under certain conditions - Fix for plugins not always being detected correctly
- Fix for
deploy
failing under certain condition with IAM - Support HTTP responses where headers exceed 8kb in size
- Fix for a race condition in
run-cluster
which could lead to CLI not detecting that a test run has completed successfully - Better detection of ECS/Fargate scheduling errors
- Various other minor performance improvements in the CLI and the backend
v1.3.3
- Artillery workers will now use 1024 CPU units (equivalent to 1 vCPU) and 2048 memory units by default on both Fargate and classic ECS to improve performance
- Fixes an issue where the CLI would sometimes hang indefinitely due to getting into an inconsistent internal state because of Fargate taking too long to provision required containers
- The CLI will now provide progress updates as Fargate provisions the required resources (number of tasks ready / being provisioned / pending)
- Improvements to the accuracy of intermediate reports
- Improves Node v11 support
v1.3.2
- Fix for cpu/memory overrides with
--launch-config
sometimes not being set correctly when running on Fargate - Fix for subscription limits sometimes not being applied correctly
- Activation with a Github account now requires the use of the
--github
flag - Artillery Pro will now print a notification when an update is available
- Upgrade worker containers to use Node.js 10
- Various improvements to error handling & providing friendlier error messages
v1.3.1
- Stability improvements for the installation/upgrade process of backend components
- Speed improvements in metric aggregation & reporting
v1.3.0
- Serverless load testing with Fargate is here! To use Fargate, add a
--fargate
flag, and tell Fargate which public subnets to use with--public-subnet-ids
, and your test will run from serverless Fargate containers rather than managed EC2 instances. Any existing Artillery test script, any scale, without having to manage any EC2 VMs. - Fixed a bug which could cause
run-cluster
to hang and never exit despite the test completing successfully. - Various internal stability and performance improvements.
Important - Artillery Pro backend components need to be upgraded with artillery setup-pro
before using v1.3.0
.
v1.2.7
- ECS worker containers can be customized now (with ECS container overrides for
cpu
,memory
, andenvironment
) with --launch-config - Custom
DEBUG
variable may be set to customize debug output in CloudWatch logs for a test run - Alpha AWS Fargate support (get in touch on team@artillery.io if you'd like to give it a go)
- Various internal improvements
v1.2.6
- Autodetect plugins used in a test script and make them available in worker containers running on ECS
- Performance improvements
v1.2.5
- File uploads:
setContentLengthHeader
option may now be set - Non-flat custom JS code dependencies are resolved properly now (for test scripts that load a custom JS module which loads other local modules in turn)