Artillery

This update is focused on improving visibility of test phases for tests in progress.

New Activity widget

• Improve current test phases visibility for tests in progress
• Outline important test information in a easy to read widget

Improve Test Logs

• Move Worker Logs next to Text log to improve logs discoverability
• Improve Text log loading state by showing CLI loader
• Add auto scroll to Text log
• Improve readability and contrast on Text log widget
• Update Text log to always show scrollbar if content is scrollable

Bug fixes & Improvements

• Improve design of error screens
• Group vusers metrics under one chart on Test Reports
• Add tooltip to display properties button
• Improve layout of Header and Load Test view on for screen sizes medium and smaller
• Animate Metadata duration to match real-time data elsewhere
• Fix Metadata duration showing 0 for tests in progress
• Fix issue where Text log would show “N/A” on page load
• Fix a bug with tags not showing for some organizations
• Fix page zoom when adding notes to tests on mobile devices
• Fix test list view alignment if user info is not available

This update is focused on many small improvements to improve overall information density and make it easier to understand load testing activity at a glance, as well as to make it easier to analyze individual load test runs.

New features

Test list view

• The results of checks can now be inspected inline. The count of notes added to a test will is now also shown inline.
• Elapsed time is now updated live for in-progress tests
• Display relative dates in the list view, and show the full timestamp on hover
• Tests that run on AWS Lambda and AWS Fargate will now be shown with the appropriate icon for those services

Load test view

• All timeseries charts will now overlay the name of the load phase and show absolute timestamps for X axis labels
• Add metric toggles to for hiding/showing specific aggregations on HTTP Latency Distribution widget
• Make metadata view more compact and easier to scan
• Improve empty states on Checks, Metrics Explorer and Errors widgets
• JSON reports can now be downloaded from the context menu

Bug fixes & Improvements

• The list of tags will now show more tags by default to increase information density
• The name tag is no longer shown in the list of tags (the name tag sets the name of the test shown in the list)
• Match styling of tags in filter dropdown to styling used elsewhere in the UI
• Fix a bug with date filter not always resetting the selected range as expected
• Fix a bug in date filter where the displayed range could be off by one day
• Fix a bug in date filter where future dates would be shown in date filter
• Fix content overflow on Errors widget when there's a lot of error data
• Fix a bug where filtering by tags would not work correctly
• Fix a bug where filtering by status would not work correctly for tests that included failed checks

New features

• Dynatrace support - the official publish-metrics plugin can now be configured to send Artillery metrics and events to Dynatrace. Read more in the docs
• config.http.defaults - default headers and cookies for all requests in a test may now be set via config.http.defaults

Fixes & improvements

Artillery Dashboard

• Send load phase start/end timings for visualization
• Send CLI exit code to display in test run results
• Fix test run status not always getting reported correctly
• Fix bug which caused test runs on AWS Lambda to not be tagged correctly

Core

• Allow variables set with --variables to be used anywhere in config
• Fix for encoding of array values in query strings
• Make sure CLI exit handlers are always run
• Fix bug affecting match groups in regexp captures

Plugins

• publish-metrics - Exit early if required config is missing for publish-metrics
• publish-metrics - Reduce memory usage of the plugin
• expect - add useRequestNamesOnly option
• ensure - fix bug where ensure config under config.plugins.ensure would not get picked up

We've reorganized the layout of individual test reports to make notes more prominent, improved readability of the metadata section, and created more room for useful visualizations in the main area.

Test view sidebar

Notes

Notes provide a way to annotate test runs with useful information such as links to external logs and dashboards, and for recording results of investigations into performance issues found by load tests.

We've now pulled them out from a separate view to make them more accessible.

Other fixes & improvements:

• Remove shadow overlap on first note
• Tweak microcopy to make actions clearer

Metadata display

• Test timing information has been moved into the sidebar to give more space for data widgets in the main area
• Make Metadata section collapsible
• Improve spacing on list of CloudWatch Log Streams for Fargate tests
• Move "Add note" to the top so notes are added right below it, better UX
• Tweak styling of rendered Markdown for better readability
• Improve handling on small screens
• Improve alignment and padding of metadata items

Load test visualizations & insights

• Fix issue that could cause all charts to rerender when not necessary and lead to charts jumping around
• Improve responsiveness of Page Performance widget
• Improve spacing on Error summary view
• Add empty state to Counters summary
• Test IDs can be copied by clicking them

Other fixes & improvements

• Fix bug that could cause new tests to show up as "Untitled Test" for a short period of time
• Fix bug that prevented deep links to charts from working
• Error dialogs and error handling. Error responses in both the UI and API will now include unique error IDs that are tied to error logs
• Fix bug that would cause some pages to show loading skeleton indefinitely for logged out users
• Fix alignment of loading indicators on buttons across the app
• Improve contrast for all popovers and dropdown menus across the app
• Make styling of all external links consistent through the app
• Remove banding on the gradient in the header

Improved page performance visualizations

See how Core Web Vitals change under load

You can now see Core Web Vitals measurements for an entire test run, and understand how user experience is affected by high load.

Switch between p99, p95 and median

We have added the ability to switch between p99, p95 and median aggregations for visualizing Core Web Vitals metrics.

Other improvements

• Improved layout and spacing for tooltips and charts
• Improved handling of overlapping text.
• Improved readability with tweaks to x axis labels and URL formatting
• Improved toggle interactions and animations

New quickstart wizard

We have added a new quickstart wizard which lets you go from first sign-in to running a load test in 3 steps.

Other fixes & improvements

• Display the user who triggered a test in the metadata sidebar
• Add option to display the user who triggered the test in the test list view
• Fix issue where dialogs could close themselves accidentally
• Fix incorrect default labels on some charts
• Improve default error pages
• Fix regression on response time widget displaying an unnecessary count metric

Frontend performance metrics

Core Web Vitals is a set of key frontend performance metrics that measure user-perceived performance of a web application.

The dashboard will now visualize the performance of every page for load tests that use Artillery's Playwright integration. This will help you understand how user-perceived performance is affected by high load.

Fixes & Improvements

• Add syntaxt highlighting on JSON report views
• Tail text logs for in-progress load tests
• Replace popover with inline editing for notes
• Notes can now be posted with Ctrl + Enter

See results of ensure checks at a glance

The main load test list now shows the results of ensure checks so that you can see at a glance which tests succeeded and which ones did not meet performance thresholds.

We recommend using ensure checks for all regular tests to help make sure that performance goals are met.

Simplified status filters

We have reduced the number of status filters down to just four: Succeeded, Failed, Cancelled, and In Progress. A test run that sets ensure checks is marked as Succeeded only if all checks are successful.

A fresh lick of paint

We have done a comprehensive overhaul across the whole interface of the application, updating component surfaces, font sizes, animations, and more, to make the UI feel more consistent, accessible, and easy on the eyes.

Settings management

The Settings area has been updated to make navigation easier and now includes the ability to edit organization and personal details.

It's now also possible to pick a "system theme" which makes our UI follow your operating system's light/dark theme.

Fixes & improvements

• Performance improvements: faster loading of individual load tests, and settings pages
• Fix bug with searching by test run ID that would cause older test runs to be omitted
• Make Settings navigation fully responsive on mobile
• Add infinite scroll for the main load test list view

Saved views

To make getting to the information you want faster, the dashboard supports Saved Views now. Any combination of tag filters, date filters, and test status filters can be named and saved as a view.

Team member and auth management

You can now manage current team members, and the GitHub allowlist that determines who may join your organization from the Settings area.

Live test updates

Overview pages are now live and can be used to follow the progress of an in-progress load test.

Denser and more scannable list view

We improved information density and scannability of the main load test list view. The default page size has also been increased from 10 to 50.

API key management

Create API keys to use with the Artillery CLI or to create custom integrations with Artillery Cloud.

Performance and reliability improvements

We spent a lot of time on refactoring and optimization. End result: a leaner, snappier and more responsive UI.

Fixes & Improvements

• Show the name of current organization in the header
• All execution platforms can send data to the dashboard now - local, AWS Lambda and AWS ECS/Fargate
• Allow flash messages to stack
• Improve responsiveness of side navigation on test reports
• Add tooltips for tags in the main list view
• Fix bug with the direction of the arrow on the user dropdown menu
• Match loading skeletons with layout across the app
• Make dialog entry/exit animations consistent
• Fix dialogs not showing if page height was larger than viewport size
• Fix bug when active tab inside load test view would not be highlighted correctly
• Improve spacing, grid styling, and tooltip styling on all charts

We have released a big update to the Artillery Dashboard with a focus on improving the Overview section and our charting engine to let you understand load test results faster.

Load Test Overview

• Show Peak RPS achieved during a load test run
• Display errors (if any) at the top of the screen
• New "Load Summary" chart to visualize request rate, response time, and VUs created across the test run

• Show a breakdown of HTTP codes by URL

• New "Metric Explorer" widget to let you chart and compare any metric (including custom metrics) generated by the test run

Fixes, Tweaks and Polish

• Layout: tweaks to spacing across the board to increase information density and make more room for what matters. Improved layout of the main test list view and individual test views.
• Charting: ability to combine multiple metrics on one chart
• Charting: show combined tooltips for each point along the time axis
• Fix for sidebar and main area overlapping at some screen sizes
• Fix issue with flash messages disappearing from view
• Improve tag list display in the main list view

New Overview section

The new Overview section provides at-a-glance view of what happened during a load test, including:

• Status of ensure checks.
• A summary view of all errors raised during the test (if any), such as request timeouts, dropped TCP connections etc.
• A summary view of HTTP codes, HTTP request rate, and response time distribution (min, max, p999, p99, and p95).

Notes

Capture the results of performance investigations and link to relevant external resources such as Datadog dashboards, Kibana logs, or JIRA tickets. With full Markdown support, and ability to post notes via the CLI.

Charting improvements

• Color palette improvements to improve readability
• Fixed an issue with displaying charts for tests that generate a lot of data, such as 8-12 hour long soak tests
• Fixes to zooming and panning

Light mode

And finally, if dark mode isn't your thing, the dashboard has a light mode now. Toggle it on via Profile -> Light Mode.

Artillery CLI

New HTML reports

We have improved the layout and styling of reports generated with artillery report.

--dotenv flag

You can now load environment variables from a file with the --dotenv flag. Those env vars are available in Artillery scenarios through the $processEnvironment variable.

Dashboard

Visual polish

• Improved homepage responsiveness
• Updated load test table with new context tags and new flexible tags
• Updated context menus
• Updated header styling in load test view.
• New and improved selected tags filter bar
• Show loading skeletons while data is being fetched

CDK installer

We released an official AWS CDK-based installer for Artillery dashboard.

https://github.com/artilleryio/installer-aws-cdk

Flexible deployment modes

The dashboard can now be deployed in more than one way to fit the exact requirements of your organization:

• As an internal-facing, inside a VPC. This is the default deployment mode, and it will require an existing VPN setup to access the app. TLS is optional with this setup.
• As a publig public-facing, which requires TLS with a pre-provisioned ACM certificate.
• Custom domain may be configured on top of the provisioned ALB DNS name.

GitHub Authentication

We added support for authentication through GitHub.

Aurora Serverless v1 Support

An Aurora Serverless database cluster may now be used as the backing store for Artillery Pro.

This requires Data API to be enabled on the cluster. Only Aurora Serverless v1 is supported right now.

Dashboard

Fixes & Improvements

• Improve loading states on the Settings screen
• Fix bug with context menu getting cut off
• Fix double-trigger on context menu buttons

Artillery Pro

Artillery Pro v3.0.0-10 has been released (full changelog)

New

• Add support for loading enviroment variables into workers from a .env file with --dotenv flag
• Include extra information in JSON reports: test metadata, test run ID, and ensure check results
• Include launch platform and subnet information in environment summary output
• Test runs are automatically tagged with test_run_id tag, which is set to the test run ID
• Print test run ID at the end of the test run

Fixes & Improvements

• Exclude raw histogram data from JSON reports to reduce file size
• Fix an issue which could prevent ensure checks from running
• Fix an issue where only one --secret flag would be taken into account
• Show auto-guessed subnet IDs when subnets aren't set explicitly with --subnet-ids in run-test

Dashboard

A long requested feature of our dashboard has been the ability to login with GitHub. A layer of authentication removing the need for VPN's on your end.

With the latest we're shipping:

• Authentication
• Fixed a bug relating to Load Test table horizontal scroll.
• Load tests now fail a little more gracefully with more robust checks for data.
• Also improvement on empty states when a Load Test is firing up.

Artillery Probe

• Added querying for XML/HTML responses with Cheerio

Kubectl

• Latest release of Kubectl plugin is out now! Find it here.

Artillery Probe

We have launched the first version of Artillery Probe, a smart and user-friendly HTTP client built into Artillery

• Send HTTP requests from the command line
• Send JSON, arbitrary request bodies, set custom headers, querystrings, forms, and more
• See request performance waterfalls
• Syntax highlighting for responses
• Query JSON responses with JMESPath
• Set expectations and checks on the command-line for quick acceptance testing and smoke testing

More details on our blog: https://www.artillery.io/blog/swiss-army-knife-for-http-testing

Artillery Probe is available in the most recent release of Artillery:

npm install -g artillery@latest

Artillery CLI

• Improve validation of test scripts
• Add wait action to WebSocket scenarios
• Fix an issue with aggregating summary metrics under certain conditions

Detailed changelog for the CLI is available on https://github.com/artilleryio/artillery/releases

Dashboard

• Improve test status indicators
• Fix issue with tooltip positioning on metric charts
• Fix keyboard navigation in tag search dropdown

Website and Docs

• Typography & readability improvements

Docs

• Documentation search is now available
• A whole lot of polish to improve readability: layout, typography, colors, layout & spacing
• Need to get to the docs quickly? Try docs.art

Dashboard

Polish & fixes:

• Add horizontal scrolling to the main load test view when the number of column exceeds display size
• Load test view: clicking on tag in the sidebar will filter for that tag
• Order of charts in the sidebar matches that on the page
• Fix Y-axis overflow in load test list view
• Fix bug with allowing duplicate tag selection
• Fix "Copy JSON" button on load test report view
• Fix for an exception shown when some metrics from a test run are missing
• Fix spacing between tags in load test view
• Fix filter icon colors
• Fix styling of "clear" button in tag filter component

Artillery CLI

• Fixed issue with saving large test run results produced by long-running (8+ hours) load tests
• Fixed issue with aggregating summary metrics at the end of a test run under certain conditions
• Add --max-duration flag to run-test command to set a limit on running time of a test
• Handle Fargate throttling when launching multiple large load tests in parallel

Artillery Operator

New kubectl plugin! Use `kubectl artillery scaffold' to scaffold tests for existing Kube services. Available from GitHub releases on https://github.com/artilleryio/artillery-operator/releases/tag/v0.1.1

Dashboard

Delete load test results

Sometimes you don't need to keep a test result around - maybe the run did not complete and you don't need the report, or perhaps you ran the test in the wrong environment and the results are meaningless. Whatever the reason, you can now delete test results you don't need.

In case you missed it in the previous changelog, this can be done from the CLI too with the - Add delete-test-run command.

Artillery CLI

Artillery v2.0.0-13 has been released:

• Fix issue with phase names sometimes not being printed
• Add support for matching multiple headers with the same name in the expectations plugin (see an example in the docs)
• The expectation plugin, endpoint metrics plugin, and external metric publisher plugin are now bundled with Artillery

Dashboard v0.5.0

(Artillery Pro Dashboard is in early beta with a small number of Artillery Pro customers. If you'd like to join the beta, please let us know via team@artillery.io)

Last week we made a few changes focused on improving querying & filtering, providing more context around test run data, and adding a bit of extra polish.

More powerful filtering

You can now search & filter by:

• tags
• date range
• status
• or a combination of the above

We also improved the tag filtering input and made tag filter query easier to read.

Customize main load test view

You can now customize the columns shown in the main load test view to be able to find what you're looking for faster.

More test metadata

We've exposed more metadata for test runs in the sidebar. You can now see more information for each test run now, such as:

• Full artillery command used to trigger the test run
• CPU and memory configuration for the workers
• Version information for Artillery CLI used to trigger the test
• Names of secrets used by the test run

Progress indicators

Sometimes filtering and searching can take a bit of time, especially for more complex queries. We've added loading indicators in a few places to make it clear that the UI is busy doing the thing you asked it to do.

Fixes

• Fixed bug which would cause the view to scroll to the top when viewing Notes
• Fixed "Finished at" time being shown incorrectly for tests which are in progress

Artillery Pro v3.0.0-6

• Add delete-test-run command
• describe-test-run returns extra metadata about a test run:
  • cliArgs - full CLI args that a test was run with
  • secrets - names of secrets passed to the test run
  • platformConfig - platform-specific configuration, e.g. CPU and memory config on Fargate/ECS

Welcome to our brand new changelog where we will keep you updated on new releases, features, fixes and improvements in our products.

Artillery Pro Dashboard

Searching & Filtering

You can now filter the list of load test results by multiple tags.

You can further refine a tag filter with test status. For example to show all test runs tagged with team:sre and service:checkout-service which did not complete successfully.

Chart navigation

You can now jump straight to a chart for a specific metric from the sidebar.

Zooming and panning

We added support for zooming and panning to metric charts.

Update notifications

When a new release of the dashboard is available, the app will show a notification in the top-right corner, along with a link to the changelog and upgrade instructions.

Kubernetes Support

We have released an early preview of the Artillery Operator for Kubernetes. The goal of the project is to make Kubernetes-specific workflows easier with Artillery. We'd love to hear your feedback, comments, and suggestions.

Read the announcement on our blog: https://www.artillery.io/blog/artillery-operator-alpha-announced