Load testing with Playwright and Artillery
Built-in sincev2.0.0-33Playwright (opens in a new tab) is a modern browser automation framework by Microsoft. Artillery supports running Playwright-based scripts as load tests, including running Playwright at scale using AWS Fargate.
Features
- Write Playwright scripts with full access to
page API
(opens in a new tab) to run as load tests - Track Core Web Vitals (opens in a new tab) metrics and how they are affected by high load
- Create no-code tests with
playwright codegen
- Launch tens of thousands of headless Chrome instances with zero infrastructure to manage
→ See Why load test with headless browsers?
Current limitations
- TypeScript support is available but experimental. See Example 3 below.
- Only Chrome is available. Restricting the integration to just one browser improves startup time performance for large load tests and does not have any consequential effects on the results of load tests themselves.
- Not compatible with Lambda. For distributed load tests, please use AWS Fargate.
Usage
The Playwright engine is built into Artillery.
Create a new Artillery script in hello-world.yml
:
config:
target: https://www.artillery.io
# Load the Playwright engine:
engines:
playwright: {}
# Path to JavaScript file that defines Playwright test functions
processor: './flows.js'
scenarios:
- engine: playwright
testFunction: 'helloFlow'
Create a test function in flows.js
:
(Note: this script was generated with playwright codegen
(opens in a new tab). page
is an instance of Playwright page (opens in a new tab).)
module.exports = { helloFlow };
async function helloFlow(page) {
//
// The code below is just a standard Playwright script:
//
// Go to https://artillery.io/
await page.goto('https://www.artillery.io/');
// Click text=Pricing
await page.click('text=Cloud');
}
Run it:
artillery run hello-world.yml
Artillery will run the test and automatically record front-end performance metrics that measure perceived load speed (opens in a new tab) such as LCP and FCP:
--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------
vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
min: ...................................................... 5911.7
max: ...................................................... 5911.7
mean: ..................................................... 5911.7
median: ................................................... 5944.6
p95: ...................................................... 5944.6
p99: ...................................................... 5944.6
browser.page.FCP.https://artillery.io/:
min: ...................................................... 1521.1
max: ...................................................... 1521.1
mean: ..................................................... 1521.1
median: ................................................... 1525.7
p95: ...................................................... 1525.7
p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
min: ...................................................... 1521.1
max: ...................................................... 1521.1
mean: ..................................................... 1521.1
median: ................................................... 1525.7
p95: ...................................................... 1525.7
p99: ...................................................... 1525.7
browser.page.FCP.https://artillery.io/cloud/:
min: ...................................................... 205.3
max: ...................................................... 205.3
mean: ..................................................... 205.3
median: ................................................... 206.5
p95: ...................................................... 206.5
p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/cloud/:
min: ...................................................... 205.3
max: ...................................................... 205.3
median: ................................................... 206.5
p95: ...................................................... 206.5
p99: ...................................................... 206.5
Configuration
The underlying Playwright instance may be configured through config.engines.playwright
, as well as other options. The following configuration options are available:
Name | Valid Options | Description |
---|---|---|
launchOptions | Playwright browserType.launch() (opens in a new tab) options object | Configure the browser instance started by Playwright |
contextOptions | Playwright browser.newContext() (opens in a new tab) options object | Configure browser contexts created for each virtual user |
defaultNavigationTimeout | Number (in seconds) | Shorthand for setting setDefaultNavigationTimeout() (opens in a new tab) and setDefaultTimeout() (opens in a new tab) |
extendedMetrics |
| Report additional metrics. |
aggregateByName |
| Group metrics by scenario name rather than by URL. |
showAllPageMetrics |
| Send Web Vital metrics for all pages. By default, Artillery only displays Web Vital metrics for a URL that starts with the config.target URL. This avoids reporting metrics for third-party pages and iframes. |
useSeparateBrowserPerVU Added inv2.0.4 |
| Use a separate browser for each VU (instead of a new browser context (opens in a new tab)). This will require a lot more CPU and memory and is not recommended for most tests. |
testIdAttribute Added inv2.0.5 | String | When set, changes the attribute used by locator page.getByTestId (opens in a new tab) in Playwright. |
config.target
as baseURL
Added inv2.0.6
The config.target
is automatically set as the baseURL
(opens in a new tab) for the Playwright test. This means that you can use relative URLs (e.g. page.goto('/docs')
) in your Playwright scripts, and they will be resolved relative to the config.target
URL.
If you are not using relative URLs in your test script, full URLs will still work as usual.
Examples
Example 1: turn off headless mode
You can turn off the default headless mode to see the browser window for local debugging by setting the headless
(opens in a new tab) option.
config:
engines:
playwright:
launchOptions:
headless: false
Notes:
- When running tests in Fargate, headless mode is enabled by default, as you cannot run in Fargate without it. Added inv2.0.5
Example 2: set extra HTTP headers
This example sets the extraHTTPHeaders
(opens in a new tab) option for the browser context that is created by the engine.
config:
engines:
playwright:
contextOptions:
extraHTTPHeaders:
x-my-header: my-value
Example 3: using TypeScript
Added inv2.0.4You can use a TypeScript file in config.processor
.
config:
target: https://www.artillery.io
engines:
playwright: {}
processor: './flows.ts'
scenarios:
- engine: playwright
testFunction: 'helloFlow'
import { Page } from 'playwright';
import { expect } from '@playwright/test';
export async function helloFlow(page: Page) {
await page.goto('https://www.artillery.io/');
await expect(page.getByText('Never Fail To Scale')).toBeVisible();
}
Known Limitations
TypeScript support is experimental and may not work in all cases.
If you run into bundling issues with particular npm packages, you can exclude them from bundling via config.bundling.external
setting.
Example 4: Aggregate metrics by scenario name
By default metrics are aggregated separately for each unique URL. When load testing the same endpoint with different/randomized query params, it can be hepful to group metrics by a common name.
To enable the option pass aggregateByName: true
to the playwright engine and give a name to your scenarios:
config:
target: https://artillery.io
engines:
playwright: { aggregateByName: true }
processor: './flows.js'
scenarios:
- name: blog
engine: playwright
testFunction: 'helloFlow'
flows.js
:
module.exports = { helloFlow };
function helloFlow(page) {
await page.goto(`https://artillery.io/blog/${getRandomSlug()}`);
}
This serves a similar purpose to the useOnlyRequestNames
option from the metrics-by-endpoint plugin.
Test function API
Page argument
By default, only the page
argument (see Playwright's page
API (opens in a new tab)) is required for functions that implement Playwright scenarios, e.g.:
module.exports = { helloFlow };
async function helloFlow(page) {
// Go to https://artillery.io/
await page.goto('https://artillery.io/');
}
Virtual user context and events arguments
The functions also have access to the virtual user context, which can be used for several purposes:
- Accessing scenario (and environment) variables for different virtual users (
vuContext.vars
); - Getting the current virtual user ID (
vuContext.vars.$uuid
); - Getting the scenario definition for the scenario currently being run by the virtual user (
vuContext.scenario
), including its name.
Additionally, the events
argument can be used to track custom metrics.
module.exports = { helloFlow };
async function helloFlow(page, vuContext, events) {
// Increment custom counter:
events.emit('counter', `user.${vuContext.scenario.name}.page_loads`, 1);
// Go to https://artillery.io/
await page.goto('https://artillery.io/');
}
test.step
argument
Added inv2.0.0-38
The final argument of the function is test
, which contains the step
property. The API for test.step
is similar to Playwright's own test.step
, which allows you to re-use similar code. The purpose in Artillery is slightly different: to emit custom metrics that represent how long each step takes.
async function loginSearchAndLogout(page, vuContext, events, test) {
//1. simply add this line to your scenario function, or use test.step below instead
const { step } = test;
const userid = vuContext.vars.userid;
const recordid = vuContext.vars.recordid;
//2. wrap any logic you have in steps (sometimes you might already have something like this done from existing playwright tests)
await step('landing_page', async () => {
await page.goto('https://internaltesturl.com/landing');
});
await step('submit_login', async () => {
await page.getByLabel('id-label').fill(`${userid}`);
await page.getByLabel('Password').fill(`${password}`);
await page.getByRole('button', { name: 'Submit' }).click();
});
await step('search_record', async () => {
await page.getByPlaceholder('enter request id').fill(`${recordid}`);
await page.getByRole('button', { name: 'Go' }).click();
await page.locator('css=button.avatar-button').click();
});
await step('logout', async () => {
await page.getByText('Logout').click();
});
}
The above code will now emit custom metrics for each step in addition to the default metrics:
browser.step.landing_page:
min: ......................................................................... 87
max: ......................................................................... 150
mean: ........................................................................ 118.5
median: ...................................................................... 89.1
p95: ......................................................................... 89.1
p99: ......................................................................... 89.1
browser.step.submit_login:
min: ......................................................................... 300
max: ......................................................................... 716
mean: ........................................................................ 571.6
median: ...................................................................... 561.2
p95: ......................................................................... 561.2
p99: ......................................................................... 561.2
browser.step.search_record:
min: ......................................................................... 287
max: ......................................................................... 801
mean: ........................................................................ 544.6
median: ...................................................................... 290.1
p95: ......................................................................... 290.1
p99: ......................................................................... 290.1
browser.step.logout:
min: ......................................................................... 52
max: ......................................................................... 334
mean: ........................................................................ 193.1
median: ...................................................................... 140.2
p95: ......................................................................... 200.4
p99: ......................................................................... 200.4
Metrics reported by the engine
In addition to the default metrics reported by Artillery, the Playwright engine reports the following metrics:
Metric | Type | Description |
---|---|---|
browser.http_requests | Counter (count) | Number of HTTP requests made by all virtual users during this time period. |
browser.page.codes.<code> Added inv2.0.4 | Counter (count) | Number of different HTTP status codes, e.g. browser.page.codes.200 is the number of 200 OK requests. |
errors.pw_failed_assertion.<assertion_type> Added inv2.0.11 | Counter (count) | When available, Artillery will display the name of failed assertions (e.g. toBeVisible ). Defaults to errors.<error.message> if not possible to parse the assertion error. |
browser.page.TTFB.<page_url>.<aggregation> | Histogram (milliseconds) | Time To First Byte (opens in a new tab) (Web Vital metric) measurement for a specific page_url . |
browser.page.FCP.<page_url>.<aggregation> | Histogram (milliseconds) | First Contentful Paint (opens in a new tab) (Web Vital metric) measurement for a specific page_url . |
browser.page.LCP.<page_url>.<aggregation> | Histogram (milliseconds) | Largest Contentful Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url . |
browser.page.FID.<page_url>.<aggregation> | Histogram (milliseconds) | First Input Delay (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
browser.page.INP.<page_url>.<aggregation> Added inv2.0.5 | Histogram (milliseconds) | Interaction to Next Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
browser.page.CLS.<page_url>.<aggregation> | Histogram (shift score) | Cumulative Layout Shift (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
Extended metrics
If extendedMetrics
is enabled, the following metrics are also reported:
Metric | Type | Description |
---|---|---|
browser.page.domcontentloaded | Counter (count) | Number of DOM Content Loaded (opens in a new tab) events across all pages. |
browser.page.domcontentloaded.<page_url> | Counter (count) | Number of DOM Content Loaded (opens in a new tab) events for a specific page_url . |
browser.page.dominteractive.<aggregation> | Histogram (milliseconds) | Measurement of time taken for DOM to become interactive (opens in a new tab), across all pages. |
browser.page.dominteractive.<page_url>.<aggregation> | Histogram (milliseconds) | Measurement of time taken for DOM to become interactive (opens in a new tab), for a specific page_url . |
browser.memory_used_mb.<aggregation> | Histogram (megabytes) | Measurement of usedJSHeapSize . |
If test.step()
API is used, the following additional histogram is reported:
browser.step.<step_name>.<aggregation>
(milliseconds) - measurement of time taken for each step in the scenario.
Playwright compatibility
Since Artillery uses the playwright
package to run the tests, the version of Playwright used by Artillery is important. The following table shows the compatibility between Artillery and Playwright versions:
Artillery version | Playwright version | Chromium version |
---|---|---|
2.0.19 (opens in a new tab) | 1.45.3 (opens in a new tab) | 127.0.6533.5 |
2.0.18 (opens in a new tab) | 1.45.2 (opens in a new tab) | 127.0.6533.5 |
2.0.17 (opens in a new tab) | 1.45.0 (opens in a new tab) | 127.0.6533.5 |
2.0.16 (opens in a new tab) - 2.0.15 (opens in a new tab) | 1.44.1 (opens in a new tab) | 125.0.6422.14 |
2.0.14 (opens in a new tab) - 2.0.12 (opens in a new tab) | 1.44.0 (opens in a new tab) | 125.0.6422.14 |
2.0.11 (opens in a new tab) - 2.0.10 (opens in a new tab) | 1.43.1 (opens in a new tab) | 124.0.6367.29 |
2.0.9 (opens in a new tab) - 2.0.7 (opens in a new tab) | 1.42.1 (opens in a new tab) | 123.0.6312.4 |
2.0.6 (opens in a new tab) | 1.41.2 (opens in a new tab) | 121.0.6167.57 |
2.0.5 (opens in a new tab) | 1.41.0 (opens in a new tab) | 121.0.6167.57 |
2.0.4 (opens in a new tab) - 2.0.0-38 (opens in a new tab) | 1.39.0 (opens in a new tab) | 119.0.6045.9 |
Older versions of Artillery were not pinned to a specific Playwright package, so they are not tracked. It's advised to use one of the versions listed above.
Why load test with headless browsers?
Load testing complex dynamic web apps can be time consuming, cumbersome, and brittle compared to load testing pure APIs and backend services. The main reason is that testing web apps requires a different level of abstraction: whereas APIs work at API endpoint level, when testing web apps pages and user flows is a much more useful abstraction that maps onto how the web app is actually used.
Without Playwright
- Figure out which HTTP APIs are used by the web page
- Figure out what actions in the UI trigger calls to which APIs
- Figure out what in-page JavaScript code does and how it interacts with the backend
- Try to mimic realistic load on the backend at protocol level or by using HAR files
- Ignore limitations with how dynamic such tests can be, and accept how brittle and time consuming maintetance is going to be
With Playwright
- Just write UI-centric code and let the web app itself call the backend
- Run lots of Playwrigth scripts to generate load on the backend
Testing HTTP APIs vs dynamic web apps
Ultimately, testing a backend HTTP API is very different from testing a web application that may use many such APIs and use client-side JavaScript to communicate with those APIs.
HTTP APIs & microservices | Web apps | |
---|---|---|
Abstraction level | HTTP endpoint | Whole page |
Surface area | Small, a handful of endpoints | Large, calls many APIs. Different APIs may be called depending on in-page actions by the user |
Formal spec | Usually available (e.g. as an OpenAPI spec) | No formal specs for APIs used and their dependencies. You have to spend time in Dev Tools to track down all API calls |
In-page JS | Ignored. Calls made by in-page JS have to be accounted for manually and emulated | Runs as expected, e.g. making calls to more HTTP endpoints |