Playwright engine
Built-in sincev2.0.0-33Overview
Artillery's Playwright engine is the glue between Artillery & Playwright. The engine takes care of setting up headless browsers, running your Playwright test code, and collecting and emitting performance metrics.
This page provides a reference for all configuration options of Artillery's Playwright integration. You can also learn more about load testing with Playwright in the detailed guide for using Playwright at scale.
Usage
The Playwright engine is built into Artillery and does not need to be installed separately.
Artillery config - hello-world.yml
Create a new Artillery test definition in hello-world.yml
:
config:
target: https://www.artillery.io
# Load the Playwright engine:
engines:
playwright: {}
# Path to JavaScript file that defines Playwright test functions
# TypeScript is also supported
processor: './flows.js'
scenarios:
- engine: playwright
testFunction: helloWorld
Playwright test functions - flows.js
Create a test function in flows.js
:
module.exports = { helloWorld };
async function helloWorld(page) {
// The "page" argument is instance of Playwright Page class:
// https://playwright.dev/docs/api/class-page/
// Go to https://artillery.io/
await page.goto('https://www.artillery.io/');
// Click text=Docs
await page.click('text=Docs');
}
Run your test
artillery run hello-world.yml
Artillery automatically records front-end performance metrics for perceived load speed (opens in a new tab) such as LCP and FCP.
--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------
vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
min: ...................................................... 5911.7
max: ...................................................... 5911.7
mean: ..................................................... 5911.7
median: ................................................... 5944.6
p95: ...................................................... 5944.6
p99: ...................................................... 5944.6
browser.page.FCP.https://artillery.io/:
min: ...................................................... 1521.1
max: ...................................................... 1521.1
mean: ..................................................... 1521.1
median: ................................................... 1525.7
p95: ...................................................... 1525.7
p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
min: ...................................................... 1521.1
max: ...................................................... 1521.1
mean: ..................................................... 1521.1
median: ................................................... 1525.7
p95: ...................................................... 1525.7
p99: ...................................................... 1525.7
browser.page.FCP.https://artillery.io/docs/:
min: ...................................................... 205.3
max: ...................................................... 205.3
mean: ..................................................... 205.3
median: ................................................... 206.5
p95: ...................................................... 206.5
p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/docs/:
min: ...................................................... 205.3
max: ...................................................... 205.3
median: ................................................... 206.5
p95: ...................................................... 206.5
p99: ...................................................... 206.5
Configuration
Playwright / browser configuration options
Name | Valid Options | Description |
---|---|---|
launchOptions | Playwright's browserType.launch() (opens in a new tab) options | Configure the browser instance started by Playwright |
contextOptions | Playwright's browser.newContext() (opens in a new tab) options | Configure browser contexts created for each virtual user |
defaultNavigationTimeout | Number (in seconds) | Shorthand for setting setDefaultNavigationTimeout() (opens in a new tab) and setDefaultTimeout() (opens in a new tab) |
extendedMetrics |
| Report additional metrics. |
aggregateByName |
| Group metrics by scenario name rather than by URL. |
showAllPageMetrics |
| Send Web Vital metrics for all pages. By default, Artillery only displays Web Vital metrics for a URL that starts with the config.target URL. This avoids reporting metrics for third-party pages and iframes. |
useSeparateBrowserPerVU Added inv2.0.4 |
| Use a separate browser for each virtual user (instead of a new browser context (opens in a new tab)). This will require a lot more CPU and memory and is not recommended for most tests. |
testIdAttribute Added inv2.0.5 | String | The attribute used by locator page.getByTestId (opens in a new tab) in Playwright. |
trace | Object (see below) | Configuration for recording Playwright Traces |
Tracing configuration
Artillery can automatically record Playwright traces for virtual users that fail. This can be helpful for debugging any issues uncovered by the load test. Recorded traces are uploaded to Artillery Cloud (opens in a new tab) and can be viewed under the Traces tab in the test report.
You will need to create an Artillery Cloud account and run your test with --record
flag to use this feature.
Name | Valid Options | Description |
---|---|---|
enabled |
| Enable or disable recording of Playwright traces |
maxConcurrentRecordings | Number (default: 3) | The number of virtual users that will be recording a trace at the same time |
For performance reasons, only a subset of active virtual users record traces at any given time. If a virtual user succeeds, the trace is discarded. This means that some tests with only a small number of failing virtual users may not produce any traces. You can tweak the maxConcurrentRecordings
setting to increase the likelihood of capturing traces for failing virtual users.
Customizing maxConcurrentRecordings
You can increase the value of maxConcurrentRecordings
to increase the likelihood of capturing traces for failing VUs.
In distributed tests this setting is applied per worker, e.g. if you run a distributed load test on Azure ACI with 20 workers like this:
artillery run-aci hello-world.yml --count 20
Each of those 20 workers will be recording up to maxConcurrentRecordings
traces at a time.
See the load testing with Playwright guide for more information on how trace recordings work.
$rewriteMetricName
hook
It's possible to customize the names of metrics emitted by Playwright load tests by defining a $rewriteMetricName
hook function.
This can be used to group metrics from distinct URLs & pages requested during the test under a single metric name in the report. A common use-case is to report metrics for a page that's loaded with many different query parameters under a single metric name.
If your custom code (loaded with config.processor
) defines a $rewriteMetricName
function, it will be called for each metric emitted by the Playwright engine.
The function should return the new metric name as a string.
Example of a $rewriteMetricName
function that groups metrics by the first part of the URL:
function $rewriteMetricName(metricName, metricType) {
// metricType is one of: histogram, counter, rate
// For more details on Artillery metric types, see: https://www.artillery.io/docs/reference/reported-metrics
if (metricName.includes('/checkout?promoid=')) {
return 'browser.page.checkout';
} else {
return metricName;
}
}
config.target
is Playwright's baseURL
Added inv2.0.6
The config.target
is automatically set as the baseURL
(opens in a new tab) for the Playwright test. This means that you can use relative URLs (e.g. page.goto('/docs')
) in your Playwright scripts, and they will be resolved relative to the config.target
URL.
If you are not using relative URLs in your test script, full URLs will still work as usual.
Test function API
Page argument
By default, only the page
argument (see Playwright's page
API (opens in a new tab)) is required for functions that implement Playwright scenarios, e.g.:
module.exports = { helloFlow };
async function helloFlow(page) {
// Go to https://artillery.io/
await page.goto('https://artillery.io/');
}
Virtual user context and events arguments
The functions also have access to the virtual user context, which can be used for several purposes:
- Accessing scenario (and environment) variables for different virtual users (
vuContext.vars
); - Getting the current virtual user ID (
vuContext.vars.$uuid
); - Getting the scenario definition for the scenario currently being run by the virtual user (
vuContext.scenario
), including its name.
Additionally, the events
argument can be used to track custom metrics.
module.exports = { helloFlow };
async function helloFlow(page, vuContext, events) {
// Increment custom counter:
events.emit('counter', `user.${vuContext.scenario.name}.page_loads`, 1);
// Go to https://artillery.io/
await page.goto('https://artillery.io/');
}
test.step
argument
Added inv2.0.0-38
The final argument of the function is test
, which contains the step
property. The API for test.step
is similar to Playwright's own test.step
, which allows you to re-use similar code. The purpose in Artillery is slightly different: to emit custom metrics that represent how long each step takes.
async function loginSearchAndLogout(page, vuContext, events, test) {
//1. add this line to your scenario function to use test.step helper:
const { step } = test;
const userid = vuContext.vars.userid;
const recordid = vuContext.vars.recordid;
//2. wrap any logic you have in steps (sometimes you might already have something like this done from existing Playwright tests)
await step('landing_page', async () => {
await page.goto('https://internaltesturl.com/landing');
});
await step('submit_login', async () => {
await page.getByLabel('id-label').fill(`${userid}`);
await page.getByLabel('Password').fill(`${password}`);
await page.getByRole('button', { name: 'Submit' }).click();
});
await step('search_record', async () => {
await page.getByPlaceholder('enter request id').fill(`${recordid}`);
await page.getByRole('button', { name: 'Go' }).click();
await page.locator('css=button.avatar-button').click();
});
await step('logout', async () => {
await page.getByText('Logout').click();
});
}
The above code will now emit custom metrics for each step in addition to the default metrics:
browser.step.landing_page:
min: ......................................................................... 87
max: ......................................................................... 150
mean: ........................................................................ 118.5
median: ...................................................................... 89.1
p95: ......................................................................... 89.1
p99: ......................................................................... 89.1
browser.step.submit_login:
min: ......................................................................... 300
max: ......................................................................... 716
mean: ........................................................................ 571.6
median: ...................................................................... 561.2
p95: ......................................................................... 561.2
p99: ......................................................................... 561.2
browser.step.search_record:
min: ......................................................................... 287
max: ......................................................................... 801
mean: ........................................................................ 544.6
median: ...................................................................... 290.1
p95: ......................................................................... 290.1
p99: ......................................................................... 290.1
browser.step.logout:
min: ......................................................................... 52
max: ......................................................................... 334
mean: ........................................................................ 193.1
median: ...................................................................... 140.2
p95: ......................................................................... 200.4
p99: ......................................................................... 200.4
Metrics reported by the engine
In addition to the default metrics reported by Artillery, the Playwright engine reports the following metrics:
Metric | Type | Description |
---|---|---|
browser.http_requests | Counter (count) | Number of HTTP requests made by all virtual users during this time period. |
browser.page.codes.<code> Added inv2.0.4 | Counter (count) | Number of different HTTP status codes, e.g. browser.page.codes.200 is the number of 200 OK requests. |
errors.pw_failed_assertion.<assertion_type> Added inv2.0.11 | Counter (count) | When available, Artillery will display the name of failed assertions (e.g. toBeVisible ). Defaults to errors.<error.message> if not possible to parse the assertion error. |
browser.page.TTFB.<page_url>.<aggregation> | Histogram (milliseconds) | Time To First Byte (opens in a new tab) (Web Vital metric) measurement for a specific page_url . |
browser.page.FCP.<page_url>.<aggregation> | Histogram (milliseconds) | First Contentful Paint (opens in a new tab) (Web Vital metric) measurement for a specific page_url . |
browser.page.LCP.<page_url>.<aggregation> | Histogram (milliseconds) | Largest Contentful Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url . |
browser.page.FID.<page_url>.<aggregation> | Histogram (milliseconds) | First Input Delay (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
browser.page.INP.<page_url>.<aggregation> Added inv2.0.5 | Histogram (milliseconds) | Interaction to Next Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
browser.page.CLS.<page_url>.<aggregation> | Histogram (shift score) | Cumulative Layout Shift (opens in a new tab) (Core Web Vital metric) measurement for a specific page_url (if available). |
Extended metrics
If extendedMetrics
is enabled, the following metrics are also reported:
Metric | Type | Description |
---|---|---|
browser.page.domcontentloaded | Counter (count) | Number of DOM Content Loaded (opens in a new tab) events across all pages. |
browser.page.domcontentloaded.<page_url> | Counter (count) | Number of DOM Content Loaded (opens in a new tab) events for a specific page_url . |
browser.page.dominteractive.<aggregation> | Histogram (milliseconds) | Measurement of time taken for DOM to become interactive (opens in a new tab), across all pages. |
browser.page.dominteractive.<page_url>.<aggregation> | Histogram (milliseconds) | Measurement of time taken for DOM to become interactive (opens in a new tab), for a specific page_url . |
browser.memory_used_mb.<aggregation> | Histogram (megabytes) | Measurement of usedJSHeapSize . |
If test.step()
API is used, the following additional histogram is reported:
browser.step.<step_name>.<aggregation>
(milliseconds) - measurement of time taken for each step in the scenario.
Auto-instrumentation of Web Vitals
Artillery automatically tracks the following Web Vitals metrics for every page load in your test:
- LCP (opens in a new tab) (Largest Contentful Paint)
- CLS (opens in a new tab) (Cumulative Layout Shift)
- INP (opens in a new tab) (Interaction to Next Paint)
- TTFB (opens in a new tab) (Time To First Byte)
- FCP (opens in a new tab) (First Contentful Paint)
- FID (opens in a new tab) (First Input Delay)
The metrics are tracked using Google's web-vitals
(opens in a new tab) library. Artillery injects the library into the context of each page and Page event handlers (opens in a new tab) are set up to collect those metrics.
If you have any code that modifies the Page
object provided as the first argument to the test function, you need to make sure that it doesn't overwrite the page context, or reset event handlers. If either of those is reset, then automatic collection of Web Vitals metrics won't work.
Examples
Example 1: turn off headless mode
You can turn off the default headless mode to see the browser window for local debugging by setting the headless
(opens in a new tab) option.
config:
engines:
playwright:
launchOptions:
headless: false
For tests executing in distributed mode, e.g. on AWS Fargate or Azure ACI, headless mode is enforced regardless of config.
Example 2: set extra HTTP headers
This example sets the extraHTTPHeaders
(opens in a new tab) option for the browser context that is created by the engine.
config:
engines:
playwright:
contextOptions:
extraHTTPHeaders:
x-my-header: my-value
Example 3: using TypeScript
Added inv2.0.4You can use a TypeScript file in config.processor
.
config:
target: https://www.artillery.io
engines:
playwright: {}
processor: './flows.ts'
scenarios:
- engine: playwright
testFunction: 'helloFlow'
import { Page } from 'playwright';
import { expect } from '@playwright/test';
export async function helloFlow(page: Page) {
await page.goto('https://www.artillery.io/');
await expect(page.getByText('Never Fail To Scale')).toBeVisible();
}
Example 4: Aggregate metrics by scenario name
By default metrics are aggregated separately for each unique URL. When load testing the same endpoint with different/randomized query params, it can be helpful to group metrics by a common name.
To enable the option pass aggregateByName: true
to the playwright engine and give a name to your scenarios:
config:
target: https://artillery.io
engines:
playwright: { aggregateByName: true }
processor: './flows.js'
scenarios:
- name: blog
engine: playwright
testFunction: 'helloFlow'
flows.js
:
module.exports = { helloFlow };
function helloFlow(page) {
await page.goto(`https://artillery.io/blog/${getRandomSlug()}`);
}
This serves a similar purpose to the useOnlyRequestNames
option from the metrics-by-endpoint plugin.
Example 5: Record Playwright Traces
config:
target: https://www.artillery.io
engines:
# Enable trace recordings:
playwright:
trace:
enabled: true
You will need to create an Artillery Cloud account and run your test with --record
flag to use this feature. Once enabled, traces for failing virtual users will be uploaded to the Artillery Cloud dashboard. Trace recording works for both tests running locally, and distributed tests (on AWS or Azure).
Playwright compatibility
Each release of Artillery includes a specific version of Playwright (the playwright
(opens in a new tab) npm package), which is the version of Playwright that's used to run load tests. Playwright releases are usually backwards-compatible, but if your Playwright code depends on a feature that was introduced in a newer version of Playwright than the one bundled with Artillery, you will need to wait until a new version of Artillery is released.