Playwright engine

Overview

Artillery's Playwright engine is the glue between Artillery & Playwright. The engine takes care of setting up headless browsers, running your Playwright test code, and collecting and emitting performance metrics.

This page provides a reference for all configuration options of Artillery's Playwright integration. You can also learn more about load testing with Playwright in the detailed guide for using Playwright at scale.

Usage

The Playwright engine is built into Artillery and does not need to be installed separately.

Create your test definition

Tests can be defined in one of two ways:

Fully in TypeScript Requires Artilleryv2.0.22
With a combination of YAML for Artillery configuration, and TypeScript/JS for Playwright code

Create test definition

Create a new Artillery test definition in hello-world.ts:

import { Page } from '@playwright/test';
 
export const config = {
  target: 'https://www.artillery.io',
  engines: {
    playwright: {}
  }
};
 
export const scenarios = [{
  engine: 'playwright',
  testFunction: helloWorld
}];
 
async function helloWorld(page: Page) {
  await page.goto('https://www.artillery.io/');
  await page.click('text=Docs');
}

Run your test

npx artillery run hello-world.ts

Artillery automatically records front-end performance metrics for perceived load speed (opens in a new tab) such as LCP and FCP for different pages navigated to during the test. See Metrics reported by the engine for a full list of metrics reported by the Playwright engine.

--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------

vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
  min: ...................................................... 5911.7
  max: ...................................................... 5911.7
  mean: ..................................................... 5911.7
  median: ................................................... 5944.6
  p95: ...................................................... 5944.6
  p99: ...................................................... 5944.6
browser.page.FCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  mean: ..................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  mean: ..................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.FCP.https://artillery.io/docs/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  mean: ..................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/docs/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5

Configuration

Playwright / browser configuration options

Name	Valid Options	Description
`launchOptions`	Playwright's `browserType.launch()` (opens in a new tab) options	Configure the browser instance started by Playwright
`contextOptions`	Playwright's `browser.newContext()` (opens in a new tab) options	Configure browser contexts created for each virtual user
`defaultNavigationTimeout`	Number (in seconds)	Shorthand for setting `setDefaultNavigationTimeout()` (opens in a new tab) and `setDefaultTimeout()` (opens in a new tab)
`extendedMetrics`	`false` (default) `true`	Report additional metrics.
`aggregateByName`	`false` (default) `true`	Group metrics by scenario name rather than by URL.
`showAllPageMetrics`	`false` (default) `true`	Send Web Vital metrics for all pages. By default, Artillery only displays Web Vital metrics for a URL that starts with the `config.target` URL. This avoids reporting metrics for third-party pages and iframes.
`useSeparateBrowserPerVU` Added inv2.0.4	`false` (default) `true`	Use a separate browser for each virtual user (instead of a new browser context (opens in a new tab)). This will require a lot more CPU and memory and is not recommended for most tests.
`testIdAttribute` Added inv2.0.5	String	The attribute used by locator `page.getByTestId` (opens in a new tab) in Playwright.
`trace`	Object (see below)	Configuration for recording Playwright Traces

Tracing configuration

Artillery can automatically record Playwright traces for virtual users that fail. This can be helpful for debugging any issues uncovered by the load test. Recorded traces are uploaded to Artillery Cloud (opens in a new tab) and can be viewed under the Traces tab in the test report.

You will need to create an Artillery Cloud account and run your test with --record flag to use this feature.

Name	Valid Options	Description
`enabled`	`false` (default) `true`	Enable or disable recording of Playwright traces
`maxConcurrentRecordings`	Number (default: 5)	The number of virtual users that will be recording a trace at the same time

For performance reasons, only a subset of active virtual users record traces at any given time. If a virtual user succeeds, the trace is discarded. This means that some tests with only a small number of failing virtual users may not produce any traces. You can tweak the maxConcurrentRecordings setting to increase the likelihood of capturing traces for failing virtual users.

Customizing `maxConcurrentRecordings`

You can increase the value of maxConcurrentRecordings to increase the likelihood of capturing traces for failing VUs.

In distributed tests this setting is applied per worker, e.g. if you run a distributed load test on Azure ACI with 20 workers like this:

artillery run-aci hello-world.yml --count 20

Each of those 20 workers will be recording up to maxConcurrentRecordings traces at a time.

See the load testing with Playwright guide for more information on how trace recordings work.

`$rewriteMetricName` hook

If you need to customize the names of metrics emitted by Playwright load test, you can use a $rewriteMetricName hook to rewrite metric names on-the-fly.

This can be used to group metrics from distinct URLs & pages requested during the test under a single metric name in the report. A common use-case is to report metrics for a page that's loaded with many different query parameters under a single metric name.

Defining the function

The $rewriteMetricName hook is a function that takes two arguments:

the original metric name, e.g. browser.page.TTFB.https://example.com/page?query=1
metric type (one of histogram, counter, or rate).

The function should return the new metric name as a string.

Example of a $rewriteMetricName function that groups metrics for all checkout pages under one metric name:

function $rewriteMetricName(metricName, metricType) {
  if (metricName.includes('/checkout?promoid=')) {
    return 'browser.page.checkout';
  } else {
    return metricName;
  }
}

Loading the function

If your test configuration is defined with YAML, export $rewriteMetricName function from a JS/TypeScript file and load it with the config.processor option.

If your test is defined fully in TypeScript, export $rewriteMetricName from the test script:

    import { Page } from '@playwright/test';
 
    export const config = {
      target: 'https://www.artillery.io',
      engines: {
        playwright: {}
      }
    };
 
    export const scenarios = [{
      engine: 'playwright',
      testFunction: helloWorld
    }];
 
    async function helloWorld(page: Page) {
      await page.goto('https://www.artillery.io/');
      await page.click('text=Docs');
    }
 
    export function $rewriteMetricName(metricName: string, metricType: string) {
      if (metricName.includes('/checkout?promoid=')) {
        return 'browser.page.checkout';
      } else {
        return metricName;
      }
    }

`config.target` is Playwright's `baseURL`

Added inv2.0.6

The config.target is automatically set as the baseURL (opens in a new tab) for the Playwright test. This means that you can use relative URLs (e.g. page.goto('/docs')) in your Playwright scripts, and they will be resolved relative to the config.target URL.

If you are not using relative URLs in your test script, full URLs will still work as usual.

Test function API

Page argument

By default, only the page argument (see Playwright's page API (opens in a new tab)) is required for functions that implement Playwright scenarios, e.g.:

module.exports = { helloFlow };
 
async function helloFlow(page) {
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

Virtual user context and events arguments

The functions also have access to the virtual user context, which can be used for several purposes:

Accessing scenario (and environment) variables for different virtual users (vuContext.vars);
Getting the current virtual user ID (vuContext.vars.$uuid);
Getting the scenario definition for the scenario currently being run by the virtual user (vuContext.scenario), including its name.

Additionally, the events argument can be used to track custom metrics.

module.exports = { helloFlow };
 
async function helloFlow(page, vuContext, events) {
  // Increment custom counter:
  events.emit('counter', `user.${vuContext.scenario.name}.page_loads`, 1);
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

`test.step` argument

Added inv2.0.0-38

The final argument of the function is test, which contains the step property. The API for test.step is similar to Playwright's own test.step, which allows you to re-use similar code. The purpose in Artillery is slightly different: to emit custom metrics that represent how long each step takes.

async function loginSearchAndLogout(page, vuContext, events, test) {
  //1. add this line to your scenario function to use test.step helper:
  const { step } = test;
 
  const userid = vuContext.vars.userid;
  const recordid = vuContext.vars.recordid;
 
  //2. wrap any logic you have in steps (sometimes you might already have something like this done from existing Playwright tests)
  await step('landing_page', async () => {
    await page.goto('https://internaltesturl.com/landing');
  });
 
  await step('submit_login', async () => {
    await page.getByLabel('id-label').fill(`${userid}`);
    await page.getByLabel('Password').fill(`${password}`);
    await page.getByRole('button', { name: 'Submit' }).click();
  });
 
  await step('search_record', async () => {
    await page.getByPlaceholder('enter request id').fill(`${recordid}`);
    await page.getByRole('button', { name: 'Go' }).click();
    await page.locator('css=button.avatar-button').click();
  });
 
  await step('logout', async () => {
    await page.getByText('Logout').click();
  });
}

The above code will now emit custom metrics for each step in addition to the default metrics:

browser.step.landing_page:
  min: ......................................................................... 87
  max: ......................................................................... 150
  mean: ........................................................................ 118.5
  median: ...................................................................... 89.1
  p95: ......................................................................... 89.1
  p99: ......................................................................... 89.1
browser.step.submit_login:
  min: ......................................................................... 300
  max: ......................................................................... 716
  mean: ........................................................................ 571.6
  median: ...................................................................... 561.2
  p95: ......................................................................... 561.2
  p99: ......................................................................... 561.2
browser.step.search_record:
  min: ......................................................................... 287
  max: ......................................................................... 801
  mean: ........................................................................ 544.6
  median: ...................................................................... 290.1
  p95: ......................................................................... 290.1
  p99: ......................................................................... 290.1
browser.step.logout:
  min: ......................................................................... 52
  max: ......................................................................... 334
  mean: ........................................................................ 193.1
  median: ...................................................................... 140.2
  p95: ......................................................................... 200.4
  p99: ......................................................................... 200.4

Metrics reported by the engine

In addition to the default metrics reported by Artillery, the Playwright engine reports the following metrics:

Metric	Type	Description
`browser.http_requests`	Counter (count)	Number of HTTP requests made by all virtual users during this time period.
`browser.page.codes.<code>` Added inv2.0.4	Counter (count)	Number of different HTTP status codes, e.g. `browser.page.codes.200` is the number of `200 OK` requests.
`errors.pw_failed_assertion.<assertion_type>` Added inv2.0.11	Counter (count)	When available, Artillery will display the name of failed assertions (e.g. `toBeVisible`). Defaults to `errors.<error.message>` if not possible to parse the assertion error.
`browser.page.TTFB.<page_url>.<aggregation>`	Histogram (milliseconds)	Time To First Byte (opens in a new tab) (Web Vital metric) measurement for a specific `page_url`.
`browser.page.FCP.<page_url>.<aggregation>`	Histogram (milliseconds)	First Contentful Paint (opens in a new tab) (Web Vital metric) measurement for a specific `page_url`.
`browser.page.LCP.<page_url>.<aggregation>`	Histogram (milliseconds)	Largest Contentful Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific `page_url`.
`browser.page.FID.<page_url>.<aggregation>`	Histogram (milliseconds)	First Input Delay (opens in a new tab) (Core Web Vital metric) measurement for a specific `page_url` (if available).
`browser.page.INP.<page_url>.<aggregation>` Added inv2.0.5	Histogram (milliseconds)	Interaction to Next Paint (opens in a new tab) (Core Web Vital metric) measurement for a specific `page_url` (if available).
`browser.page.CLS.<page_url>.<aggregation>`	Histogram (shift score)	Cumulative Layout Shift (opens in a new tab) (Core Web Vital metric) measurement for a specific `page_url` (if available).

Extended metrics

If extendedMetrics is enabled, the following metrics are also reported:

Metric	Type	Description
`browser.page.domcontentloaded`	Counter (count)	Number of DOM Content Loaded (opens in a new tab) events across all pages.
`browser.page.domcontentloaded.<page_url>`	Counter (count)	Number of DOM Content Loaded (opens in a new tab) events for a specific `page_url`.
`browser.page.dominteractive.<aggregation>`	Histogram (milliseconds)	Measurement of time taken for DOM to become interactive (opens in a new tab), across all pages.
`browser.page.dominteractive.<page_url>.<aggregation>`	Histogram (milliseconds)	Measurement of time taken for DOM to become interactive (opens in a new tab), for a specific `page_url`.
`browser.memory_used_mb.<aggregation>`	Histogram (megabytes)	Measurement of `usedJSHeapSize`.

If test.step() API is used, the following additional histogram is reported:

browser.step.<step_name>.<aggregation> (milliseconds) - measurement of time taken for each step in the scenario.

Auto-instrumentation of Web Vitals

Artillery automatically tracks the following Web Vitals metrics for every page load in your test:

LCP (opens in a new tab) (Largest Contentful Paint)
CLS (opens in a new tab) (Cumulative Layout Shift)
INP (opens in a new tab) (Interaction to Next Paint)
TTFB (opens in a new tab) (Time To First Byte)
FCP (opens in a new tab) (First Contentful Paint)
FID (opens in a new tab) (First Input Delay)

The metrics are tracked using Google's web-vitals (opens in a new tab) library. Artillery injects the library into the context of each page and Page event handlers (opens in a new tab) are set up to collect those metrics.

⚠️

If you have any code that modifies the Page object provided as the first argument to the test function, you need to make sure that it doesn't overwrite the page context, or reset event handlers. If either of those is reset, then automatic collection of Web Vitals metrics won't work.

Examples

Example 1: turn off headless mode

You can turn off the default headless mode to see the browser window for local debugging by setting the headless (opens in a new tab) option.

For tests executing in distributed mode, e.g. on AWS Fargate or Azure ACI, headless mode is enforced regardless of config.

export const config = {
  engines: {
    playwright: {
      launchOptions: {
        headless: false
      }
    }
  }
};

Example 2: set extra HTTP headers

This example sets the extraHTTPHeaders (opens in a new tab) option for the browser context that is created by the engine.

export const config = {
  engines: {
    playwright: {
      contextOptions: {
        extraHTTPHeaders: {
          'x-my-header': 'my-value'
        }
      }
    }
  }
};

Example 3: Record Playwright Traces

You will need to create an Artillery Cloud account and run your test with --record flag to use this feature. Once enabled, traces for failing virtual users will be uploaded to the Artillery Cloud dashboard. Trace recording works for both tests running locally, and distributed tests (on AWS or Azure).

export const config = {
  target: 'https://www.artillery.io',
  engines: {
    playwright: {
      trace: {
        enabled: true
      }
    }
  }
};

Playwright compatibility

Each release of Artillery includes a specific version of Playwright (the playwright (opens in a new tab) npm package), which is the version of Playwright that's used to run load tests. Playwright releases are usually backwards-compatible, but if your Playwright code depends on a feature that was introduced in a newer version of Playwright than the one bundled with Artillery, you will need to wait until a new version of Artillery is released.

Artillery version	Playwright version
2.0.23 (opens in a new tab)	1.52.0 (opens in a new tab)
2.0.22 (opens in a new tab)	1.49.1 (opens in a new tab)
2.0.21 (opens in a new tab)	1.48.0 (opens in a new tab)
2.0.20 (opens in a new tab)	1.45.3 (opens in a new tab)
2.0.19 (opens in a new tab)	1.45.3 (opens in a new tab)
2.0.18 (opens in a new tab)	1.45.2 (opens in a new tab)
2.0.17 (opens in a new tab)	1.45.0 (opens in a new tab)
2.0.16 (opens in a new tab) - 2.0.15 (opens in a new tab)	1.44.1 (opens in a new tab)
2.0.14 (opens in a new tab) - 2.0.12 (opens in a new tab)	1.44.0 (opens in a new tab)
2.0.11 (opens in a new tab) - 2.0.10 (opens in a new tab)	1.43.1 (opens in a new tab)
2.0.9 (opens in a new tab) - 2.0.7 (opens in a new tab)	1.42.1 (opens in a new tab)
2.0.6 (opens in a new tab)	1.41.2 (opens in a new tab)
2.0.5 (opens in a new tab)	1.41.0 (opens in a new tab)
2.0.4 (opens in a new tab) - 2.0.0-38 (opens in a new tab)	1.39.0 (opens in a new tab)

HTTP Socket.IO