Reference
Playwright

Load testing with Playwright and Artillery

Overview

Playwright (opens in a new tab) is a modern browser automation framework by Microsoft. Artillery supports running Playwright-based scripts as load tests, including running Playwright at scale using AWS Fargate.

Features

→ See Why load test with headless browsers?

Current limitations

  • Test functions have to be written in Javascript (if you have existing Playwright code in TypeScript, it will need to be transpiled to JS first)
  • Only Chrome is available. Restricting the integration to just one browser improves startup time performance for large load tests and does not have any consequential effects on the results of load tests themselves.

Usage

The Playwright engine is built into Artillery.

Create a new Artillery script in hello-world.yml:

config:
  target: https://www.artillery.io
  # Load the Playwright engine:
  engines:
    playwright: {}
  # Path to JavaScript file that defines Playwright test functions
  processor: "./flows.js"
scenarios:
  - engine: playwright
    testFunction: "helloFlow"

Create a test function in flows.js:

(Note: this script was generated with playwright codegen (opens in a new tab). page is an instance of Playwright page (opens in a new tab).)

module.exports = { helloFlow };
 
async function helloFlow(page) {
  //
  // The code below is just a standard Playwright script:
  //
  // Go to https://artillery.io/
  await page.goto("https://www.artillery.io/");
  // Click text=Pricing
  await page.click("text=Cloud");
}

Run it:

artillery run hello-world.yml

Artillery will run the test and automatically record front-end performance metrics that measure perceived load speed (opens in a new tab) such as LCP and FCP:

--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------

vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
  min: ...................................................... 5911.7
  max: ...................................................... 5911.7
  median: ................................................... 5944.6
  p95: ...................................................... 5944.6
  p99: ...................................................... 5944.6
browser.page.FCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.FCP.https://artillery.io/cloud/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/cloud/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5

Configuration

The underlying Playwright instance may be configured through config.engines.playwright.

You can pass the following options in:

Example 1: turn off headless mode

You can turn off the default headless mode to see the browser window for local debugging by setting the headless (opens in a new tab) option.

config:
  engines:
    playwright:
      launchOptions:
        headless: false

Example 2: set extra HTTP headers

This example sets the extraHTTPHeaders (opens in a new tab) option for the browser context that is created by the engine.

config:
  engines:
    playwright:
      contextOptions:
        extraHTTPHeaders:
          x-my-header: my-value

Aggregate metrics by scenario name

By default metrics are aggregated separately for each unique URL. When load testing the same endpoint with different/randomized query params, it can be hepful to group metrics by a common name.

To enable the option pass aggregateByName: true to the playwright engine and give a name to your scenarios:

config:
  target: https://artillery.io
  engines:
    playwright: { aggregateByName: true }
  processor: "./flows.js"
scenarios:
  - name: blog
    engine: playwright
    testFunction: "helloFlow"

flows.js:

module.exports = { helloFlow };
 
function helloFlow(page) {
  await page.goto(`https://artillery.io/blog/${getRandomSlug()}`);
}

This serves a similar purpose to the useOnlyRequestNames option from the metrics-by-endpoint plugin.

Extended Metrics

If enabled, Artillery will collect the following additional metrics:

  • browser.page.domcontentloaded / browser.page.domcontentloaded.<url>: Count of DOM Content Loaded (opens in a new tab) events.
  • browser.page.dominteractive / browser.page.dominteractive.<url>: Histogram measurement of time it takes for dom to become interactive.
  • browser.memory_used_mb: Histogram measurement of usedJSHeapSize.

You can enable them by setting extendedMetrics to true in your config:

config:
  engines:
    playwright: { extendedMetrics: true }

Test function API

By default, only the page argument (see Playwright's page API (opens in a new tab)) is required for functions that implement Playwright scenarios, e.g.:

module.exports = { helloFlow };
 
async function helloFlow(page) {
  // Go to https://artillery.io/
  await page.goto("https://artillery.io/");
}

The functions also have access to virtual user context and events arguments, which can be used to access scenario variables for different virtual users, or to track custom metrics.

module.exports = { helloFlow };
 
async function helloFlow(page, vuContext, events) {
  // Increment custom counter:
  events.emit("counter", "user.page_loads", 1);
  // Go to https://artillery.io/
  await page.goto("https://artillery.io/");
}

Why load test with headless browsers?

Load testing complex dynamic web apps can be time consuming, cumbersome, and brittle compared to load testing pure APIs and backend services. The main reason is that testing web apps requires a different level of abstraction: whereas APIs work at API endpoint level, when testing web apps pages and user flows is a much more useful abstraction that maps onto how the web app is actually used.

Without Playwright

  • Figure out which HTTP APIs are used by the web page
  • Figure out what actions in the UI trigger calls to which APIs
  • Figure out what in-page JavaScript code does and how it interacts with the backend
  • Try to mimic realistic load on the backend at protocol level or by using HAR files
  • Ignore limitations with how dynamic such tests can be, and accept how brittle and time consuming maintetance is going to be

With Playwright

  • Just write UI-centric code and let the web app itself call the backend
  • Run lots of Playwrigth scripts to generate load on the backend

Testing HTTP APIs vs dynamic web apps

Ultimately, testing a backend HTTP API is very different from testing a web application that may use many such APIs and use client-side JavaScript to communicate with those APIs.

HTTP APIs & microservicesWeb apps
Abstraction levelHTTP endpointWhole page
Surface areaSmall, a handful of endpointsLarge, calls many APIs. Different APIs may be called depending on in-page actions by the user
Formal specUsually available (e.g. as an OpenAPI spec)No formal specs for APIs used and their dependencies. You have to spend time in Dev Tools to track down all API calls
In-page JSIgnored. Calls made by in-page JS have to be accounted for manually and emulatedRuns as expected, e.g. making calls to more HTTP endpoints