Playwright E2E testing

Overview

Your existing Playwright E2E test suite can be integrated with Artillery Cloud to provide a reporting dashboard, and tools for debugging and collaboration for your Playwright E2E tests:

• A central dashboard for your team’s Playwright test reports
• Real-time streaming of test results – this is useful for monitoring large test suites as they’re running in CI
• Automatic tracking of Web Vitals  metrics for all pages in your tests
• GitHub Actions integration with PR comments
• AI-assisted analysis & debugging
• Shareable URLs
• Notes & comments on test results

Getting Started

The Artillery Playwright reporter is a standard Playwright Test Runner reporter  which sends test results to Artillery Cloud. You need to install it as a dependency in your Playwright project, and enable it it your Playwright config file.

Install Artillery Playwright Reporter

npm install -D @artilleryio/playwright-reporter

pnpm add -D @artilleryio/playwright-reporter

yarn add --dev @artilleryio/playwright-reporter

bun add --dev @artilleryio/playwright-reporter

Enable the reporter

Enable the reporter in your Playwright config:

export default defineConfig({
  reporter: [
    ['@artilleryio/playwright-reporter', { name: 'My Test Suite' }],
  ],
  // ... rest of your config
});

Set your Artillery Cloud API key

Sign up for Artillery Cloud and create an API key , and then export it as ARTILLERY_CLOUD_API_KEY:

export ARTILLERY_CLOUD_API_KEY=YOUR_API_KEY

Run your Playwright tests

Run your Playwright test suite as normal, e.g. with:

npx playwright test

When the test starts the reporter will print the URL of the test run to the console. The URL will look like this:

https://app.artillery.io/orgidkdowwbwpjiyxn/playwright/pwzxhq_abcdefabcdefghi_7atf

The report will be updated in real time as the test is running.

Web Vitals

You can configure your tests to track & report Web Vitals  automatically for every page in your tests (LCP, CLS, FCP, TTFB, and INP).

To enable performance tracking, use the withPerformanceTracking() method to extend Playwright’s built-in test function:

// Rename the "test" import from @playwright/test to "base":
import { test as base } from '@playwright/test';
// Import the withPerformanceTracking function:
import { withPerformanceTracking } from '@artilleryio/playwright-reporter';
 
// Extend the base test function with performance tracking. This
// needs to be at the top before using test()
const test = withPerformanceTracking(base);
 
// The rest of your tests remain the same, e.g.:
 
test('has title', async ({ page }) => {
  await page.goto('https://playwright.dev/');
  await expect(page).toHaveTitle(/Playwright/);
});

GitHub Actions

When the reporter is used in a GitHub Actions workflow, it can post comments to the PR with the test results. To enable PR comments, set GITHUB_TOKEN environment variable in your GitHub Actions job that runs Playwright tests:

jobs:
  test:
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
 
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'
 
      - name: Run Playwright tests
        env:
          ARTILLERY_CLOUD_API_KEY: "${{ secrets.ARTILLERY_CLOUD_API_KEY }}" # API key to send results to Artillery Cloud
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Enable PR comments
        run: npx playwright test

If the test suite runs multiple times, the comment will be updated with the link to each run, and its results.

Share URLs

When you need to share a test report with someone who’s not a member of your Artillery Cloud workspace, you can generate a share URL via Share -> Share test publicly on the report page.

Anyone with the link will be able to see the report. Share URLs are not crawled by Google or other search engines by default, but if the link is posted on the public internet it may get crawled. Playwright test reports may contain sensitive information so only share them with trusted parties.

Share URLs can be disabled again at any time via Share dialog.

Current limitations

• Sharded mode - Playwright test suites running in sharded mode are not fully supported yet. Each run will produce multiple partial reports in Artillery Cloud, one per shard.