@clipboard-health/playwright-reporter-llm

Playwright reporter that outputs structured JSON for LLM agents. Minimal console output, flat schema, easy to filter to failures.

Table of contents

• Install
• Usage
  • Options
  • Console output
  • What to read, by task
  • Minimal failure example
  • Flaky test example (pass-vs-fail comparison)
  • Full example report
  • Field reference
• Why not Playwright's built-in JSON reporter?
• Local development commands

Install

npm install @clipboard-health/playwright-reporter-llm
Copy

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  reporter: [
    ["@clipboard-health/playwright-reporter-llm", { outputFile: "test-results/llm-report.json" }],
  ],
});
Copy

Usage

Options

Console output

.......F..........F.S....
26 tests | 23 passed | 2 failed | 1 skipped (4.2s)
Report: test-results/llm-report.json
Copy

What to read, by task

Minimal failure example

Walk attempts[].timeline[] (steps + network + console, sorted by offsetMs) to reconstruct the failure. Each timeline network entry carries a networkId you can resolve against attempts[].network.instances[] for full per-request detail and attempts[].network.groups[] / bodies[] for shared shape/payload context:

{
  "schemaVersion": 3,
  "summary": {
    "total": 10,
    "passed": 9,
    "failed": 1,
    "flaky": 0,
    "skipped": 0,
    "timedOut": 0,
    "interrupted": 0
  },
  "tests": [
    {
      "id": "abc123",
      "title": "Checkout > applies discount code",
      "status": "failed",
      "flaky": false,
      "location": { "file": "tests/checkout.spec.ts", "line": 42, "column": 5 },
      "errors": [
        {
          "message": "Expected: 90\nReceived: 100",
          "diff": { "expected": "90", "actual": "100" },
          "location": { "file": "tests/checkout.spec.ts", "line": 58, "column": 7 }
        }
      ],
      "attempts": [
        {
          "attempt": 1,
          "status": "failed",
          "failureArtifacts": { "screenshotBase64": "iVBORw0KGgo...", "videoPath": "video.webm" },
          "timeline": [
            {
              "kind": "step",
              "offsetMs": 1050,
              "title": "click Apply",
              "category": "test.step",
              "durationMs": 40,
              "depth": 0
            },
            {
              "kind": "network",
              "offsetMs": 1100,
              "networkId": "n3",
              "method": "POST",
              "url": "https://api.example.com/discount",
              "status": 500
            },
            {
              "kind": "console",
              "offsetMs": 1240,
              "type": "pageerror",
              "text": "Uncaught TypeError: discount is undefined"
            },
            {
              "kind": "step",
              "offsetMs": 1300,
              "title": "expect total to equal 90",
              "category": "pw:api",
              "durationMs": 5,
              "depth": 0,
              "error": "Expected: 90\nReceived: 100"
            }
          ]
        }
      ]
    }
  ]
}
Copy

Reads as: click Apply → backend returned 500 → page threw → assertion failed. Resolve networkId: "n3" against attempts[0].network.instances for per-request traceId / spanId / requestId / body refs; groups[instance.groupId] holds the shape and aggregate counts (occurrences, first/last offset).

Flaky test example (pass-vs-fail comparison)

For a test that fails on attempt 1 and passes on attempt 2, the divergence between the two timelines is the diagnosis. Find the first entry that differs:

{
  "tests": [
    {
      "title": "Checkout > shows confirmation",
      "status": "passed",
      "flaky": true,
      "attempts": [
        {
          "attempt": 1,
          "status": "failed",
          "timeline": [
            {
              "kind": "step",
              "offsetMs": 800,
              "title": "goto /checkout",
              "category": "test.step",
              "durationMs": 120,
              "depth": 0
            },
            {
              "kind": "network",
              "offsetMs": 950,
              "networkId": "n0",
              "method": "GET",
              "url": "https://api.example.com/cart",
              "status": 200
            },
            {
              "kind": "step",
              "offsetMs": 1400,
              "title": "expect banner visible",
              "category": "pw:api",
              "durationMs": 5000,
              "depth": 0,
              "error": "Timeout 5000ms exceeded"
            }
          ]
        },
        {
          "attempt": 2,
          "status": "passed",
          "timeline": [
            {
              "kind": "step",
              "offsetMs": 800,
              "title": "goto /checkout",
              "category": "test.step",
              "durationMs": 120,
              "depth": 0
            },
            {
              "kind": "network",
              "offsetMs": 950,
              "networkId": "n0",
              "method": "GET",
              "url": "https://api.example.com/cart",
              "status": 200
            },
            {
              "kind": "network",
              "offsetMs": 1100,
              "networkId": "n1",
              "method": "GET",
              "url": "https://api.example.com/inventory",
              "status": 200
            },
            {
              "kind": "step",
              "offsetMs": 1350,
              "title": "expect banner visible",
              "category": "pw:api",
              "durationMs": 40,
              "depth": 0
            }
          ]
        }
      ]
    }
  ]
}
Copy

Divergence: the passing attempt made an extra /inventory call that the failing attempt didn't. That's either a stale frontend cache or a race — not a flaky test, a real bug.

Full example report

See docs/example-report.json for a complete report with representative optional fields populated (network timings, redirect chains, headers, attachments, step nesting, multi-attempt retries).

Field reference

• summary -- quick pass/fail counts
• tests[].errors[].message -- ANSI-stripped, clean error text
• tests[].errors[].diff -- extracted expected/actual from assertion errors
• tests[].errors[].location -- exact file and line of failure
• tests[].errors[].{apiName,selector,actionLog} / tests[].attempts[].error.{apiName,selector,actionLog} -- failing Playwright action context from trace logs when available, including failed attempts in flaky tests. Useful for locator and detached-DOM failures; action log messages are capped at 512 characters, max 20 entries per error with an omission marker
• tests[].flaky -- true if test passed after retry
• tests[].attempts[] -- full retry history with per-attempt status, timing, stdio, attachments, steps, and network
• tests[].attempts[].consoleMessages[] -- warning/error/pageerror/page-closed/page-crashed trace entries only (2KB text cap with [truncated] marker, max 50 per attempt, high-signal entries prioritized over low-signal)
• tests[].steps / tests[].network / tests[].timeline -- convenience aliases from the final attempt
• tests[].attempts[].timeline[] -- unified, sorted-by-offsetMs array of all retained events (kind: "step" | "network" | "console"). Slimmed-down entries for quick temporal scanning; full details remain in the source arrays
• offsetMs -- milliseconds since the attempt's startTime. Always present on steps (from TestStep.startTime). Optional on network entries (from trace _monotonicTime or startedDateTime, converted via the trace's context-options anchor) and console entries (from trace monotonic time field + anchor). Absent when the trace lacks a context-options event. Entries without offsetMs are excluded from the timeline
• tests[].attempts[].network -- a three-layer NetworkReport that separates instances (what happened), groups (shared shape), and bodies (payloads). Access patterns:
  • Filter network.instances[] by status, redirectFromId, traceId, etc. to find specific occurrences
  • Scan network.groups[groupId] for aggregate shape info (resourceType, failureText, wasAborted, occurrenceCount, retainedInstanceCount, suppressedInstanceCount, evictedInstanceCount, firstOffsetMs, lastOffsetMs)
  • Resolve network.bodies[instance.requestBodyRef] / [instance.responseBodyRef] for JSON/text payloads (2KB cap with [truncated] marker, canonicalized: false in v3.0)
  • network.summary gives end-to-end accounting: observedInstances === retainedInstances + instancesDroppedByFilter + instancesDroppedByGroupCap + instancesDroppedByInstanceCap + instancesSuppressedAsDuplicate + instancesEvictedAfterAdmission. For every group, occurrenceCount === retainedInstanceCount + suppressedInstanceCount + evictedInstanceCount
• Retention policy -- instances capped at 500, groups at 200, bodies at 100 per attempt. Low-signal static assets (script/stylesheet/image/font/media with 2xx and no failure) are dropped at the filter. Duplicates of the same shape are sampled: first 3 always admit, then 1-in-10. Under pressure, eviction is by priority tier (5xx > actionable connect/DNS/TLS failure > 4xx > successful xhr/fetch > plain aborted > unknown > other known > static asset) with strict < — ties reject rather than churn.
• tests[].attempts[].network.instances[].{traceId,spanId,requestId,correlationId} -- traceId and spanId come from the response W3C traceparent when available. If the response is unreadable, Datadog request headers (x-datadog-trace-id / x-datadog-parent-id) take precedence over a conflicting request traceparent and are normalized from unsigned decimal to zero-padded hexadecimal. requestId and correlationId come from x-request-id / x-correlation-id. Malformed and all-zero trace identifiers are rejected.
• tests[].attempts[].network.instances[].clientLifecycle -- additive schema-v3 browser lifecycle diagnostics from a browser-network-lifecycle JSON attachment. The reporter joins records by method, origin, sanitized path template, and nearest request-start time. The lifecycle distinguishes no_response_headers, headers_without_body_completion, network_failure, and completed, and can retain supplied request/response/completion/failure times, pending-at-timeout state, opaque correlation identifiers, protocol/connection metadata, and encoded byte counts.
• Redirects -- instance.redirectFromId / instance.redirectToId reference sibling instance ids; walk the graph to reconstruct chains.
• tests[].attempts[].failureArtifacts -- for failing/timed-out/interrupted attempts: screenshotBase64 (base64-encoded screenshot, max 512KB), videoPath (first video attachment path). Omitted entirely when neither screenshot nor video is available
• tests[].attachments[].path -- relative to Playwright outputDir
• tests[].stdout / tests[].stderr -- capped at 4KB with [truncated] marker

Browser lifecycle attachment contract

Producers should import encodeBrowserLifecycleAttachment instead of defining their own schema or size limit. The encoder sanitizes records, orders no_response_headers, headers_without_body_completion, and network_failure records before routine completed records, and returns a body guaranteed not to exceed the shared 64 KiB limit:

import {
  encodeBrowserLifecycleAttachment,
  type BrowserLifecycleRecord,
} from "@clipboard-health/playwright-reporter-llm";

const records: BrowserLifecycleRecord[] = [];
const encoded = encodeBrowserLifecycleAttachment({ records });

await testInfo.attach("browser-network-lifecycle", {
  body: encoded.body,
  contentType: "application/json",
});
Copy

BROWSER_LIFECYCLE_ATTACHMENT_SCHEMA exposes the version, maximum bytes, and maximum records. encoded.attachment.truncated is set when the caller reports earlier truncation or the encoder drops invalid or over-limit records; the result also reports observed, included, and dropped record counts.

The encoder emits schema version 2. Each version identifies one wire shape; version 2 uses the producer's nested lifecycle events:

{
  "schemaVersion": 2,
  "truncated": false,
  "records": [
    {
      "method": "GET",
      "origin": "https://api.example.com",
      "pathTemplate": "/api/v1/:workplaceId/cases",
      "playwrightRequestKey": "request-17",
      "cdpRequestId": "1234.56",
      "loaderId": "loader-1",
      "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
      "spanId": "00f067aa0ba902b7",
      "apiGatewayRequestId": "A0XEghTXPHcEScg=",
      "requestStarted": {
        "cdp": {
          "utc": "2026-07-20T18:35:43.100Z",
          "monotonicMilliseconds": 12345.1
        }
      },
      "responseReceived": {
        "cdp": {
          "utc": "2026-07-20T18:35:43.168Z",
          "monotonicMilliseconds": 12413.1
        }
      },
      "loadingFinished": {
        "cdp": {
          "utc": "2026-07-20T18:35:43.170Z",
          "monotonicMilliseconds": 12415.1
        }
      },
      "protocol": "h2",
      "connection": {
        "id": 17,
        "reused": true,
        "remoteEndpoint": {
          "ipAddress": "10.0.0.12",
          "port": 443
        }
      },
      "encodedBytes": {
        "data": 421,
        "responseHeaders": 256,
        "total": 677
      },
      "classification": "completed"
    }
  ]
}
Copy

For a failure, loadingFailed can supply a nested CDP or Playwright timestamp plus errorText (restricted to Chromium net::ERR_* values), canceled, blockedReason, and corsErrorStatus. The reporter prefers the CDP timestamp and falls back to Playwright. Unknown fields are discarded. Origins are reduced to scheme/host/port, query and fragment text is removed from pathTemplate, identifiers are format/length checked, and only the documented allowlist is emitted.

Fixtures can attach the encoded body as browser-network-lifecycle or browser-network-lifecycle.json with content type application/json, either directly in body or through path. The reporter independently enforces the same byte and record limits; clientLifecycle.truncated marks producer- or reporter-side truncation. The legacy flat version-1 reader remains for existing attachments, but new producers should only emit version 2 through the shared encoder.

Why not Playwright's built-in JSON reporter?

This library is specialized for agents:

• JSON instead of markdown. Other LLM-focused reporters emit a markdown summary, which is easy to read and hard to post-process. Agents can't cheaply filter to "just the failed tests" or "just 4xx/5xx requests on the failing attempt" without re-parsing prose. JSON with a flat, documented schema lets agents jq or index into exactly the fields they need.
• Better flaky test diagnosis. Full per-attempt retry history with a unified, time-ordered timeline[] of steps, network, and console on every attempt. The divergence between the failing and passing attempts is usually the diagnosis (see the flaky test example).
• Better backend trace correlation. traceId and spanId are parsed from response W3C traceparent headers, with a Datadog request-context fallback for unreadable responses, so an agent can jump from a failing test to the context the backend actually selected.
• Better signal filtering. Network priority retention, console filtered, and headers allowlisted. An agent reading an unfiltered trace burns tokens on noise; this reporter does the filtering up front.

Local development commands

See package.json scripts for a list of commands.