test-harness-kit

Get started

You need Node ≥ 18. The CLI has zero runtime dependencies. To run recipes against a live app you also need one browser engine — Playwright, or a Chrome on a debugging port.

$ npm install -g test-harness-kit        # global CLI
$ npx test-harness-kit start              # or run without installing — guided tour
# to run recipes live, one engine:
$ npm i -g playwright && npx playwright install chromium
# …or a Chrome:  chrome --remote-debugging-port=9222   (THK_CDP_URL)

Quick start (60 seconds)

Explore the bundled sample cookbooks:

$ git clone https://github.com/huytieu/test-harness-kit && cd test-harness-kit
$ node cli/test-harness-kit.mjs list cookbooks
  wikipedia     v1.0.1  recipes:8   surfaces:3   freshness:ok
  the-internet  v1.0.2  recipes:16  surfaces:11  freshness:ok
  … (11 sites)
$ node cli/test-harness-kit.mjs show cookbooks/wikipedia.trif TC-01   # read one recipe
$ node cli/test-harness-kit.mjs run  cookbooks/wikipedia.trif TC-01 --adapter playwright
  TC-01  PASS/clean  2.1s  [intent:playwright]
  report:  cookbooks/wikipedia.trif/results/<ts>/report.html

On your own app: init scaffolds a skeleton, then you explore the app once and write the real recipes (authoring).

$ npx test-harness-kit init my-app --url https://my-app.example
$ npx test-harness-kit validate my-app.trif

How it works

→ See the one-page infographic — the loop, the five levels, and how a recipe becomes a script.

A cookbook (<app>.trif/) is a versioned artifact with five disclosure levels, so an executor loads only what one test needs:

Explore once (funded), write the cookbook; then any executor runs one recipe at O(1-test) context cost — read the router, then exactly one recipe. Zero-to-one-executed-test stays under 1,500 tokens.

The four verdicts

Every run returns exactly one — drift is named, never silently absorbed:

Honesty rules: retry once after settle before any FAIL; an anchor miss is FAIL-STALE, never an improvised path; a missing capability is BLOCKED, never a fabricated FAIL.

Binding-first runtime

Pay to learn the app, not to re-run it. A recipe that PASSes on the intent lane and is deterministic-ready graduates to a compiled script binding (Playwright and/or Katalon). The next run executes the binding — no model, no key, $0. Freshness is fingerprinted over (recipe source + resolved anchors); on miss/stale/failure the binding is dropped as suspect and the run falls back to the intent lane and heals. A binding failure is a signal, never a verdict.

$ test-harness-kit run app.trif TC-01 --adapter playwright   # graduates on PASS
$ test-harness-kit run app.trif TC-01 --adapter playwright   # now $0 via the binding
  flags: --no-binding · --binding-engine <e> · --no-graduate

The four doors

One artifact, four consumption paths — each a thin wrapper over the same files.

1. Agent Skills — a cookbook's SKILL.md router loads into Claude Code / Codex / Cursor.
2. MCP server — test-harness-kit mcp; see below.
3. CLI — npx test-harness-kit for CI and any shell-capable agent.
4. Raw files — it's markdown + JSON; read a recipe and run it by hand.

MCP server any agent

Run test-harness-kit mcp and any MCP-capable agent drives the whole workflow as tools. Bundled in the CLI — upgrade the CLI, the server upgrades with it.

Claude Code

$ claude mcp add test-harness-kit -- npx test-harness-kit mcp --dir "$PWD"
$ claude mcp list        # → test-harness-kit: … ✔ Connected

Then, in a session: "list the TRIF cookbooks", "show recipe TC-01 in wikipedia.trif", "run TC-01 with the playwright adapter and give me the verdict".

Claude Desktop / Cursor / other clients

{
  "mcpServers": {
    "test-harness-kit": {
      "command": "npx",
      "args": ["test-harness-kit", "mcp", "--dir", "/abs/path/to/cookbooks"]
    }
  }
}

Tools

CLI reference

Evidence & reports native

Every run on the Playwright engine leaves a per-recipe evidence bundle — the way a native tool records — and a detailed report is generated automatically.

The report shows, per recipe: the verdict, the resolved lane, a step trace, oracle checks vs observed, timing, and the evidence inline/linked. See a real report →

IA map

test-harness-kit map <cookbook> renders a self-contained, interactive map of a cookbook's information architecture — surfaces as nodes carrying their anchors, recipes, and freshness, with a detail drawer and filters. The fastest way to make a cookbook legible to a human. Open the live map →

Authoring a cookbook

Authoring is the one step that benefits from an agent: explore the app once and turn an init skeleton into real recipes.

1. Map the surfaces. Walk the app; record each surface (page/state) in map/surfaces/*.md and every element in map/anchors.json (role + accessible name + a stable hint + a note on the non-obvious bits).
2. Write recipes. One test per recipes/TC-*.md: setup, steps (12-verb grammar), and an explicit oracle — prefer deterministic assertions over judged ones.
3. Be honest about coverage. Record what you exercised (and didn't) in coverage.json.
4. Validate + run. validate, then run; a PASS graduates to a binding.

Solutions

TRIF is an open, vendor-neutral standard. Kit Apache-2.0 · spec CC-BY-4.0 · by Huy Tieu.