Telling a bug from drift — and healing without hiding it.
Every browser test eventually fails because the app changed, not because it broke. Traditional tooling can't tell the difference — so it either cries wolf or silently adapts. TRIF makes drift a first-class, named outcome and repairs it through an explicit, reviewable loop that can never absorb a real regression.
The problem: one red X hides three different things
When a step fails, three completely different situations look identical to a normal test runner — so reds get ignored, "flaky" becomes an excuse, and real regressions slip through.
How drift is detected
Detection is layered — four independent mechanisms, each cheap, that together decide "the element the recipe expects is no longer where/what it was" before anything is called a bug.
a. The anchor resolution ladder
Every element reference (an anchor) resolves by trying progressively more robust strategies. Which rung succeeded is recorded per step as resolvedVia.
#firstHeading. Fastest; the one bindings compile to.role=heading name="Article title" — survives most markup churn.b. Freshness fingerprint
A binding is trusted only while a fingerprint over (recipe source + resolved anchors) is unchanged. Edit the recipe, or let an anchor resolve differently, and the fingerprint breaks — the binding retires until the recipe re-earns it. stale reports this per surface, dated, before a run.
c. A binding failure is a signal, never a verdict
If a compiled script stops passing, TRIF does not report FAIL. It drops the binding as suspect and falls back to the intent lane, which re-derives the answer from the recipe. The deterministic fast-path can be wrong; the recipe is the source of truth.
d. The retry-once honesty gate
Before any FAIL, the executor retries once after the page settles — so a slow render never masquerades as drift or a bug.
The decision: which verdict, and why
Once a step can't be satisfied, the outcome is decided by a strict, auditable rule — not a guess.
The healing loop
Drift doesn't fix itself in the dark. It becomes a machine-readable patch proposal that a human reviews and applies — three roles, one writer.
Executor
Runs recipes, grades oracles, records friction. Can never write to the cookbook — the discipline that catches mis-modeled tests instead of papering over them.
Healer
Turns friction into a typed patch proposal (a diff), with the evidence that motivates it. Proposes only — applies nothing.
Maintainer
Reviews, applies the patch, bumps the version, writes the CHANGELOG. The recipe re-verifies and re-graduates to a binding.
Patch types
| Patch | When | Guard |
|---|---|---|
anchor-fix | an element moved / its hint went stale — update the anchor | auto-applyable |
new-quirk | a newly-observed behavior worth recording in the map | auto-applyable |
recipe-step-fix | a step needs adjusting to match a changed flow | reviewed |
binding-stale | a compiled script no longer matches — retire/recompile it | auto-applyable |
oracle-drift | the expected result itself changed (not the path to it) | requires --human-signed |
oracle-drift is gated hardest on purpose: changing what "correct" means is exactly where a silent auto-heal would erase a real regression. A human must sign it.
# drift surfaced on a run → inspect and apply, explicitly $ test-harness-kit stale wikipedia.trif # what's suspect, and since when $ test-harness-kit heal wikipedia.trif list # pending patch proposals $ test-harness-kit heal wikipedia.trif apply patch-anchor.json $ test-harness-kit verify wikipedia.trif --surface article # re-drive, re-shoot, refresh ledger
Worked example: a Wikipedia skin change
A real case from the reference cookbook — the difference between a false alarm and a caught change.
#vector-toc (the Vector-2022 skin). Wikipedia rolls the page back to the legacy skin, where the TOC is #toc.#vector-toc selector misses. Binding failure → signal, not verdict → drop to intent lane. The anchor's rung-1 hint misses; the healer notes the documented fallback pattern (#toc present) — a skin-version event, not a missing feature.anchor-fix proposal updates the TOC anchor's hint to prefer #vector-toc, fall back to #toc. A maintainer applies it; the recipe re-verifies and re-graduates.Why not just "self-healing locators"?
Auto-adapting selectors are common — and dangerous — because they optimize for green, not for truth.
| Silent self-heal | TRIF stale + heal | |
|---|---|---|
| on drift | silently re-binds to a best-guess element | names it FAIL-STALE and proposes a reviewable patch |
| audit trail | none — the test just keeps "passing" | a versioned diff + evidence in the CHANGELOG |
| regression risk | high — a removed/changed feature is absorbed | low — a human signs off on any change to expected results |
| who writes | the runner, at run time | only the maintainer, after review |