Empirical Proof Benchmark — the measuring instrument (ruler)
Harvest the benchmark harness/scorer from the never-deploy `feature/devday-scan` branch onto `main` as a **deterministic A/B measuring instrument** — a "ruler" that takes two Claude Code session transcripts (one produced with Anatomia, one vanilla) and emits a tamper-evident, reproducible comparison: tokens, tool calls, cost, and the reliability metrics that carry the product's "verified over trusted" claim.
verdict PASSscore 38 / 38findings 6 (0 risk · 3 debt · 3 obs)duration 1h 43mrejection cycles 0shipped Jun 19, 2026surface cli
Pipeline timeline
Intent to proven code in 1h 43m across Think, Plan, Build, and Verify.
Think25m
Plan12m
Build17m
Verify24m
Assertion ledger
38 claims, each independently verified. Showing 8 — show all →
| ID | Says | Matcher | |
|---|---|---|---|
| A001 | The benchmark suite is OFF by default and only turns on when explicitly enabled | verified | ok |
| A002 | The benchmark suite turns on only when the enable switch is set to exactly on | verified | ok |
| A003 | A stray or partial value never accidentally turns the benchmark suite on | verified | ok |
| A004 | The benchmark scorer never depends on the never-deploy branch it was ported from | verified | ok |
| A005 | The fixed localization task loads with its pinned commit and relevant-file set | verified | ok |
| A006 | Asking for a task that does not exist returns nothing instead of crashing | verified | ok |
| A007 | Scoring a real session transcript succeeds rather than abstaining | verified | ok |
| A008 | The scorer counts exactly how many distinct files the agent opened | verified | ok |
Findings 6 total
debtpackages/cli/tests/benchmark/scorer.ts→ scope
Dead export `totalTokens` — defined with full JSDoc, never called anywhere (cost uses derived.tokens; the edit-token walk inlines its own input+output sum)
obspackages/cli/tests/benchmark/aggregate.ts→ monitor
aggregate treats 'lower is better' for every numeric metric, including distinctFilesRead/inputTokens/turns — an arm that reads fewer files (even by missing relevant ones) scores a 'win'. Spec-endorsed simplification, but a semantic foot-gun for the next reader.
obspackages/cli/tests/benchmark/harness.test.ts→ monitor
AC7 guard (A036/A037) is a textual substring assertion on scorer.ts source — a future read of a best-effort field via destructuring-rename or computed key would slip past. Legitimate enforcement test, but textual not semantic.
obspackages/cli/tests/benchmark/scorer.ts→ monitor
The three to-first-correct-edit metrics can disagree on null-ness: wallClockMsToFirstCorrectEdit is null when the editing line lacks a parseable timestamp, even while turnsToResolution/tokensToFirstCorrectEdit are populated. Latent inconsistency; not triggered by the committed fixtures.
debtpackages/cli/tests/benchmark/aggregate.test.ts→ monitor
Determinism test (A034) only asserts same-input idempotence, not stability across input permutations. Implementation genuinely sorts by (task,metric,arm) so cross-permutation determinism holds, but the test would pass even if the explicit sort were removed.
+1more findings
Provenance
Who ran what, and what it cost. Recomputable estimates from the shared price table — subordinate to the verdict, never gating.
model claude-opus-4-8
| session | turns | tools | in | out | cache | cost |
|---|---|---|---|---|---|---|
| ana | 51 | 21 | 7.4k | 26.0k | 1.4M | $1.74 |
| plan | 70 | 30 | 7.3k | 53.1k | 1.9M | $2.98 |
| build | 115 | 52 | 7.8k | 62.6k | 5.1M | $4.92 |
| verify | 120 | 53 | 8.1k | 37.3k | 4.6M | $3.97 |
| TOTAL 4 sessions | $13.61 (table 2026-06-14) | |||||
churn 9 files · +1638/−0
completeness ✓ complete (plan 1/1 · build 1/1 · verify 1/1)
Session attestation
How each agent session behaved, coverage-aware. Evidence, not a gate — unverifiable is honest abstention, not a failure.
core v0.4.0 · framework anatomia
plan · 49 claims
1 satisfied · 0 violated · 48 unverifiable
coverage 1/49 checked · 48 unverifiable
⚠ ana-plan:skill:testing-standards unverifiable (absent-signal)
⚠ contract:file-scope:packages/cli/tests/benchmark/scorer.ts unverifiable (channel-coverage-incomplete)
⚠ contract:file-scope:packages/cli/tests/benchmark/harness.ts unverifiable (channel-coverage-incomplete)
mandate sha256:98bbd781e1… · transcript sha256:7b625612d5…
⚠ incomplete coverage
build · 49 claims
1 satisfied · 0 violated · 48 unverifiable
coverage 1/49 checked · 48 unverifiable
⚠ ana-build:verify-independence unverifiable (channel-coverage-incomplete)
⚠ contract:file-scope:packages/cli/tests/benchmark/scorer.ts unverifiable (channel-coverage-incomplete)
⚠ contract:file-scope:packages/cli/tests/benchmark/harness.ts unverifiable (channel-coverage-incomplete)
mandate sha256:00ebf6cca1… · transcript sha256:42004c0f2c…
⚠ incomplete coverage
verify · 52 claims
2 satisfied · 0 violated · 50 unverifiable
coverage 2/52 checked · 50 unverifiable
⚠ ana-verify:verify-independence unverifiable (channel-coverage-incomplete)
⚠ ana-verify:no-code-branch-mutation:git-rebase unverifiable (command-unresolvable)
⚠ ana-verify:no-code-branch-mutation:git-push---force unverifiable (command-unresolvable)
mandate sha256:d09a15fdb4… · transcript sha256:45bf7834d8…
⚠ incomplete coverage
verdict veto: not applied — verify did not read build_report.md
veto is forward-only; pre-veto verdicts were self-reported.
Integrity seal
scopesha256:a1d0c74563fcf...
contractsha256:16f4c23ccc2af...
plansha256:3f1eb93b09a02...
specsha256:2fdb300af0867...
build-reportsha256:ae25482f383d7...
build-datasha256:c54658f79ab79...
verify-reportsha256:dbd4865ea6351...
verify-datasha256:298df85eb4cb8...
audit cmd$ ana proof audit proof-benchmark-harness → all hashes match