Captured Test Evidence — engine-captured, seal-gated test evidence
The single most-trusted number a customer reads from a build — *did the tests pass, how many* — is today the **agent's self-report**, and nothing mechanically verifies the agent pasted real output. `validateBuildReportFormat` (`artifact-validators.ts:582-600`) checks only section *headers*; the test evidence inside is whatever the agent typed. This is the one place "verified over trusted" is currently only *hoped*: an agent that types "all tests pass" seals exactly like one that ran them.
verdict PASSscore 36 / 36findings 11 (2 risk · 4 debt · 5 obs)duration 4h 39mrejection cycles 0shipped Jun 6, 2026surface cli
Pipeline timeline
Intent to proven code in 4h 39m across Think, Plan, and 2 Build→Verify phases.
Think40m
Plan88m
Build 138m
Verify 19m
Build 239m
Verify 233m
Assertion ledger
36 claims, each independently verified. Showing 8 — show all →
| ID | Says | Matcher | |
|---|---|---|---|
| A001 | Running tests through capture writes the full raw test output to a capture file | verified | ok |
| A002 | Capture returns a compact marker instead of the raw test scrollback | verified | ok |
| A003 | A plain test command is parsed into a program and arguments without a shell | verified | ok |
| A004 | A command wrapped in a directory change is parsed, recovering the directory and command | verified | ok |
| A005 | Leading environment assignments are lifted into the environment, not treated as the program | verified | ok |
| A006 | A test command containing a pipe is refused with an error naming the pipe | verified | ok |
| A007 | A test command using shell substitution or backticks is refused, never run through a shell | verified | ok |
| A008 | A spawn failure or output overflow throws before any capture file is written, so truncated output is never sealed | verified | ok |
Findings 11 total
debt.claude/agents/ana-build.md→ closed
Repo-root dogfood agent files (.claude/.codex/agents/ana-build|verify.md) edited — spec Gotcha 'Templates, not dogfood' explicitly excludes them; propagation to the dogfood is owned by the prerequisite re-init scope. Edits are byte-identical to the template edits.
riskpackages/cli/src/commands/test.ts→ scope
Checkpoint passthrough is joined with spaces and re-parsed by resolveCommand, losing original argv quoting. A multi-token checkpoint command whose args contain spaces/parens/metacharacters is misparsed or refused (verified live: parens in an arg triggered a subshell refusal). Mitigated by degrade-to-raw so it never blocks, but counts/verdict are lost.
riskpackages/cli/src/utils/capture-runner.ts→ monitor
deriveCounts falls through to every parser when no hint matches; the rspec parser regex /(\d+) examples?, (\d+) failures?/ is loose enough to match unrelated output, risking a false count (and a false 'pass' when passed>0 at exit 0) on an unknown runner. Counts are fail-open by design, but a coincidental match defeats ABSTAIN-ON-UNKNOWN for that input.
debtpackages/cli/src/utils/capture-marker.ts→ monitor
validateCapturePresent uses parseMarkers (per-line scan) which does NOT skip inlined block content, unlike the integrity validators that use eachMarker. A capture marker embedded in preserved output could falsely satisfy 'present.' Harmless when a real top-level marker exists; the asymmetry is worth recording.
debtpackages/cli/tests/capture-corpus/invariants.test.ts→ acknowledge
Corpus errorToken is the generic string 'Error' for 7 of 8 stacks (vitest uses the specific 'AssertionError'). It is present in each fail fixture, but a generic token is a weak assertion for ERROR-NEVER-STRIPPED — it would pass even if a different error string were the one preserved.
+6more findings
Integrity seal
scopesha256:c3c83aee7ef23...
contractsha256:23c7ef1e5e873...
plansha256:563de8d90aad7...
spec-1sha256:e2c30ba365b27...
spec-2sha256:8adc3d45885b3...
build-report-1sha256:4f9936f42812e...
build-data-1sha256:62321e15f069c...
verify-report-1sha256:134b411e5530b...
verify-data-1sha256:e09c32cffc0fb...
build-report-2sha256:8da844530b770...
build-data-2sha256:f7cc5d7247ff6...
verify-report-2sha256:45d71b968573d...
verify-data-2sha256:0767137494e86...
audit cmd$ ana proof audit captured-test-evidence → all hashes match