ADR 0258: Evidence lives in the evidence store

Status: accepted (2026-10-09; proposed 2026-10-09). Tracks VUH-1902. The first slice of the built-in tracker ("Clankie Work"); its evidence bundles reference the records defined here.

Context

ADR 0221 asks for proof by the real thing, recorded as evidence. Workers record it as dated folders under docs/testing/: a README.md narrative with screenshots, raw run JSON and logs committed beside it. On 2026-10-09 clankie's docs/testing held 20 MB and about 165k lines in git: 103k lines of JSON/JSONL, 19k of Markdown, 123 PNGs. James's audit puts clankie-app at about 294 MB of evidence media in artifacts/ and docs/ (841 files, a 202 MiB pack) and clankie-ops at about 43 MB.

Large raw JSON pollutes agent search results and context. Every worktree checks out every capture. On 2026-10-08, dozens of worktrees plus fseventsd and Spotlight indexing were a real cost on this Mac (VUH-1871, VUH-1899). The tracker can only point at repo paths; it cannot hold the media.

Decision

Git keeps narratives, conclusions and pointers. The evidence store keeps the bytes and the records that describe them.

Scope. Each repository may declare its evidence roots in .clankie/evidence.json; an absent declaration defaults to docs/testing. Only files under a repository's declared evidence roots move: docs/testing/** in clankie and in clankie-app, plus clankie-app's artifacts/**. Product assets stay in git wherever they live: branding, app icons and images, persona art, App Store screenshots, fixtures. Inside an evidence root, evidence objects are media (png, jpg, gif, mp4, mov, webm) of any size, and any other non-Markdown file of 16 KiB or more: raw eval/run JSON, JSONL journals and logs. Markdown and smaller text stay in git. At 16 KiB, 46 non-media files in clankie's docs/testing move, carrying 89k of their 118k lines; about 600 small receipts and fixtures stay readable in place.

Service. The store is a TypeScript module inside the clankie service (apps/clankie/src/evidence-store.ts). Its routes live on the service's existing HTTP API behind its existing operator authentication, so there is no second process, toolchain or supervisor. (2026-10-09, James: performance doesn't matter for a blob store; no second toolchain.) Blob storage and metadata storage each sit behind an interface, so the hosted S3/R2 and Postgres backends replace the local ones without changing the API. Blobs are keyed by their sha256, so identical files are stored once and every upload is idempotent. Blobs are immutable, and nothing deletes them in this slice. They live on local disk when self-hosted and in S3 or R2 when hosted. Hosted buckets, databases and their provisioning live only in clankie-ops, so a managed user sets up nothing. Metadata lives in Postgres, or SQLite for single-machine self-hosting, and never in the bucket. Bucket credentials come from the credential broker and never leave the service.

Records. Each upload creates a record: file name, sha256, size, content type, issue key, commit, actor (operator, worker or seat, with its on-behalf-of chain), caption, and created-at. Many records may share one blob. commit is the HEAD the evidence was produced against; the commit that lands the manifest is found from git.

API.

• Upload takes the hash, size, content type and record fields, plus a client idempotency key. It answers immediately with a receipt and, only when the blob is missing, a presigned upload URL. The service checks sha256 and size before it accepts a blob. The sender can look the receipt up at once and sees applied, pending or refused, so a timed-out upload is reconciled, never resent blind. This is requirement A2 (exactly-once writes) in miniature and the pattern for VUH-1898.
• Fetch resolves an object to a signed URL that expires within 15 minutes: presigned S3/R2 URLs, or service-signed routes on local disk.
• List returns records by issue key or by commit. The recent projection (VUH-1933) reads the same records newest first with a (createdAt,id) cursor, project/repo, issue, actor, media and inclusive time filters. Optional project/repo/model/outcome metadata lives in the existing record, not another store. Historical records retain unknown fields. Bearer-authenticated raster previews cap bytes at 64 KiB; larger images use placeholders in the feed.

Callers authenticate with Clankie's own identities, the operator, worker and seat credentials the service already issues. The store has no separate accounts, and no link, manifest, log or comment carries a long-lived credential.

Callers. clankie evidence push|fetch and MCP tools (evidence_push, evidence_fetch, evidence_list) through clankie mcp. Both resolve paths from the working directory and need no per-repo configuration, so they work in any repo.

• clankie evidence push [path] [--issue KEY] [--caption TEXT] hashes the in-scope files under path (default ., but only within an evidence root: outside one it refuses and names the roots), uploads only missing blobs, and writes or updates the folder's manifest. It prints what it added and changed, with their links. The pushed raw files then move into the .local/ mirror described below. A second run uploads nothing, leaves the manifest byte-identical and says so. A listed file absent locally keeps its entry; removing one is an explicit manifest edit.
• clankie evidence fetch [path] downloads listed objects into .local/evidence/<repo-relative folder>/<path> and verifies each sha256 before moving it into place. A local file whose hash differs is reported and left untouched. Anything it cannot fetch or verify is named with its reason, and the command exits non-zero.

Amendment (VUH-2923, 2026-10-27): linked machines push over the fleet link. A fleet machine such as the Windows PC has no Clankie CLI and no operator credential, only Node and the worker plugin. The plugin now carries the push walk (worker/bin/evidence-push.mjs, which Clankie's own CLI also runs) and two ways in: a clankie command in its bin/, which Claude Code appends to PATH so an installed CLI still wins, and an evidence_push bridge tool for harnesses without that PATH. Both need --issue; key inference stays on Clankie's machine. The upload and receipt reads go to /v1/fleet/seats/{pane}/evidence-uploads, where the seat proves itself as a peer sender does and becomes the record's actor (seat, its worker name, and its lead as onBehalfOf). The bytes then go to the store's signed upload path on the same link; its signature authorizes them, so no remote proof runs while they stream. The store still refuses any blob whose sha256 differs from the one the pane hashed.

Manifest. evidence.json sits beside the folder's README.md, so agents grepping the repo still find every pointer:

{
  "version": 1,
  "objects": [
    {
      "path": "evidence/iphone-report-share.png",
      "size": 865655,
      "sha256": "<64 lowercase hex>",
      "url": "clankie://evidence/sha256/<64 lowercase hex>"
    }
  ]
}

Entries are sorted by path, which is relative to the manifest and uses / separators, so a change is a one-object diff. The rest of the metadata lives in records, not in git. The format is JSON because the CLI, the docs/testing viewer and the tracker already parse it without a dependency, and the repo formatter keeps it stable. A nested folder uses its ancestor's manifest.

Links. One object is referenced as clankie://evidence/sha256/<64 lowercase hex>, in the clankie:// scheme the app already handles. It names content, not a location, so it survives a backend move or bucket rename. The service resolves it to a fresh signed URL for an authorized reader.

Migration. READMEs and conclusions stay in git. Raw files are fetched on demand into the root .local/, which is already ignored. History is not rewritten, because clankie is public and clones depend on it; pack sizes therefore do not shrink. Two consumers read archive files today: apps/tui/test/claude-tool-catalog.test.ts and scripts/evals/lead.mjs. Each gets a trimmed fixture it owns before its folder migrates. Then each repository moves its existing evidence in one commit: push the folders, untrack the moved files, commit the manifests. Media in evidence roots (docs/testing/** here, clankie-app's artifacts/ and docs/testing/) gets ignore rules. .gitignore cannot match on size, so a cheap repository check in check:landing rejects any newly added in-scope file under those roots.

Alternatives

Git LFS still materializes every file in every worktree, binds each clone to one LFS server and its quota, and has no issue or actor records. A tracker-only attachment store would leave repo evidence in git. Keeping everything in git is the cost above.

Proposed hosted credential follow-up

ADR 0297 proposes default-off tenant credential brokerage and renewal for VUH-1913. It is awaiting James's decision and authorizes no hosted rollout or owner activation.

Consequences

Implementation note (VUH-1913, partial): the public S3 blob adapter keeps the existing authenticated service upload and JSON receipt contract, checks size and SHA256 on private temporary disk before PUT, and resolves recorded blobs to S3-signed GETs capped at 15 minutes. Local remains the default. Phase B adds owner GET/POST and clankie evidence storage for strict non-secret destination configuration through the revision-fenced settings API. Configured and active remain separate: a save does not hot apply, restart, migrate or activate storage. Captured writes and private historical/pending associations remain unchanged. Arbitrary local roots, endpoints, credential fields and settings-file fallbacks are refused. Existing invalid/unknown settings fail closed without overwrite; this slice does not broaden the settings schema. Full startup proof remains open. This foundation uses the standard AWS provider chain and SQLite metadata. It does not complete the broker-backed renewable tenant credentials, private hosted infrastructure, backup policy, migration, or required real-hosted push/fetch proof. The hosted metadata decision above remains unchanged; Postgres is not added by this slice.

Phase A (VUH-1913) preserves destinations in private SQLite columns on pending uploads and records. CLAIM captures the fixed service-local root identity or the non-secret S3 bucket/region/prefix; settlement copies the stored claim, even when fixture configuration changes or the store reopens. Fetch, device ranges and the service-signed blob GET resolve that recorded identity. A SHA with several records selects the earliest (createdAt,id) association deterministically; all records and issue/actor bindings remain intact. Reads never substitute today's destination or fall back to another association. The resolver caches at most 32 historical adapters, evicts and reconstructs descriptors without limiting retained destinations, and releases its owned SDK clients when operations/streams complete or cancel. .blobs remains the active upload backend.

Legacy destinations remain unknown. S3 startup refuses unknown records or pending uploads. Unchanged local reads verify SHA and size from the fixed original local root without tagging or inferring a location from settings. The internal explicit preparation helper verifies original bytes before tagging each association and preserves originals and successful partial progress. It accepts at most 100 named identifiers, 256 MiB declared bytes and 30 seconds for association progress per batch. Missing, unknown, oversized or slow inputs leave their associations unresolved; this is bounded preparation, not a completed migration. The deadline stops association progress; preparation does not currently thread the adapter's optional abort signal into SDK HEAD/GET calls, and their eventual streams are cancelled without tagging. No public destination/path/endpoint input or owner switch control is introduced.

An unknown pending upload with no original bytes can instead receive an explicit private local continuation admission for its exact actor key, receipt ID and full immutable request. Admission permits newly supplied bytes into the fixed local root; it does not assert historical origin or tag metadata. Every incoming body is fully SHA-and-size verified in private staging, even if a correct same-SHA original already exists. A transaction checks the upload is still pending, NULL and unchanged before exclusive installation and atomic metadata destination, record and applied settlement. Existing originals are never replaced. Wrong, short, cancelled or conflicting bodies leave the upload unresolved; unadmitted uploads remain refused. S3 continuation and public admission controls are absent. Admissions cap 100 identities and 256 MiB declared bytes with a 15-minute lifetime. Private guards remain through another 15 minutes to cover every refreshed signed PUT; changed tagged/applied rows cannot fall through to ordinary backend handling. Restart invalidates old links with the existing fresh process signing key.

The metadata transaction is atomic; filesystem installation and SQLite commit are not one atomic storage operation. A genuine later database failure may leave verified unrecorded local bytes while metadata rolls back to pending/NULL. Originals remain intact, record-bound reads hide those bytes, and explicit retry must verify the full incoming body again. This describes preserved partial state, not a claimed or forced failure proof.

Destination metadata never enters public receipts, records/details, device DTOs, signed endpoint bodies or logs. The existing authorized S3 presigned GET contract still returns its short-lived URL, which necessarily identifies an object location. The two preview implementations remain on the active backend; historical preview routing requires its separately cleared follow-up. API/CLI selection and activation, full service startup proof, tenant/broker delivery and real-hosted acceptance remain outside Phase A. Its manual integration proof uses production evidence routes, credential-backed authentication with a real disposable file broker, real SQLite and files, the production SDK and an owned loopback MinIO over actual HTTP; it neither starts nor switches the owner service.

Paired owner devices may read recorded media in bounded byte ranges through their authenticated device/relay path (evidence_fetch), with the record’s content type; they never receive service URLs or store-signed links (VUH-1967, also enabling large-image reads for VUH-1954).

• Workers publish evidence by writing the folder's README.md, running clankie evidence push <folder> --issue <key> (or the MCP tool), and committing only the README and evidence.json. Tracker comments cite the printed clankie://evidence links. The clankie skill, the linear-issues media rules and the testing records guide change when the commands ship.
• Worktrees and agent search stop carrying raw captures. Readers fetch only the folders they open. pnpm testing:view must learn to read the .local/ mirror.
• The built-in tracker's evidence bundles (requirement A8) are lists of record IDs and clankie://evidence links, queried by issue or commit. The tracker stores references, not bytes. Its own uploads and the Linear import use the same upload path.
• Manifests in this public repo expose paths, sizes and hashes. Captions live in private records. Store access follows Clankie's identities.
• The store ships inside the clankie service, so it adds no native toolchain and no process to supervise; clankie doctor reports its backends and data directory (2026-10-09, James: performance doesn't matter for a blob store; no second toolchain). Store growth is unbounded until a retention decision.