Footage index contract

This contract keeps search results tied to the media and timing they describe. Use it when creating a new implementation, changing artifact schemas, or debugging stale or shifted results.

Dependency graph

Game-profile artifacts (hud/<region>.mp4, audio16k.wav, hud_events.json, audio_events.json, rank.json) follow the same sidecar, provenance, and invalidation rules as every other stage. Event and window records carry t_sec/end_sec source-relative float seconds — the same interval contract as dialogue cues — so search joins and select manifests treat them identically. Window hits deliberately skip the shot join: a continuous gameplay camera has no cuts, so the ranked window itself is the cuttable unit and flows into the select manifest with shot_index: null.

Visual retrieval

Visual search compares text and representative shot frames in one embedding space. The current implementation uses OpenCLIP ViT-B-32 with the laion2b_s34b_b79k weights, but the artifact contract records the exact model and pretrained variant so another implementation cannot silently mix vectors.

Build time performs the expensive image pass once: extract one interior frame per shot, encode it, L2-normalize it, and store the vector with its shot_index. Search time encodes the query with the matching text encoder, normalizes it, and ranks stored vectors with a matrix-vector dot product. Because both sides are normalized, the score is cosine similarity. It is a relative ranking signal, not a probability or calibrated confidence value.

The default midpoint represents the shot economically and avoids boundary residue, but it does not model motion or the whole duration. A long or changing shot may need multiple interior samples; when present, compare every sample and pool the scores back to one shot before global ranking. Text queries describe what should appear in an image; richer phrases usually disambiguate a single word without changing the retrieval contract.

The index directory defaults to <media_dir>/.index so artifacts travel with their media; a workspace may relocate it (index_dir in footage.json) when the media volume cannot hold the artifacts or a faster disk should. A relocated index is rebuildable state, not part of the media's identity.

Silhouette samples

silhouette.npz holds one record per proxy frame at sample_fps (default 4): person-matte area, normalized top-left-origin bbox and centroid (NaN when nobody is found), and the matte pooled onto a coarse grid (default 16×9) as mask (uint8). The producer is Apple's Vision person segmentation via scripts/vision-person.swift, recorded as model with the quality level in params; the stage depends on the validated proxy.

Samples are whole-film, not per shot, because the aligned frame is usually mid-shot; consumers join a sample to its shot through shots.tsv and slide the in-point. continuity() and best_in_point() in _footagelib.py are the single implementation; music-edit imports them rather than re-deriving the score. A sample's direction of travel is the centroid delta to the next sample, so the outgoing side reads dir[k] and the incoming side dir[k-1].

Source identity

Keep these concepts separate:

• asset_id identifies media content independently of its display name.
• name is a human label and is not unique identity.
• path is current operational state and can change when media is relinked.
• source_fingerprint detects replacement or mutation and invalidates derived artifacts.

A full content hash is strongest. For multi-gigabyte sources, a named sampled hash plus file size is acceptable for asset_id; combine it with size and nanosecond mtime for change detection. Record the hashing method. Never let two different source files share an artifact directory merely because their stems match—namespace by asset ID or reject the collision before doing work.

Time and frame semantics

• Float seconds are authoritative and source-relative: zero is the first decoded program frame, not an arbitrary container PTS.
• Intervals are half-open: [in_sec, out_sec).
• Record the container start time even when timestamps are normalized.
• Store exact rational rates such as 24000/1001 and the stream time base. A float rate is a convenience rendering, not the conversion authority.
• For a verified CFR source, convert a half-open seconds interval to inclusive Resolve bounds by flooring the start and using ceil(end × fps) - 1 for the end. Clamp to the source extent and test the API's inclusivity after Resolve upgrades.
• VFR has no single seconds × fps mapping. A nominal/average rate comparison is a cheap screen, not proof of CFR. Suspected or known VFR never enters the direct frame-based import path; normalize only selected ranges or use a timestamp-aware mapping that has been independently validated.

Proxy timing gate

The proxy is a rebuildable cache, not an edit master. Record the source fingerprint, encoder, dimensions, bitrate, cadence mode, ffmpeg version, and output signature. Before downstream use, validate at least:

• The proxy starts at normalized time zero.
• Source and proxy durations agree within a stated frame-based tolerance.
• A CFR candidate retains its exact average rate.
• The proxy artifact and its metadata match the current source and parameters.

Container duration and rate checks catch broad shifts but do not prove every PTS. Spot-check proxy-derived boundaries against the original for each new container/codec path; automate representative PTS comparisons where the tooling can expose them without a full software decode.

Artifact provenance and invalidation

Every stage sidecar contains:

{
  "contract_version": 2,
  "stage": "shots",
  "source": {
    "asset_id": "content-derived-id",
    "size_bytes": 0,
    "mtime_ns": 0,
    "sampled_sha256": "..."
  },
  "params": {},
  "dependencies": {},
  "outputs": {},
  "observations": {}
}

An artifact is current only when its source fingerprint, parameters, dependency signatures, schema version, tool/model identity, and output signatures match. Rebuild from the first stale stage. Write manifests and sidecars atomically so an interrupted pass cannot look complete.

Stage parameters use JSON-native shapes before comparison. A tuple written as a JSON list otherwise differs on the next in-memory comparison and forces an identical stage to rebuild forever.

Embedding files additionally carry their model identifier, preprocessing or pretrained variant, vector dimensions, asset ID, and contract version. Search rejects an index built by a different model rather than discovering the mismatch through a matrix error or silently incomparable vectors.

Dialogue and visual selection

For embedded subtitles, record codec, global stream index, language, title, and disposition. Prefer the requested language and full-dialogue/default streams; avoid forced-only and commentary tracks unless requested. For ASR, record the chosen audio stream, language, engine, and model. Spot-check cue offsets.

Dialogue cues are intervals. Literal and semantic results both join each cue to every overlapping shot. Visual results already originate from a shot, but a long shot may need several interior samples; pool those scores back to one shot before ranking results.

The shot index remains comprehensive; editorial curation is a derived selection surface, not a destructive filter. curation.json records within-asset percentile scores for each prompt-bank lane and contiguous multi-shot moments with source-relative bounds, constituent shot indices, a representative shot, and labels. Its sidecar depends on current shots, frames, and visual embeddings. Consumers mix films with cross-asset diversity and temporal suppression while retaining an explicit raw-shot browse path for full recall.

Select manifest decision

Each select records asset_id, current path, source-relative seconds, handles, detected bounds, import mode, and color metadata. Direct CFR selects also carry the exact source rate and derived inclusive frame bounds. A consumer verifies the asset identity and import mode before touching Resolve.