▋
creative/footage-index/SKILL.md

name: footage-index description: >- Index a large footage, film, or gameplay-recording library so moments can be found by dialogue, visual content, or detected gameplay events (kills, multikills, hype), then imported into DaVinci Resolve without blanket transcoding. Owns the footage-* pipeline scripts in scripts/. Use for BOTH phases: building the index, and every later query against one — "find the best action/fight/close-up shots", "find where they say X", picking clips for a montage, supercut, fan edit, or video essay, or assembling lanes from an already-indexed corpus. Never hand-roll queries against the footage API or index files; load this skill first.

footage-index

Make hours of long-form footage searchable, then hand cuttable source ranges to an NLE without producing edit masters for material the edit never uses.

When to use this

  • A montage, supercut, video essay, or fan edit draws from feature-length media.
  • The request is “find where someone says Y” or “find the shot where X happens.”
  • The source library is far larger than the finished piece.
  • The sources are gameplay recordings and the request is “find my best plays” — the game-profile stages below detect kills, deaths, and hype deterministically.
  • The corpus is already indexed and the ask is to select from it. This is the common repeat case and the easy one to miss — an indexed workspace makes the request look like plain API work. Load this skill for those queries too rather than starting the API and hand-rolling calls; the scripts already own the query, lane-building, and Resolve-import conventions.

For a handful of supplied clips, inspect them directly and skip library indexing; normalize only if their timing or decode behavior requires it.

For creative edit direction, use music-edit. Search ranks, curation lanes, and representative frames supply candidates; review the actual source interval and transition handles before choosing a gesture, cut point, or moving crop.

Scripts and workspaces

This skill owns the implementation: scripts/footage-* with shared conventions in scripts/_footagelib.py. Scripts run against the workspace named by FOOTAGE_ROOT (default: the current directory). A footage.json descriptor at the workspace root selects the layout — media_dir, project_dir, optional index_dir (default <media_dir>/.index; relocate when the media volume is full or slow), optional game_profile; without a descriptor the legacy numbered layout (06-films/, 05-project/) applies. Projects keep thin bin/ wrappers that export FOOTAGE_ROOT and exec these scripts.

footage-selftest exercises the film contract on synthetic media; footage-game-selftest does the same for the gameplay stages; footage-silhouette --selftest and footage-matchcut --selftest cover the match-cut stage. Run them after touching scripts/.

footage-corpus-ui starts the reusable local editorial desk and API against the active workspace; footage-corpus-api starts the agent-facing JSON surface alone. The implementation lives in web/. Workspace-specific branding belongs under ui in footage.json; curation lane labels, prompts, negative weights, and the lanes feeding best belong under curation. Neither belongs in copied project code or a hardcoded reusable prompt bank.

The workspace repository owns corpus identity and state: original media, footage.json, authored collections, and derived indexes (normally ignored by Git). This skill owns all reusable probing, indexing, search, browse, and export implementation. A corpus may add configuration and thin launch wrappers, but it does not copy these stages.

Define the project vocabulary explicitly when using footage-curate:

{
  "curation": {
    "prompt_version": 1,
    "best_description": "The strongest moments across the primary lanes.",
    "best_lanes": ["action", "movement"],
    "negative_prompts": ["credits, logos, or black frames"],
    "negative_weight": {"action": 0.25, "movement": 0.2},
    "lanes": {
      "action": {
        "label": "Action",
        "description": "Physical action and impact.",
        "prompts": ["a dynamic cinematic action scene"]
      },
      "movement": {
        "label": "Movement",
        "description": "Traversal and expressive motion.",
        "prompts": ["fast cinematic movement through an environment"]
      }
    }
  }
}

The rule that drives the design

Index first. Select second. Transcode only to satisfy a proven conform requirement.

Blanket ProRes conversion inverts the economics of long-form source libraries: 4K24 ProRes 422 HQ is roughly 315 GB/hour, while an edit may use only minutes of a multi-hour library. Preserve originals, build small cached indexing proxies, and normalize only selected ranges or sources that fail the direct-import contract, such as VFR media.

Before designing manifests or changing pipeline code, read references/index-contract.md. Before importing selects into Resolve, read references/resolve-import.md.

Pipeline

  1. Probe — bind a stable asset identity; record exact rates, time base, duration, codecs, HDR transfer, cadence classification, and subtitle/audio tracks; decide the import mode.
  2. Proxy — build a cached, rebuildable ~640px H.264 proxy and validate its timing against the original.
  3. Shots — detect boundaries on the validated proxy.
  4. Dialogue — prefer a suitable embedded text track; use ASR when none is usable.
  5. Frames + sheets — extract representative frames and labelled contact sheets from the proxy.
  6. Embeddings (optional) — build text first; add visual search only when it earns its heavier dependencies.
  7. Curation (optional) — score existing visual vectors against reusable editorial lanes, suppress low-signal material, and group strong adjacent shots into instant-play moments without decoding the source again.
  8. Silhouettes (optional) — footage-silhouette samples a person matte a few times per second through the whole proxy (Apple Vision, no torch) so footage-matchcut and the music-edit planners can find an incoming shot and in-point whose subject sits where the outgoing one was, at the same size, moving the same way. See the match-cut section of the contract.
  9. Search → selects — join timestamped hits to every overlapping shot, add handles, and emit either direct sub-ranges or a normalization requirement.

Every derived artifact carries its source fingerprint, parameters, dependency signatures, schema version, and tool/model identity. “The output exists” never means “the output is current.”

Report a stage as done only against evidence from this session — the manifest it wrote, the selftest output, or the timing validation — and name any stage that is still unverified.

Dialogue index earns its keep first

Dialogue is usually the fastest route to a memorable moment. Prefer embedded text subtitles (subrip, mov_text, ass, and similar) because they are fast and usually cleaner than ASR, but do not assume the first text stream is the right one. Record and select by stream index, language, title, and disposition; avoid forced-only, commentary, and wrong-language tracks unless explicitly requested. Spot-check timing.

Bitmap subtitles such as PGS and VobSub are images, not searchable text. Detect their codec and choose OCR or ASR explicitly. For ASR, choose and record the audio stream and language rather than relying on ffmpeg's default mapping.

Join timestamps back to shots

A dialogue timestamp is not a cuttable range. Join each cue interval against the shot list and return every overlapping shot. A line delivered across a reverse-shot produces two useful clips. Literal and semantic search share this same interval join.

Match cuts come from samples, not shot heads

A match cut lands wherever two silhouettes line up, which is rarely a detected boundary. footage-matchcut "FILM:SECONDS" takes the outgoing frame's source time and ranks every other shot by the best in-point inside it; the hit's start_sec is that in-point, not the shot start. Query with the last CLEAN frame you intend to show: a whip, flash, or dissolve frame has no usable matte, and a close-up whose matte fills the frame is rejected as a figure — it would "match" every other close-up and drag a chain into talking heads. Position dominates the score, CLIP similarity (--clip) keeps the incoming subject the same kind of thing, and --sheet writes out/in frame pairs — judge the pairs, the number only orders them.

Gotchas that cost real cycles

A comprehensive index is not an editorial browse feed. Shot detection should preserve recall, including ordinary dialogue, objects, and connective material; randomly paging that inventory makes a healthy corpus look bad. Keep raw shots available for search and agents, then run footage-curate after visual embeddings to produce scored lanes and contiguous curation.json moments for human discovery. Apply cross-asset diversity and temporal suppression in the consumer, because those policies depend on the set being browsed.

Lower priority and bound concurrency for every batch pass. On this 18-core machine, an unbounded decode or detection pass can starve Resolve's real-time audio thread; playback then stutters and pitch-shifts in a way that resembles corrupt media. Use nice -n 15, cap parallel jobs, and check load before diagnosing playback or codec faults.

A PySceneDetect downscale does not avoid source decode. Its -d reduction happens after decode. A hardware-built 640px proxy made a measured 4K HEVC sample use about 10x less CPU with identical detected boundaries. Keep and reuse the proxy for detection, frame extraction, and scrubbing. Automatically validate duration, normalized time origin, and cadence before trusting its timestamps; spot-check boundaries against the original when first wiring a new container/codec path.

Film chapters can survive a short cut as a long data track. A clip can have correct video and container durations while an inherited chapter track makes browser players report the film length. For standalone silent cuts, explicitly use -map 0:v:0 -map_metadata -1 -map_chapters -1; repair existing copies with those options and -c copy without re-encoding pixels. Validate all streams and chapters with ffprobe -show_streams -show_chapters -show_format, not just v:0 or the container duration.

Parse PySceneDetect CSV by header name. Releases differ on whether the scene list includes preamble rows. Skipping a fixed number of lines can silently drop the first shot; use Start Time (seconds) and End Time (seconds) columns.

macOS xargs -I{} fails on long substituted paths with command line cannot be assembled, too long. Use a helper executable with xargs -P N -n 2, or use a bounded worker pool without shell substitution.

ImageMagick may have no configured default font on macOS. Labelled contact sheets then fail with unable to read font ''. Probe a real font path (for example an installed system font), pass it with montage -font, and include the choice in artifact parameters so a font change invalidates the sheets.

Treat color per consumer. Probe PQ (smpte2084) and HLG (arib-std-b67). An untonemapped proxy can be adequate for recognizing a shot, but it is not grade-reference media and it degrades visual embeddings trained on normal SDR imagery. Resolve imports need the appropriate input color transform. Before designing an ffmpeg tonemap path, probe the active build for zscale or libplacebo support; if absent, choose another build or another color pipeline.

Score the whole shot, not the window someone picked. A hand-chosen in/out is a guess about where the good part is, and it is usually wrong. Resolve the shot bounds first, then slide a window across the whole thing and rank by measurement. One swing shot scored 56.72% on its hand-picked cut and 96.90% on its best window — same shot, same footage, nothing changed but where inside it we looked; the original cut had taken its last two seconds, where the subject is smallest. Report where inside the shot the winner fell: if the best window is repeatedly not where the cuts were, hand-picked timecodes are the problem.

Validate selection criteria against a known-good example before you filter on them. Criteria written from failures describe the failures, not the target. Five conjunctive rules — single sustained action, large subject, steady camera, minimum length, no occlusion — returned "no qualifying candidates" across six recordings and then across 175 curated film moments. The one clip that had actually worked would have failed three of them. Stack enough all-of constraints and any corpus goes to zero, and the zero looks like a property of the footage. Run the known-good sample through the filter first; if it does not survive, the filter is wrong. When screening is cheap, measure broadly and rank rather than excluding up front.

Use the shot midpoint as the default representative frame. Boundary frames often contain dissolve residue. For long or compositionally changing shots, sample multiple interior frames and pool visual-search scores rather than pretending one midpoint describes the whole shot.

Source paths are operational state. Direct Resolve subclips refer to the original. Keep canonical library paths stable or retain enough asset identity and manifest data to relink them deliberately.

An offline source is not a stale index. stage_record_status reports current, stale, or source-offline: intact local artifacts whose source is unreachable are offline, and only a source that is present and changed is stale. Read-only consumers ask stage_record_is_usable, so footage-search serves a relocated index_dir with the media volume unmounted (one warning naming the volume). Writers ask the strict stage_record_is_current, and footage-selects blocks per clip with source unavailable — anything referring back to the original still needs the drive mounted.

Validate a stage record against its own outputs, not a subset. Omit outputs and stage_record_* compares every artifact the record lists. Passing a partial list (proxy writes five) can never match, so the call returns False for a reason that has nothing to do with what you meant to check, and any assertion resting on it is vacuous.

Local tooling

Whisper models on this machine live in ~/Models/whisper.cpp/. A fallback glob must exclude colocated VAD and other-architecture models or whisper-cli fails with a misleading model-load error.

Local Ollama text embeddings are cheap and small. CLIP visual search pulls in Torch and a much larger model stack; build it only when contact sheets and text search do not cover the edit.

Scientific Python stages select ~/.venvs/footage-index automatically when the bare interpreter lacks NumPy, Pillow, Torch, or OpenCLIP. Keep those dependencies in that shared venv; callers and project wrappers should not need to manipulate PATH or remember a special Python command.

Game profiles: deterministic events, then model judgment

With "game_profile" set in footage.json, footage-proxy's single decode pass also emits native-resolution low-fps HUD region crops and a 16 kHz mono analysis WAV. Downstream:

  • footage-hud — template-NCC on the profile's KO emblem (yellow-mask, FFT-based) plus Apple Vision OCR (compiled on demand from scripts/vision-ocr.swift) over kill feed and status strip. Emits typed events: kill, death, feed_kill, multikill, low_hp, ult_ready.
  • footage-audio-events — speech-band hype intervals, spectral-flatness gated so explosions don't read as excitement. Loud ≠ hype.
  • footage-rank — fuses events + motion into padded candidate windows, then a local vision model (ollama) judges filmstrips of the top windows only — a handful of calls per clip, never per-frame. Descriptions become searchable.
  • footage-search --events / footage-selects — ranked windows carry their own in/out (no shot join: a gameplay camera has no cuts) and flow into the same Resolve sub-range manifest as film selects.

Profiles live in profiles/<game>/ — region fractions (resolutions vary per clip; never pixels), template assets with their reference frame width, and fallback player handles. Handles are per-account, not per-person: the detector infers the player's tag per clip from feed rows that co-occur with KO banners, and profile handles are only fallbacks.

Hype is relative to each clip's own median loudness, so pre-trimmed highlight clips (all action, no baseline) legitimately yield none; the signal earns its keep on long uncut sessions.

Not built yet, deliberately: announcer/SFX audio fingerprinting (needs curated cue samples), per-track OBS audio (mic vs game), and live Overwolf event capture (Windows-side).

Gotchas specific to these scripts

This machine's ffmpeg lacks drawtext. Compose synthetic text by overlaying image assets instead (the game selftest overlays the captured killfeed screenshot — which also tests OCR on real feed chrome).

montage fails without an explicit -font even for plain tiling on this ImageMagick build; footage-rank builds filmstrips with ffmpeg hstack for that reason.

FFT correlation layout: F·conj(T) puts valid lags at [0 : f−t+1], not the convolution slice [t−1 : f]. A self-template must score 1.0 — assert that when touching ncc_score.

Reference implementations

~/Movies/spiderman-edit (films; thin bin/film-* wrappers over these scripts) and ~/Movies/rivals-edit (Marvel Rivals gameplay; external-drive media with a local index_dir). Each project README owns its concrete layout, commands, and storage budget.