FFmpeg beat-edit pipeline

Read this reference when using or changing the edit-* scripts. These are the bundled planner and renderer contracts; they do not constrain edits made in Resolve/Fusion or another compositor. Script paths below are relative to ~/.claude/skills/music-edit/.

Automatic source selection reads footage-index artifacts. Song-analysis stages can be used independently of that selection pipeline.

Setup in a workspace

Two descriptors, two owners. footage.json (footage-index's) names the layout; this engine reads it rather than restating it. edit.json names what is true of this edit — every key optional:

{
  "slug": "project",
  "audio": "02-audio/song.wav",
  "lead_film": "Lead Film",
  "lead_boost": 1.35,
  "character_rx": "Featured Character",
  "subject_scores": "subject_scores.json",
  "subject_score": {
    "seed_lane": "hero",
    "seed_sources": ["Trusted Source"],
    "seeds_per_source": 20
  },
  "lane_roles": {
    "movement": "movement",
    "action": "action",
    "hero": "hero",
    "emotion": "emotion",
    "spectacle": "spectacle",
    "transition": "transition"
  }
}

lead_film is the film the edit leads with (a substring); character_rx earns an emotional-lane boost for shots whose surrounding dialogue names that character. lane_roles maps the planner's editorial roles onto the corpus's workspace-defined curation lanes. subject_score tells the generic prototype builder which trusted sources and lane define the edit's featured subject; the subject itself remains workspace data. Without edit.json the engine still runs — no film leads, no character is special, and renders are named edit_*.

Stages resolve the workspace from EDIT_ROOT. Projects expose thin wrappers, or invoke a stage directly:

EDIT_ROOT=/absolute/workspace ~/.claude/skills/music-edit/scripts/edit-plan-epic \
  @/absolute/workspace/05-project/intent-v01.txt

Stages

Pair the reframe and render shapes: --square with --square (1:1, delivering 1080×1080, or 540×540 with --preview), and edit-reframe --wide with edit-render --wide-fill (16:9, delivering 1920×1080, or 960×540). Both crop the scope frame through the crop-centre path, so both need a reframe file. --wide is the different one: it letterboxes the whole 2.40:1 frame and needs no centres, so choose it when the source composition matters more than filling the frame, and --wide-fill when the reference is full-bleed 16:9. Inspect the rendered crops and author corrections in the reframe JSON. Delivery uses VideoToolbox H.264; per-slot decoding and filtering each use two threads. python3 scripts/edit-render-selftest renders a synthetic marker to check crop shape, framing, and visible zoom across delivery sizes, and checks the delivery colour contract below.

Delivery is Rec.709 SDR, and build_filter states that on the frame right after the HDR LUT — setparams for the tags, sidedata=mode=delete for the HDR10 mastering-display and content-light metadata. Do not move this to the encoder. lut3d rewrites pixels but not colour properties, so an HDR source's PQ/BT.2020 tags outlive the conversion that invalidated them, and at the encoder -color_trc and -color_primaries lose to those inherited properties while -colorspace wins. That split is worse than losing outright: the RGB is matrixed to YUV with BT.2020 coefficients and then labelled bt709, so every player dematrixes it wrongly and the result reads as oversaturated and crushed. The relabel is not a transform — the RGB reaching the encoder is bit-identical with and without it — so it cannot tone-map anything a second time. The static metadata needs deleting separately because the muxer writes it as mp4 mdcv/clli boxes that some players trust ahead of the tags, and deleting a type the source never had is a no-op. When a delivery looks harsh, probe its colour tags AND its side data before touching a grade.

The fx vocabulary

_editlib.py holds the tag set, edit-render owns what each one means, and edit-lint rejects a tag no renderer implements — a silent no-op is the worst way to find out. The automatic planners derive tags from the song. Authored plans may choose them per slot; the renderer supports only the vocabulary below. Amplitudes, decays and frequencies live in the FX constants block in edit-render, with the FX ROSTER docstring above it describing each tag — read those rather than a number quoted here.

Slot kind drives the renderer alongside fx: impact implies a zoom punch and a ramp-in with no tag at all, and burst implies push. A slot's grade follows its section — loud runs and the finale punchier, breathers and intro softer.

What is tunable without touching code: --fx-profile cinematic|social selects a coherent density ladder while preserving cut and source choices. Social uses lower bass/treble gates, directional cut kicks, one-frame glitches, and short trail phrases; dialogue and breathers remain clean. --pulse-strength, --tap-strength, --spark-strength, and --glitch-strength can override its gates individually. These are thresholds on the audio hit, not amplitudes, so the direction is the opposite of what the name suggests — lowering one lets weaker hits qualify and emits more accents. Each planner run prints the resulting counts for inspection; judge density in the rendered sequence. Amplitude lives in the constants block; SLAM scales every impact at once.

--transition-profile classic|capcut|capcut-bold|capcut-dynamic is independent of rhythmic density. Classic keeps the single outgoing whip; CapCut replaces selected impact cuts with paired outgoing/incoming halves. Both halves remain clip-local and meet at the existing hard cut, so transitions never alter timing or need overlapping source handles. capcut is deliberately restrained: 100 ms halves on impact cuts only. --transition-profile capcut-bold uses 217 ms halves, four-direction swipes, and secondary bass-led bridges spaced away from structural impacts so the transition is unmistakable at playback speed without touching dialogue. capcut-dynamic assigns effects by musical role instead of cycling one preset: slam drops use crash zoom/flash, bass stops use the echo-black shutter, bass re-entries use the chromatic split, and ordinary impacts use swipes whose direction follows the nearest varied bass response. --transition-strength sets the strong-hit floor for secondary stop/re-entry cues; the default follows --tap-strength. Incoming slots record transition_cue, transition_style, and any transition_direction, while edit-lint fails an unpaired half. edit-plan-action --matchcut W changes which shot and where it starts: each cut after the first picks, from every film's top-ranked unused shots, the shot and in-point whose person silhouette best continues the outgoing frame (continuity in footage-index: position, size, coarse shape, direction), with rank multiplied by 1 + W × continuity. The plan records matchcut per slot; cuts at or above --matchcut-min alternate trans-blur pairs and trans-thresh-out. Everything else about the passage — beat grid, film rotation fallback, gates — is unchanged. --slowmo SLOT --slowmo-rate RATE changes how much source that slot consumes; check the selected action and its source bounds after retiming.

--bass-bounce flat|weighted controls whether bass motion is a binary preset or an auditable musical mapping. Weighted tags retain the detected peak strength; the renderer squares it before mapping to zoom, so weaker qualifying hits stay small while the top peaks separate decisively. A slam cut carries a 1.55× structural multiplier. Use --pulse-strength for qualifying cut hits and --tap-strength for in-slot hits; weighted mode keeps at most two non-adjacent in-slot peaks per slot, never every detector output.

For reliable sync, pair weighted mode with --bounce-timing peak. The planner rounds each measured peak once in global 60 fps coordinates, assigns it to the slot owning that frame, and records a clip-local offset that resolves back to the same global frame. --bounce-variation seeded --bounce-seed N distributes events across eight camera responses without immediate repetition; the seed is part of plan argv, so “random” remains reproducible. Gate the result with edit-bounce-audit --plan PLAN --video OUTPUT: peak scheduling must stay within half a frame and the rendered fps/frame count must match the plan.

For per-slot choices, edit a new plan JSON generation and validate its tags and paired transition halves with edit-lint. Keep the authoring changes alongside the generated parent: regenerating from planner argv alone loses manual edits. Clearing fx does not neutralize the renderer's kind-based motion or section grade. If those assumptions conflict with the intended sequence, choose a compositor or change the renderer explicitly with a regression check.

The contract that makes iteration work

1. Plans are generations, not drafts. editplan-v01.json stays on disk when v02 lands. A/B is always possible.
2. Intent lives in a file, not in shell history. intent-NN.txt holds one flag per line with comments, fed as @path. Planner flag changes belong here; authored plan changes, crop centres, graphics, and finishing commands also need to be retained. Plans record expanded argv, which reproduces the generated parent, not later edits.
3. Pins address structure, not position. Use scoped ordinals such as --pin "kind/breather#2:FILM:SHOT" or section/action-run-2#5. Slot ids are positions in a derived list; any change to derivation renumbers them and silently re-aims every pin. A bare NAME#N is accepted only when that name is not shared by both a kind and a section. Explicit pins are the editor's override: they bypass rank gates and automatic exclude ranges; unpinned selection still honors every exclusion. A --pin-range start needs the slot's source duration plus --handle (0.355 s by default) before the shot ends, and edit-plan-epic calls sys.exit when one does not — it does not warn and continue. Piping the planner into grep or tail reports the filter's status instead of its own, so the failure reads as success and every later stage silently runs on the previous plan. Check its exit status, then confirm the pins landed in the written plan.
4. Slot ownership is a field. Each slot carries fill: moment (sliced from a contiguous curated scene) or shot (a single pick), set once where the slot is created. Inferring it at each use site is how slots end up filled twice or not at all.
5. Retain render provenance. Keep the plan, source identity, audio window, reframe file, assets, render command, and script revision together. These reproduce the intended edit; encoder output need not be byte-identical.

Things that cost real time to learn

• Frame counts come from timeline boundaries, never durations. Rounding each duration independently accumulates error across 90 cuts and walks the picture off the beat by the end.
• Correct frame counts are not sufficient. The concat demuxer positions each segment by the previous segment's container duration, which a mov header stores in milliseconds. State each duration explicitly in the concat list and re-stamp PTS from the frame index, or cuts drift half a frame by slot 45 and the encoder freezes frames to fill gaps.
• Retime source consumption must fit the selected range. The bundled slow ramps consume less source than they play; fast ramps can consume more. Check the time map and handles so neither crosses an unintended source cut.
• For beat-locked planner slots, use tracked beats. Arithmetic tempo division drifts from the performance. Deliberately off-beat authored cuts still use exact output-frame boundaries.
• Transition presence does not prove readability. Compare the outgoing tail and incoming head at playback speed. Adjust duration, direction, or source choice to the intended move; preset names and effect counts do not establish reference fidelity. Preserve old-plan semantics when changing a shared profile and pair-lint each outgoing/incoming half.
• Bass bounce needs measurement in the plan. A constant pulse makes a medium hit and the drop look identical. bounce@OFF:STRENGTH:DROP_MULT preserves the detector peak and structural multiplier, while renderer math owns the final amplitude and decay.
• The bass-hit window is a selection gate, not a firing time. Accepting a peak within the window of a cut and then firing the camera at offset zero misplaces the accent by up to the whole window. Preserve the exact detected peak, snap it once to the nearest GLOBAL output frame, and assign that frame to its owning slot — even when the peak belongs to the outgoing picture just before a cut.
• Audit timing in global frame coordinates. Adding a clip-local offset to the slot's float t0_sec double-counts rounding and can falsely report almost a full frame of error. The renderer starts the slot at round(t0*FPS); audit round(t0*FPS) + round(offset*FPS). At 60 fps the correct hard limit is 8.333 ms. Also verify mux alignment separately: correct visual math cannot compensate for an audio encoder or filter delay.
• Render-to-detector sync is not detector-to-ear sync. edit-song-bass's causal 20–160 Hz Butterworth filter has frequency-dependent group delay (about 4–16 ms across the band). Replacing it with zero-phase filtering is not a safe timing patch: on the reference edit it changed the detected hit set materially. Preserve a generation's detector identity; investigate detector latency as its own versioned signal change rather than subtracting an unmeasured constant or weakening the half-frame render audit.
• A temporary output override changes plan provenance. Plans retain fully expanded argv, including --out; regenerating with --out /tmp/x.json therefore produces an expected argv-only diff against an in-tree plan. For a byte-identity regression, run the canonical intent invocation and compare its before/after hash. Use a field-level diff with a temporary output when localizing semantic regressions.
• zoompan uses source coordinates and divides both input dimensions by z. Feed it the delivery aspect ratio, use z ≥ 1, and centre with x=cx-iw/(2*zoom), y=cy-ih/(2*zoom). Output pixels are not source pixels; changing preview resolution must not change framing. crop evaluates its width and height at configuration time, so animated zoom belongs in zoompan.
• Contact-sheet tiles must be byte-identical in size or ffmpeg's tile= silently emits black, which reads as a broken render rather than a bad call.
• HDR is a color-management problem, not a grade problem. Untone-mapped PQ next to SDR reads washed-out and grey. Fit the LUT against native-SDR sources; an eyeballed curve rendered one film entirely black.
• Run edit-lint before rendering a plan with edit-render. It exits nonzero on the failures that render wrong, and edit-lint --selftest checks the checks against a deliberately broken plan with no media at all.
• Identify a featured subject by example, not colour. Colour heuristics rank costumes, signage, and grade differences instead of identity. Seed edit-signal-subject from trusted sources in a useful curation lane, reuse the existing CLIP vectors, and inspect its --sheet before trusting the gate.
• Write the story arc before asking the ranker to choose. For a narrative cut, establish the intended sequence and any spoken lines first. Bind each musical phrase to its source scene with --story-range "PHRASE:FILM:T0-T1"; constrained phrases refill forward in source order so a scene never jumps backward when one curated moment is too short.
• Shot pins do not locate dialogue inside a long shot. Use --pin-range "SLOT:FILM:START" for the exact spoken beat. If ASR grouped the words into a cue longer than the musical slot, --force-voice SLOT still brings the pinned source audio under ducked music while retaining the overlapping cue as provenance.
• Picture and dialogue may need different source windows. --voice-range "SLOT:FILM:T0-T1" leaves the visual pin alone and records the exact spoken interval. A line longer than its musical slot is tempo-fitted gently; a shorter line ends naturally and the remainder is music-only instead of leaking the film mix.
• Separate dialogue once, then bind it to the plan. Install the Apple- Silicon path with uv venv ~/.venvs/edit-plan --python 3.12 and uv pip install --python ~/.venvs/edit-plan/bin/python 'mlx-audio-separator[convert]', then run edit-voice-isolate --plan PLAN. It extracts one second of context around every voice range, runs the BS-Roformer once across the batch, and writes a manifest containing the plan and model hashes. Render with edit-render --voice-stems MANIFEST; a changed plan or missing stem fails rather than falling back to the leaking film mix.
• Verify voice in the rendered mix, not just the plan. Compare the output audio against the music bed: each intended voice window must contain a non-music residual and lower music gain. A voice: true field only proves the graph was requested, not that the chosen source slice contains audible words.