▋
clankvox/docs/go-live.md

Go Live: Native Screen Watch And Self Publish

This document consolidates the Discord Go Live transport inside clankvox.

ClankVox is Clanky's main native package for voice and media transport. Go Live is the first video-heavy platform implementation living there, not the whole scope of the crate.

It covers:

  • inbound native screen watch (stream_watch)
  • outbound native self publish (stream_publish)
  • how Clanky and the selfbot gateway feed stream credentials into the subprocess

Mental Model

Discord Go Live is not “extra fields on the normal voice socket.”

The main voice connection and the Go Live stream connection are separate legs:

  • main voice leg: normal audio send/receive, speaking, voice session identity
  • stream leg: video receive or send, stream-specific SSRCs, stream-server credentials

That is why the Discord transport models Go Live as extra transport roles instead of trying to force everything through the primary voice slot.

Control Plane Vs Media Plane

Clanky and the selfbot gateway own the control plane:

  • raw gateway dispatch handling
  • stream discovery
  • OP18 STREAM_CREATE
  • OP19 STREAM_DELETE
  • OP20 STREAM_WATCH
  • OP22 STREAM_SET_PAUSED
  • deciding which session should attach to which stream

clankvox owns the Discord media plane:

  • stream-server WebSocket connection
  • UDP media send/receive
  • codec advertisement and selection
  • DAVE and transport encryption
  • inbound frame forwarding
  • outbound H264 packetization

Shared Stream Facts

For both watch and publish, Clanky eventually supplies:

  • stream endpoint
  • stream token
  • rtc_server_id
  • main voice session_id
  • self user id
  • DAVE channel id

The DAVE channel derivation for stream connections is:

BigInt(rtc_server_id) - 1

That value is computed in Clanky and passed to clankvox over IPC.

stream_watch Flow

Inbound native watch works like this:

  1. Clanky discovers an active Go Live stream for a target user
  2. Clanky sends OP20 STREAM_WATCH
  3. Discord returns STREAM_CREATE and STREAM_SERVER_UPDATE
  4. Clanky calls stream_watch_connect
  5. clankvox opens the stream-server transport
  6. Discord sends video state and media
  7. clankvox decrypts/depacketizes frames and emits:
    • user_video_state
    • user_video_frame
    • user_video_end
  8. clankvox decodes H264 access units to JPEG in-process (decoded_video_frame); Clanky decodes sampled VP8 keyframes to JPEG and feeds the higher-level screen-watch pipeline

The receiver path supports H264 and VP8 receive.

stream_publish Flow

Outbound self publish works like this:

  1. Clanky decides to publish a self-owned stream
  2. if needed, Clanky sends OP18 STREAM_CREATE
  3. Clanky sends OP22 STREAM_SET_PAUSED { paused: false }
  4. Discord returns self stream discovery and credentials
  5. Clanky calls:
    • stream_publish_connect
    • stream_publish_play for URL-backed publish, or
    • stream_publish_browser_start followed by repeated stream_publish_browser_frame
  6. clankvox opens the sender-side stream transport
  7. clankvox advertises H264 sender capability and announces active video state
  8. clankvox turns the active source into H264 access units:
    • URL-backed publish uses ffmpeg/yt-dlp
    • browser-session publish feeds PNG frames into ffmpeg over stdin
  9. each access unit is DAVE-encrypted, RTP-packetized, and sent over UDP

Pause/resume/stop are split cleanly:

  • pause: Clanky sends OP22 paused true and stream_publish_pause
  • resume: Clanky reuses the existing stream when possible and sends OP22 paused false plus stream_publish_resume
  • stop: Clanky sends OP19 STREAM_DELETE and stream_publish_stop / stream_publish_disconnect

Sender Boundary

The Discord sender path exists, but it is not yet a general-purpose arbitrary video publisher.

Rollout:

  • publish lifecycle is tied to Clanky-owned source orchestration
  • source support is intentionally narrow and centered on:
    • YouTube-backed music/video URLs
    • browser-session PNG frames captured on the Clanky side (../clanky-agent agent/lib/headless-browser.ts + agent/tools/web_capture_frames.ts)
  • sender codec is H264
  • transport is the native Discord stream server path, not the share-link fallback path

Why voice_conn.rs Is So Large

../src/voice_conn.rs owns the protocol-heavy work for the Discord implementation: both normal voice and Go Live.

  • role-aware identify and select-protocol payloads
  • READY parsing and stream SSRC extraction
  • OP12/OP18/OP15 handling
  • speaking and video-state announcements
  • RTP packetization for outbound video
  • inbound video depacketization handoff
  • transport encryption mode handling

That file is effectively the protocol core of the crate.

Key Files

Transport Crypto: rtpsize AAD Rules

Discord's aead_aes256_gcm_rtpsize and aead_xchacha20_poly1305_rtpsize modes authenticate different slices of the packet depending on packet type:

  • RTP media packets: AAD = RTP fixed header (12 bytes) + CSRC list (cc * 4 bytes) + 4-byte extension header prefix (profile + length). The extension body is part of the ciphertext, not the AAD. parse_rtp_header returns a header_size that includes the full extension body — this value is correct for locating the payload start but must NOT be used as the AAD boundary. decrypt() recomputes the AAD from the raw packet bytes.
  • RTCP packets: AAD = the 4-byte RTCP fixed header. decrypt_with_aad() is used directly with RTCP_HEADER_LEN.

Inbound RTCP packets (payload types 72-76 after masking, corresponding to RTCP types 200-204 per RFC 5761 mux) are filtered early in the UDP recv loop before any decrypt attempt. They are silently skipped because we do not process inbound RTCP feedback.

H264 Keyframe and SPS Strategy

Discord's raw UDP protocol path does not honour PLI or FIR RTCP feedback for keyframe requests. PLI/FIR only works through the WebRTC protocol path used by reference implementations like Discord-video-stream. Since clankvox uses protocol: "udp", we cannot request keyframes on demand. PLI/FIR packets are still sent as a best-effort hint in three scenarios: periodic reassertion (every 2s), after decoder reset (50 consecutive errors), and after DAVE ready.

To compensate:

  • Cached SPS+PPS are prepended to every emitted frame after DAVE decrypt. prepend_cached_parameter_sets is a no-op when the frame already contains inline parameter sets. The prepend happens after DAVE decrypt (not during depacketization) so that DAVE trailer byte offsets remain correct.
  • Only IDR slices (NAL type 5) are treated as keyframes for rate-limiting and forwarding purposes.
  • The persistent OpenH264 decoder processes all frames (IDR + P-frames) for reference state accumulation with error concealment enabled. The first decoded frame may have visual artifacts, but subsequent frames improve as P-slice prediction converges. After 50 consecutive decode errors the decoder auto-resets and requests PLI.

DAVE Video Decrypt

DAVE video decrypt works at near 100% on the main voice connection. strip_rtp_padding() in rtp.rs strips RTP padding bytes before depacketization, so AES-GCM tag verification succeeds on padded FU-A mid-fragments.

Go Live streams are a separate, still-open problem. The davey crate reports UnencryptedWhenPassthroughDisabled for every Go Live video frame, but the frame bodies are actually encrypted — feeding them to a decoder produces deblocking_filter_idc out of range and reference count overflow errors characteristic of encrypted data being parsed as H264.

Behavior: video frames that fail DAVE decrypt with UnencryptedWhenPassthroughDisabled are validated by looks_like_valid_h264() and dropped when they appear to be encrypted. Audio passthrough is still allowed.

The first few frames after DAVE session commit sometimes arrive genuinely unencrypted (the sender hasn't started encrypting yet). When those frames contain a real IDR (nal_types=[7, 8, 6, 5]), they decode successfully and bootstrap the screen watch pipeline. After that initial window, DAVE-encrypted frames are dropped and no new frames reach Clanky until the next unencrypted IDR or a session reconnect.

Fixing Go Live DAVE decrypt requires either:

  • updating the davey crate to correctly handle Go Live video DAVE framing
  • or connecting to Go Live streams using the WebRTC protocol path instead of raw UDP (which would also fix PLI/FIR)

ffmpeg Video Decode

H264 decode is handled entirely in-process by clankvox's persistent OpenH264 decoder (video_decoder.rs), running on a dedicated decode thread (video_decode_worker.rs) behind a bounded drop-oldest frame lane so decode and JPEG encode never stall the event loop. The fps gate runs between decode and JPEG encode: every frame feeds the decoder's reference state, but rate-limited frames skip the encode entirely. H264 frames do not use ffmpeg.

VP8 still uses per-frame ffmpeg decode on the Clanky side. The ffmpeg raw demuxer hangs on single-frame input; Clanky works around this by piping through cat | ffmpeg -fflags +genpts -f ivf -i pipe:0 which guarantees clean pipe close and EOF delivery.