Skip to main content

​
What is frame metadata?

A sendonly video frame can carry a small tag of opaque bytes alongside its pixels — user_data. A model that derives its output from that frame (an echo or video-to-video model, for instance) can mirror the same bytes onto the frame it produces, so your app can tell which outbound frame a given inbound one came from, without a side channel to track the pairing itself.
Currently Python-only. The capability itself is negotiated automatically — reactor-webrtc advertises it in the WebRTC offer and the runtime’s answer mirrors it — but only the Python SDK exposes user_data today.
Nothing configures this on your side: a model that reads and echoes tags does, and one that doesn’t produces frames with empty user_data — check for that rather than assuming every frame carries a tag.
Video only. The wire format has no metadata trailer for audio — passing user_data to push_frame() on an audio track raises TypeError rather than silently dropping it.

​
Tagging an outbound frame

Pass user_data to push_frame() as raw bytes — encode whatever you need to recover later:

​
Reading it back

user_data arrives as the last argument to both on_frame() and on_raw_frame() — the NumPy conversion the first one does only touches the pixel data, so the tag comes through unchanged either way:
frame_id and timestamp_us arrive on every frame, tagged or not — they aren’t something you set. frame_id is a per-track counter assigned as each frame is decoded, useful for spotting gaps or reordering on the track you’re reading; timestamp_us is that frame’s capture time in microseconds. Neither is preserved from an outbound frame to whatever inbound frame a model derived from it — user_data is the one thing that survives the round trip unchanged, which is why it’s what a correlation tag belongs in.

​
Putting it together

examples/frame_metadata_roundtrip.py tags each outbound webcam frame with a sequence number and send time, matches the same tag on whatever comes back on the model’s output track, and reports how many round-tripped, in what order, and how long each took:
If nothing comes back at all, that means the model doesn’t echo frame metadata — not that anything is misconfigured on the client side.