> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reactor.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Frame Metadata

> Tag an outbound video frame and read the tag back on whatever the model produces from it

## What is frame metadata?

A `sendonly` video frame can carry a small tag of opaque bytes alongside its pixels —
`user_data`. A model that derives its output from that frame (an echo or video-to-video model,
for instance) can mirror the same bytes onto the frame it produces, so your app can tell which
outbound frame a given inbound one came from, without a side channel to track the pairing itself.

<Note>
  Currently Python-only. The capability itself is negotiated automatically — `reactor-webrtc`
  advertises it in the WebRTC offer and the runtime's answer mirrors it — but only the Python SDK
  exposes `user_data` today.
</Note>

Nothing configures this on your side: a model that reads and echoes tags does, and one that
doesn't produces frames with empty `user_data` — check for that rather than assuming every frame
carries a tag.

<Warning>
  Video only. The wire format has no metadata trailer for audio — passing `user_data` to
  [`push_frame()`](/sdk-reference/python/track#push_frame) on an audio track raises `TypeError`
  rather than silently dropping it.
</Warning>

***

## Tagging an outbound frame

Pass `user_data` to [`push_frame()`](/sdk-reference/python/track#push_frame) as raw bytes — encode
whatever you need to recover later:

```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
import json

tag = json.dumps({"seq": seq, "sent_us": int(time.time() * 1e6)}).encode()
track.push_frame(frame, width=640, height=480, user_data=tag)
```

***

## Reading it back

`user_data` arrives as the last argument to both
[`on_frame()`](/sdk-reference/python/track#on_frame) and
[`on_raw_frame()`](/sdk-reference/python/track#on_raw_frame) — the NumPy conversion the first one
does only touches the pixel data, so the tag comes through unchanged either way:

```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
@output.on_raw_frame
def handle(bgra, width, height, frame_id, timestamp_us, user_data):
    if not user_data:
        return  # this model doesn't echo tags, or this particular frame carried none
    tag = json.loads(user_data)
    print("frame", tag["seq"], "returned")
```

<Note>
  `frame_id` and `timestamp_us` arrive on every frame, tagged or not — they aren't something you
  set. `frame_id` is a per-track counter assigned as each frame is decoded, useful for spotting
  gaps or reordering on the track you're reading; `timestamp_us` is that frame's capture time in
  microseconds. Neither is preserved from an outbound frame to whatever inbound frame a model
  derived from it — `user_data` is the one thing that survives the round trip unchanged, which is
  why it's what a correlation tag belongs in.
</Note>

***

## Putting it together

[`examples/frame_metadata_roundtrip.py`](https://github.com/reactor-team/reactor-client-sdks/blob/main/sdks/python/examples/frame_metadata_roundtrip.py)
tags each outbound webcam frame with a sequence number and send time, matches the same tag on
whatever comes back on the model's output track, and reports how many round-tripped, in what
order, and how long each took:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
python -m examples.frame_metadata_roundtrip --frames 60 --verbose
```

If nothing comes back at all, that means the model doesn't echo frame metadata — not that
anything is misconfigured on the client side.