> ## Agent Instructions > Reactor hosts multiple models, each with its own connect slug (modelName) and command/event schema. The catalog of every model — slug, typed SDK package, and links to its schema — is at /model-api-reference/overview. Some models expose one slug per experience (e.g. HappyOyster); always take the slug from the model's own pages, never guess it. > Fastest path to a working app: `npx create-reactor-app my-app --model=` scaffolds a complete app with secure auth wired up. Typed TypeScript SDKs are published as @reactor-models/; Python uses the base reactor-sdk package. > Auth: exchange an API key (rk_...) for a JWT via POST https://api.reactor.inc/tokens from your server. Never put the API key in client-side code. > Append .md to any docs URL for clean Markdown. Search these docs via the MCP server at https://docs.reactor.inc/mcp. # Authentication Source: https://docs.reactor.inc/authentication How authentication works in Reactor Reactor uses API keys to authenticate requests. JavaScript apps exchange the key for a short-lived token; Python apps pass the key directly and the SDK does the exchange for them. ## Get your API key Sign up at [reactor.inc](https://reactor.inc). Open the **[Dashboard](https://reactor.inc/dashboard)**, click your user icon, then navigate to **API Keys** to create a new key. Your key starts with `rk_`. Store it securely and never commit it to source control. ## JavaScript ### How it works Your server exchanges the API key for a short-lived token, which the browser uses to connect. Always mint **session-scoped** tokens: pass `authorization_details` naming the models the token may start sessions for. A scoped token can only create and operate its own sessions — it cannot touch other sessions, other models, or any account API. If it leaks, the blast radius is a handful of sessions on the models you listed, for at most the token's lifetime (1 hour by default). Server exchanges API key for a token, passes token to browser, browser connects to Reactor ### Generate a session-scoped token Exchange your API key for a short-lived token by making a `POST` request to the `/tokens` endpoint with your API key in the `Reactor-API-Key` header. **Always scope the token** with `authorization_details`, naming the models it may start sessions for: ```typescript Session-scoped (1-hour expiry) theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} const result = await fetch("https://api.reactor.inc/tokens", { method: "POST", headers: { "Reactor-API-Key": process.env.REACTOR_API_KEY, "Content-Type": "application/json", }, body: JSON.stringify({ authorization_details: [ { type: "session", resources: { models: { match: ["your-model-name"] } }, constraints: { max_sessions: 5 }, }, ], }), }); const { jwt, expires_at } = await result.json(); ``` ```typescript Custom expiry theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} const result = await fetch("https://api.reactor.inc/tokens", { method: "POST", headers: { "Reactor-API-Key": process.env.REACTOR_API_KEY, "Content-Type": "application/json", }, body: JSON.stringify({ expires_after: Number(process.env.TOKEN_LIFETIME_SECONDS), authorization_details: [ { type: "session", resources: { models: { match: ["your-model-name"] } }, constraints: { max_sessions: 5 }, }, ], }), }); const { jwt, expires_at } = await result.json(); ``` The `authorization_details` entry has three parts: * **`type: "session"`** (required) — the only supported type today. The token can create sessions and do everything those sessions need (transport negotiation, uploads, clips, session logs), and nothing else. * **`resources.models.match`** (required) — a non-empty list of models the token may start sessions for. Every listed model must be visible to your API key. There is no wildcard; list each model explicitly. * **`constraints.max_sessions`** (optional) — how many sessions the token may create in total, from 1 to 500. Defaults to 5. This counts sessions ever created by the token, not concurrent ones: closing a session does not restore capacity. Session-scoped tokens live for **1 hour** by default. Pass `expires_after` (in seconds) to shorten or extend the lifetime, up to the server ceiling of 6 hours; values at or above the ceiling are silently clamped. The server still returns 200, so always check `expires_at` (a Unix epoch timestamp) on the response to confirm the actual expiry. Omitting `authorization_details` mints an **unscoped** token that can call every API your key's roles allow — sessions on any model, account data, key management. Never hand an unscoped token to a browser. Reserve unscoped tokens for trusted server-to-server calls (for example the [Platform API](/resources/platform-api-overview)), and don't store your API key in client-side code either: use your server as a proxy to mint scoped tokens, as shown below. ### Server-side proxy Set up an API route on your server that mints a session-scoped token and returns it to your frontend: ```typescript Next.js theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { NextResponse } from "next/server"; export async function POST() { const result = await fetch("https://api.reactor.inc/tokens", { method: "POST", headers: { "Reactor-API-Key": process.env.REACTOR_API_KEY!, "Content-Type": "application/json", }, body: JSON.stringify({ authorization_details: [ { type: "session", resources: { models: { match: ["your-model-name"] } }, }, ], }), }); const { jwt } = await result.json(); return NextResponse.json({ jwt }); } ``` ```typescript Express theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} app.post("/api/token", async (_req, res) => { const result = await fetch("https://api.reactor.inc/tokens", { method: "POST", headers: { "Reactor-API-Key": process.env.REACTOR_API_KEY!, "Content-Type": "application/json", }, body: JSON.stringify({ authorization_details: [ { type: "session", resources: { models: { match: ["your-model-name"] } }, }, ], }), }); const { jwt } = await result.json(); res.json({ jwt }); }); ``` Then fetch the token from your frontend and pass it to the SDK: ```tsx app/page.tsx theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} "use client"; import { use } from "react"; import { ReactorProvider, ReactorView } from "@reactor-team/js-sdk"; async function getToken() { const result = await fetch("/api/token", { method: "POST" }); const { jwt } = await result.json(); return jwt; } const tokenPromise = getToken(); export default function App() { const token = use(tokenPromise); return ( ); } ``` ### Tokens for a queue or admission-control server If you run an admission-control layer in front of a fixed pool of sessions — a waiting room gating limited GPU capacity, for example — mint a separate token per admitted slot instead of one shared token for the whole pool. Scope each token to `max_sessions: 1` so it can create exactly the one session it is being admitted into: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} curl -X POST https://api.reactor.inc/tokens \ -H "Reactor-API-Key: $REACTOR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "expires_after": 900, "authorization_details": [ { "type": "session", "resources": { "models": { "match": ["reactor/helios"] } }, "constraints": { "max_sessions": 1 } } ] }' ``` ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} async function mintSlotToken(model: string, slotLifetimeSeconds: number) { const result = await fetch("https://api.reactor.inc/tokens", { method: "POST", headers: { "Reactor-API-Key": process.env.REACTOR_API_KEY!, "Content-Type": "application/json", }, body: JSON.stringify({ expires_after: slotLifetimeSeconds, authorization_details: [ { type: "session", resources: { models: { match: [model] } }, constraints: { max_sessions: 1 }, }, ], }), }); const { jwt, expires_at } = await result.json(); return { jwt, expires_at }; } ``` Size `slotLifetimeSeconds` to the slot's whole lifetime: the time an admitted user might take to connect, plus the full session duration. Omitting it defaults to 1 hour, and 6 hours is the [ceiling](#generate-a-session-scoped-token); a session-scoped token can't be topped up mid-session, so undersizing this strands the session it created. Hand the resulting `jwt` to the admitted client the same way as the [server-side proxy](#server-side-proxy) above; its `connect()` call creates the session and binds it to this token's grant. If your API key is compromised, rotate it immediately from the **[Dashboard](https://reactor.inc/dashboard)**. Rotating does not affect active sessions. Need help? Email us at [support@reactor.inc](mailto:support@reactor.inc). **Adopting an existing session.** If your backend creates a session and hands the `sessionId` to a client, that client must connect with a token that can access the session. Session-scoped tokens are bound to the sessions they create, so the simplest approach is to hand the client the same scoped JWT your backend created the session with. See [Sessions](/concepts/sessions#multiple-connections-per-session). ## Python ### How it works **Python** runs server-side, so you pass your API key straight to the `Reactor` constructor. On `connect()`, the SDK exchanges it for a short-lived, **session-scoped** token — scoped to the model you built the `Reactor` with — and authenticates every request with that token, never the raw key. If that token leaks, the blast radius is a handful of sessions on that one model, not your whole account. Scoping is automatic; there is nothing extra to configure. Your app sends the API key directly to Reactor Pass your API key directly to the `Reactor` constructor: ```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import os from reactor_sdk import Reactor reactor = Reactor( model_name="your-model-name", api_key=os.environ["REACTOR_API_KEY"], ) # Mints a token scoped to "your-model-name", then connects with it. await reactor.connect() ``` Never hardcode `rk_...` values in your code or commit them to source control. Use environment variables. ### Minting a token yourself Need a token for something other than a `Reactor` connection — a JWT broker that hands tokens to browser clients, for instance? Call `fetch_jwt` and scope it with `models`: ```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import asyncio import os from reactor_sdk import DEFAULT_API_URL, fetch_jwt # fetch_jwt is synchronous — run it in a thread from async code so it doesn't # block the event loop. jwt = await asyncio.to_thread( fetch_jwt, api_key=os.environ["REACTOR_API_KEY"], api_url=DEFAULT_API_URL, models=["your-model-name"], # session-scoped to these models ) ``` Pass `max_sessions` to cap how many sessions the token may create (1–500, default 5), and `expires_after` (in seconds) to shorten its lifetime below the 6-hour ceiling. Omitting `models` mints an **unscoped** token that can call every API your key's roles allow — reserve that for trusted server-to-server calls such as the [Platform API](/resources/platform-api-overview), and never hand one to a client. **Adopting a session someone else created.** `connect(session_id=...)` attaches to a session that already exists rather than creating one. Because a session-scoped token can only act on the sessions it created itself, the SDK mints an unscoped token for this path. See [Sessions](/concepts/sessions#multiple-connections-per-session). Need help? Email us at [support@reactor.inc](mailto:support@reactor.inc). *** # Changelog Source: https://docs.reactor.inc/changelog/overview What's new in the Reactor SDKs and models. Rolling red sand dunes at dusk New releases, breaking changes, and notable improvements across the Reactor platform: JavaScript SDK, Python SDK, models, and API. ### Breaking * **`send_command()` now waits for its reply.** Previously fire-and-forget — the call resolved as soon as the command was queued, and any reply from the model arrived later as a separate `message` event. It now returns the model's correlated reply directly: `{"type": ..., "data": ...}`, or `None` if the handler acknowledged the command without returning a message (an auto-generated `set_` setter, for instance). To fire a command without waiting on the reply, schedule the call instead of awaiting it directly — `asyncio.create_task(reactor.send_command(...))` — keeping a reference to the task so it isn't garbage-collected before it completes. * **Errors are now typed exceptions, and [`ReactorError`](/sdk-reference/python/types#reactorerror) is the one class for both.** Every failure used to arrive as one bare exception carrying only a sentence, and separately, `on_error` handed you a same-shaped-but-different dataclass. `ReactorError` is now both: the base of 16 concrete subclasses (`UnauthorizedError`, `ConflictError`, `RateLimitedError`, and more — see [Error codes](/sdk-reference/python/types#error-codes)) that `on_error` and a raised call hand you identically, each with a stable `code`, `recoverable`, `status`, `operation`, `retry_after_ms`, and (event-only) `timestamp_ms`. The old `component` field (`api`/`gpu`) is gone — it split codes by platform tier, which wasn't actionable and could report the same failure under two different codes. `except ReactorError` still catches everything; code relying on `error.component` needs updating. * **Audio devices are synthetic-only.** The constructor's `adm_mode` argument is gone; no `Reactor` instance ever opens a real microphone or speaker on its own. A `sendonly` audio track only ever carries PCM you push yourself with [`Track.push_frame()`](/sdk-reference/python/track#push_frame), and a model's audio must be played back explicitly via [`Track.on_frame()`](/sdk-reference/python/track#on_frame) / [`on_raw_frame()`](/sdk-reference/python/track#on_raw_frame) — or with `Speaker`/`Microphone`, below. * **`reactor.on_frame` (client-wide) is removed.** It only ever worked for video, and a single handler fed every `recvonly` video track couldn't tell them apart. Register on the [`Track`](/sdk-reference/python/track#on_frame) instead, found by name (`reactor.track(name)`) or by filtering [`reactor.tracks`](/sdk-reference/python/types#tracklist) (`reactor.tracks.with_direction("recvonly").with_kind("video").one()`). * **The client-wide `on("frame", ...)` / `on("audio", ...)` events are removed too**, for the same reason. `on()` raises `ValueError` at registration for either name, naming the [`Track`](/sdk-reference/python/track) method to use instead — [`on_frame()`](/sdk-reference/python/track#on_frame) for decoded frames, [`on_raw_frame()`](/sdk-reference/python/track#on_raw_frame) for the same bytes with no NumPy conversion. * **[`publish_track()`](/sdk-reference/python/reactor#publish_track) / [`unpublish_track()`](/sdk-reference/python/reactor#unpublish_track) are the only name-based track methods still public on `Reactor`.** `push_video_frame()`, `push_audio_frame()`, `pause_track()`, and `resume_track()` are now internal — reach the same operations through the [`Track`](/sdk-reference/python/track) object instead: `track.push_frame()`, `track.pause()`, `track.resume()`. `publish_track()` already hands back that same `Track`. * **[`Track.push_frame()`](/sdk-reference/python/track#push_frame) now raises [`InvalidStateError`](/sdk-reference/python/types#reactorerror) on an unpublished `sendonly` track**, rather than silently dropping the frame. Call [`publish()`](/sdk-reference/python/track#publish) first — [`Track.published`](/sdk-reference/python/track#published) says whether that's already happened. * **[`reconnect()`](/sdk-reference/python/reactor#reconnect) resumes `recvonly` tracks automatically, but does not restore a `sendonly` track's publish.** Call [`publish()`](/sdk-reference/python/track#publish) again after reconnecting for anything you were sending before it. * **[`@reactor.on_track`](/sdk-reference/python/decorators#reactor-on-track) hands over the [`Track`](/sdk-reference/python/track) itself, not a bare `(name, mid)`.** The `Track` is already resolved — `reactor.track(name)` done for you — so a handler that wants to push frames or register `on_frame` no longer has to look it up first; `track.mid` still carries the WebRTC media stream ID. * **[`unpublish_track()`](/sdk-reference/python/reactor#unpublish_track) / [`Track.unpublish()`](/sdk-reference/python/track#unpublish) return `None`, not an `int`.** A failure is now logged (`reactor_sdk` at `WARNING`) rather than left for the caller to notice by checking a return value nobody checked — unpublish is commonly the last call in a `finally` block, so it still does not raise. * **[`Reactor()`](/sdk-reference/python/reactor#constructor)'s positional arguments are `model_name`, `api_key` — `api_url` moved to keyword-only.** Matches the old `py-sdk`'s own constructor order exactly, so `Reactor(model, key)` ported from there needs no rewrite to keyword arguments at all. Previously `api_url`/`model_name` were positional (in either order, sniffed by which one looked like a URL); any call relying on that specifically needs updating, but a plain `Reactor("your-model-name")` is unaffected. * **[`upload_file()`](/sdk-reference/python/reactor#upload_file) now requires `ready`.** Previously allowed as soon as the coordinator created a session, before the WebRTC handshake finished — raises [`InvalidStateError`](/sdk-reference/python/types#reactorerror) now if called earlier, the same guard `request_clip()` and [`Track.pause()`](/sdk-reference/python/track#pause) already used. See [File Uploads](/concepts/file-uploads). ### New * **`reactor.tracks` is now a `TrackList`.** Still a plain `list[Track]` for iterating and indexing, with `with_kind()` / `with_direction()` filters that chain, and `.one()` for the common case of "the track I mean, and an error if that's ambiguous." See [`TrackList`](/sdk-reference/python/types#tracklist). * **The `Track` object.** [`reactor.track(name)`](/sdk-reference/python/reactor#track) and [`reactor.tracks`](/sdk-reference/python/reactor#tracks) return a [`Track`](/sdk-reference/python/track) scoped to one named slot, with `publish()`, `push_frame()`, `on_frame()`/`on_raw_frame()`, and `pause()`/`resume()` that raise on a direction mismatch instead of a silent no-op. * **[`Track.published`](/sdk-reference/python/track#published).** Whether a `sendonly` track is currently activated — readable state for something the session itself doesn't track, cleared automatically whenever the connection leaves `ready`. * **[`download_clip()`](/sdk-reference/python/types#download_clip).** Fetches every segment a `Clip`'s `playlist_url` names. Given a `path`, streams straight to it (returns `None`, never holds more than one segment in memory — the one to use for `request_recording()`); without one, returns the assembled bytes instead. Takes an optional `on_progress` callback either way. Closes a real gap left by the rewrite off the old `py-sdk`, which had this as `download_clip_as_file()`. * **[`reactor.download_clip(seconds)`](/sdk-reference/python/reactor#download_clip) / [`reactor.download_recording()`](/sdk-reference/python/reactor#download_recording).** `request_clip()` / `request_recording()` plus the download above, in one call, for when the file is all you want. `request_clip()` / `request_recording()` still return just the `Clip` for anyone who also wants its `session_id`, markers, or `predicted_ready_at_ms`. * **`connect(connection_id=...)`.** Adopts a WebRTC connection slot a backend already registered for a session, the connection-level analogue of `session_id` adoption — see [Multiple connections per session](/concepts/sessions#multiple-connections-per-session). Also closes a gap left by the rewrite: the old `py-sdk` had this, and the JavaScript SDK still does. * **[`reconnect()`](/sdk-reference/python/reactor#reconnect) now works from `ready`, not only after a drop.** It tears the live connection down itself first — without ending the session, unlike `disconnect()` — so there was never a need to call `disconnect()` immediately before it; that combination is also no longer required. `disconnect()`'s own behavior is unchanged — it has always ended the session server-side, with no recoverable option — a few places in these docs previously described it backwards, now corrected. See [Sessions](/concepts/sessions#disconnecting). * **`Speaker` / `Microphone` (`reactor_sdk.audio_devices`).** Play a `recvonly` audio track through real speakers, or capture a real microphone into a `sendonly` track — context managers wrapping a `sounddevice` stream around a `Track`: `with Speaker(output_track), Microphone(mic_track): ...`. Not re-exported from `reactor_sdk` — the only part of this SDK with a runtime dependency, so `pip install "reactor-sdk[audio]"` and import from `reactor_sdk.audio_devices` directly. `Speaker` promotes what `examples/pygame_app` had hand-rolled; `Microphone` is new. * **`fetch_jwt(api_key, api_url=...)`'s `api_url` is optional now**, defaulting to the same production coordinator `Reactor()` itself defaults to. Pass it explicitly only when minting a token against a different one. ### Breaking * **Publishing an input track is now explicit.** Output tracks still stream automatically on connect, but `sendonly` (input) tracks send nothing until your app calls [`publishTrack()`](/sdk-reference/reactor-class#publishtrack) (or renders [``](/sdk-reference/react-components#webcamstream)). Apps that relied on input media flowing at connect must now publish explicitly once the connection is `ready`. See [Tracks](/concepts/tracks). ### New * **Output track pause/resume.** New [`pauseTrack()`](/sdk-reference/reactor-class#pausetrack) and [`resumeTrack()`](/sdk-reference/reactor-class#resumetrack) methods, plus an `autoResumeTracks` connect option (default `true`), control when each output track streams. * **Multi-client sessions.** A single session can now back several connections at once. `connect()` accepts `sessionId` and `connectionId` options to attach to a session or connection your backend created. ### Changed * **Per-second and per-hour pricing in dollars.** Model pricing is now displayed as US-dollar rates on the [Pricing & Billing](/resources/billing) page. ### New * **Pricing & billing docs.** Published [Pricing & Billing](/resources/billing), covering session pricing, idle vs. active billing, and recoverable-state behavior. * **Beta status callouts.** Added a banner across docs and the [Overview](/overview) noting that Reactor is in beta. APIs and pricing may evolve. ### New * **Token TTL with `expires_after`.** Auth tokens now carry an `expires_after` field so clients can refresh proactively instead of waiting for a 401. See [Authentication](/authentication). ### New * **File uploads in the Python SDK.** `reactor-sdk` now supports `upload_file()` returning a `FileRef` that can be passed into `send_command()`. * **File uploads in the JavaScript SDK.** `uploadFile()` and `FileRef` are available in `@reactor-team/js-sdk` for passing files (images, audio, blobs) as command arguments. * **New concept page:** [File Uploads](/concepts/file-uploads) documents the upload-then-reference pattern in both SDKs. ### Breaking * **Token endpoint is now POST.** The auth-token endpoint moved from `GET` to `POST` to allow request-body parameters. Update any direct API callers; the SDKs handle this automatically once upgraded. ### Breaking * **New SDK protocol.** Wire-level protocol overhaul shipped in `@reactor-team/js-sdk` 2.7 and `reactor-sdk` 0.4. Older SDK versions will no longer connect. Upgrade to the latest release. * **Removed `fetchInsecureToken()` and JWT helpers.** The development-only insecure token helpers were removed; all auth now flows through the standard token endpoint. ### Improved * **Maple theme refresh.** Docs site updated to Mintlify's Maple theme with Lucide icons, new codeblock styling, and refreshed colors. * **Navigation split** into separate "SDK" and "API Reference" tabs. ### New * **Image conditioning for Helios.** `set_image()` accepts a reference image to steer generation, with cut or blend transitions for swapping mid-stream. See [Helios](/model-api-reference/helios/overview). ### Improved * Image size guidance clarified: 64KB limit with JPEG quality 0.5 recommended for set-image calls. ### Breaking * **API renames in `@reactor-team/js-sdk` 2.6.0.** Several public methods and types were renamed for consistency with the Python SDK. Update import paths and call sites. ### New * **Helios launch.** Reactor's first interactive real-time video generation model, built on a 14B-parameter Diffusion Transformer. Produces a continuous infinite stream you can steer with prompts and reference images. * **Helios prompt guide.** A dedicated [Prompt Guide](/model-api-reference/helios/prompt-guide) covering scene composition, transitions, and steering techniques. ### Improved * **Renamed "Realtime AI" → "Realtime Video AI"** across the docs to better reflect what Reactor does. No API or SDK changes. ### New * **Python SDK launched.** `reactor-sdk` is now available on PyPI, async-first, with parity for sessions, commands, and message handling. See [Using the SDK](/sdk-reference/using-the-sdk). ### Breaking * **`sendCommand()` replaces `sendMessage()`.** The command API was renamed to clarify the distinction between commands (client → model) and messages (model → client). Update all `sendMessage` call sites. See [Commands and messages](/concepts/commands-and-messages). * **Removed message queueing.** Commands now have explicit `waiting` state instead of implicit queueing. Applications can decide whether to drop, replace, or buffer in-flight commands. ### Breaking * **JWT auth replaces `insecureApikey`.** All client connections now require a short-lived JWT minted server-side from your API key. The `insecureApikey` development flow is removed. * **New API Reference section.** Published a complete [API Reference](/sdk-reference/using-the-sdk) covering the `Reactor` class, React components, hooks, types, and events. ### New * **Reactor docs site launched.** Initial release with [Overview](/overview) and [Quickstart](/quickstart). Ported from the legacy readme.io site. # Commands & Messages Source: https://docs.reactor.inc/concepts/commands-and-messages How two-way communication works between your app and a model Alongside the video stream, your app and the model exchange data over a real-time channel. You control the model and receive feedback while it generates. * **Commands** are instructions your app sends to the model. They control what the model generates in real time. * **Messages** are updates the model sends back to your app. They report state, signal events, or deliver data. For example, a video model might accept a `set_prompt` command to change what it generates, and respond with periodic `state` messages reporting the current frame number and active prompt. Commands flow from your app to the model, messages flow back *** ## Commands A command is a named action with a payload. ```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} await reactor.sendCommand("set_prompt", { prompt: "a forest at dawn" }); ``` ```tsx React theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { useReactor } from "@reactor-team/js-sdk"; function PromptButton() { const sendCommand = useReactor((s) => s.sendCommand); return ( ); } ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} await reactor.send_command("set_prompt", {"prompt": "a forest at dawn"}) ``` Every model defines its own commands with typed parameters. A prompt-driven model might expose `set_prompt`; a model with camera controls might expose `move` or `rotate`. Command names, parameter names, and types are all model-specific. **Prefer typed methods over string commands?** Models with a published typed SDK (like [Helios](/model-api-reference/helios/overview) or [LingBot](/model-api-reference/lingbot/overview)) expose every command as a typed method with autocomplete, parameter hints, and compile-time validation. No hand-written JSON, no string-typed command names. ```typescript Helios typed SDK theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { HeliosModel } from "@reactor-models/helios"; const helios = new HeliosModel(); await helios.setPrompt({ prompt: "a forest at dawn" }); ``` See [Two layers: base SDK and typed SDKs](/sdk-reference/using-the-sdk#two-layers-base-sdk-and-typed-sdks) for the full picture. Commands can only be sent when the connection is in the `ready` state. *** ## Messages Reactor models can send structured JSON messages to your app at any time during a session. Each model defines its own message types. ```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} reactor.on("message", (msg) => { console.log(msg); }); ``` ```tsx React theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { useReactorMessage } from "@reactor-team/js-sdk"; function MessageLogger() { useReactorMessage((msg) => { console.log(msg); }); return null; } ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} @reactor.on_message def on_message(msg): print(msg) ``` For the full list of commands and messages per model, see the [Model API Reference](/model-api-reference/overview). # File Uploads Source: https://docs.reactor.inc/concepts/file-uploads How to upload files and pass them to model commands Some models accept file inputs alongside regular parameters. The SDK handles file uploads in two steps: upload the file to get a reference, then pass that reference into a command. 1. **Upload** the file with `uploadFile()` to get a `FileRef`. 2. **Pass** the `FileRef` into `sendCommand()` as a regular parameter. *** ## Uploading a file Call `uploadFile()` with a `File` or `Blob`. It returns a [`FileRef`](/sdk-reference/types#fileref) you can pass into `sendCommand()`, alongside regular arguments. The example below uses Helios's `set_image` command. The commands and file parameters a model accepts vary by model, so check your [model's reference](/model-api-reference/overview) for what it expects. File upload flow: your app calls uploadFile() to get a presigned PUT URL and a FileRef from Reactor, PUTs the file bytes directly to object storage, then sends the FileRef in a command to the model, which fetches the bytes from storage itself. ```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} const input = document.querySelector("input[type=file]") as HTMLInputElement; const file = input.files![0]; const ref = await reactor.uploadFile(file); await reactor.sendCommand("set_image", { image: ref }); ``` ```tsx React theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { useReactor } from "@reactor-team/js-sdk"; function ImageUpload() { const { uploadFile, sendCommand, status } = useReactor((s) => s); const handleFile = async (e: React.ChangeEvent) => { const file = e.target.files?.[0]; if (!file) return; const ref = await uploadFile(file); await sendCommand("set_image", { image: ref }); }; return ( ); } ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} ref = await reactor.upload_file("scene.jpg") await reactor.send_command("set_image", {"image": ref}) ``` `uploadFile()` / `upload_file()` can only be called when the connection status is `"ready"` — Python raises [`InvalidStateError`](/sdk-reference/python/types#reactorerror) otherwise, the same guard `request_clip()` uses. Upload URLs expire after 15 minutes. For more information, see [Rate Limits](/resources/rate-limits#file-uploads). *** ## The FileRef type `uploadFile()` returns a `FileRef` with metadata about the uploaded file. You never construct one directly. ```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} class FileRef { readonly uploadId: string; // Upload identifier readonly name: string; // Filename readonly mimeType: string; // MIME type (e.g. "image/jpeg") readonly size: number; // File size in bytes } ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} @dataclass(frozen=True) class FileRef: upload_id: str # Upload identifier name: str # Filename mime_type: str # MIME type (e.g. "image/jpeg") size: int # File size in bytes ``` *** ## Complete example For a full image-to-video walkthrough that uploads a reference image and conditions the first frame on it, see Helios's [image-to-video example](/model-api-reference/helios/schema#complete-example-image-to-video). # Frame Metadata Source: https://docs.reactor.inc/concepts/frame-metadata Tag an outbound video frame and read the tag back on whatever the model produces from it ## What is frame metadata? A `sendonly` video frame can carry a small tag of opaque bytes alongside its pixels — `user_data`. A model that derives its output from that frame (an echo or video-to-video model, for instance) can mirror the same bytes onto the frame it produces, so your app can tell which outbound frame a given inbound one came from, without a side channel to track the pairing itself. Currently Python-only. The capability itself is negotiated automatically — `reactor-webrtc` advertises it in the WebRTC offer and the runtime's answer mirrors it — but only the Python SDK exposes `user_data` today. Nothing configures this on your side: a model that reads and echoes tags does, and one that doesn't produces frames with empty `user_data` — check for that rather than assuming every frame carries a tag. Video only. The wire format has no metadata trailer for audio — passing `user_data` to [`push_frame()`](/sdk-reference/python/track#push_frame) on an audio track raises `TypeError` rather than silently dropping it. *** ## Tagging an outbound frame Pass `user_data` to [`push_frame()`](/sdk-reference/python/track#push_frame) as raw bytes — encode whatever you need to recover later: ```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import json tag = json.dumps({"seq": seq, "sent_us": int(time.time() * 1e6)}).encode() track.push_frame(frame, width=640, height=480, user_data=tag) ``` *** ## Reading it back `user_data` arrives as the last argument to both [`on_frame()`](/sdk-reference/python/track#on_frame) and [`on_raw_frame()`](/sdk-reference/python/track#on_raw_frame) — the NumPy conversion the first one does only touches the pixel data, so the tag comes through unchanged either way: ```python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} @output.on_raw_frame def handle(bgra, width, height, frame_id, timestamp_us, user_data): if not user_data: return # this model doesn't echo tags, or this particular frame carried none tag = json.loads(user_data) print("frame", tag["seq"], "returned") ``` `frame_id` and `timestamp_us` arrive on every frame, tagged or not — they aren't something you set. `frame_id` is a per-track counter assigned as each frame is decoded, useful for spotting gaps or reordering on the track you're reading; `timestamp_us` is that frame's capture time in microseconds. Neither is preserved from an outbound frame to whatever inbound frame a model derived from it — `user_data` is the one thing that survives the round trip unchanged, which is why it's what a correlation tag belongs in. *** ## Putting it together [`examples/frame_metadata_roundtrip.py`](https://github.com/reactor-team/reactor-client-sdks/blob/main/sdks/python/examples/frame_metadata_roundtrip.py) tags each outbound webcam frame with a sequence number and send time, matches the same tag on whatever comes back on the model's output track, and reports how many round-tripped, in what order, and how long each took: ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} python -m examples.frame_metadata_roundtrip --frames 60 --verbose ``` If nothing comes back at all, that means the model doesn't echo frame metadata — not that anything is misconfigured on the client side. # Recordings Source: https://docs.reactor.inc/concepts/recordings Save clips of a live session, preview them, and download them as MP4 files ## What is a clip? Once a model has recording enabled, your app can ask Reactor for two types of clip from a live session: * **Snap clip**: the last `N` seconds of the session. * **Full recording**: everything from the start of the session up to now. Both return a [`Clip`](/sdk-reference/types#clip) object you can hand to the SDK's player to preview, or to its download helpers to save as a single MP4. The same object works either way. Reactor does not host clips. The URL you receive from Reactor expires after 24 hours, so it is not suitable for sharing. If you want users to keep a clip, download it immediately and host the resulting MP4 yourself. Reactor holds clips for 24 hours and then deletes them. None of this data is used for training. *** ## Capturing a clip Recording lives on the `Reactor` instance. In React, reach it via `useReactor()` inside a `ReactorProvider`. ```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} const snap = await reactor.requestClip(10); const full = await reactor.requestRecording(); ``` ```tsx React theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} import { useReactor } from "@reactor-team/js-sdk"; function SnapButton() { const { status, reactor } = useReactor((s) => ({ status: s.status, reactor: s.internal.reactor, })); return ( ); } ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}} snap = await reactor.request_clip(10) full = await reactor.request_recording() ``` `requestClip` / `request_clip` takes a duration in seconds and grabs the live session up to that point in the past. It is capped server-side at 5 minutes by default. `requestRecording` / `request_recording` takes no arguments and grabs the full session from the start of recording up to now. Both resolve with the same `Clip` object. Both methods can only be called when the connection status is `"ready"`. Otherwise they throw (JavaScript: a [`RecordingError`](/sdk-reference/types#recordingerror) with code `DISCONNECTED`; Python: an [`InvalidStateError`](/sdk-reference/python/types#reactorerror)) — and a request still in flight when the session disconnects raises [`DisconnectedError`](/sdk-reference/python/types#reactorerror) the same way. *** ## Previewing a clip Play a captured clip in place. In React, drop a [`ClipPlayer`](/sdk-reference/react-components#clipplayer) into your UI and pass the clip; it renders a native `