Skip to main content
Genie 4 produces video and audio in a Reactor session. Send commands with reactor.sendCommand() or the typed SDK. This page describes the reactor/genie-4 contract. See the overview for a quick start.

Tracks

There are no input tracks. Attach both outputs before starting a world. Handle blocked audio playback with a user control. A connected track alone does not establish that the world is live or that the browser plays its sound.

Session lifecycle

One session holds at most one world at a time. The phases are idle, starting, live, ending, ended, and failed. Call start_world from idle, ended, or failed. The first generated picture moves the world from starting to live. A world lasts up to 60 seconds live. end_world stops it early and leaves the Reactor session open. Starting again creates a new world. Supply the prompt or image again, because a world cannot resume. Subscribe to session_state before connecting, then await get_state for a fresh snapshot. Clear local snapshots when disconnected. After reconnecting, read get_state again before enabling world controls.

Commands

The JavaScript SDK reports refused commands through error, or React lastError, with operation: "sendCommand". Calls resolve undefined on failure and do not reject. For the three commands without a reply body, success also resolves undefined. The Python SDK raises ReactorError for a refusal. See Refusals. The React examples below render inside Genie4Provider. Its provider declares the model and tracks. See the tutorial for authentication and media playback.

start_world

Start from idle, ended, or failed. Supply a nonblank prompt, an image, or both. The prompt and image apply only at world start. Use instruct for later scene changes. Upload the image through uploadFile() and use the returned reference. Reference images work best at a 16:9 aspect ratio; other shapes are center-cropped. Do not pass a file path or a URL in reference_image. For an image-only start, omit prompt or pass prompt: "". Do not set it to undefined or null.
Watch for phase: "live". A later failure appears as phase: "failed" with last_error. A refused start uses busy, invalid_input, or invalid_image.

set_keys

Replace every key this client holds. Valid keys are w, a, s, d, e, literal space " ", arrowup, arrowdown, arrowleft, and arrowright. Send an update whenever a key is pressed or released. Send [] when the final key is released. Release held keys when someone switches tabs or apps, or clicks away from the controls. Also release them when a mouse or touch press ends or is canceled. Ignore keyboard presses while someone types in a text field. Keys sent during starting take effect when the world becomes live. Keys sent before starting a world or after it ends are ignored.
Disconnecting releases held keys.

instruct

Change the scene or give a character a line while the world is live. A new instruction replaces one still being delivered.
Refuses with not_live outside live, or invalid_input for blank text.

end_world

Stop the current world and await its session_state reply. This command accepts an empty object. It is valid in every phase and safe to repeat. If no world runs, it returns the settled state. When it stops a world, the phase becomes ended with end_reason: "ended_by_client".
Typed replies expose the snapshot fields directly: read reply.phase. The base JavaScript result uses reply.data.phase. Use the awaited reply for this request. Keep subscriptions for broadcasts.

get_state

Read the current session_state snapshot in any phase. This command accepts an empty object.

Refusals

Schema validation can also report invalid_command before the handler runs, for example for a value with the wrong type. The four codes above are model-handler refusals. These refusals use the SDK error path. There is no command_error broadcast. Keep them separate from connection status and from later world failures in last_error. See Surfacing refusals for React handling.

Messages

Broadcasts reach every connected client. A command reply goes to the client that sent the command. Base SDK messages use { type: "session_state", data: snapshot }. Typed hooks and typed replies expose snapshot fields directly, such as state.phase.

session_state payload

The receiving flags describe the model media output. The browser can still block playback. For a countdown, use Math.max(0, max_seconds - elapsed_seconds).

end_reason values

last_error fields

World failure codes include upstream_capacity, upstream_unavailable, upstream_rejected, upstream_interrupted, upstream_auth, upstream_protocol, content_policy, no_stream, media_stalled, and platform_failed. Show reason and use retryable when offering another start.

Complete example

The tutorial provides a complete Next.js app with scoped authentication, video and audio playback, snapshots, held keys, live instructions, and end/restart controls. For a shorter base SDK connection, see the overview.