Managing State

A real-time model is not called once with everything it needs. The runtime calls generate() over and over, and each call is one step that produces the next piece of output: a frame, a chunk of frames, a stretch of audio. Between steps the client’s inputs change: a new prompt, a key pressed, a camera turned. The state is where those inputs live. A client sets a field from wherever it is, the runtime stores the value, and the next step reads it. Your model never handles the message itself.

A timeline. A state bar along the top reads move = idle, then move = forward from the point where a client sends set_move. Below it, four generate() boxes, one per step, each with an arrow from the state bar above it: the first two read idle, the last two read forward. — One value over time. A client changes it between two steps, and every step after that reads the new value.

The examples on this page are a world model in the shape of SANA-WM: a client types a prompt for the scene and then walks through it with camera controls, and every generate() call produces the next chunk of frames.

Declaring the state

The state is a class that extends InputState, annotated on your model as state:. Each field gets a type, a default, and optionally some bounds:

sana_wm.py

class SanaWMState(InputState):
    prompt: str = InputField(
        default="a misty valley at sunrise",
        max_length=500,
        moderate=True,
        description="The scene to generate. A new prompt starts a fresh rollout.",
    )
    move: str = InputField(
        default="idle",
        choices=["idle", "forward", "back", "strafe_left", "strafe_right"],
        description="Camera movement, held until changed.",
    )
    yaw: int = InputField(
        default=0, ge=-1, le=1, description="Turn the camera: -1 left, 0 still, 1 right."
    )
    pitch: int = InputField(
        default=0, ge=-1, le=1, description="Tilt the camera: -1 down, 0 still, 1 up."
    )


class SanaWMOutput(Output):
    main_video: Video


class SanaWM(ReactorApp):
    state: SanaWMState

    def load(self, config_path: Path | None) -> None:
        self.pipe = SanaWMPipeline.from_pretrained(get_weights_path())

    def generate(self, input: SanaWMState) -> SanaWMOutput:
        chunk = self.pipe.next_chunk(
            prompt=input.prompt,
            move=input.move,
            yaw=input.yaw,
            pitch=input.pitch,
        )
        return SanaWMOutput(main_video=chunk)  # (24, H, W, 3) uint8

When a session starts, the runtime builds a fresh instance from the defaults. That instance is what generate() receives, and it is also available as self.state anywhere else in the class.

What a field becomes

Every public field turns into a command named set_<field>. The client sends the new value, the runtime checks it against the field’s type and bounds, and if it passes, the field changes.

A SanaWMState box listing prompt, move, yaw, and pitch with their current values. Arrows labeled set_prompt, set_move, set_yaw, and set_pitch come in from the left, and an arrow on the right leads to generate(). — Clients set individual fields from outside. generate() reads the whole object on every step.

The four fields above give a client these four commands:

prompt: str

default:"a misty valley at sunrise"

The scene to generate. A new prompt starts a fresh rollout. Up to 500 characters.

move: str

default:"idle"

Camera movement, held until changed. One of idle, forward, back, strafe_left, strafe_right.

yaw: int

default:"0"

Turn the camera: -1 left, 0 still, 1 right.

pitch: int

default:"0"

Tilt the camera: -1 down, 0 still, 1 up.

The description you wrote on the field is the description the client sees. This is what a call looks like from each side:

await reactor.send_command("set_prompt", {"prompt": "a city street at night"})
await reactor.send_command("set_move", {"move": "forward"})

await reactor.sendCommand("set_prompt", { prompt: "a city street at night" });
await reactor.sendCommand("set_move", { move: "forward" });

{ "type": "set_prompt", "data": { "prompt": "a city street at night" } }
{ "type": "set_move", "data": { "move": "forward" } }

A value stays set until the client changes it. The model reads the current value on every step, so a held key maps onto one command when it goes down and one when it comes up: set_move with forward on key down, set_move with idle on key up, and the camera keeps moving through every chunk in between.

Validation

The constraints you write into a field are what the runtime checks before the value reaches your model. A value outside them is refused with invalid_command, the field keeps its old value, and your code is never called: a set_move with "jump" never gets through. Bounds on numbers (ge, le), string lengths (min_length, max_length), a fixed set of (choices), and a moderate mark for free text all live on the field declaration, alongside the default and the description a client sees.

InputField()

Every argument the field declaration accepts, with what each one enforces.

Private fields

A field whose name starts with an underscore is yours alone. No command is generated for it, the client never sees it, and it resets with the rest of the state when a session ends. Use it for values the model needs to remember between steps but that a client should not set directly. SANA-WM starts every rollout from a first frame the client uploads. The upload arrives through a hand-written command, the handler decodes it and stores the result in a private field, and generate() reads it from there:

sana_wm.py

class SanaWMState(InputState):
    prompt: str = InputField(default="a misty valley at sunrise", max_length=500, moderate=True)
    move: str = InputField(default="idle", choices=["idle", "forward", "back"])

    # Private: no command, invisible to the client.
    _first_frame: np.ndarray | None = None
    _first_frame_id: int = 0

From the client’s side, this state has exactly two commands:

prompt: str

default:"a misty valley at sunrise"

Up to 500 characters.

move: str

default:"idle"

One of idle, forward, back.

_first_frame and _first_frame_id do not appear. Keep private fields to plain data, such as arrays, strings, and ids. GPU tensors belong on the model, not on the state.

Overriding a generated command

Sometimes setting a value should also do something else: reply to the client, cut the video, or remember that something changed. Write an @event handler with the same name as the generated command and it takes over. The field stays a field; only what happens when the client sets it changes. A new prompt starts a fresh rollout, and the model needs to know that on its next step. The override below writes the prompt as before, bumps a private counter so generate() can tell a new prompt from a repeated one, and drops the frames still queued for playout so the client does not receive the tail of the old scene:

sana_wm.py

class SanaWMState(InputState):
    prompt: str = InputField(default="a misty valley at sunrise", max_length=500, moderate=True)
    # ...

    _rollout_id: int = 0  # bumped on every new prompt


class SanaWM(ReactorApp):
    state: SanaWMState

    @event(name="set_prompt", description="The scene to generate. Starts a fresh rollout.")
    def set_prompt(self, prompt: str = InputField(max_length=500, moderate=True)) -> None:
        self.state.prompt = prompt
        self.state._rollout_id += 1
        self.output.flush()

    def generate(self, input: SanaWMState) -> SanaWMOutput:
        if input._rollout_id != self.rollout_id:
            self.pipe.reset(prompt=input.prompt)
            self.rollout_id = input._rollout_id
        chunk = self.pipe.next_chunk(move=input.move, yaw=input.yaw, pitch=input.pitch)
        return SanaWMOutput(main_video=chunk)

From the client’s side nothing changed: set_prompt is still one command with one string. The private field is how the handler and generate() talk to each other without the client seeing it. You can also define entirely new commands that the client can call, for any action you want to expose. Events & Messages shows how.

When the state resets

The runtime builds the state from the defaults when a session starts and drops it when the session ends. A client that disconnects and reconnects within the same session finds the values it left. A new session starts clean.

Events & Messages

Hand-written commands, replies, and messages from the model to the client.

Video & Audio Tracks

Receive the client’s camera and microphone as input tracks.

Get Started

CLI & Platform

Build a Model

Media & Transport

Releases

Declaring the state

What a field becomes

Validation

InputField()

Private fields

Overriding a generated command

When the state resets

Next

Events & Messages

Video & Audio Tracks

​Declaring the state

​What a field becomes

​Validation

InputField()

​Private fields

​Overriding a generated command

​When the state resets

​Next

Events & Messages

Video & Audio Tracks

Declaring the state

What a field becomes

Validation

Private fields

Overriding a generated command

When the state resets

Next