Build your model with the Reactor Runtime

The Reactor Runtime serves your model to clients in real time. Output reaches each client as it is generated, and whatever the client sends back, such as a prompt, a key press, a camera feed, or a sensor reading, reaches the model before its next step. You write one Python class: how to load the weights, and how to produce the next output from the current inputs. The runtime handles everything between that class and its clients.

Clients on the left, such as a web app, a script, or a robot, send commands and media tracks to the Reactor Runtime on the right, which holds your model and streams video, audio, and messages back. — Clients send controls and media in. The runtime streams your model's output back.

What the runtime handles

Real-time streaming

Frames and audio reach every client over WebRTC as your model produces them, paced so playback stays smooth.

Controls and messages

Clients change the settings you declare, each one validated before your model reads it. Your model answers with typed messages: a status, a prediction, an action.

Media input

A client’s camera, microphone, or other media source arrives as a track your model reads, for video-to-video and audio-driven models.

Sessions

One instance serves one session after another, several clients can share a session, and your model is told when they arrive and leave.

The reactor CLI packages your model into a container, and the same container runs on your machine and on Reactor.

Model outline

Here is what a model looks like in practice. The example is Waypoint, Overworld’s world model, which you explore with a keyboard and mouse. The complete version is in the runtime repository under examples/waypoint.

model.py

# imports omitted


class WaypointState(InputState):
    action: str = InputField(
        default="idle",
        choices=["idle", "forward", "back", "left", "right"],
        description="Movement key, held until changed.",
    )
    mouse_x: float = InputField(
        default=0.0, description="Look left or right."
    )
    mouse_y: float = InputField(
        default=0.0, description="Look up or down."
    )


class WaypointOutput(Output):
    main_video: Video


class Waypoint(ReactorApp):
    state: WaypointState

    def load(self, config_path):
        self.engine = WorldEngine("Overworld/Waypoint-1.5-1B")
        self.engine.append_frame(SEED_FRAME)

    def generate(self, input: WaypointState) -> WaypointOutput:
        ctrl = CtrlInput(
            button=KEYS[input.action],
            mouse=(input.mouse_x, input.mouse_y),
        )
        frames = self.engine.gen_frame(ctrl=ctrl).cpu().numpy()
        return WaypointOutput(main_video=frames)

Here is how the runtime drives it:

Inside the Reactor Runtime, the client's commands write into WaypointState at the top, the Waypoint model in the center reads it in generate() and returns a WaypointOutput at the bottom, which is streamed back to the client as main_video. — load() runs once. Then the runtime loops: read the state, call generate(), stream the output.

To deploy this on Reactor, run reactor model deploy from the workspace. The quickstart walks you through that. You can also test it locally; see Test locally. If your model already supports inference across several GPUs, see the Multi-GPU inference.

Model Anatomy

Every member of a ReactorApp, read through one working model.

The Step Loop

How the runtime drives generate(), when to skip a step, and how frames are paced.

Get Started

CLI & Platform

Build a Model

Media & Transport

Releases

Build your model with the Reactor Runtime

What the runtime handles

Real-time streaming

Controls and messages

Media input

Sessions

Model outline

Next

Model Anatomy

The Step Loop

​What the runtime handles

Real-time streaming

Controls and messages

Media input

Sessions

​Model outline

​Next

Model Anatomy

The Step Loop

What the runtime handles

Model outline

Next