Integration

Reactor predicts actions. Your application acquires observations, validates replies, and drives the controller. This variant is trained for LIBERO’s Panda setup; no physical robot driver or hardware validation is supplied with this guide.

Map the observations

Use the exact camera and state contract. Capture both views and state together. Camera dimensions alone do not establish compatibility: preserve the reference views, orientation, field of view, robot coordinates, and gripper joint ordering. The input orientation is axis-angle, not Euler angles or the robot’s joint angles. For a LIBERO observation, use robosuite’s axis-angle conversion, matching the upstream evaluator. The simulator environment supplies robosuite.

import numpy as np
from robosuite.utils.transform_utils import quat2axisangle


def encode_observation(obs):
    frames = {
        "exterior_view_1": np.ascontiguousarray(obs["agentview_image"]),
        "wrist_view": np.ascontiguousarray(obs["robot0_eye_in_hand_image"]),
    }
    orientation = quat2axisangle(np.array(obs["robot0_eef_quat"], copy=True))
    proprio = np.concatenate([
        obs["robot0_eef_pos"], orientation, obs["robot0_gripper_qpos"]
    ])
    if proprio.shape != (8,) or not np.isfinite(proprio).all():
        raise ValueError("Expected eight finite LIBERO state values")
    return frames, proprio.tolist()

Request and execute

Keep one request outstanding. Publish its current observations, send state_json, and continue supplying camera frames while waiting. Validate the returned step, finite (32, 7) array, and controller limits before scheduling any actions. Discard duplicate or obsolete replies. Use the controller scaling documented in the reference. Do not interpret the first six columns as physical displacement without that controller mapping. The gripper is already converted to LIBERO’s convention. Choose how many rows to execute before observing and replanning. The upstream evaluator uses 10; all 32 rows are predictions, and there is no seed prefix or executed-action acknowledgement. For the next prediction, replace the observations and increment chunk_id.

Timing and recovery

Track observation age, request-to-reply time, and actual execution time separately. Camera publishing rate, model prediction time, and controller frequency are distinct. Set an observation-age limit and response deadline appropriate to your controller; an expired request must not enqueue actions later. If a timeout leaves request ownership uncertain, stop consuming its results and start a fresh session. Retry only when your client can recognize and discard duplicate replies. Never blindly replay returned actions after reconnecting. At episode end, clear the local pending action plan and reset the simulator or robot through your own controller. Reset the model separately, set the new instruction if needed, and send fresh frames with a new request ID. The model’s reset does not interrupt physical motion.

Get started

Models

Map the observations

Request and execute

Timing and recovery

​Map the observations

​Request and execute

​Timing and recovery

Map the observations

Request and execute

Timing and recovery