aloha-agilex; it does not validate transfer to a physical robot or another embodiment.
Observation adapter
Provide all three camera views as RGBuint8 arrays, with the view names and image geometry in the
reference. Align camera capture and state
measurement. Preserve the checkpoint’s camera geometry and framing when evaluating compatibility.
Pack measured end-effector positions, orientations, and gripper openings into proprio. Joint
positions alone are insufficient; your adapter must compute the end-effector poses. Use unit
quaternions in wxyz order and map the gripper to the RoboTwin opening convention.
Coordinate mapping
The upstream adapter defines two fixed transforms for RoboTwin. With column-vector positions and rotation matrices:B changes the base coordinate convention; E changes the end-effector axes. A physical robot
needs its own calibrated mapping into the checkpoint frame. Do not apply these simulator-specific
matrices to arbitrary hardware without establishing the frame relationship.
Action execution
For each arm, start at the state used in the request and integrate the returned rows sequentially:proprios are diagnostic and must not replace
measured state.
The reference simulator executes the full 32-step chunk. If your controller executes only a prefix,
discard the remainder and obtain a new observation before replanning. Never restart a relative
action chunk from its first row after partially executing it.
Control loop and recovery
Run one prediction at a time. Associate it with the captured state, accept only its matchingstep,
and reject duplicate or expired chunks. The model’s reply identifies the request, but does not
include a sensor capture timestamp or an execution deadline; track both in your client.
Camera publishing frequency, inference latency, and controller frequency are separate quantities.
Choose the execution cadence for your controller and validate policy behavior at that cadence. The
reference client’s 15 fps tracks do not specify a 15 Hz robot control rate.
Set a maximum observation age and a response deadline for your robot. The replay client’s default
30-second response timeout and two retries are debugging defaults, not a hardware control budget. On
timeout or disconnect, stop applying old predictions according to your controller’s stop policy.
Reconnect with a fresh client, discard old replies, and resend the task and current observations.
Always close the client in finally, as the quickstart does. Measure capture-to-execution time in
your application; prediction.latency_ms measures only request send to matching reply and excludes
the client’s frame-delivery wait.