Skip to main content
Matrix-Game-2.0 is an interactive world foundation model for real-time long video generation. Built upon an auto-regressive diffusion-based image-to-world framework, it can generate real-time long videos conditioned on keyboard and mouse inputs, enabling fine-grained control and dynamic scene evolution.

​
Key Features

Real-Time Control

Navigate through generated environments using keyboard (WASD) and mouse controls (IJKL)

Custom Starting Images

Initialize video generation from any image, creating explorable worlds from static scenes

Interactive World Building

Dynamically explore and generate content as you move through the environment
The model responds to movement and camera controls in real-time, creating an immersive, game-like experience where users can explore AI-generated worlds with immediate visual feedback.

​
Quick Start

Get started with Matrix-2 in seconds:
  • npm
  • pnpm
npx create-reactor-app my-matrix-app matrix-2
This creates a Next.js application from our open-source example. You can also try the live demo first.

​
Model Name

matrix-2

​
Commands

Your application communicates with the model through commands sent via reactor.sendMessage(). These commands enable real-time control of the generation—from steering the camera and movement to initializing custom scenes and managing state. Below are all available commands:
  • control
  • set_starting_image
  • set_show_icon
  • reset

​
control

Description: Steer the video generation in real-time using keyboard and mouse controls.Parameters:
  • keyboard_key (string, required): Movement control
    • W = Move forward
    • A = Move left
    • S = Move backward
    • D = Move right
    • Q = No movement action
  • mouse_key (string, required): Camera/look control
    • J = Turn left
    • K = Look down
    • L = Turn right
    • I = Look up
    • U = No action
Behavior:
  • Both parameters must be provided with each control command
  • Use Q and U together to reset to a no-operation state (no movement or camera change)
  • Controls are applied immediately to the ongoing generation
When you send a control, that control will kept active until you “unset” it. To unset the control, send the no-input command for the associated control field. (Q for movement, U for camera). This system allows you to build application in which you don’t have to constantly send a new control input, but instead you can react only to the changes in inputs by the user (press and release of buttons.)Example Use Case: Create an interactive, game-like experience where users can navigate through generated environments in real-time.Example:
// Move forward while looking up
await reactor.sendMessage({
  type: "control",
  data: {
    keyboard_key: "W",
    mouse_key: "I"
  }
});

// Move left while turning left
await reactor.sendMessage({
  type: "control",
  data: {
    keyboard_key: "A",
    mouse_key: "J"
  }
});

// No movement or camera change
await reactor.sendMessage({
  type: "control",
  data: {
    keyboard_key: "Q",
    mouse_key: "U"
  }
});

​
Credits

Matrix-Game-2.0 is developed by Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Cyrus Wu, Wei Li, Xuchen Song, Yang Liu, Eric Li, and Yahui Zhou Project Page - View on GitHub
⌘I