📜 Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

Jiuming Liu¹ ChaoJun Ni² Mengmeng Liu³ Chensheng Peng⁴ Fangjinhua Wang⁵ Sitian Shen⁶
Marc Pollefeys⁵ Masayoshi Tomizuka⁴ Ayush Tewari¹ Per Ola Kristensson¹

¹University of Cambridge ²Peking University ³University of Twente
⁴University of California, Berkeley ⁵ETH Zurich ⁶University of Oxford

This is a curated awesome list of papers, codebases, websites for interactive video world modeling. This repository follows the organization our survey paper and supports three ways of browsing:

by application domains: open-world exploration, game engines, autonomous driving, embodied AI.
by industry products: Genie, Happy Oyster, Odyssey, Mirage, etc.
by technical themes: trends, controllability, long-horizon, memory, and real-time efficiency.

🚩 News & Updates

⭐ [Ongoing Update] We are updating this repo progressively with novelly open-sourced papers, products, datasets, and benchmarks. The most recent few works are:

[2026-8-10]Sekai2 - Sekai2: From World Exploration to Interactive World Modeling.
[2026-8-9]Khora - Population-Scalable Multi-Agent World Modeling.
[ICML'26 Workshop] [2026-8-7]WorldTrace - Addressable Memory for Video World Models.
[2026-8-6]GeniWorld - GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions.
[2026-8-6]MASS - MASS: Multiplayer World Models with Authoritative Shared State.
[2026-8-6]UA-NWM - Uncertainty-Aware World Model for Aerial Image-Goal Navigation.
[2026-8-5]WorldClaw - WorldClaw: Agentic 3D Open-World Generation at Scale.
[2026-8-5]HelloWorld - HelloWorld: Enabling Socially Interactive Characters in Video World Models.
[2026-7-29]StatePlay - StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation.
[2026-7-28]Wonder - Wonder: Video World Model Done Better.
[2026-7-23]WorldWeaver - Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers.
[2026-7-21]ABot-World-0 - ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU.
[2026-7-20]AlayaWorld - AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report.
[2026-7-20]GeoWorldAD - GeoWorldAD: Geometry World Action Model for Autonomous Driving.
[2026-7-17]Orbis 2 - Orbis 2: A Hierarchical World Model for Driving.
[2026-7-15]From Pixels to States - From Pixels to States: Rethinking Interactive World Models as Game Engines.
[2026-7-15]M4World - M4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming.
[ECCV'26] [2026-7-13]Cycle-World - Cycle-World: Mitigating Error Accumulation in Long-term Video World Models via Reverse-Prediction Cycle Consistency.
[2026-7-10]PanoWorld - PanoWorld: Real-World Panoramic Generation.
[2026-7-9]Canvas360 - Enhancing In-context Panoramic Generation via Geometric-aware Pretraining.
[2026-7-8]LingBot-World 2.0 - Infinite Worlds with Versatile Interactions.
[2026-7-7]RynnWorld-Teleop - RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation.
[2026-7-7]AlayaWorld - AlayaWorld: Long-Horizon and Playable Video World Generation.
[2026-7-6]MIRA - Multiplayer Interactive World Models with Representation Autoencoders.
[2026-7-3]Vidu S1 - Vidu S1: A Real-Time Interactive Video Generation Model.
[2026-7-2]WorldDirector - WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory.
[ECCV'26] [2026-6-30]MemLearner - MemLearner: Learning to Query Context memory for Video World Models.
[2026-6-30]WorldRoamBench - WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models.
[ECCV'26] [2026-6-29]NeuWorld - Walking in the Implicit: Interactive World Exploration via Neural Scene Representation.
[ECCV'26] [2026-6-29]A2World - Learning Transferable Dynamics Priors from Action to World Modeling.
[2026-6-29]Orca - Orca: The World is in Your Mind.
[2026-6-29]DreamForge-World - DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model.

⭐ [News] We also refer to our recent work ''VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs.'' accepted by ICML 2026 as Spotlight. 🥳🥳🥳 and ''DriveVA:Video Action Models are Zero-Shot Drivers.'' accepted by ECCV 2026. 🥳🥳🥳

⭐ [Ongoing Update] Welcome to contribute to the curated list by contacting us. If you find this repository is useful in your research, please cite the paper by:

📝 Citation

@article{liu2026towards,
  title={Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends},
  author={Liu, Jiuming and Ni, Chaojun and Liu, Mengmeng and Peng, Chensheng and Wang, Fangjinhua and Shen, Sitian and Pollefeys, Marc and Tomizuka, Masayoshi and Tewari, Ayush and Kristensson, Per Ola},
  journal={arXiv preprint arXiv:2606.01164},
  year={2026}
}

Entries may appear in multiple sections when one paper contributes to several research axes.

📚 Overview

🗺️ Application Domains
- 🌍 Open-world Exploration
- 🎮 Game Engines
- 🚗 Autonomous Driving
- 🤖 Embodied AI and Robotics
🚀 Industry Products
🔭 Recent Research Trends
- 🌐 Specialist to Generalist
- 🌀 Static to Dynamic and Multi-agent Worlds
- 🔊 Single-sensory to Multi-sensory Interfaces
📖 User Action Controllability
- 🎯 One-shot Loose Control
- 🕹️ Frame-level Fine-grained Control
- ⌨️ Action Injection and Control Interfaces
🧠 Long-Horizon Interactions and Memory
- 🕰️ History as Condition
- 🗃️ Memory Construction
- 🧱 Explicit 3D Reconstruction
- 🧪 Forcing, Noise, and AR Diffusion Training
⚡ Real-time Interactivity
- ⚖️ Conflict: Responsiveness vs Coherence
- 🚀 Efficient Rollout
📊 Datasets and Benchmarks
- 🌍 Open-world Benchmarks
- 🚗 Autonomous Driving Benchmarks
- 🤖 Embodied AI Datasets and Benchmarks
- 🎮 Game Dataset and Benchmarks
🙏 Acknowledgements

🗺️ Application Domains

🌍 Open-world Exploration

Canvas360 - Enhancing In-context Panoramic Generation via Geometric-aware Pretraining.
WorldDirector - WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory.
[ECCV'26] NeuWorld - Walking in the Implicit: Interactive World Exploration via Neural Scene Representation.
Holo-World - Holo-World: Unified Camera, Object and Weather Control for Video World Model.
ActWorld - ActWorld: From Explorable to Interactive World Model via Action-Aware Memory.
BadWorld - BadWorld: Adversarial Attacks on World Models.
MoVerse - MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold.
BiWM - BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression.
Prisma-World - Prisma-World: Camera-Controllable Multi-Agent Video World Model.
AnchorWorld - AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization.
HomeWorld
MetaWorld - MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data.
minWM - minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models.
[ICML'26] SphericalDreamer - SphericalDreamer: Generating Navigable Immersive 3D Worlds with Panorama Fusion.
Happy Oyster
HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
Lyra 2.0 - Lyra 2.0: Explorable Generative 3D Worlds.
Inspatio-world - Inspatio-world: A real-time 4D world simulator via spatiotemporal autoregressive modeling.
[SIGGRAPH'26] OmniRoam - OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation.
[CVPR'26] SonoWorld - SonoWorld: From One Image to a 3D Audio-Visual Scene.
Omni-WorldBench - Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models.
MosaicMem - MosaicMem: Hybrid Spatial Memory for Controllable Video World Models.
SWM - Grounding World Simulation Models in a Real-World Metropolis.
LiveWorld - LiveWorld: Simulating Out-of-sight Dynamics in Generative Video World Models.
RealWonder - RealWonder: Real-Time Physical Action-Conditioned Video Generation.
Shareverse - Shareverse: Multi-agent Consistent Video Generation for Shared World Modeling.
[ICML'26] Olaf-World - Olaf-world: Orienting Latent Actions for Video World Modeling.
[ICML'26] WorldCompass - WorldCompass: Reinforcement Learning for Long-Horizon World Models.
MIND - MIND: Benchmarking Memory Consistency and Action Control in World Models.
[CVPR'26] PerpetualWonder - PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation.
[ICML'26] LIVE - LIVE: Long-horizon Interactive Video World Modeling.
[ICML'26] Infinite-world - Infinite-world: Scaling Interactive World Models to 1000-frame Horizons via Pose-free Hierarchical Memory.
Lingbot-World - Advancing Open-source World Models.
EgoWM - Walk through Paintings: Egocentric World Models from Internet Priors.
Pixelverse-R1 - Pixelverse-r1.
[CVPR'26] VerseCrafter - VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control.
[CVPR'26] NeoVerse - NeoVerse: Enhancing 4D World Model with In-the-wild Monocular Videos.
Teleworld - Teleworld: Towards Dynamic Multimodal Synthesis with a 4D World Model.
WorldCanvas - The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text.
[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
[CVPR'26] WonderZoom - Wonderzoom: Multi-scale 3D World Generation.
[ICLR'26] Astra - Astra: General Interactive World Model with Autoregressive Denoising.
RELIC - RELIC: Interactive Video World Model with Long-Horizon Memory.
MagicWorld - MagicWorld: Towards Long-Horizon Stability for Interactive Video World Exploration.
Pan - Pan: A World Model for General, Interactable, and Long-Horizon World Simulation.
[ICLR'26] MotionStream - MotionStream: Real-Time Video Generation with Interactive Motion Controls.
[NeurIPS'25] IaaW - Image as a World: Generating Interactive World from Single Image via Panoramic Video Generation.
[ICLR'26] FantasyWorld - FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction.
Genie 3 - Genie 3: A new frontier for world models.
HY-World 1.0 - HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels.
Yume - Yume: An Interactive World Generation Model.
WonderFree - Wonderfree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration.
[ICCV'25] VMem - VMem: Consistent Interactive Video Scene Generation with Surfel-indexed View Memory.
[NeurIPS'25] Spmem - Video World Models with Long-term Spatial Memory.
DeepVerse - Deepverse: 4D Autoregressive Video Generation as a World Model.
[ICLR'26] Vid2World - Vid2World: Crafting Video Diffusion Models to Interactive World Models.
[NeurIPS'25] WORLDMEM - Worldmem: Long-term Consistent World Simulation with Memory.
[ICCV'25] WonderTurbo - Wonderturbo: Generating Interactive 3D World in 0.72 Seconds.
WorldScore - WorldScore: A Unified Evaluation Benchmark for World Generation.
[ICCV'25] AETHER - Aether: Geometric-Aware Unified World Modeling.
[CVPR'25] WonderWorld - Wonderworld: Interactive 3D Scene Generation from a Single Image.
[CVPR'24] Video2Game - Video2game: Real-time Interactive Realistic and Browser-compatible Environment from a Single Video.
[ICML'24] Genie - Genie: Generative Interactive Environments.
[CVPR'24] WonderJourney - Wonderjourney: Going from Anywhere to Everywhere.
[ICLR'24] UniSim - Learning Interactive Real-World Simulators.

🎮 Game Engines

From Pixels to States - From Pixels to States: Rethinking Interactive World Models as Game Engines.
AlayaWorld - AlayaWorld: Long-Horizon and Playable Video World Generation.
MIRA - Multiplayer Interactive World Models with Representation Autoencoders.
GameCraft-Bench - GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
BadWorld - BadWorld: Adversarial Attacks on World Models.
Light Interaction - Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models.
minWM - minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models.
Gamma-World - Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players.
SCOPE - SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models.
SANA-WM - SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer.
MultiWorld - MultiWorld: Scalable Multi-Agent Multi-View Video World Models.
Happy Oyster
HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
Lyra 2.0 - Lyra 2.0: Explorable Generative 3D Worlds.
Matrix-Game 3.0 - Matrix-game 3.0 Real-time and Streaming Interactive World Model with Long-Horizon Memory.
GameWorld - Gameworld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents.
ActionParty - Actionparty: Multi-subject Action Binding in Generative Video Games.
WorldCam - Worldcam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation.
PERSIST - Beyond Pixel Histories: World Models with Persistent 3D State.
Solaris - Solaris: Building a Multiplayer Video World Model in Minecraft.
[ICML'26] LIVE - LIVE: Long-horizon Interactive Video World Modeling.
Lingbot-World - Advancing Open-source World Models.
StableWorld - StableWorld: Towards Stable and Consistent Long Interactive Video Generation.
[CVPR'26] NeoVerse - NeoVerse: Enhancing 4D World Model with In-the-wild Monocular Videos.
[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
RELIC - RELIC: Interactive Video World Model with Long-Horizon Memory.
WorldGen - WorldGen: From Text to Traversable and Interactive 3D Worlds.
PAN - PAN: A World Model for General, Interactable, and Long-Horizon World Simulation.
Memory Forcing - Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft.
Dreamer 4 - Training Agents Inside of Scalable World Models.
[ICLR'26] FantasyWorld - FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction.
Matrix-Game 2.0 - Matrix-game 2.0: An Open-source Real-time and Streaming Interactive World Model.
Matrix-3D - Matrix-3D: Omnidirectional Explorable 3D World Generation.
Genie 3 - Genie 3: A new frontier for world models.
HY-World 1.0 - HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels.
Yume - Yume: An Interactive World Generation Model.
RealPlay - From Virtual Games to Real-World Play.
Matrix-Game - Matrix-game: Interactive World Foundation Model.
Hunyuan-GameCraft - Hunyuan-gamecraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition.
[ICLR'26] Vid2World - Vid2World: Crafting Video Diffusion Models to Interactive World Models.
[NeurIPS'25] WorldMem - Worldmem: Long-term Consistent World Simulation with Memory.
MineWorld - MineWorld: A Real-time and Open-source Interactive World Model on Minecraft.
[ICCV'25] GameFactory - Gamefactory: Creating New Games with Generative Interactive Videos.
[NeurIPS'25] The Matrix - The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control.
Genie 2 - Genie 2: A Large-scale Foundation World Model.
Oasis - Oasis: A Universe in a Transformer.
[ICLR'25] GameNGen - Diffusion Models are Real-time Game Engines.
[NeurIPS'24] DIAMOND - Diffusion for World Modeling: Visual Details Matter in Atari.
[CVPR'24] Video2Game - Video2game: Real-time Interactive Realistic and Browser-compatible Environment from a Single Video.
[ICML'24] Genie - Genie: Generative Interactive Environments.
[CVPR'20] GameGAN - Learning to Simulate Dynamic Environments with GameGAN.

🚗 Autonomous Driving

GeoWorldAD - GeoWorldAD: Geometry World Action Model for Autonomous Driving.
Orbis 2 - Orbis 2: A Hierarchical World Model for Driving.
M4World - M4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming.
OmniDrive - OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation.
[ICML'26] VectorWorld - VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs.
Vega - Vega: Learning to Drive with Natural Language Instructions.
[NeurIPS'25] ReSim - ReSim: Reliable World Simulation for Autonomous Driving.
[NeurIPS'25] FutureSightDrive - Futuresightdrive: Thinking Visually with Spatio-temporal CoT for Autonomous Driving.
GAIA-2 - GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving.
DrivingWorld - DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT.
DrivingGPT - DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers.
Doe-1 - Doe-1: Closed-loop Autonomous Driving with Large World Model.
[ICCV'25] MagicDrive-V2 - MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control.
[AAAI'25] DriveDreamer-2 - DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation.
[CVPR'24] Drive-WM - Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving.
GAIA-1 - GAIA-1: A Generative World Model for Autonomous Driving.
[ECCV'24] DriveDreamer - DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving.

🤖 Embodied AI and Robotics

RynnWorld-Teleop - RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation.
IOI - IOI: Decoupling Kinematics and Physics for Interactive World Models.
Mem-World - Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation
w-EVA - w-EVA: Envision, Verify, and Act with Latent Interactive World Models.
HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
Lyra 2.0 - Lyra 2.0: Explorable Generative 3D Worlds.
GigaWorld-Policy - GigaWorld-Policy: An Efficient Action-Centered World--Action Model.
Fast-WAM - Fast-WAM: Do World Action Models Need Test-time Future Imagination?
Interactive World Simulator - Interactive World Simulator for Robot Policy Training and Evaluation.
MVISTA-4D - MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation.
LingBot-VA - Causal World Modeling for Robot Control.
Motus - Motus: A Unified Latent Action World Model.
GigaWorld-0 - GigaWorld-0: World Models as Data Engine to Empower Embodied AI.
Ctrl-World - Ctrl-World: A Controllable Generative World Model for Robot Manipulation.
EMMA - EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfer.
MimicDreamer - MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training.
EmbodieDreamer - EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling.

🚀 Industry Products

[2026-7-3]Vidu S1 - Vidu S1: A Real-Time Interactive Video Generation Model.
[2026-7-8][Robbyant] LingBot-World 2.0 - Infinite Worlds with Versatile Interactions.
[2026-6-1][VAST] Project Eden
[2026-5-27][PixVerse]PixVerse R1
[2026-5-18][Odyssey] Agora-1
[2026-5-13][DreamX Team] DreamX-World
[2026-4-23][Odyssey] Odyssey-2 Max
[2026-4-16][Alibaba] Happy Oyster
[2025-8-21][Dynamics Lab] Mirage 2
[2025-8-5][Google] Genie 3
[2024-12-4][Google] Genie 2
[2024-2-23][Google] Genie

🔭 Recent Research Trends

🌐 Specialist to Generalist

Happy Oyster
Lyra 2.0 - Lyra 2.0: Explorable Generative 3D Worlds.
[ICLR'26] Astra - Astra: General Interactive World Model with Autoregressive Denoising.
[ICML'25] AdaWorld - AdaWorld: Learning Adaptable World Models with Latent Actions.
Genie 3 - Genie 3: A new frontier for world models.
[ICLR'24] UniSim - Learning Interactive Real-World Simulators.
GAIA-1 - GAIA-1: A Generative World Model for Autonomous Driving.
[ECCV'24] DriveDreamer - DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving.

🌀 Static to Dynamic and Multi-agent Worlds

Dynamics (Out of Sight Dynamics and Self-evolution):

[ECCV'26] A2World - Learning Transferable Dynamics Priors from Action to World Modeling.
IOI - IOI: Decoupling Kinematics and Physics for Interactive World Models.
Holo-World - Holo-World: Unified Camera, Object and Weather Control for Video World Model.
Happy Oyster
Inspatio-world - Inspatio-world: A real-time 4d world simulator via spatiotemporal autoregressive modeling.
HM-World - Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models.
LiveWorld - Liveworld: Simulating out-of-sight dynamics in generative video world models.
PERSIST - Beyond Pixel Histories: World Models with Persistent 3D State.
PerpetualWonder - Perpetualwonder: Long-horizon action-conditioned 4d scene generation.
[CVPR'26] VerseCrafter - VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control.
[CVPR'26] NeoVerse - NeoVerse: Enhancing 4D World Model with In-the-wild Monocular Videos.
Teleworld - Teleworld: Towards dynamic multimodal synthesis with a world model.
WorldCanvas - The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text.
[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
Hunyuan-GameCraft - Hunyuan-gamecraft: High-dynamic interactive game video generation with hybrid history condition.
DeepVerse - Deepverse: 4d autoregressive video generation as a world model.

Multi-Agent Interaction:

MIRA - Multiplayer Interactive World Models with Representation Autoencoders.
OmniDrive - OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation.
Prisma-World - Prisma-World: Camera-Controllable Multi-Agent Video World Model.
MetaWorld - MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data.
Gamma-World - Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players.
MultiWorld - MultiWorld: Scalable Multi-Agent Multi-View Video World Models.
ActionParty - ActionParty: Multi-Subject Action Binding in Generative Video Games.
[CVPR'26 Workshop] F3DGS - F3DGS: Federated 3D Gaussian Splatting for Decentralized Multi-Agent World Modeling.
SWM - Grounding World Simulation Models in a Real-World Metropolis.
MultiGen - MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines.
Shareverse - Shareverse: Multi-agent Consistent Video Generation for Shared World Modeling.
Solaris - Solaris: Building a Multiplayer Video World Model in Minecraft.

🔊 Single-sensory to Multi-sensory Interfaces

Happy Oyster
[CVPR'26] SonoWorld - SonoWorld: From One Image to a 3D Audio-Visual Scene.
GAIA-2 - GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving.
RealWonder - RealWonder: Real-Time Physical Action-Conditioned Video Generation.
[CVPR'26] PerpetualWonder - PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation.
Teleworld - Teleworld: Towards Dynamic Multimodal Synthesis with a 4D World Model.

📖 User Action Controllability

🎯 One-shot Loose Control

Here we mainly list papers with one-shot camera trajectory as controls.

[Siggraph Asian 2025] Uni3C - Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation.
SEVA - Stable Virtual Camera: Generative View Synthesis with Diffusion Models.
[TPAMI'25] ViewCrafter - ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis.
CameraCtrl - CameraCtrl: Enabling Camera Control for Video Diffusion Models.
[SIGGRAPH 2024] MotionCtrl - MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

🕹️ Frame-level Fine-grained Control

Actually, most of interactive world models can achieve frame-level control. Here, we only list some representative works.

[ICML'26] Olaf-World - Olaf-world: Orienting Latent Actions for Video World Modeling.
[CVPR'26] VerseCrafter - VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control.
[CVPR'26] NeoVerse - NeoVerse: Enhancing 4D World Model with In-the-wild Monocular Videos.
WorldCanvas - The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text.
MagicWorld - MagicWorld: Towards Long-Horizon Stability for Interactive Video World Exploration.
[NeurIPS'24] iVideoGPT - iVideoGPT: Interactive VideoGPTs are Scalable World Models.
[ICLR'24] UniSim - Learning Interactive Real-World Simulators.

⌨️ Action Injection and Control Interfaces

Direction Concatenation:

Most of recent works belong to this category, so we just list some representation works in early stage.

Matrix-Game 2.0 - Matrix-game 2.0: An Open-source Real-time and Streaming Interactive World Model.
[ICLR'26] Vid2World - Vid2World: Crafting Video Diffusion Models to Interactive World Models.
[ICML'25] AdaWorld - AdaWorld: Learning Adaptable World Models with Latent Actions.
[ICCV'25] GameFactory - GameFactory: Creating New Games with Generative Interactive Videos.
[ICML'24] Genie - Genie: Generative Interactive Environments.

Scaling and Shifting:

[ICLR'26] FantasyWorld, "FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction".
[ICCV'25] SSM-WM, "Long-Context State-Space Video World Models".
[NeurIPS'25] WorldMem, "Worldmem: Long-term Consistent World Simulation with Memory".
GameGEN-X, "GameGen-X: Interactive Open-world Game Video Generation".

Camera-Controlled Rendering or Simulation:

LiveWorld - LiveWorld: Simulating Out-of-sight Dynamics in Generative Video World Models.
RealWonder - RealWonder: Real-Time Physical Action-Conditioned Video Generation.
[CVPR'26] PerpetualWonder - PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation.
[ICCV'25] VMem - VMem: Consistent Interactive Video Scene Generation with Surfel-indexed View Memory.

Matrix Transformation:

PanoWorld - PanoWorld: Real-World Panoramic Generation.
[NeurIPS'25] IaaW - Image as a World: Generating Interactive World from Single Image via Panoramic Video Generation.
GenEx GenEx: Generating an Explorable World.

🧠 Long-Horizon Interactions and Memory

🕰️ History as Condition

Most of interactive world models adopt an autoregressive paradigm, so we just list some representation works.

[ICLR'26] Astra - Astra: General Interactive World Model with Autoregressive Denoising.
PAN - PAN: A World Model for General, Interactable, and Long-Horizon World Simulation.
Hunyuan-GameCraft - Hunyuan-gamecraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition.
MineWorld - MineWorld: A Real-time and Open-source Interactive World Model on Minecraft.
[NeurIPS'24] iVideoGPT - iVideoGPT: Interactive VideoGPTs are Scalable World Models.
[NeurIPS'24] DIAMOND - Diffusion for World Modeling: Visual Details Matter in Atari.
[ICML'24] Genie - Genie: Generative Interactive Environments.
[ICLR'24] UniSim - Learning Interactive Real-World Simulators.

🗃️ Memory Construction

WorldDirector - WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory.
[ECCV'26] MemLearner - MemLearner: Learning to Query Context memory for Video World Models.
Mem-World - Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation
ActWorld - ActWorld: From Explorable to Interactive World Model via Action-Aware Memory.
HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
MosaicMem - MosaicMem: Hybrid Spatial Memory for Controllable Video World Models.
WorldCam - WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation.
[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
RELIC - RELIC: Interactive Video World Model with Long-Horizon Memory.
[ICCV'25] VMem - VMem: Consistent Interactive Video Scene Generation with Surfel-indexed View Memory.
[NeurIPS'25] Spmem - Video World Models with Long-term Spatial Memory.
[NeurIPS'25] VRAG - Video Retrieval Augmented Generation for World Models.
[NeurIPS'25] WorldMem - WorldMem: Long-Term Consistent World Simulation with Memory.

🧱 Explicit 3D Reconstruction

Representative works are listed here.

HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
[CVPR'26] WonderZoom - Wonderzoom: Multi-scale 3D World Generation.
WorldGen - WorldGen: From Text to Traversable and Interactive 3D Worlds.
[ICLR'26] FantasyWorld - FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction.
HY-World 1.0 - HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels.
WonderFree - Wonderfree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration.
[ICCV'25] WonderTurbo - Wonderturbo: Generating Interactive 3D World in 0.72 Seconds.
[ICCV'25] AETHER - Aether: Geometric-Aware Unified World Modeling.
[CVPR'25] WonderWorld - Wonderworld: Interactive 3D Scene Generation from a Single Image.
[CVPR'24] WonderJourney - Wonderjourney: Going from Anywhere to Everywhere.

🧪 Forcing or Noise Augmentation in AR Diffusion Training

HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
[ICML'26] LIVE - LIVE: Long-horizon Interactive Video World Modeling.
Lingbot-World - Advancing Open-source World Models.
[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
[ICLR'26] Astra - Astra: General Interactive World Model with Autoregressive Denoising.
RELIC - RELIC: Interactive Video World Model with Long-Horizon Memory.
Memory Forcing - Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft.
Matrix-Game 2.0 - Matrix-game 2.0: An Open-source Real-time and Streaming Interactive World Model.
Geometric Forcing - Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling
[ICLR'26] Vid2World - Vid2World: Crafting Video Diffusion Models to Interactive World Models.
Oasis - Oasis: A Universe in a Transformer.
[ICLR'25] GameNGen - Diffusion Models are Real-time Game Engines.

⚡ Real-time Interactivity

⚖️ Conflict between Responsiveness vs Coherence

[ICML'26] WorldPlay - WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
[ICLR'26] Astra - Astra: General Interactive World Model with Autoregressive Denoising.
ReSim - ReSim: Reliable World Simulation for Autonomous Driving.
[NeurIPS 2025] FutureSightDrive - Futuresightdrive: Thinking Visually with Spatio-temporal CoT for Autonomous Driving.

🚀 Efficient Rollout

Cycle-World - Cycle-World: Mitigating Error Accumulation in Long-term Video World Models via Reverse-Prediction Cycle Consistency.
ABot-World-0 - ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU.
Causal-rCM - Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models.
Light Interaction - Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models.
minWM - minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models.
SANA-WM - SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer.
HY-World 2.0 - HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds.
Matrix-Game 3.0 - Matrix-game 3.0 Real-time and Streaming Interactive World Model with Long-Horizon Memory.
RELIC - RELIC: Interactive Video World Model with Long-Horizon Memory.
[ICLR'26] MotionStream - MotionStream: Real-Time Video Generation with Interactive Motion Controls.
HY-World 1.0 - HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels.
Yume - Yume: An Interactive World Generation Model.
Hunyuan-GameCraft - Hunyuan-gamecraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition.
MineWorld - MineWorld: A Real-time and Open-source Interactive World Model on Minecraft.

📊 Datasets and Benchmarks

🌍 Open-world Benchmarks

WBench - WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation.
[ICML'26] iWorldBench - iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework.
WorldMark - WorldMark: A Unified Benchmark Suite for Interactive Video World Models.
Omni-WorldBench - Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models.
MIND - MIND: Benchmarking Memory Consistency and Action Control in World Models.
[ICCV'25] WorldScore - WorldScore: A Unified Evaluation Benchmark for World Generation.
[CVPR'25] T2V-CompBench - T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation.
[NeurIPS D&B'24] ChronoMagic-Bench - ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation.
TC-Bench - TC-Bench: Benchmarking Temporal Compositionality in Conditional Video Generation.
[CVPR'24] EvalCrafter - EvalCrafter: Benchmarking and Evaluating Large Video Generation Models.
[CVPR'24] VBench - VBench: Comprehensive Benchmark Suite for Video Generative Models.

The survey also highlights older generic video-generation suites such as FETV, plus WorldScopeDataset as a large-scale training-oriented interactive dataset.

🎮 Game Dataset and Benchmarks

GameCraft-Bench - GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
WBench - WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation.
[ICML'26] iWorldBench - iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework.
WorldMark - WorldMark: A Unified Benchmark Suite for Interactive Video World Models.
GameWorld - Gameworld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents.
WorldCam-50h - 50-hour gameplay dataset introduced with WorldCam.
Sekai-Real-HQ / Sekai - Real-world video data used by Yume and related game-world models.
GF-Minecraft - The Minecraft dataset released with GameFactory.
OGameData - Open-world game video data released with GameGen-X.

The survey also points to Atari 100k, VPT, Matrix-Game-MC, and Sekai-Real-HQ as important training or evaluation resources for interactive game-world research.

🚗 Autonomous Driving Benchmarks

[ICLR'26] DrivingGen - DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving.
ACT-Bench - ACT-Bench: Towards Action Controllable World Models for Autonomous Driving.
[NeurIPS D&B'24] NAVSIM - NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking.
[NeurIPS D&B'24] Bench2Drive - Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving.

The survey further discusses nuScenes, INTERACTION, ReconDreamer-RL, and SimScale as useful evaluation or synthetic-data resources for controllable driving world models.

🤖 Embodied AI Datasets and Benchmarks

WorldArena - WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models.
RoboChallenge - RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies.
RoboTwin 2.0 - RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation.
VLABench - VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks.
CALVIN - CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks.
LIBERO - LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning.

🙏 Acknowledgements

This repository is largely inspired by related community resources:

Name		Name	Last commit message	Last commit date
Latest commit History 225 Commits
assets		assets
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation