🌍 Towards fied orld odels for Visual Navigation via Memory-Augmented Planning and Foresight

¹UW, ²NUS, ³Apple, ⁴Microsoft Research, ⁵CMU

👁 Image
👁 Image
👁 Image
👁 Image

👁 task

UniWM introduce a unified, memory-augmented world model paradigm integrating egocentric visual foresight and planning within a single multimodal autoregressive backbone. Unlike modular frameworks, UniWM explicitly grounds action decisions in visually imagined outcomes, ensuring tight alignment between visualization and planning. A hierarchical memory mechanism further integrates detailed short-term perceptual cues with longer-term trajectory context, enabling stable, coherent reasoning over extended horizons.

You are also welcome to explore our previous work, including GOViG, which introduces a new task that we leverage multimodal LLM reasoning to generate navigation instructions directly from egocentric visual observations of the initial and goal states and HA-VLN, where we introduce HA-VLN 2.0, a unified benchmark coupling discrete (DE) and continuous (CE) navigation paradigms with explicit social-awareness constraints.

Data

We host the UniWM dataset on Hugging Face: fly1113/UniWM_Dataset.

• go_stanford, recon, sacson, scand used for both training and evaluation.
• tartandrive reserved for unseen evaluation only.
• 1XHumanoid reserved for humanoid navigation.

The directory structure will look like:

data/
├── go_stanford/
│ ├── traj_0000/
│ │ ├── 0.jpg
│ │ ├── 1.jpg
│ │ ├── ...
│ │ ├── n.jpg
│ │ └── traj_data.pkl
│ ├── traj_0001/
│ └── ...
└── ...

Each traj_xxxx/ folder contains a sequence of egocentric frames (0.jpg, 1.jpg, ..., n.jpg) and a traj_data.pkl file storing the per-step metadata (e.g., actions, poses) for that trajectory. The other splits follow the same layout.

Contributing

We welcome contributions to this project! Please contact yfeidong@uw.edu or fyiwu@uw.edu.

Acknowledgement

We would like to thank ReCon, Go Stanford, SACSon, SCAND and 1XHumanoid for their publicly available datasets.

🌟 Citation

If you find this repository or our paper useful, please consider starring this repository and citing our paper:

@misc{dong2026unifiedworldmodelsvisual,
 title={Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight}, 
 author={Yifei Dong and Fengyi Wu and Guangyu Chen and Lingdong Kong and Xu Zhu and Qiyu Hu and Yuxuan Zhou and Jingdong Sun and Jun-Yan He and Qi Dai and Alexander G. Hauptmann and Zhi-Qi Cheng},
 year={2026},
 eprint={2510.08713},
 archivePrefix={arXiv},
 primaryClass={cs.AI},
 url={https://arxiv.org/abs/2510.08713}, 
}