TILOS Seminar: What makes a good world model?
Nicklas Hansen, New York University
Abstract: World models are quickly becoming a central ingredient in systems for decision-making. However, several fundamental questions remain largely unresolved: what’s the right learning objective, what’s the right training data, and how do we evaluate the quality of a world model? In this talk, I will present a line of work that approaches these questions from a data-centric perspective. I will first cover TD-MPC2, a highly data-driven approach to world modeling that learns a latent predictive model optimized for decision-making rather than image generation, and show how it enables robots to acquire new skills directly in the real world. Second, I will turn to data and evaluation, exploring “hallucination” in world models: producing rollouts that remain visually fluent yet are inconsistent with the true dynamics. Using a new 427-hour dataset for visual world modeling, I will show that hallucination is, first and foremost, a data coverage problem, and discuss both data- and model-centric mitigation strategies. Finally, I will conclude the talk with an overview of open problems for the field.
Nicklas Hansen is a postdoctoral researcher at NYU Courant working with Lerrel Pinto and Yann LeCun. Nick’s research spans world models, reinforcement learning, and robot learning, with the goal of developing generalist AI agents that learn from interaction. They completed their PhD at UC San Diego advised by Xiaolong Wang and Hao Su, and have also spent time at NVIDIA Research, Meta AI (FAIR), and UC Berkeley. Nick received their BS and MS degrees from the Technical University of Denmark, and is a recipient of the 2024 NVIDIA Graduate Fellowship. Webpage: www.nicklashansen.com