PVG Seminar Series

Abstract

We all know video generative models still struggle with the physics of a scene. Objects can multiply, deform, and float around. We are still waiting for video generation to have the same moment that image generation reached, and chances are that with scale, we will get close sometime soon (~a year?). But where would that leave us academics with limited data and compute? In this talk, I will discuss some works I have done on benchmarking, understanding, and generating physics in video models, and share my perspective on where academia’s focus will be most worthwhile as companies scale bigger and bigger models.

Bio

Saman Motamed is a 4th-year PhD candidate at INSAIT, Sofia University, where he is advised by Prof. Luc Van Gool and co-advised by Dr. Iro Laina at the Visual Geometry Group (VGG), University of Oxford. His research focuses primarily on video generation and understanding. He has worked on similar topics as a student researcher at Google DeepMind and a machine learning research intern at Netflix.

Abstract

Video generation models are getting impressively realistic, but the worlds they produce are often geometrically inconsistent and forgetful. A camera that pans away and comes back sees a different room, and objects drift over time because the model has no real sense of 3D space or lasting memory. My topic today is building video world models that respect 3D structure and remember what they have already seen, so they can serve as reliable interactive world simulators. In this talk I will share our two recent works toward this goal, World-R1 and Latent Spatial Memory. World-R1 aligns text-to-video generation with 3D constraints through reinforcement learning, improving geometric consistency while preserving visual quality and motion diversity. Latent Spatial Memory then stores persistent 3D scene content directly as latent tokens, giving the model an efficient and spatially consistent memory that keeps scenes stable across long rollouts. I will close with our ongoing project on agentic video world modeling, including some early demos, which I hope can be a good starting point for our discussion.

Bio

Weijie Wang received the B.E. degree with Honors from the Chu Kochen Honors College, Zhejiang University. He is currently a first-year Ph.D. student in Computer Science at Zhejiang University, advised by Prof. Bohan Zhuang, and a Research Intern with ByteDance Seed. He was previously a Research Intern with Microsoft Research Asia. His research interests lie in efficient large 3D models and video world models. He is also the first author of TriSplat, VolSplat and ZPressor.

Abstract

3D reconstruction and generation have come a long way, but the 3D they produce is often hard to actually use. A point cloud, the 3D Gaussians, or a freshly generated asset is not something a robot can stand on, or a physics engine can read. My broader goal here is simple: build 3D that is efficient to create and ready to simulate, so it can serve as a substrate for embodied AI. In this talk I will share our two recent works toward this goal, Depth Anything 3 and TriSplat. Depth Anything 3 is a foundation model that recovers geometry from any number of views, with a minimal design of just depth and rays on a plain Transformer. TriSplat then builds on this kind of geometry prior to predict oriented triangles, so a single feed-forward pass exports a simulation-ready texture mesh in about one second, with no post-processing. I will close with our ongoing project on agentic, simulation-ready 3D scene generation, including some early demos, which I hope can be a good starting point for our discussion.

Bio

[Donny Y. Chen](https://donydchen.github.io/) received the B.Eng. and M.Eng. degrees from Sun Yat-sen University, and the Ph.D. degree from Monash University. He was previously a Research Assistant with Nanyang Technological University. He is currently a Research Scientist with ByteDance Seed, Singapore, where he develops 3D world models. His research interests include 3D generation, feed-forward novel view synthesis, and multi-view geometry. He is a lead author of MVSplat (ECCV 2024 Oral) and Depth Anything 3 (ICLR 2026 Oral).

Abstract

Training state-of-the-art latent diffusion models requires complex staging: a tokenizer must first be trained before a diffusion model can operate in its frozen latent space. We propose UNITE — an architecture for unified tokenization and latent diffusion. A single Generative Encoder serves as both image tokenizer and latent generator via weight sharing, trained in a single stage that jointly optimizes both tasks. UNITE learns a common latent language for tokenization and generation.

Bio

Xingjian Bai (https://xingjianbai.com/) is a second-year PhD student at MIT. He received his master's and bachelor's degree in Mathematics and Computer Science from the University of Oxford. His research focuses on generative models in Computer Vision.

Abstract

In this talk, I will discuss our progress in making 3D a first-class citizen in AI. I will argue that understanding 3D structures is a prerequisite for understanding, acting, and creating in the physical world. I will then explore how a new generation of 3D foundation models has changed the way we approach 3D reconstruction. I will show that these ideas extend to novel-view synthesis as well, benefiting from transformers trained for 3D reconstruction. Next, I will consider 4D reconstruction, where the goal is to recover 3D shape and motion from videos, and demonstrate how transformers can be further extended to this task via dynamic point maps. I will then examine the role of scale in learning these models, introducing VGGT-Omega, a new, more efficient architecture trained on 15× more data than our previous model. I will discuss the importance and challenges of 3D data engineering and demonstrate significant performance improvements on benchmarks. Finally, I will address the problem of 3D generation for content creation, simulation, and design. I will emphasize that generating 3D shape alone is insufficient, and discuss approaches to compositional and articulated generation as well.

Bio

Andrea Vedaldi is Professor of Computer Vision and Machine Learning at the University of Oxford, where he co-leads the Visual Geometry Group since 2012. He is also a research scientist and technical lead at Meta. Andrea is a Fellow of the Royal Academy of Engineering (FREng) and a Royal Society’s Faraday Discovery Fellow. He researches generative AI in computer vision, applied to the generation of 3D content from text and images and to image understanding. He is the author of more than 240 peer-reviewed publications in computer vision and machine learning. He is the recipient of the IEEE Thomas Huang Memorial Prize, the IEEE Mark Everingham Prize, and the Test of Time Award by the ACM, and two best paper awards from the Conference on Computer Vision and Pattern Recognition. He is the recipient of the ERC Starting and Consolidator Grants and co-I in two EPSRC Programme Grants.

Abstract

In this talk I will give an overview of our recent V-DPM: 4D Video Reconstruction with Dynamics Point Maps. In this work we tackle the goal of 4D feed-forward reconstruction with an emphasis on versatility, the ability to work in diverse in-the-wild videos. I will go through the evolution of designs to reach this point, starting with the original version on Dynamic Point Maps which built on top of DUSt3R and processed pairs of frames. I will motivate why we want and end-to-end model for 4D reconstructions compared to decoupled designs common in 3D trackers. I will then describe how we build on top of a pre-trained model for static scenes VGGT, to adapt it to 4D reconstructions with a training recipe that does not need massive data, showing some of the early designs that failed. Finally will show results highlighting the versatility of the method but also the current limitations for future work.

Bio

Edgar Sucar is a postdoctoral research fellow at the University of Oxford in the Visual Geometry Group (VGG). His research lies on developing 3D/4D perception systems with an emphasis on developing representations that enable robots to have intelligent interactions with their environment. He completed his PhD at Imperial College with Prof. Andrew Davison, where he developed innovative SLAM algorithms such as NodeSLAM, iMAP, and iLabel. iMAP pioneered the use of neural fields as an underlying representation for SLAM with over 900+ citations. Currently his research has focused on developing methods for understanding dynamics in videos with 4D reconstruction. He also worked at the robotics startup SLAMCore where he helped implement core algorithms for industry robotics.