CS Colloquium - James Tompkin, Brown University

Abstract:

Understanding the dynamic 3D world from images is a foundational pursuit in computer vision with a rich literature stretching back decades. One perspective frames this problem as *measurement*: photogrammetry triangulates features across multiple views to reconstruct 3D geometry, and these techniques underpin modern mapping and VFX even though they can be brittle. Via advances in deep learning, the rise of differentiable image formation provided new optimization-based methods like Neural Radiance Fields (NeRFs) and Gaussian Splatting that use self-supervised learning to create views that are photorealistically consistent with the input images but might not be geometrically accurate. A more recent trend and an ideological shift is toward *prediction*. Large neural network models like DUST3R or VGGT use large datasets to directly predict camera parameters and geometry from just a few images in a single forward pass, with even stronger priors predicting hidden parts of a scene. Such models may provide the basis for real-world 3D reasoning for AI applications like scene analysis and robotics.

This talk will provide an overview of the interplay between measurement and prediction in dynamic 3D reconstruction. We will see shiny state-of-the-art results to demonstrate the progress that the visual computing community has made in this pursuit. We will explore the reasons why classical methods are brittle to help explain why modern neural approaches still "hallucinate" scenes even in visible areas. And, we will consider the task-specific trade-offs we must make to find balance between measuring the world as it is and predicting the world as we've seen it before. Finally, if you'd like some hot takes, you can also ask me about world models in AI.

Bio:

James Tompkin is an Associate Professor of Computer Science at Brown University. His research at the intersection of computer vision, computer graphics, and human-computer interaction helps develop new visual computing tools and experiences from cameras. For this, his lab creates techniques for 3D scene reconstruction from multi-camera systems and for dynamics. His doctoral work at University College London on large-scale video processing and exploration techniques led to creative exhibition work in the Museum of the Moving Image in New York City. Postdoctoral work at Max-Planck-Institute for Informatics and Harvard University helped create new methods to edit content within images and videos. Recent research has developed new techniques for low-level reconstruction of dynamic scenes, view synthesis for VR, and AI content editing and generation.