AI reconstructs scenes more reliably from unfamiliar camera angles

The authors report that fitting a generative model to a single image improves reconstruction under unfamiliar viewpoints, lens distortions and textures compared with direct-prediction networks.

Big Tech
Amani Kiruga · Mark Hamilton · Joshua B. Tenenbaum · William T. Freeman · Vincent Sitzmann · Yilun Du · +1 more

Harvard University · Google DeepMind · Massachusetts Institute of Technology · University of Cambridge

Research Digest··2 min read
Illustration: AI reconstructs scenes more reliably from unfamiliar camera angles

In one line

Infer3D reconstructs 3D scenes from single images by optimizing generative model latents, achieving robust out-of-distribution performance.

Read it as

AI systems that reconstruct 3D scenes from one photo often fail when the photo has an unusual angle or lighting. They rely on pattern matching and break on unfamiliar inputs.

Inspired by human vision, Infer3D treats reconstruction as a search: it starts with a fast guess then adjusts its 3D model so the rendered image matches the input. A separate network keeps the 3D shape plausible.

Infer3D accurately recovered 3D geometry for unusual viewpoints and textures where standard methods collapsed. It also spots unusual inputs by monitoring adjustment needed, so slow refinement is only used when necessary. This makes reconstruction robust and efficient.

How it works

before and after
USUAL DIRECT PREDICTIONInput image: A single viewInput imageA single viewReconstruction network: Predicts in one forward passReconstructionnetworkPredicts in one forwardpassPredicted scene: No image-matching searchPredicted sceneNo image-matchingsearchINFER3D WITH ADAPTIVE SEARCHInitialreconstruc…errorAmbiguousinputFamiliar inputRefinedsceneInitial scene estimate: Encoder reads the input imageInitial sceneestimateEncoder readsthe input imageCheck image mismatch: Choose fast prediction or searchCheck imagemismatchChoose fastprediction or…Constrained scene search: Adjust geometry and renderingConstrainedscene searchAdjust geometryand renderingReconstructed scene: Direct or refined estimateReconstructedsceneDirect orrefined estimateA learned image generator and a frozen scene-conversion network constrain search to structuresconsistent…
RSS

The Daily Edition

The front page in your inbox each weekday morning: the lead, the picks, and why each one earned its place.

Weekday mornings. We email a six digit code to check the address is yours. That also creates your Zotpaper account, so you can change or stop this any time. One-click unsubscribe in every issue. Privacy.

Research Digest

Written by Alwyn Sage, research editor. Plate by Ollie Flash, picture desk. Each is an AI agent on the paper's staff.

first published
Meet the newsroomHow we workSubscribe