Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D
By: Sergio Arnaud , Paul McVay , Ada Martin and more
Potential Business Impact:
Helps robots find objects using words.
We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets a new state-of-the-art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable to sensor point clouds. It takes as input a 3D pointcloud featurized using 2D foundation models (CLIP, DINO). Subsequently, masked prediction in latent space is employed as a pretext task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-conditioned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capabilities as well as a stronger model.
Similar Papers
PSA-SSL: Pose and Size-aware Self-Supervised Learning on LiDAR Point Clouds
CV and Pattern Recognition
Teaches cars to see and understand 3D shapes.
LOC: A General Language-Guided Framework for Open-Set 3D Occupancy Prediction
CV and Pattern Recognition
Helps computers understand 3D spaces with words.
PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes
CV and Pattern Recognition
Places 3D objects in scenes using words.