ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
By: Jaewook Lee , Filippo Aleotti , Diego Mazala and more
Potential Business Impact:
Creates AR scenes from your spoken ideas.
While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually defined VR environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, the first mobile tool for AI-assisted AR authoring to combine offline scene understanding, fast 3D asset generation, and LLMs -- enabling users to create outdoor scenes through natural language interaction. For example, saying "a dragon enjoying a campfire" (P7) prompts the system to generate and arrange relevant assets, which can then be refined manually. Our technical evaluation shows that our custom pipelines produce more accurate outdoor scene graphs and generate 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI, how users create in freeform use, and design implications for future AR authoring tools. ImaginateAR takes a step toward empowering anyone to create AR experiences anywhere -- simply by speaking their imagination.
Similar Papers
A Vision for AI-Driven Adaptation of Dynamic AR Content to Users and Environments
Human-Computer Interaction
Makes virtual things in games move smartly.
ARbiter: Generating Dialogue Options and Communication Support in Augmented Reality
Human-Computer Interaction
Lets glasses help you talk better.
AdjustAR: AI-Driven In-Situ Adjustment of Site-Specific Augmented Reality Content
Human-Computer Interaction
Keeps virtual objects in place as the real world changes.