TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech
By: Weiyan Shi , Sunaya Upadhyay , Geraldine Quek and more
Potential Business Impact:
Draw and talk to create design ideas faster.
Sketching is a widely used medium for generating and exploring early-stage design concepts. While generative AI (GenAI) chatbots are increasingly used for idea generation, designers often struggle to craft effective prompts and find it difficult to express evolving visual concepts through text alone. In the formative study (N=6), we examined how designers use GenAI during ideation, revealing that text-based prompting disrupts creative flow. To address these issues, we developed TalkSketch, an embedded multimodal AI sketching system that integrates freehand drawing with real-time speech input. TalkSketch aims to support a more fluid ideation process through capturing verbal descriptions during sketching and generating context-aware AI responses. Our work highlights the potential of GenAI tools to engage the design process itself rather than focusing on output.
Similar Papers
TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech
Human-Computer Interaction
Draw and talk to create design ideas faster.
SketchConcept: Sketching-based Concept Recomposition for Product Design using Generative AI
Human-Computer Interaction
Helps designers create products with drawings and words.
Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI
Human-Computer Interaction
Lets game makers draw characters without being artists.