Words That Make Language Models Perceive
By: Sophie L. Wang, Phillip Isola, Brian Cheung
Potential Business Impact:
Makes text-only AI "see" and "hear" with words.
Large language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by multimodal regularities encoded in language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational alignment with specialist vision and audio encoders. When a sensory prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.
Similar Papers
Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
Machine Learning (CS)
Computers learn to "see" from reading words.
Exploring Multimodal Prompt for Visualization Authoring with Large Language Models
Human-Computer Interaction
Draw pictures to help computers make charts.
The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance
Artificial Intelligence
Teaches AI to understand pictures and words better.