Gemma 3 Technical Report
By: Gemma Team , Aishwarya Kamath , Johan Ferret and more
Potential Business Impact:
Lets computers understand pictures and many languages.
We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
Similar Papers
EmbeddingGemma: Powerful and Lightweight Text Representations
Computation and Language
Helps computers understand words better, faster, and cheaper.
T5Gemma 2: Seeing, Reading, and Understanding Longer
Computation and Language
Helps computers understand pictures and many languages.
Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction
Machine Learning (CS)
Predicts wildfires better using AI's "memory."