Token Homogenization under Positional Bias
By: Viacheslav Yusupov , Danil Maksimov , Ameliia Alaeva and more
Potential Business Impact:
Makes AI understand words better by fixing how it sees them.
This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms.
Similar Papers
Token Sampling Uncertainty Does Not Explain Homogeneity Bias in Large Language Models
Computation and Language
Fixes AI to treat all people fairly.
Layer-wise Positional Bias in Short-Context Language Modeling
Computation and Language
Shows how computer language learning changes over time.
Training Language Models with homotokens Leads to Delayed Overfitting
Computation and Language
Makes AI understand words better, even with different spellings.