Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models
By: Linhao Zhong , Linyu Wu , Bozhen Fang and more
Potential Business Impact:
Lets computers write better by fixing mistakes early.
Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.
Similar Papers
Soft-Masked Diffusion Language Models
Machine Learning (CS)
Helps computers write better code, faster.
Diffusion Language Models are Provably Optimal Parallel Samplers
Machine Learning (CS)
Lets computers write faster and better.
Learning Unmasking Policies for Diffusion Language Models
Machine Learning (CS)
Teaches computers to write better and faster.