Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
By: Yuerong Song , Xiaoran Liu , Ruixiao Li and more
Potential Business Impact:
Makes AI models remember more without using more memory.
Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding but suffer from prohibitive quadratic computational complexity and memory overhead during inference. Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage that limit long-context applications. Our analysis of attention patterns in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining salient across decoding steps and low-relevance tokens staying unimportant, motivating selective cache eviction. We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention via delayed bidirectional sparse caching. By leveraging the stability of token saliency over steps, it retains critical tokens and dynamically evicts unimportant prefix/suffix entries using an attention-guided strategy. Extensive experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to 10$\times$ higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs, outperforming previous methods in efficiency and effectiveness.
Similar Papers
dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
Machine Learning (CS)
Makes AI text generators work much faster.
Accelerating Diffusion LLM Inference via Local Determinism Propagation
Computation and Language
Makes AI write faster without losing quality.
Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibration
Machine Learning (CS)
Makes AI write and create much faster.