Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
By: Yue Guan , Changming Yu , Shihan Fang and more
Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to $3.98\times$ speedup over state-of-the-art baselines across multiple hardware setups.
Similar Papers
Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
Distributed, Parallel, and Cluster Computing
Makes AI talk faster when shared.
Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
Computation and Language
Makes AI talk and write much faster.
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
Distributed, Parallel, and Cluster Computing
Makes AI answer faster by guessing better.