Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation
By: Zhenyu Liu , Yunzhen Liu , Zehao Fan and more
Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput.
Similar Papers
BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
Machine Learning (CS)
Lets AI learn more without needing more computer memory.
Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
Machine Learning (CS)
Makes AI faster by sharing computer brain parts.
MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
Machine Learning (CS)
Makes smart AI run faster on less powerful computers.