Challenges and Research Directions for Large Language Model Inference Hardware
By: Xiaoyu Ma, David Patterson
Potential Business Impact:
Makes AI think faster by improving its memory.
Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication. While our focus is datacenter AI, we also review their applicability for mobile devices.
Similar Papers
Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need
Hardware Architecture
Makes AI understand questions faster and cheaper.
System-performance and cost modeling of Large Language Model training and inference
Hardware Architecture
Makes big AI models train and run cheaper.
AI Accelerators for Large Language Model Inference: Architecture Analysis and Scaling Strategies
Hardware Architecture
Finds best computer chips for AI tasks.