Towards Efficient and Practical GPU Multitasking in the Era of LLM
By: Jiarong Xing , Yifan Qiao , Simon Mo and more
Potential Business Impact:
Lets computers do many jobs at once.
GPU singletasking is becoming increasingly inefficient and unsustainable as hardware capabilities grow and workloads diversify. We are now at an inflection point where GPUs must embrace multitasking, much like CPUs did decades ago, to meet the demands of modern AI workloads. In this work, we highlight the key requirements for GPU multitasking, examine prior efforts, and discuss why they fall short. To advance toward efficient and practical GPU multitasking, we envision a resource management layer, analogous to a CPU operating system, to handle various aspects of GPU resource management and sharing. We outline the challenges and potential solutions, and hope this paper inspires broader community efforts to build the next-generation GPU compute paradigm grounded in multitasking.
Similar Papers
RTGPU: Real-Time Computing with Graphics Processing Units
Hardware Architecture
Makes computers do hard jobs on time.
LithOS: An Operating System for Efficient Machine Learning on GPUs
Operating Systems
Makes computer chips work better and save power.
Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
Distributed, Parallel, and Cluster Computing
Lets computers finish big jobs faster, cheaper.