DMA Collectives for Efficient ML Communication Offloads

Offloading machine learning (ML) communication collectives to direct memory access (DMA) engines has emerged as an interesting and low-cost solution to efficiently overlap computation and communication in inference and training. Doing so delivers superior concurrent performance by freeing up all GPU cores for computation and also lowers interference in the memory sub-system (caches). While DMA collectives show strong promise, prior works have only studied them in limited context (bandwidth-bound transfer sizes only, performance-only). To address this, we provide a comprehensive performance, power/energy and synchronization costs analysis of offloading ML communication collectives (all-gather, all-to-all) to DMA engines on state-of-the-art AMD Instinct MI300X GPUs. Our analysis reveals that, compared to the state-of-the-art RCCL communication collectives library, DMA collectives are at-par or better for large sizes (10s of MB to GB) in terms of both performance (16% better) and power (32% better). However, they significantly lag for latency-bound small sizes; 4.5X and 2.5X slower for all-gather and all-to-all, respectively. We provide a detailed latency breakdown of a DMA transfer and identify that DMA command scheduling and synchronization costs can limit DMA collective performance. To tackle this, we harness existing DMA architecture innovations, hitherto untapped, to build optimized DMA collectives and demonstrate their efficacy on real hardware. Our optimized implementations considerably close the performance gap for DMA collectives at smaller sizes (30% slower and 20% faster all-gather and all-to-all, respectively) and further improves performance (by 7%) and power savings at larger sizes (3-10%). Overall, this work represents a significant step toward making DMA collectives suitable for adoption in mainstream collective libraries.