A Study on Messaging Trade-offs in Data Streaming for Scientific Workflows

Memory-to-memory data streaming is essential for modern scientific workflows that require near real-time data analysis, experimental steering, and informed decision-making during experiment execution. It eliminates the latency bottlenecks associated with file-based transfers to parallel storage, enabling rapid data movement between experimental facilities and HPC systems. These tightly coupled experimental-HPC workflows demand low latency, high throughput, and reliable data delivery to support on-the-fly analysis and timely feedback for experimental control. Off-the-shelf messaging frameworks are increasingly considered viable solutions for enabling such direct memory streaming due to their maturity, broad adoption, and ability to abstract core messaging and reliability functionalities from the application layer. However, effectively meeting the workflows' requirements depends on utilizing the framework's capabilities and carefully tuning its configurations. In this paper, we present a study that investigates the messaging parameters, and their configuration choices that impact the streaming requirements of two representative scientific workflows. We specifically characterize throughput trade-offs associated with reliable message transmission for these workflows. Our study is conducted through streaming simulations using synthetic workloads derived from the Deleria and LCLS workflows, employing the RabbitMQ messaging framework within the context of the Data Streaming to HPC infrastructure at OLCF. Our simulations reveal several key observations and practical insights that help users understand which configurations best meet the needs of their streaming workloads.