Intelligent Experiments Through Real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and Future EIC Detectors

This proposal builds on the successful achievements and experiences the team has acquired in the previous grant period of the Fast-ML project (DE-FOA-0002490, FY21) in designing and implementing a real-time AI and autonomous control for the sPHENIX experiment, as well as on experts' inputs from new collaborators (Jo and Callie). In the next two years, 2024-2025, the team aims to achieve the following goals: 1) deploy the AI heavy flavor trigger and autonomous detector control demonstrator in the upcoming sPHENIX experiment p+p run (2024) to collect high statistic samples of heavy flavor events to address key physics questions in QGP and Cold-QCD to complete the science mission of RHIC program; 2) design and develop a state-of-the-art AI-based readout and control system for the EIC detector(ePIC), providing technical solutions for the ePIC experiment that can both enhance the real-time readout capabilities of the proposed experiment.

With an ever-increasing demand for high-precision data for discovery science and precision measurements, all major high-energy nuclear and particle experiments are facing the challenge of how to deal with the large volume of raw data generated from sophisticated, high-rate detectors. These goals need to be balanced with available hardware and cost limits on DAQ (Data AcQuisition system) bandwidth and offline computing resources to capture, store and process the signal events. Two prototypical examples are the upcoming sPHENIX experiment, the DOE next-generation heavy ion physics experiment at the Relativistic Heavy Ion Collider, and the future EIC experiments that are planned to be online circa 2030 at BNL.

To meet this challenge, we propose to develop a selective streaming readout and control system comprising state-of-the-art AI-based fast data processing and autonomous detector control systems, to effectively sample the full collision rate delivered by the accelerators while maintaining final data throughput for offline storage at a manageable level within available DAQ bandwidth and storage and computing capacity.

This proposal designs and deploys real-time AI-based algorithms operating on high-rate data streams, allowing the identification of important rare physics events from abundant background collisions in the sPHENIX's p+p runs, as well as in the future EIC experiments, such as the EIC day-1 detector ePIC. We will co-design physics-aware high-speed deep neural networks that perform complex event reconstruction and analysis tasks, monitor and calibrate the beam interaction points, and align detectors in real-time.

We aim to ensure the adaptability, high speed and real-time requirements in our proposed online AI algorithms. Notably, we will integrate two complimentary co-processors: Nvidia GPU (training) and FPGA (inference) and use industry open-source tools and our community-developed \hlsfml to translate high-level machine learning and neural networks models into GPU kernels and FPGA bitstreams. Our proposed solutions minimize the complexity of hardware implementations and maximally sample the delivered high luminosity collisions for discovery science. To build an autonomous feedback loop, an anomaly detection algorithm will recognize when detector or beam conditions have changed and thus require a re-optimization to the ML model running online. Demonstrating such a full system integration will be a first step in autonomous control loops of powerful online AI algorithms for large-scale, complex nuclear physics experiments.