Beyond growing data volumes, researchers are now having to deal with increasing data velocity. For example, instruments are generating data faster than can be consumed by downstream processes and AI/ML-guided research is relying on near real-time feedback for optimizing experiments. Such data streaming applications require different tools for data management and computation than those used for traditional file-based data.
Building on the established and widely used Globus file transfer service, we present newly released capabilities that enable researchers to stream data securely across wide area networks in support of real-time data processing. We will provide an overview of the Globus data streaming service and describe how to incorporate it into your research applications.
