StreamFlow: A System for Summarizing and Learning Over Industrial Big Data Streams

The growing need for predictive analytics over streaming data in the industry requires a flexible and continuously scalable big data system. In real-time big data applications (cybersecurity, AIOps, anomaly detection, predictive maintenance, IoT etc.), efficient machine learning models must be trained and industrialized within existing data processing plat-forms and industrial tools. This requires interoperability between various components: data collection, processing, summarization, modelling and analytics. Existing works focus on building AI models for big data, neglecting real-world challenges when integrating such models into an existing industrial production framework. In this paper, we propose StreamFlow, an operational data pipeline to address industrial challenges for continuous learning over big data streams. We also propose an online method using sliding windows to summarize high-velocity data. The final result of the framework is a feature vector that describes the underlying processes and is ready to use in machine learning tasks. Moreover, we showcase real-world applications such as automated feature engineering for real-time monitoring and online machine learning for event classification. The proposed system has been deployed within production in a banking system, processing billions of daily traffic operations. Our experiments demonstrate the effectiveness and performance of our approach by evaluating it at different levels: processing, summarization, improvement of machine learning performance and effectiveness in an industrial setting. In the case of downstream machine learning tasks, using summarized data generated by StreamFlow results in up to 2 orders of magnitude speedups in training time without compromising predictive performance.