StreamDM: Advanced Data Mining in Spark Streaming

Albert Bifet; Silviu Maniu; Jianfeng Qian; Guangjian Tian; Cheng He; Wei Fan

doi:10.1109/ICDMW.2015.140

StreamDM: Advanced Data Mining in Spark Streaming

Albert Bifet
, Silviu Maniu
, Jianfeng Qian
, Guangjian Tian
, Cheng He
, Wei Fan

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Real-time analytics are becoming increasingly important due to the large amount of data that is being created continuously. Drawing from our experiences at Huawei Noah's Ark Lab, we present and demonstrate here StreamDM, a new open source data mining and machine learning library, designed on top of Spark Streaming, an extension of the core Spark API that enables scalable stream processing of data streams. StreamDM is designed to be easily extended and used, either practitioners, developers, or researchers, and is the first library to contain advanced stream mining algorithms for Spark Streaming.

Original language	English
Title of host publication	Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015
Editors	Xindong Wu, Alexander Tuzhilin, Hui Xiong, Jennifer G. Dy, Charu Aggarwal, Zhi-Hua Zhou, Peng Cui
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1608-1611
Number of pages	4
ISBN (Electronic)	9781467384926
DOIs	https://doi.org/10.1109/ICDMW.2015.140
Publication status	Published - 29 Jan 2016
Event	15th IEEE International Conference on Data Mining Workshop, ICDMW 2015 - Atlantic City, United States Duration: 14 Nov 2015 → 17 Nov 2015

Publication series

Name	Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015

Conference

Conference	15th IEEE International Conference on Data Mining Workshop, ICDMW 2015
Country/Territory	United States
City	Atlantic City
Period	14/11/15 → 17/11/15

Keywords

Software
Spark Streaming
open-source
stream mining

Access to Document

10.1109/ICDMW.2015.140

Cite this

Bifet, A., Maniu, S., Qian, J., Tian, G., He, C., & Fan, W. (2016). StreamDM: Advanced Data Mining in Spark Streaming. In X. Wu, A. Tuzhilin, H. Xiong, J. G. Dy, C. Aggarwal, Z.-H. Zhou, & P. Cui (Eds.), Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015 (pp. 1608-1611). Article 7395869 (Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDMW.2015.140

Bifet, Albert ; Maniu, Silviu ; Qian, Jianfeng et al. / StreamDM : Advanced Data Mining in Spark Streaming. Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015. editor / Xindong Wu ; Alexander Tuzhilin ; Hui Xiong ; Jennifer G. Dy ; Charu Aggarwal ; Zhi-Hua Zhou ; Peng Cui. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 1608-1611 (Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015).

@inproceedings{e0e8e58ca2114b06adb638de3ba022cd,

title = "StreamDM: Advanced Data Mining in Spark Streaming",

abstract = "Real-time analytics are becoming increasingly important due to the large amount of data that is being created continuously. Drawing from our experiences at Huawei Noah's Ark Lab, we present and demonstrate here StreamDM, a new open source data mining and machine learning library, designed on top of Spark Streaming, an extension of the core Spark API that enables scalable stream processing of data streams. StreamDM is designed to be easily extended and used, either practitioners, developers, or researchers, and is the first library to contain advanced stream mining algorithms for Spark Streaming.",

keywords = "Software, Spark Streaming, open-source, stream mining",

author = "Albert Bifet and Silviu Maniu and Jianfeng Qian and Guangjian Tian and Cheng He and Wei Fan",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015 ; Conference date: 14-11-2015 Through 17-11-2015",

year = "2016",

month = jan,

day = "29",

doi = "10.1109/ICDMW.2015.140",

language = "English",

series = "Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1608--1611",

editor = "Xindong Wu and Alexander Tuzhilin and Hui Xiong and Dy, \{Jennifer G.\} and Charu Aggarwal and Zhi-Hua Zhou and Peng Cui",

booktitle = "Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015",

}

Bifet, A, Maniu, S, Qian, J, Tian, G, He, C & Fan, W 2016, StreamDM: Advanced Data Mining in Spark Streaming. in X Wu, A Tuzhilin, H Xiong, JG Dy, C Aggarwal, Z-H Zhou & P Cui (eds), Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015., 7395869, Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015, Institute of Electrical and Electronics Engineers Inc., pp. 1608-1611, 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015, Atlantic City, United States, 14/11/15. https://doi.org/10.1109/ICDMW.2015.140

StreamDM: Advanced Data Mining in Spark Streaming. / Bifet, Albert; Maniu, Silviu; Qian, Jianfeng et al.
Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015. ed. / Xindong Wu; Alexander Tuzhilin; Hui Xiong; Jennifer G. Dy; Charu Aggarwal; Zhi-Hua Zhou; Peng Cui. Institute of Electrical and Electronics Engineers Inc., 2016. p. 1608-1611 7395869 (Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - StreamDM

T2 - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015

AU - Bifet, Albert

AU - Maniu, Silviu

AU - Qian, Jianfeng

AU - Tian, Guangjian

AU - He, Cheng

AU - Fan, Wei

PY - 2016/1/29

Y1 - 2016/1/29

N2 - Real-time analytics are becoming increasingly important due to the large amount of data that is being created continuously. Drawing from our experiences at Huawei Noah's Ark Lab, we present and demonstrate here StreamDM, a new open source data mining and machine learning library, designed on top of Spark Streaming, an extension of the core Spark API that enables scalable stream processing of data streams. StreamDM is designed to be easily extended and used, either practitioners, developers, or researchers, and is the first library to contain advanced stream mining algorithms for Spark Streaming.

AB - Real-time analytics are becoming increasingly important due to the large amount of data that is being created continuously. Drawing from our experiences at Huawei Noah's Ark Lab, we present and demonstrate here StreamDM, a new open source data mining and machine learning library, designed on top of Spark Streaming, an extension of the core Spark API that enables scalable stream processing of data streams. StreamDM is designed to be easily extended and used, either practitioners, developers, or researchers, and is the first library to contain advanced stream mining algorithms for Spark Streaming.

KW - Software

KW - Spark Streaming

KW - open-source

KW - stream mining

UR - https://www.scopus.com/pages/publications/84964789930

U2 - 10.1109/ICDMW.2015.140

DO - 10.1109/ICDMW.2015.140

M3 - Conference contribution

AN - SCOPUS:84964789930

T3 - Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015

SP - 1608

EP - 1611

BT - Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015

A2 - Wu, Xindong

A2 - Tuzhilin, Alexander

A2 - Xiong, Hui

A2 - Dy, Jennifer G.

A2 - Aggarwal, Charu

A2 - Zhou, Zhi-Hua

A2 - Cui, Peng

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 14 November 2015 through 17 November 2015

ER -

Bifet A, Maniu S, Qian J, Tian G, He C, Fan W. StreamDM: Advanced Data Mining in Spark Streaming. In Wu X, Tuzhilin A, Xiong H, Dy JG, Aggarwal C, Zhou ZH, Cui P, editors, Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015. Institute of Electrical and Electronics Engineers Inc. 2016. p. 1608-1611. 7395869. (Proceedings - 15th IEEE International Conference on Data Mining Workshop, ICDMW 2015). doi: 10.1109/ICDMW.2015.140

StreamDM: Advanced Data Mining in Spark Streaming

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this