Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes

Maxime Lanvin; Pierre François Gimenez; Yufei Han; Frédéric Majorczyk; Ludovic Mé; Éric Totel

doi:10.1007/978-3-031-31108-6_2

Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes

Maxime Lanvin
, Pierre François Gimenez
, Yufei Han
, Frédéric Majorczyk
, Ludovic Mé
, Éric Totel

IRISA

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Among the difficulties encountered in building datasets to evaluate intrusion detection tools, a tricky part is the process of labelling the events into malicious and benign classes. The labelling correctness is paramount for the quality of the evaluation of intrusion detection systems but is often considered as the ground truth by practitioners and is rarely verified. Another difficulty lies in the correct capture of the network packets. If it is not the case, the characteristics of the network flows generated from the capture could be modified and lead to false results. In this paper, we present several flaws we identified in the labelling of the CICIDS2017 dataset and in the traffic capture, such as packet misorder, packet duplication and attack that were performed but not correctly labelled. Finally, we assess the impact of these different corrections on the evaluation of supervised intrusion detection approaches.

Original language	English
Title of host publication	Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers
Editors	Slim Kallel, Mohamed Jmaiel, Ahmed Hadj Kacem, Mohammad Zulkernine, Frédéric Cuppens, Nora Cuppens
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	18-33
Number of pages	16
ISBN (Print)	9783031311079
DOIs	https://doi.org/10.1007/978-3-031-31108-6_2
Publication status	Published - 1 Jan 2023
Event	17th International Conference on Risks and Security of Internet and Systems, CRiSIS 2022 - Sousse, Tunisia Duration: 7 Dec 2022 → 9 Dec 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13857 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th International Conference on Risks and Security of Internet and Systems, CRiSIS 2022
Country/Territory	Tunisia
City	Sousse
Period	7/12/22 → 9/12/22

Keywords

dataset labelling
intrusion detection
machine learning

Access to Document

10.1007/978-3-031-31108-6_2

Cite this

Lanvin, M., Gimenez, P. F., Han, Y., Majorczyk, F., Mé, L., & Totel, É. (2023). Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. In S. Kallel, M. Jmaiel, A. Hadj Kacem, M. Zulkernine, F. Cuppens, & N. Cuppens (Eds.), Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers (pp. 18-33). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13857 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-31108-6_2

Lanvin, Maxime ; Gimenez, Pierre François ; Han, Yufei et al. / Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers. editor / Slim Kallel ; Mohamed Jmaiel ; Ahmed Hadj Kacem ; Mohammad Zulkernine ; Frédéric Cuppens ; Nora Cuppens. Springer Science and Business Media Deutschland GmbH, 2023. pp. 18-33 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0ebd3134b6984768bd1f0df8a9e0cb23,

title = "Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes",

abstract = "Among the difficulties encountered in building datasets to evaluate intrusion detection tools, a tricky part is the process of labelling the events into malicious and benign classes. The labelling correctness is paramount for the quality of the evaluation of intrusion detection systems but is often considered as the ground truth by practitioners and is rarely verified. Another difficulty lies in the correct capture of the network packets. If it is not the case, the characteristics of the network flows generated from the capture could be modified and lead to false results. In this paper, we present several flaws we identified in the labelling of the CICIDS2017 dataset and in the traffic capture, such as packet misorder, packet duplication and attack that were performed but not correctly labelled. Finally, we assess the impact of these different corrections on the evaluation of supervised intrusion detection approaches.",

keywords = "dataset labelling, intrusion detection, machine learning",

author = "Maxime Lanvin and Gimenez, \{Pierre Fran{\c c}ois\} and Yufei Han and Fr{\'e}d{\'e}ric Majorczyk and Ludovic M{\'e} and {\'E}ric Totel",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th International Conference on Risks and Security of Internet and Systems, CRiSIS 2022 ; Conference date: 07-12-2022 Through 09-12-2022",

year = "2023",

month = jan,

day = "1",

doi = "10.1007/978-3-031-31108-6\_2",

language = "English",

isbn = "9783031311079",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "18--33",

editor = "Slim Kallel and Mohamed Jmaiel and \{Hadj Kacem\}, Ahmed and Mohammad Zulkernine and Fr{\'e}d{\'e}ric Cuppens and Nora Cuppens",

booktitle = "Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers",

}

Lanvin, M, Gimenez, PF, Han, Y, Majorczyk, F, Mé, L & Totel, É 2023, Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. in S Kallel, M Jmaiel, A Hadj Kacem, M Zulkernine, F Cuppens & N Cuppens (eds), Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13857 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 18-33, 17th International Conference on Risks and Security of Internet and Systems, CRiSIS 2022, Sousse, Tunisia, 7/12/22. https://doi.org/10.1007/978-3-031-31108-6_2

Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. / Lanvin, Maxime; Gimenez, Pierre François; Han, Yufei et al.
Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers. ed. / Slim Kallel; Mohamed Jmaiel; Ahmed Hadj Kacem; Mohammad Zulkernine; Frédéric Cuppens; Nora Cuppens. Springer Science and Business Media Deutschland GmbH, 2023. p. 18-33 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13857 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes

AU - Lanvin, Maxime

AU - Gimenez, Pierre François

AU - Han, Yufei

AU - Majorczyk, Frédéric

AU - Mé, Ludovic

AU - Totel, Éric

PY - 2023/1/1

Y1 - 2023/1/1

N2 - Among the difficulties encountered in building datasets to evaluate intrusion detection tools, a tricky part is the process of labelling the events into malicious and benign classes. The labelling correctness is paramount for the quality of the evaluation of intrusion detection systems but is often considered as the ground truth by practitioners and is rarely verified. Another difficulty lies in the correct capture of the network packets. If it is not the case, the characteristics of the network flows generated from the capture could be modified and lead to false results. In this paper, we present several flaws we identified in the labelling of the CICIDS2017 dataset and in the traffic capture, such as packet misorder, packet duplication and attack that were performed but not correctly labelled. Finally, we assess the impact of these different corrections on the evaluation of supervised intrusion detection approaches.

AB - Among the difficulties encountered in building datasets to evaluate intrusion detection tools, a tricky part is the process of labelling the events into malicious and benign classes. The labelling correctness is paramount for the quality of the evaluation of intrusion detection systems but is often considered as the ground truth by practitioners and is rarely verified. Another difficulty lies in the correct capture of the network packets. If it is not the case, the characteristics of the network flows generated from the capture could be modified and lead to false results. In this paper, we present several flaws we identified in the labelling of the CICIDS2017 dataset and in the traffic capture, such as packet misorder, packet duplication and attack that were performed but not correctly labelled. Finally, we assess the impact of these different corrections on the evaluation of supervised intrusion detection approaches.

KW - dataset labelling

KW - intrusion detection

KW - machine learning

U2 - 10.1007/978-3-031-31108-6_2

DO - 10.1007/978-3-031-31108-6_2

M3 - Conference contribution

AN - SCOPUS:85161178971

SN - 9783031311079

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 18

EP - 33

BT - Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers

A2 - Kallel, Slim

A2 - Jmaiel, Mohamed

A2 - Hadj Kacem, Ahmed

A2 - Zulkernine, Mohammad

A2 - Cuppens, Frédéric

A2 - Cuppens, Nora

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th International Conference on Risks and Security of Internet and Systems, CRiSIS 2022

Y2 - 7 December 2022 through 9 December 2022

ER -

Lanvin M, Gimenez PF, Han Y, Majorczyk F, Mé L, Totel É. Errors in the CICIDS2017 Dataset and the Significant Differences in Detection Performances It Makes. In Kallel S, Jmaiel M, Hadj Kacem A, Zulkernine M, Cuppens F, Cuppens N, editors, Risks and Security of Internet and Systems - 17th International Conference, CRiSIS 2022, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2023. p. 18-33. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-31108-6_2