Malware Detection Through Windows System Call Analysis

Detecting malware remains a significant challenge, as malware authors constantly develop new techniques to evade traditional signature-based and heuristic-based detection methods. This paper proposes a novel approach to malware detection that analyzes patterns in Windows system calls sequences to identify malicious behaviors. We use a voting classifier, a machine learning model that aggregates predictions from multiple individual models. It determines the final output based on the class that receives the highest likelihood or majority vote from the ensemble of models. We trained the model on large datasets of benign and malicious system call traces to detect anomalies indicative of malware. By focusing on system call behavior rather than static code characteristics, the approach is able to identify novel malware variants without requiring prior knowledge of their signatures. Experiments using a dataset of 42,797 API call sequences from malware samples and 1,079 sequences from benign software demonstrate that voting classifier can achieve high detection rates while maintaining low false positive rates. This type of Machine Learning-based malware detection could be integrated into an Endpoint Detection and Response (EDR) tool to provide advanced, behavior-based malware detection capabilities.