DDoS Attacks Detection and Prevention on Real-Time Streaming Data Using Machine Learning Methods With Kafka and Apache Spark

초록

Distributed Denial of Service (DDoS) attacks in the constantly expanding sphere of cybersecurity are highly important to detect and respond to promptly. This paper proposes an Intrusion Detection System (IDS) that integrate Machine Learning (ML), Apache Kafka and Apache Spark to enable real-time detection and mitigation of DDoS attacks. The proposed framework collects live network traffic data, performs preprocessing and feature extraction, and applies machine learning algorithms within a streaming architecture supported by Apache Kafka and Apache Spark. Kafka enables real-time data ingestion, while Spark provides distributed processing for efficient model training and inference. The methodology supports adaptability to emerging attack patterns through training and evaluation on publicly available benchmark datasets.Machine learning models, including Random Forest and Support Vector Machine (SVM), were evaluated within the proposed streaming framework. Experimental results show that the integrated use of Apache Kafka and Apache Spark achieved an accuracy of up to 95%. The real-time streaming capability of Kafka, combined with the distributed processing power of Spark, enables efficient handling of high-volume network traffic and supports timely detection of DDoS attacks.

키워드