ExplainabilityFairnessSecuritycs.LG
Analysis of Zero Day Attack Detection Using MLP and XAI
Ashim Dahal, Prabin Bajgai, Nick Rahimi
January 28, 2025
arXiv.org
11 citations
3 influential
Key Takeaways
-
High accuracy doesn't guarantee fairness: The truncated ML model achieved 99.62% accuracy but showed bias in class representation, while the weighted truncated model sacrificed accuracy (97.26%) to reduce bias—demonstrating that organizations must choose between raw performance metrics and equitable treatment across different attack types.
-
Explainability is essential for security systems: The researchers used SHAP (Shapley Additive exPlanations) to identify which features the model relied on for decisions, highlighting that security teams need to understand why a system flags a threat as zero-day, not just that it does.
-
Dataset limitations affect real-world deployment: The study relied on the KDD99 dataset, which the authors note has been extensively researched but may not reflect current attack patterns, suggesting that AI-based intrusion detection systems require regular updates with fresh data to remain effective against evolving threats.
-
Trade-offs between performance metrics matter for responsible deployment: The comparison between different model variants shows that practitioners must explicitly decide what matters most—catching all attacks accurately, treating all attack classes fairly, or balancing both—rather than defaulting to the highest accuracy score.
Abstract
Any exploit taking advantage of zero-day is called a zero-day attack. Previous research and social media trends show a massive demand for research in zero-day attack detection. This paper analyzes Machine Learning (ML) and Deep Learning (DL) based approaches to create Intrusion Detection Systems (IDS) and scrutinizing them using Explainable AI (XAI) by training an explainer based on randomly sampled data from the testing set. The focus is on using the KDD99 dataset, which has the most research done among all the datasets for detecting zero-day attacks. The paper aims to synthesize the dataset to have fewer classes for multi-class classification, test ML and DL approaches on pattern recognition, establish the robustness and dependability of the model, and establish the interpretability and scalability of the model. We evaluated the performance of four multilayer perceptron (MLP) trained on the KDD99 dataset, including baseline ML models, weighted ML models, truncated ML models, and weighted truncated ML models. Our results demonstrate that the truncated ML model achieves the highest accuracy (99.62%), precision, and recall, while weighted truncated ML model shows lower accuracy (97.26%) but better class representation (less bias) among all the classes with improved unweighted recall score. We also used Shapely Additive exPlanations (SHAP) to train explainer for our truncated models to check for feature importance among the two weighted and unweighted models.
Paper Info
arXiv ID2501.16638
PublishedJanuary 28, 2025
Citations
11
Influential
3
Relevance72%
VenuearXiv.org
TypeJournalArticle
Categorycs.LG
Responsible AI Topics
ExplainabilityFairnessSecurity
Links
Citation data from Semantic Scholar