A Multi-Dimensional Evaluation Framework for IoT Intrusion Detection: Balancing Accuracy, Efficiency and Real-World Deployment Constraints

Abstract

The rapid proliferation of Internet of Things (IoT) devices has introduced unprecedented security vulnerabilities, with botnet attacks representing persistent threats to network infrastructure. While machine learning-based intrusion detection systems (IDS) show promise in laboratory settings, real-world deployment faces challenges including extreme class imbalance, resource constraints, and false alarm minimization. Traditional evaluation approaches focus primarily on accuracy, overlooking deployment factors such as execution time, memory consumption, and false alarm rates. This study introduces a comprehensive multi-dimensional evaluation framework that assesses IDS performance across 14 distinct metrics, including two novel composite scores, Efficiency Score and Deployment Score, for quantifying deployment readiness and operational efficiency. Applied to the BoT-IoT dataset (3.6 million flows, 1:7,682 imbalance), we evaluated six algorithms using six feature selection methods and a novel two-stage balancing approach combining undersampling and SMOTE. Critically, all experiments were conducted within a 12GB RAM constraint to reflect realistic resource limitations in edge computing and IoT gateway deployments. Random Forest, XGBoost, and Decision Tree achieved 99.97% accuracy with zero false positives on 733,705 test samples, maintaining practical training times (0.06–1.98 s) and memory footprints (0.07–2.00 MB). Our framework provides use-case-specific recommendations: XGBoost for resource-constrained devices (0.07 MB), Decision Tree for real-time applications (0.04 s prediction), and Random Forest for balanced deployment (0.9999 deployment score). This framework enables IoT security practitioners to make informed, context-aware model selections aligned with specific deployment constraints—whether prioritizing minimal memory footprint for edge sensors, ultra-low latency for real-time prevention systems, or rapid retraining for adaptive security—thereby accelerating the transition from laboratory research to operational IoT security solutions.