AI-Driven Observability for Managing Security Complexity in Cloud-native Microservices and Containerized Environments

Abstract

This study investigates security-visibility challenges in cloud-native microservices by analysing adversarial behaviours using the MITRE ATT&CK container-techniques dataset, assessing observability limitations through time-series telemetry-coverage quantification applied to the OpenTelemetry Demo dataset, and proposing an AI-driven observability model for security anomaly detection. The proposed method employs an unsupervised deep autoencoder trained on integrated telemetry from the Google Cloud Microservices Demo and Kubernetes audit logs to learn normal system behaviour and identify anomalies, defined as statistically significant deviations from established baseline telemetry patterns associated with validated security events. Model performance was evaluated using a hold-out train–test validation protocol across combined multi-source telemetry datasets. Results show that the AI-driven observability model improves detection effectiveness compared with a baseline rule-based monitoring approach, increasing recall from 54% to 84%, reducing false negatives from 690 to 240, and substantially lowering average detection latency under comparable operational conditions. These findings demonstrate that AI-enabled observability enhances detection accuracy, reduces monitoring blind spots and supports more timely operational decision-making through improved multi-layer telemetry correlation. Accordingly, the study recommends unified telemetry pipelines, consistent instrumentation across microservices, adoption of unsupervised anomaly-detection models and the development of AI-observability standards to improve the reliability and security of cloud-native systems.