Comparative Evaluation of Machine Learning Algorithms for Real-Time Quality Alert Classification in a Metalworking Manufacturing Environment

Abstract

Background: Real-time quality monitoring in metalworking manufacturing is essential to minimize defects, reduce downtime, and maintain product consistency in dynamic production environments. Machine learning algorithms provide robust tools for classifying quality alerts by capturing complex patterns in process data, enabling faster and more accurate decision-making. Aims: This paper critically assesses eight trained machine learning (ML) classifiers to predict the level of quality alert priority in a metalworking factory, and suggests the best-suited algorithm to use in real-time quality control systems. Study Design:  Empirical, comparative ML analysis based on a 5-year retrospective industrial dataset. Location and Time of Study: The data were acquired through Mendeley Data (Ramírez-Velásquez & Restrepo 2025; DOI: 10.17632/8kb7h5rnnf.1), which consisted of 374 quality alert reports in a metalworking company, between March 2020 and December 2024. Methodology: Eight classifiers (Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, Support Vector Machine (RBF kernel), K-Nearest Neighbours, Naive Bayes, and Multilayer Perceptron Neural Network) were trained on eight engineered features after SMOTE-based class balancing. Accuracy, Weighted Precision, Recall, F1-Score, AUC-ROC, 5-fold, and 10-fold cross-validation were used to evaluate performance. Results: The MLP Neural Network achieved the highest performance (Accuracy = 0.8970, F1 = 0.8964, AUC = 0.9719, CV-F1 = 0.8946), followed by Random Forest (F1 = 0.8900, AUC = 0.9666). Alert Cost and Process Type were the most discriminative features. Logistic Regression and Naive Bayes returned the least F1 values (0.6632 and 0.7115, respectively), indicating the lack of suitability of linear assumptions in manufacturing with complexity. Conclusion: The MLP can be suggested to be implemented in AI-driven real-time quality control systems, and Random Forest is an interpretation-friendly alternative. The work adds empirical data from real industrial data to the increasing body of literature on ML-based quality assurance in Industry 4.0 manufacturing settings.