Minimalistic Fault Detection In System Logs Using Unsupervised Learning

Abstract

In a computing system, the system logs are important for detecting the issues and failures and anomaly. The modern systems are producing massive amount of log daily. The traditional detecting methods are usually unable to detect complex issues. This study aims for a machine learning based method that will detect anomalies in the system logs. Log data is the first transformed into structured templates and represented them using TF-IDF method. The anomaly detectiion models, with Isolation Forest and Local Outlier Factor (LOF), are then applied to detect issues and failures. The Experiments shows that the LOF (Local Outlier Factor) achieves higher accuracy and recall compared to IF (Isolation Forest), which is showing the strong potential for practical assessment. This study results highlight that with the combination of feature engineering along with lightweight machine learning methods enables efficient and automated log anomaly detection, and improving system reliability with reducing manual monitoring efforts.