Graph Neural Networks (GNNs) and Parallel Autoencoders for Heterogeneous Enterprise Log Mapping
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V5I3P125Keywords:
Graph Neural Networks (GNNs), Parallel Autoencoders, Unsupervised Reconstruction Loss, Heterogeneous Enterprise Log Analysis, Compliance-Risk Auditing, Siloed Network Log Parsing, Cross-Domain Network TopologiesAbstract
These numerous kinds of logs cover all the various applications, infrastructure, security and enterprise Information Systems. But more and more often you will get nights plagued with lots of cross domain, chaotic data structures and clunky log format for situational awareness, anomaly detection and compliance based auditing. Considering the need of intelligent mapping, reconstructing and analysing the various enterprise logs, this research proposes Integrated Graph Neural Networks (GNNs) and Parallel Autoencoders (PAEs) for Mapping Heterogeneous Enterprise Logs. In this paper, the concept is to leverage the parallel autoencoders to model the structure and the relationships between context elements from the different sources and fuse with the graph representation learning approach to obtain the logs. The learnt log undergoes the unsupervised reconstruction loss and this approach is capable of capturing the variant behaviour of log patterns, and doesn't need a large amount of labelled data to cover all these possibilities. Moreover, the inter dependencies between systems that cross multiple platforms/cross domain networks can be identified using GNN based mapping. The analytics base can be used to build logs from the networks and store them in silos, and can detect abnormal behaviors, with the risk assessment for compliance purposes. The information of distributed features reconstruct is to be merged with the information in relational graph learning to be fed in the samplable conceptual structure to improve enterprise log visibility, security intelligent monitoring & audits. The vineyard highlights how the heterogeneous systems can share a common enterprise management of logs, which can be achieved by leveraging and synergies brought by unsupervised learning and graph-based analytics.
References
[1] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
[2] Du, M., Li, F., Zheng, G., & Srikumar, V. (2017). DeepLog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (pp. 1285–1298). Association for Computing Machinery. https://doi.org/10.1145/3133956.3134015
[3] Guo, H., Yuan, S., & Wu, X. (2021). LogBERT: Log anomaly detection via BERT. In 2021 International Joint Conference on Neural Networks (pp. 1–8). IEEE. https://doi.org/10.1109/IJCNN52387.2021.9534113
[4] He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017). Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE International Conference on Web Services (pp. 33–40). IEEE. https://doi.org/10.1109/ICWS.2017.13
[5] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
[6] Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020). HitAnomaly: Hierarchical transformers for anomaly detection in system log. IEEE Transactions on Network and Service Management, 17(4), 2064–2076. https://doi.org/10.1109/TNSM.2020.3034647
[7] Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023). Deep learning for anomaly detection in log data: A survey. Machine Learning with Applications, 12, Article 100470. https://doi.org/10.1016/j.mlwa.2023.100470
[8] Le, V.-H., & Zhang, H. (2021). Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (pp. 492–504). IEEE. https://doi.org/10.1109/ASE51524.2021.9678773
[9] Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., & Sun, P. (2019). LogAnomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (pp. 4739–4745). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2019/658
[10] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv. https://doi.org/10.48550/arXiv.1301.3781
[11] Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., & Kao, O. (2020). Self-attentive classification-based anomaly detection in unstructured logs. In 2020 IEEE International Conference on Data Mining (pp. 1196–1201). IEEE. https://doi.org/10.1109/ICDM50108.2020.00148
[12] Oliner, A. J., & Stearley, J. (2007). What supercomputers say: A study of five system logs. In Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (pp. 575–584). IEEE. https://doi.org/10.1109/DSN.2007.103
[13] Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (pp. 1532–1543). Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1162
[14] Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., & Chen, J. (2019). Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 807–817). Association for Computing Machinery. https://doi.org/10.1145/3338906.3338931
