Feedback‑Aware Anomaly Detection Through Logs for Large‑Scale Software Systems

doi:10.12142/ZTECOM.202103011

ZTE Communications ›› 2021, Vol. 19 ›› Issue (3): 88-94.DOI: 10.12142/ZTECOM.202103011

• Research Paper • Previous Articles

Feedback‑Aware Anomaly Detection Through Logs for Large‑Scale Software Systems

HAN Jing¹(), JIA Tong², WU Yifan², HOU Chuanjia², LI Ying²

^1.ZTE Corporation, Shenzhen 518057, China
^2.Peking University, Beijing 100091, China

Received:2021-02-04 Online:2021-09-25 Published:2021-10-11
About author:HAN Jing (han.jing28@zte.com.cn) received her master’s degree from Nanjing University of Aeronautics and Astronautics, China. She has been with ZTE Corporation since 2000, where she had been engaged in 3G/4G key technologies from 2000 to 2016. She has become a technical director responsible for intelligent operation of cloud platforms and wireless networks since 2016. Her research interests include machine learning, data mining, and signal processing.|JIA Tong is a doctoral researcher at Department of Computer Science and Technology, Peking University, China. His research interests include distributed computing and algorithmic IT operations.|WU Yifan is pursuing his doctorate at School of Software and Microelectronics in Peking University, China. His research mainly focuses on distributed computing and algorithmic IT operations.|HOU Chuanjia is pursuing his master’s degree at School of Software and Microelectronics in Peking University, China. His research focuses on algorithmic IT operations.|LI Ying is a researcher at National Engineering Research Center for Software Engineering, Peking University, China. She is also a professor of School of Software and Microelectronics, Peking University. Her research interests include distributed computing and trusted computing.
Supported by:
ZTE Industry-University-Institute Cooperation Funds(20200492)

Abstract

Abstract:

One particular challenge for large?scale software systems is anomaly detection. System logs are a straightforward and common source of information for anomaly detection. Existing log?based anomaly detectors are unusable in real?world industrial systems due to high false?positive rates. In this paper, we incorporate human feedback to adjust the detection model structure to reduce false positives. We apply our approach to two industrial large?scale systems. Results have shown that our approach performs much better than state?of?the-art works with 50% higher accuracy. Besides, human feedback can reduce more than 70% of false positives and greatly improve detection precision.

Key words: human feedback, log?based anomaly detection, system log

HAN Jing, JIA Tong, WU Yifan, HOU Chuanjia, LI Ying. Feedback‑Aware Anomaly Detection Through Logs for Large‑Scale Software Systems[J]. ZTE Communications, 2021, 19(3): 88-94.

Figures/Tables 6

Figure 1 Approach overview

Figure 2 Log templates and TCFG model

Figure 3 Template transition distributions of an industrial software system Ada

Table 1 Computations of transition likelihood for power-law model

Computation Entity	Computation Method
Log survival function: $l o g S t i \| t k, α k, i$	$- α j, i l o g ? (t i - t j δ)$
Hazard function: $H t i \| t j, α j, i$	$α j, i ? 1 t i - t j$
Gradient for unobserved ones in $? c$ : $? α j, i L c Α$	$l o g ? (T - t j c δ)$
Gradient for observed ones in $c$ : $? α j, i L c Α$	$l o g ? t i c - t j c δ - (t i c - t j c) - 1 ∑ k : t k c < t i c α k, i (t i c - t k c) - 1$

Table 1 Computations of transition likelihood for power-law model

Computation Entity	Computation Method
Log survival function: $l o g S t i \| t k, α k, i$	$- α j, i l o g ? (t i - t j δ)$
Hazard function: $H t i \| t j, α j, i$	$α j, i ? 1 t i - t j$
Gradient for unobserved ones in $? c$ : $? α j, i L c Α$	$l o g ? (T - t j c δ)$
Gradient for observed ones in $c$ : $? α j, i L c Α$	$l o g ? t i c - t j c δ - (t i c - t j c) - 1 ∑ k : t k c < t i c α k, i (t i c - t k c) - 1$

Table 2 Evaluation results of Ada

Approaches	Precision/%	Recall/%	#Human Feedback
LogSed^[18]	0.34	1.00	/
DeepLog^[6]	0.45	1.00	/
Our approach	0.86(0.42)	1.00	28

Table 3 Evaluation results of Bob

Approaches	Precision/%	Recall/%	#Human Feedback
LogSed^[18]	0.04	0.89	/
DeepLog^[6]	0.09	0.99	/
Our approach	0.77(0.07)	0.96	52

References 25

1	LOU J G, FU Q, YANG S Q, et al. Mining invariants from console logs for system problem detection [C]//USENIX Annual Technical Conference. Berkeley, USA: USENIX, 2010
2	OLINER A J, AIKEN A. Online detection of multi‑component interactions in production systems [C]//2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN). Hong Kong, China: IEEE, 2011: 49–60. DOI: 10.1109/DSN.2011.5958206 DOI
3	CHEN C, SINGH N, YAJNIK S. Log analytics for dependable enterprise telephony [C]//2012 Ninth European Dependable Computing Conference. Sibiu, Romania: IEEE, 2012: 94–101. DOI: 10.1109/EDCC.2012.14 DOI
4	DU S Z, JIAN C. Behavioral anomaly detection approach based on log monitoring [C]//2015 International Conference on Behavioral, Economic and Socio‑cultural Computing (BESC). Nanjing, China: IEEE, 2015: 188–194. DOI:10.1109/BESC.2015.7365981 DOI
5	NANDI A, MANDAL A, ATREJA S, et al. Anomaly detection using program control flow graph mining from execution logs [C]//The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Francisco, USA: ACM, 2016: 215–224. DOI:10.1145/2939672.2939712 DOI
6	JIA T, YANG L, CHEN P F, et al. LogSed: anomaly diagnosis through mining time‑weighted control flow graph in logs [C]//2017 IEEE 10th International Conference on Cloud Computing (CLOUD). Honololu, USA: IEEE, 2017: 447–455. DOI:10.1109/CLOUD.2017.64 DOI
7	JIA T, CHEN P F, YANG L, et al. An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services [C]//2017 IEEE International Conference on Web Services (ICWS). Honolulu, USA: IEEE, 2017: 25–32. DOI:10.1109/ICWS.2017.12 DOI
8	FU Q, LOU J G, WANG Y, et al. Execution anomaly detection in distributed systems through unstructured log analysis [C]// The 9th IEEE International Conference on Data Mining. Miami Beach, USA: IEEE, 2009: 149–158. DOI:10.1109/ICDM.2009.60 DOI
9	BABENKO A, MARIANI L, PASTORE F. AVA: automated interpretation of dynamically detected anomalies [C]//The 18th International Symposium on Software Testing and Analysis. Chicago, USA: ISSTA, 2009: 237–248. DOI:10.1145/1572272.1572300 DOI
10	YEN T F, OPREA A, ONARLIOGLU K, et al. Beehive: large‑scale log analysis for detecting suspicious activity in enterprise networks [C]//The Annual Computer Security Applications Conference. New Orleans, USA: ACM, 2013: 199–208. DOI:10.1145/2523649.2523670 DOI
11	ZHAOX, Y.ZHANG, LIOND, et al. lprof: a non‑intrusive request flow profiler for distributed systems [C]//Usenix Symposium on Operating System Implementation & Design. Broomfield, USA: OSDI, 2014, 629–644
12	YU X, JOSHI P, XU J W, et al. CloudSeer [J]. ACM SIGPLAN notices, 2016, 51(4): 489–502. DOI:10.1145/2954679.2872407 DOI
13	TAK B C, TAO S, YANG L, et al. LOGAN: problem diagnosis in the cloud using log‑based reference models [C]//2016 IEEE International Conference on Cloud Engineering (IC2E). Berlin, Germany: IEEE, 2016: 62–67. DOI:10.1109/IC2E.2016.12 DOI
14	AALST WVAN DER, WEIJTERS T, MARUSTER L. Workflow mining: discovering process models from event logs [J]. IEEE transactions on knowledge and data engineering, 2004, 16(9): 1128–1142. DOI:10.1109/TKDE.2004.47 DOI
15	LOU J G, FU Q, YANG S Q, et al. Mining program workflow from interleaved traces [C]//Proceedings of the 16th ACM SIGKDD International Conference on Knowledge discovery and data mining. Washington, USA: ACM, 2010: 613–622. DOI:10.1145/1835804.1835883 DOI
16	YUAN D, MAI H H, XIONG W W, et al. SherLog [J]. ACM SIGARCH computer architecture news, 2010, 38(1): 143–154. DOI:10.1145/1735970.1736038 DOI
17	FU Q, LOU J G, LIN Q W, et al. Contextual analysis of program logs for understanding system behaviors [C]//The 2013 10th Working Conference on Mining Software Repositories (MSR). San Francisco, USA: IEEE, 2013: 397–400. DOI:10.1109/MSR.2013.6624054 DOI
18	DU M, LI F F, ZHENG G N, et al. DeepLog: anomaly detection and diagnosis from system logs through deep learning [C]//The 2017 ACM SIGSAC Conference on Computer and Communications Security. Dallas, USA: ACM, 2017: 1285–1298. DOI:10.1145/3133956.3134015 DOI
19	MENG W B, LIU Y, ZHU Y C, et al. LogAnomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs [C]//The 28th International Joint Conference on Artificial Intelligence. Macao, China: IJCAI, 2019: 4739–4745. DOI:10.24963/ijcai.2019/658 DOI
20	LIN Q W, ZHANG H Y, LOU J G, et al. Log clustering based problem identification for online service systems [C]//Proceedings of the 38th International Conference on Software Engineering Companion. Austin, USA: ACM, 2016: 102–111. DOI:10.1145/2889160.2889232 DOI
21	SIDDIQUI M A, FERN A, DIETTERICH T G, et al. Feedback‑guided anomaly discovery via online optimization [C]//The 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London, United Kingdom: ACM, 2018: 2200–2209. DOI:10.1145/3219819.3220083 DOI
22	DAS S, WONG W K, DIETTERICH T, et al. Incorporating expert feedback into active anomaly discovery [C]//16th International Conference on Data Mining (ICDM): IEEE, 2017, 853–858
23	DAS S, WONG W K, FERN A, et al. Incorporating feedback into tree‑based anomaly detection [EB/OL]. [2021‑01‑20].
24	HE P J, ZHU J M, ZHENG Z B, et al. Drain: an online log parsing approach with fixed depth tree [C]//2017 IEEE International Conference on Web Services (ICWS). Honolulu, USA: IEEE, 2017: 33–40. DOI: 10.1109/ICWS.2017.13 DOI
25	GOMEZ RODRIGUEZ M, LESKOVEC J, SCHÖLKOPF B. Structure and dynamics of information pathways in online media [C]//The sixth ACM international conference on Web search and data mining ‑ WSDM’13. Rome, Italy: ACM, 2013: 23–32. DOI: 10.1145/2433396.2433402 DOI

Feedback‑Aware Anomaly Detection Through Logs for Large‑Scale Software Systems

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 25

Related Articles 1

Recommended Articles

Metrics