Approach to Anomaly Detection in Microservice System with Multi- Source Data Streams

doi:10.12142/ZTECOM.202203011

ZTE Communications ›› 2022, Vol. 20 ›› Issue (3): 85-92.DOI: 10.12142/ZTECOM.202203011

• Research Paper • Previous Articles Next Articles

Approach to Anomaly Detection in Microservice System with Multi- Source Data Streams

ZHANG Qixun¹, HAN Jing²(), CHENG Li², ZHANG Baisheng², GONG Zican²

^1.Peking University, Beijing 100091, China
^2.ZTE Corporation, Shenzhen 518057, China

Received:2022-01-24 Online:2022-09-13 Published:2022-09-14
About author:ZHANG Qixun is currently an assistant professor in School of Software and Microelectronics in Peking University, China. He received his PhD in 2022. His research interests include distributed systems, AIOps, etc.|HAN Jing (han.jing28@zte.com.cn) joined ZTE Corporation in 2000. She is an expert in AIOps. She has been putting effort into natural language process for over 10 years and has published several papers.|CHENG Li joined ZTE Corporation in 2006. He is an expert in AIOps and wireless communications. He has much experience in analyzing variety of types of data. He has a lot of experience in problem-solving and methodology.|ZHANG Baisheng joined ZTE Corporation in 2011. His work has been devoted to cell-phone terminal techniques for over 10 years. Besides, he is interested in the research of auto-driving technology.|GONG Zican joined ZTE Corporation in 2020. He received his master’s degree in computing from Australian National University, Australia in 2019. His research interests include AIOps and natural language processing.
Supported by:
ZTE Industry-University-Institute Cooperation Funds(HF-CN-202008200001)

Abstract

Abstract:

Microservices have become popular in enterprises because of their excellent scalability and timely update capabilities. However, while fine-grained modularity and service-orientation decrease the complexity of system development, the complexity of system operation and maintenance has been greatly increased, on the contrary. Multiple types of system failures occur frequently, and it is hard to detect and diagnose failures in time. Furthermore, microservices are updated frequently. Existing anomaly detection models depend on offline training and cannot adapt to the frequent updates of microservices. This paper proposes an anomaly detection approach for microservice systems with multi-source data streams. This approach realizes online model construction and online anomaly detection, and is capable of self-updating and self-adapting. Experimental results show that this approach can correctly identify 78.85% of faults of different types.

Key words: anomaly detection, data stream, microservice, monitored indicator, system log

ZHANG Qixun, HAN Jing, CHENG Li, ZHANG Baisheng, GONG Zican. Approach to Anomaly Detection in Microservice System with Multi- Source Data Streams[J]. ZTE Communications, 2022, 20(3): 85-92.

Figures/Tables 5

References 13

1	WANG P, XU J M, MA M, et al. CloudRanger: root cause identification for cloud native systems [C]//18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, 2018: 492–502. DOI: 10.1109/CCGRID.2018.00076 DOI
2	MA M, LIN W L, PAN D S, et al. MS-rank: multi-metric and self-adaptive root cause diagnosis for microservice applications [C]//IEEE International Conference on Web Services. IEEE, 2019: 60–67. DOI: 10.1109/ICWS.2019.00022 DOI
3	JIA T, LI Y, WU Z H. Survey of state-of-the-art log-based failure diagnosis (in Chinese) [J]. Journal of software, 2020, 31(7): 1997–2018. DOI: 10.13328/j.cnki.jos.006045 DOI
4	CHUAH E, KUO S H, HIEW P, et al. Diagnosing the root-causes of failures from cluster log files [C]//International Conference on High Performance Computing. IEEE, 2010: 1–10. DOI: 10.1109/HIPC.2010.5713159 DOI
5	CHEN C, SINGH N, YAJNIK S. Log analytics for dependable enterprise telephony [C]//Ninth European Dependable Computing Conference. IEEE, 2012: 94–101. DOI: 10.1109/EDCC.2012.14 DOI
6	ZHOU X, PENG X, XIE T, et al. Latent error prediction and fault localization for microservice applications by learning from system trace logs [C]//27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2019: 683–694
7	SILVERMAN B. Density estimation for statistics and data analysis [M]. London, UK: Routledge, 2018
8	PARZEN E, BROWN R G. Smoothing, forecasting and prediction of discrete time series [J]. Journal of the American statistical association, 1964, 59(307): 973. DOI: 10.2307/2283122 DOI
9	HE P J, ZHU J M, ZHENG Z B, et al. Drain: an online log parsing approach with fixed depth tree [C]//IEEE International Conference on Web Services. IEEE, 2017: 33–40. DOI: 10.1109/ICWS.2017.13 DOI
10	JIA T, WU Y F, HOU C J, et al. LogFlash: real-time streaming anomaly detection and diagnosis from system logs for large-scale software systems [C]//IEEE 32nd International Symposium on Software Reliability Engineering. IEEE, 2021: 80–90. DOI: 10.1109/ISSRE52982.2021.00021 DOI
11	ZHAO N W, CHEN J J, WANG Z, et al. Real-time incident prediction for online service systems [C]//28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2020: 315–326. DOI: 10.1145/3368089.3409672 DOI
12	WU L, TORDSSON J, BOGATINOVSKI J, et al. MicroDiag: fine-grained performance diagnosis for microservice systems [C]//IEEE/ACM International Workshop on Cloud Intelligence (CloudIntelligence). IEEE, 2021: 31–36. DOI: 10.1109/CloudIntelligence52565.2021.00015 DOI
13	YANG Y, LI Y, WU Z H. Survey of state-of-the-art distributed tracing technology (in Chinese) [J]. Journal of software, 2020, 31(7): 2019–2039

Virtual Machine Number	Virtual Machine Function	Resource Configuration	Virtual Machine Location
1	Master 1	8 Core CPU, 16 G Memory, 200 G Disk Space	Server_1
2	Master 2	8 Core CPU, 32 G Memory, 200 G Disk Space	Server_2
3	Worker 1	8 Core CPU, 32 G Memory, 200 G Disk Space	Server_1
4	Worker 2	8 Core CPU, 16 G Memory, 200 G Disk Space	Server_1
5	Worker 3	4 Core CPU, 16 G Memory, 200 G Disk Space	Server_2
6	ELK	8 Core CPU, 32 G Memory, 1 T Disk Space	Server_2
7	Others	4 Core CPU, 8 G Memory, 200 G Disk Space	Server_2

Virtual Machine Number	Virtual Machine Function	Resource Configuration	Virtual Machine Location
1	Master 1	8 Core CPU, 16 G Memory, 200 G Disk Space	Server_1
2	Master 2	8 Core CPU, 32 G Memory, 200 G Disk Space	Server_2
3	Worker 1	8 Core CPU, 32 G Memory, 200 G Disk Space	Server_1
4	Worker 2	8 Core CPU, 16 G Memory, 200 G Disk Space	Server_1
5	Worker 3	4 Core CPU, 16 G Memory, 200 G Disk Space	Server_2
6	ELK	8 Core CPU, 32 G Memory, 1 T Disk Space	Server_2
7	Others	4 Core CPU, 8 G Memory, 200 G Disk Space	Server_2

	Front-End	User	Catalogue	Carts	Orders	Queue-Master	Payment	Shipping
CPU/m	300	300	200	300	500	300	200	300
Memory/Mi	1000	200	200	500	500	500	200	500

	Front-End	User	Catalogue	Carts	Orders	Queue-Master	Payment	Shipping
CPU/m	300	300	200	300	500	300	200	300
Memory/Mi	1000	200	200	500	500	500	200	500

Fault Type	Fault Description
Application fault	Caused by null value errors in the code, short-circuit of exception statements, condition reversal, default values missing in switch statements, etc.
System resource fault	Caused by high node CPU utilization, insufficient memory, network delay or packet loss, disk I/O blocking, etc.

Approach to Anomaly Detection in Microservice System with Multi- Source Data Streams

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 5

References 13

Related Articles 3

Recommended Articles

Metrics

	Indicator-Based Anomaly Detection	Log-Based Anomaly Detection	Fault Identification
Null value	1.00	1.00	1.00
Unexpected value	1.00	1.00	1.00
Short circuit of the exception statement	0.35	0.45	0.45
Condition reversed	0.10	0.55	0.55
Switch statement lacking a default value	0.15	0.75	0.45
Exception uncaught	1.00	1.00	1.00
Requested memory unreleased	1.00	1.00	1.00
High node CPU load	1.00	0.00	0.70
High container CPU load	1.00	0.00	0.75
Insufficient node memory	1.00	1.00	1.00
Insufficient container memory	1.00	1.00	1.00
Node disk I/O obstructed	0.65	0.00	0.65
Network delay in the node/package loss	0.90	0.00	0.70

[1]	HAN Jing, JIA Tong, WU Yifan, HOU Chuanjia, LI Ying. Feedback‑Aware Anomaly Detection Through Logs for Large‑Scale Software Systems [J]. ZTE Communications, 2021, 19(3): 88-94.
[2]	LIU Jianwei, YUAN Yifei, HAN Jing. A Case Study on Intelligent Operation System for Wireless Networks [J]. ZTE Communications, 2019, 17(4): 19-26.
[3]	DONG Zhenjiang, LUO Shengmei, WEN Tao, ZHANG Fayang, LI Lingjuan. Random Forest Based Very Fast Decision Tree Algorithm for Data Stream [J]. ZTE Communications, 2017, 15(S2): 52-57.