Incident and Problem Ticket Clustering and Classification Using Deep Learning

doi:10.12142/ZTECOM.202304009

ZTE Communications ›› 2023, Vol. 21 ›› Issue (4): 69-77.DOI: 10.12142/ZTECOM.202304009

收稿日期:2022-12-19 出版日期:2023-12-07 发布日期:2023-12-07

Incident and Problem Ticket Clustering and Classification Using Deep Learning

FENG Hailin¹, HAN Jing²(), HUANG Leijun¹, SHENG Ziwei³, GONG Zican²

^1.Zhejiang A&F University, Hangzhou 310007, China
^2.ZTE Corporation, Shenzhen 518057, China
^3.Huazhong University of Science and Technology, Wuhan 430074, China

Received:2022-12-19 Online:2023-12-07 Published:2023-12-07
About author:FENG Hailin received his PhD in computer science from the University of Science and Technology of China in June 2007. Since 2007, he has been working in the School of Information Engineering of Zhejiang A&F University, China. From 2013 to 2014, he was a visiting professor at Forest Products Laboratory, USA. He is currently a professor in the School of Mathematics and Computer Science and School of Information Engineering of Zhejiang A&F University. His main interest areas include computer vision, intelligent information processing, and Internet of Things.
HAN Jing (han.jing28@zte.com.cn) received her master’s degree from Nanjing University of Aeronautics and Astronautics, China. She has been with ZTE Corporation since 2000, where she worked on 3G/4G key technologies from 2000 to 2016 and has become a technical director responsible for intelligent operation of cloud platforms and wireless networks since 2016. Her research interests include machine learning, data mining, and signal processing.
HUANG Leijun received his PhD in computer science from George Mason University, USA in 2008. Since 2010, he has been working in the School of Information Engineering of Zhejiang A&F University, China. He is currently a lecturer in the School of Mathematics and Computer Science. His main interest areas include computer networks, Internet of Things and data mining.
SHENG Ziwei received her BS degree in software engineering from Huazhong University of Science and Technology, China in 2022. She is currently pursuing her MS degree in electrical and computer engineering at Carnegie Mellon University, USA. In her master’s program, she primarily focuses on the fields of engineering development and system design. Her ultimate goal is to advance technology and foster innovation in these domains.
GONG Zican received his master’s degree in professional computing and artificial intelligence from the Australian National University in 2019. He has been a machine learning engineer in ZTE Corporation since 2020. His research interests include machine learning, professional computing and system architecture.

摘要/Abstract

Abstract:

A holistic analysis of problem and incident tickets in a real production cloud service environment is presented in this paper. By extracting different bags of words, we use principal component analysis (PCA) to examine the clustering characteristics of these tickets. Then K-means and latent Dirichlet allocation (LDA) are applied to show the potential clusters within this Cloud environment. The second part of our study uses a pre-trained bidirectional encoder representation from transformers (BERT) model to classify the tickets, with the goal of predicting the optimal dispatching department for a given ticket. Experimental results show that due to the unique characteristics of ticket description, pre-processing with domain knowledge turns out to be critical in both clustering and classification. Our classification model yields 86% accuracy when predicting the target dispatching department.

Key words: problem ticket, ticket clustering, ticket classification

. [J]. ZTE Communications, 2023, 21(4): 69-77.

FENG Hailin, HAN Jing, HUANG Leijun, SHENG Ziwei, GONG Zican. Incident and Problem Ticket Clustering and Classification Using Deep Learning[J]. ZTE Communications, 2023, 21(4): 69-77.

图/表 14

参考文献 22

1	FORELL T, MILOJICIC D, TALWAR V. Cloud management: challenges and opportunities [C]//IEEE International Symposium on Parallel and Distributed Processing Workshops and PhD Forum. IEEE, 2011: 881–889. DOI: 10.1109/IPDPS.2011.233
2	MARTIN-FLATIN J P. Challenges in cloud management [J]. IEEE cloud computing, 2014, 1(1): 66–70. DOI: 10.1109/MCC.2014.4
3	PEREIRA R, SILVA M M DA. Towards an integrated IT governance and IT management framework [C]//The 16th International Enterprise Distributed Object Computing Conference. IEEE, 2012: 191–200. DOI: 10.1109/EDOC.2012.30
4	FARIA N E, SILVA M M. Selecting a software tool for ITIL using a multiple criteria decision analysis approach [EB/OL]. [2022-12-10].
5	BOTEZATU M M, BOGOJESKA J, GIURGIU I, et al. Multi-view incident ticket clustering for optimal ticket dispatching [C]//The 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015: 1711–1720. DOI: 10.1145/2783258.2788607
6	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL]. [2022-12-10].
7	ROELLEKE T, WANG J. TF-IDF uncovered: a study of theories and probabilities [C]//The 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2008: 435–442. DOI: 10.1145/1390334.1390409
8	JELODAR H, WANG Y L, YUAN C, et al. Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey [J]. Multimedia tools and applications, 2019, 78(11): 15169–15211. DOI: 10.1007/s11042-018-6894-4
9	MO Z L. Stopwords [EB/OL]. (2019-12-18) [2022-12-10].
10	LARS Y C. Stopwords [EB/OL]. (2011-12-06) [2022-12-10].
11	DU M, LI F F, ZHENG G N, et al. DeepLog: anomaly detection and diagnosis from system logs through deep learning [C]//The 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017: 1285–1298. DOI: 10.1145/3133956.3134015
12	ZHANG X, XU Y, LIN Q W, et al. Robust log-based anomaly detection on unstable log data [C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 2019: 807–817. DOI: 10.1145/3338906.3338931
13	LIN J Y, ZHANG Q, BANNAZADEH H, et al. Automated anomaly detection and root cause analysis in virtualized cloud infrastructures [C]//IEEE/IFIP Network Operations and Management Symposium. IEEE, 2016: 550–556. DOI: 10.1109/NOMS.2016.7502857
14	HARUTYUNYAN, A N, GRIGORYAN N M, POGHOSYAN A V, et al. Intelligent troubleshooting in data centers with mining evidence of performance problems [EB/OL]. (2020-09-20) [2022-12-10].
15	NASSIF A BOU, TALIB M ABU, NASIR Q, et al. Machine learning for cloud security: a systematic review [J]. IEEE access, 2021, 9: 20717–20735. DOI: 10.1109/ACCESS.2021.3054129
16	GRZONKA D, JAKÓBIK A, KOŁODZIEJ J, et al. Using a multi-agent system and artificial intelligence for monitoring and improving the cloud performance and security [J]. Future generation computer systems, 2018, 86: 1106–1117. DOI: 10.1016/j.future.2017.05.046
17	SHAO Q H, CHEN Y, TAO S, et al. Efficient ticket routing by resolution sequence mining [C]//Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 2008: 605–613. DOI: 10.1145/1401890.1401964
18	AGARWAL S, SINDHGATTA R, SENGUPTA B. SmartDispatch: enabling efficient ticket dispatch in an IT service environment [C]//The 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2012: 1393–1401. DOI: 10.1145/2339530.2339744
19	LIN D, RAGHU R, RAMAMURTHY V, et al. Unveiling clusters of events for alert and incident management in large-scale enterprise it [C]//The 20th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 2014: 1630–1639. DOI: 10.1145/2623330.2623360
20	MANI S, SANKARANARAYANAN K, SINHA V S, et al. Panning requirement nuggets in stream of software maintenance tickets [C]//The 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 2014: 678–688. DOI: 10.1145/2635868.2635897
21	AGARWAL S, AGGARWAL V, AKULA A R, et al. Automatic problem extraction and analysis from unstructured text in IT tickets [J]. IBM journal of research and development, 2017, 61(1):41–52. DOI: 10.1147/JRD.2016.2629318
22	JAN E E, CHEN K Y, IDÉ T. Probabilistic text analytics framework for information technology service desk tickets [C]//IFIP/IEEE International Symposium on Integrated Network Management (IM). IEEE, 2015: 870–873. DOI: 10.1109/INM.2015.7140397

Keywords	Frequency	Keywords	Frequency
node	538	tecs	328
defect	479	provider	198
version	463	daisy	150
symptom	329	nova	149
upgrade	317	dvs	139
alert	309	compute	123
conclusion	291	cinder	90
description	282	neutron	90
progress	275	sdn	79
operation	258	error	76
topology	256	host	69
note	253	nfv	69
cause	252	ip	64
site	235	agent	64
failure	229	ceph	62

Keywords	Frequency	Keywords	Frequency
node	538	tecs	328
defect	479	provider	198
version	463	daisy	150
symptom	329	nova	149
upgrade	317	dvs	139
alert	309	compute	123
conclusion	291	cinder	90
description	282	neutron	90
progress	275	sdn	79
operation	258	error	76
topology	256	host	69
note	253	nfv	69
cause	252	ip	64
site	235	agent	64
failure	229	ceph	62

Topic	Keywords and Probabilities in Each Topic
Topic 0	interface	daisy	NIC	file	maintain	virtualization	platform	zone	mode	increase
Topic 0	0.107	0.055	0.034	0.033	0.025	0.023	0.023	0.022	0.021	0.020
Topic 1	OS	start	capacity	install	stack	service	blade	VM	mac	down
Topic 1	0.113	0.090	0.072	0.054	0.029	0.027	0.025	0.017	0.017	0.017
Topic 2	provider	performance	device	login	director	memory	query	recover	bandwidth	occupy
Topic 2	0.218	0.048	0.038	0.037	0.037	0.031	0.029	0.026	0.021	0.019
Topic 3	dvs	gateway	project	support	manage	business	resource	pool	add	thread
Topic 3	0.081	0.041	0.038	0.038	0.033	0.032	0.031	0.028	0.027	0.023
Topic 4	network	blocked	power on	udm	update	management	information	change	packet loss	endpoint
Topic 4	0.148	0.061	0.035	0.032	0.03	0.029	0.024	0.023	0.022	0.013

Topic	Keywords and Probabilities in Each Topic
Topic 0	interface	daisy	NIC	file	maintain	virtualization	platform	zone	mode	increase
Topic 0	0.107	0.055	0.034	0.033	0.025	0.023	0.023	0.022	0.021	0.020
Topic 1	OS	start	capacity	install	stack	service	blade	VM	mac	down
Topic 1	0.113	0.090	0.072	0.054	0.029	0.027	0.025	0.017	0.017	0.017
Topic 2	provider	performance	device	login	director	memory	query	recover	bandwidth	occupy
Topic 2	0.218	0.048	0.038	0.037	0.037	0.031	0.029	0.026	0.021	0.019
Topic 3	dvs	gateway	project	support	manage	business	resource	pool	add	thread
Topic 3	0.081	0.041	0.038	0.038	0.033	0.032	0.031	0.028	0.027	0.023
Topic 4	network	blocked	power on	udm	update	management	information	change	packet loss	endpoint
Topic 4	0.148	0.061	0.035	0.032	0.03	0.029	0.024	0.023	0.022	0.013

ID	Title Description	ClusterID
BC20200229-0052	Backup of version 6.10.10.08 failed during upgrading	5
BC20200307-0094	Problems of 3.17.15.06P2 trusted resource pool	5
BC20200309-0105	Backup of 6.10.10.P8tecs6.0 environment failed	5
BC20200402-0034	Nova service failed due to version 3.17.15.06 license problem during vHSS capacity increase	5
BC20200404-0051	Provider MariaDB failed to start after rebooting one of the control node in group one	5
BC20200409-0023	Keystone service abnormal in the 5GC test node in 3.17.14 control environment	5
BC20200411-0045	Two of the disks failed when batch creating 32 35T cloud disks reporting not sufficient space	5
BC20200507-0012	Mutual trust failed during Northeast 3.17.15.06P4 upgrading	5
BC20200511-0058	Part of the related information not updated after changing configuration of 3.17.15.06P3 on Daisy	5
BC20200520-0066	One of the VMs failed during start reporting volume not found after NFVINMA1Z station upgrade	5
BC20200603-0025	Failure of V03.17.15.06P4HP3 upgrading	5
BC20200603-0027	Multi-node upgrade failed due to 3.17.15.06P4 17.15.06P4HP3 mutual trust lost	5
BC20200618-0066	3.17.15.07_T2-daisy upgrade failed	5
BC20200629-0247	V3.17.14.P2Provider auto-backup service hang	5
BC20200630-0257	Network issues from two VMs on 3.17.15.06P4HP3	5
BC20200702-0060	Mirror file upload failed after V03.17.15.07T2-Provider upgrade	5
BC20200705-0002	Nova service down after 3.17.15.06P4HP3 upgrade	5
BC20200710-0121	3.17.15.08-OS distribution failed reporting mutual-trust issue	5
BC20200710-0123	40 VMs in shutoff status using 3.17.15.08 startup script	5
BC20200716-0095	Not able to apply new license after 3.17.15.06P6-license failed	5
BC20200722-0207	Tenant resource abnormal after tenants with the same name created	5

Incident and Problem Ticket Clustering and Classification Using Deep Learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 22

相关文章 0

编辑推荐

Metrics

Category	Number of Records
Infrastructure	177 040
Operation product line	64 869
OA product line	55 570
EPMS intelligent service	22 454
iCenter application	19 849
PLM product line	15 870
AIOps group	14 121
Technical platform	1 232
Others	171
IT Wizard	19
Operation NOC	17
Network	5
Communication	2
Middleware	1
Security	1

Classification Label	Numbers of Records	ID
AIOps - itsp service website	16 724	0
desktop cloud - linux desktop cloud	11 222	1
desktop cloud - OS issues	7 162	2
PC side zmail - operation issues	7 110	3
ifol finance - oerp enterprise resource planning	6 543	4
desktop cloud - intranet client-side login	6 531	5
ifol finance - fol finance online	6 381	6
network proxy - usage issues	5 511	7
iscp supply chain - iwms.wms cloud storage	5 086	8
iccp customer relationship - msm marketing	4 994	9
iccp customer relationship - ccg contract configuration	4 779	10
PC side zmail - account issues	4 622	11
iscp supply chain - iwms.mcs manufacture management	4 207	12
im instant message - usage issues	4 156	13
PC side zmail - account creation & login	4 043	14
ifol finance - cms contract management web	4 019	15
uds failure - security check	3 930	16
ifol finance - cms contract management form	3 886	17
icenter - ts team coordination	3 878	18
desktop cloud - blue or black screen	3 872	19
engineering domain - cca cloud code review	3 584	20
desktop cloud - client side login failed	3 551	21
OS issues - installation	3 531	22
ibcp human resource - hol online	3 258	23
iscp supply chain - isrm.srm supplier management	3 254	24
Mobil application - icenter Mobile side	3 234	25
individual network issue - restriction	3 213	26
ibcp human resource - elearning academy	3 178	27
uds failure - usage issues	3 008	28

	Precision	Recall	F1-score	Support
0	0.91	0.91	0.91	4 951
1	0.85	0.85	0.85	3 289
2	0.67	0.65	0.66	2 071
3	0.80	0.85	0.83	2 146
4	0.88	0.88	0.88	1 931
5	0.73	0.75	0.74	1 974
6	0.93	0.93	0.93	1 872
7	0.91	0.93	0.92	1 619
8	0.93	0.92	0.93	1 543
9	0.91	0.93	0.92	1 470
10	0.91	0.90	0.91	1 500
11	0.84	0.80	0.82	1 381
12	0.90	0.91	0.91	1 231
13	0.91	0.93	0.92	1 266
14	0.80	0.82	0.81	1 188
15	0.80	0.80	0.80	1 206
16	0.89	0.90	0.90	1 228
17	0.79	0.77	0.78	1 159
18	0.93	0.94	0.93	1 119
19	0.67	0.71	0.69	1 173
20	0.95	0.95	0.95	980
21	0.87	0.89	0.88	1 046
22	0.89	0.88	0.89	1 060
23	0.90	0.89	0.90	925
24	0.91	0.91	0.91	978
25	0.95	0.93	0.94	994
26	0.87	0.77	0.81	960
27	0.95	0.96	0.95	955
28	0.79	0.69	0.74	968
Accuracy			0.86	44 183
Macro avg	0.86	0.86	0.86	44 183
Weighted avg	0.86	0.86	0.86	44 183

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28
0	4 520	5	6	22	4	14	8	20	1	1	7	41	2	28	42	5	16	1	26	2	0	5	58	24	7	35	13	4	34
1	12	2 789	166	24	1	118	1	16	1	3	0	0	1	26	16	1	0	0	4	60	4	13	5	1	4	0	7	4	12
2	8	216	1 352	6	0	233	0	2	0	2	1	0	1	5	2	1	2	1	0	203	1	15	8	0	0	0	7	0	5
3	17	21	6	1 826	1	1	3	5	0	1	2	139	1	1	106	1	0	1	6	1	1	0	1	1	0	0	0	3	1
4	2	2	0	0	1 706	0	49	0	19	2	2	1	56	1	1	37	0	33	0	0	7	0	0	4	6	0	1	1	1
5	10	70	242	2	0	1 473	0	0	0	0	1	0	0	1	2	0	3	0	1	113	2	47	0	0	0	0	6	0	1
6	6	5	0	3	61	0	1 745	1	4	5	3	1	2	0	0	9	0	2	3	0	2	0	0	6	9	0	0	3	2
7	21	27	1	3	0	0	2	1 510	2	0	2	0	0	1	4	1	1	0	5	0	3	1	1	1	2	0	10	5	16
8	4	1	1	2	22	0	8	2	1 421	1	6	1	8	0	0	15	1	11	4	0	0	0	0	0	27	1	1	2	1
9	7	0	1	2	2	0	6	1	3	1 374	39	1	0	1	1	7	0	5	5	0	1	0	2	7	3	0	0	2	0
10	5	2	1	3	3	0	2	3	2	66	1 352	0	5	3	1	7	2	30	1	0	7	0	1	0	2	0	0	1	1
11	28	6	1	196	1	1	1	0	1	0	1	1 103	1	1	27	0	1	1	2	0	1	0	0	1	2	1	1	2	1
12	6	3	2	0	29	0	2	0	25	3	4	0	1 117	1	0	4	0	12	1	0	4	0	3	4	8	0	2	0	1
13	14	24	3	7	1	0	3	1	0	1	0	3	2	1 173	2	4	0	1	1	0	1	1	4	5	4	2	2	2	5
14	31	16	2	125	0	5	1	2	0	0	1	12	0	3	969	0	0	0	2	0	1	2	0	3	1	2	4	0	6
15	1	0	0	3	51	0	7	1	10	13	7	0	1	3	0	966	0	132	6	0	1	0	0	0	3	0	0	1	0
16	30	4	6	2	1	4	0	1	0	0	0	2	2	3	0	0	1 107	1	0	0	3	1	4	1	1	0	4	1	50
17	0	0	0	0	40	0	0	0	15	13	29	1	23	1	1	140	0	892	0	0	2	0	0	0	1	0	1	0	0
18	19	0	1	12	2	0	6	1	2	1	0	4	1	3	1	0	0	2	1 050	0	1	3	0	4	2	2	0	1	1
19	1	42	174	4	0	93	0	0	0	1	0	0	1	2	0	0	0	0	0	832	1	17	3	0	0	0	2	0	0
20	5	9	1	6	3	0	2	3	2	2	4	1	2	3	0	0	2	1	1	0	928	0	0	2	1	0	2	0	0
21	7	11	15	1	0	50	0	3	1	0	0	0	0	1	2	0	0	0	0	17	0	926	2	1	0	1	4	0	4
22	78	5	9	2	2	1	0	0	0	1	2	0	3	2	1	0	6	1	0	6	0	1	933	0	1	0	2	1	4
23	42	2	0	2	2	1	12	0	0	3	2	3	0	6	0	0	0	0	7	0	0	1	0	826	5	5	2	2	3
24	6	4	0	4	6	1	12	1	16	5	9	1	7	2	2	2	0	5	4	0	0	0	2	1	886	1	0	1	0
25	35	5	0	3	0	0	0	0	0	0	1	1	0	3	1	0	0	0	4	0	0	0	1	7	1	927	0	4	1
26	25	9	18	3	1	28	1	71	0	2	0	1	0	1	6	1	2	1	1	6	5	5	6	2	0	0	736	3	26
27	4	4	3	4	0	1	5	2	0	1	2	2	0	1	1	0	0	0	0	0	0	0	1	6	1	0	3	914	0
28	44	11	5	7	1	5	6	6	1	1	1	1	1	12	17	0	99	0	1	2	1	22	8	6	0	1	40	3	666

Ground Truth Category/Erroneous Category		Total Samples	Error Samples
desktop cloud - linux desktop cloud		3 289
	• desktop cloud - OS issues		166	5.0%
desktop cloud - OS issues		2 071
	• desktop cloud - linux desktop cloud		216	10.4%
	• desktop cloud - intranet client-side login		233	11.3%
	• desktop cloud - blue or black screen		203	9.8%
PC side zmail - operation issues		2 146
	• PC side zmail - account issues		139	6.5%
desktop cloud - intranet client-side login		1 974
	• desktop cloud - OS issues		242	12.3%
	• desktop cloud - blue or black screen		113	5.7%
PC side zmail - account issues		1 381
	• PC side zmail - account creation & login		196	14.2%
PC side zmail - account creation & login		1 188
	• PC side zmail - operation issues		125	10.5%
ifol finance - cms contract management web		1 206
	• ifol finance - cms contract management form		132	10.9%
ifol finance - cms contract management form		1 159
	• ifol finance - cms contract management web		140	12.1%
desktop cloud - blue or black screen		1 173
	• desktop cloud - OS issues		174	14.8%
	• desktop cloud - intranet client-side login		93	7.9%
OS issues- installation		1 060
	• AIOps - itsp service website		78	7.4%
individual network issue - restriction		960
	• network proxy - usage issues		71	7.4%
uds failure- usage issues		968
	• uds failure - sercurity check		99	10.2%