ZTE Communications ›› 2026, Vol. 24 ›› Issue (3): 59-69.DOI: 10.12142/ZTECOM.202603008

• Industry-Academia Co-Research • Previous Articles     Next Articles

Constructing Large-Scale Real-World Benchmark Datasets for AIOps

Li Zeyan1, Han Jing2(), Zhao Nengwen1, Zhang Shenglin3, Sun Yongqian3, Chen Pengfei4, Wen Xidao1, Ma Minghua5, Pei Dan1   

  1. 1.Tsinghua University, Beijing 100084, China
    2.ZTE Corporation, Shenzhen 518057, China
    3.Nankai University, Tianjin 300071, China
    4.Sun Yat?sen University, Guangzhou 510275, China
    5.Microsoft Research, Redmond, WA 98052, USA
  • Received:2025-02-21 Online:2026-09-03 Published:2026-09-20
  • About author:Li Zeyan received his PhD degree in computer science and technology from Tsinghua University, China in 2023, where he was a member of the Tsinghua NetMan Lab under the supervision of Prof. Pei Dan. He received his bachelor's degree in computer science and technology from Tsinghua University in 2018. His research interests focus on Artificial Intelligence for IT Operations (AIOps), particularly intelligent fault localization, root cause analysis, and reliability management for large-scale online service systems.
    Han Jing (han.jing28@zte.com.cn) is a researcher and engineer at ZTE Corporation, where she develops next-generation intelligent network architectures, 6G wireless intelligence, and automated software maintenance systems.
    Zhao Nengwen received her PhD degree in computer science and technology from Tsinghua University, China in 2022 under the supervision of Prof. Pei Dan, and her bachelor's degree in computer science and technology from Wuhan University, China in 2017. Her research interests include Artificial Intelligence for IT Operations (AIOps), software maintenance, and reliability management, with particular emphasis on anomaly detection, alert management, log anomaly detection, and software change analysis in large-scale online service systems.
    Zhang Shenglin is a Professor at the College of Software, Nankai University, China and an affiliated faculty member of the Tsinghua NetMan Lab. He received his PhD degree in computer science and technology from Tsinghua University, China under the supervision of Prof. Pei Dan and Prof. Liu Ying, and his bachelor's degree in network engineering from Xidian University, China in 2012. During his doctoral studies, he was a visiting scholar at the School of Computer Science, Georgia Institute of Technology, USA. His research focuses on Artificial Intelligence for IT Operations (AIOps), including failure detection, diagnosis and prediction, root cause analysis, log and multimodal observability analysis, and reliability management for large-scale service and cloud systems.
    Sun Yongqian is an Associate Professor at Nankai University, China and an affiliated faculty member of the Tsinghua NetMan Lab. He received his PhD degree from the Department of Computer Science and Technology at Tsinghua University, China under the supervision of Prof. Pei Dan and Prof. Zhao Youjian, and his bachelor's degree in statistics from Northwestern Polytechnical University, China in 2012. His research interests center on Artificial Intelligence for IT Operations (AIOps), including anomaly detection, failure diagnosis, root cause analysis, failure prediction, and intelligent operations for microservice and large-scale software systems.
    Chen Pengfei is a Professor and doctoral supervisor at the School of Computer Science and Engineering, Sun Yat-sen University, China. He received his PhD degree in computer science from Xi'an Jiaotong University, China in 2016. His research interests include Artificial Intelligence for IT Operations (AIOps), distributed systems, cloud computing, microservices, software reliability, and computer networks, with particular focus on automated performance diagnosis, root cause analysis, anomaly detection, and intelligent management of cloud-native and microservice systems.
    Wen Xidao was a postdoctoral researcher at Tsinghua University, China and a member of the Tsinghua NetMan Lab, completing his postdoctoral work in 2022. His research interests include Artificial Intelligence for IT Operations (AIOps), data mining, machine learning, time-series analysis, anomaly detection, fault diagnosis, and intelligent operations for large-scale service systems. He has collaborated extensively on research in multivariate time-series anomaly detection, software change analysis, root cause diagnosis, AIOps benchmarks, and large language models for IT operations.
    Ma Minghua is a Senior Researcher at Microsoft Research in Redmond, USA. He received his PhD degree in computer science and technology from Tsinghua University, China in 2021 under the supervision of Prof. Pei Dan. From 2019 to 2020, he was a visiting scholar at the School of Computer Science, Georgia Institute of Technology, USA. His research interests lie at the intersection of software engineering, systems, and artificial intelligence, with a particular focus on Cloud Intelligence and Artificial Intelligence for IT Operations (AIOps).
    Pei Dan is a Professor in the Department of Computer Science and Technology, Tsinghua University, China and the leader of the Tsinghua NetMan Lab. His current research focuses on Artificial Intelligence for IT Operations (AIOps) for critical infrastructure and embodied systems, as well as time-series intelligence. Before joining Tsinghua University, he was a researcher at AT&T Labs–Research in the United States. He is a Senior Member of both ACM and IEEE and has extensive research experience in networked systems, large-scale service reliability, intelligent operations, root cause analysis, anomaly detection, and time-series intelligence.
  • Supported by:
    ZTE Industry?University?Institute Cooperation Funds(IA20231009003)

Abstract:

Artificial Intelligence for IT Operations (AIOps) has emerged as a critical approach for managing complex IT systems, yet its advancement has been hindered by the lack of comprehensive public datasets for evaluation. This study addresses this challenge by introducing three large-scale, real-world benchmark datasets specifically designed for AIOps research and development. We present datasets covering key AIOps scenarios: key performance indicator (KPI) anomaly detection with diverse patterns from multiple Internet companies, multi-dimensional root cause localization with 400 labeled cases, and failure discovery and diagnosis containing 169 injected failures across various system components. These datasets are distinguished by their scale, authenticity, and detailed ground-truth labels, enabling rigorous evaluation of AIOps algorithms. The practical value of our datasets has been validated through three annual AIOps algorithm competitions (2018–2020), which attracted over 400 participating teams and led to numerous research publications. By providing these comprehensive benchmarks, our work establishes a foundation for reproducible AIOps research and facilitates fair comparison of different approaches, serving a role for AIOps analogous to that of ImageNet in computer vision.

Key words: AIOps, benchmark datasets, KPI anomaly detection, root cause localization, failure discovery and diagnosis