ZTE Communications ›› 2022, Vol. 20 ›› Issue (3): 35-42.DOI: 10.12142/ZTECOM.202203005

• Special Topic • Previous Articles     Next Articles

MSRA-Fed: A Communication-Efficient Federated Learning Method Based on Model Split and Representation Aggregate

LIU Qinbo1,2, JIN Zhihao1, WANG Jiabo1, LIU Yang1,3(), LUO Wenjian1,3   

  1. 1.School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
    2.Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Shenzhen 518055, China
    3.Peng Cheng Laboratory, Shenzhen 518055, China
  • Received:2022-06-20 Online:2022-09-13 Published:2022-09-14
  • Contact: LIU Yang
  • About author:LIU Qinbo received his BS degree in mathematics and physics basic science from the School of Mathematical Sciences, University of Electronic Science and Technology of China in 2021. He is currently pursuing his ME degree with the School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China. His research interests include federated learning and GNNs.|JIN Zhihao received his BE degree in computer science and technology from the School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China in 2022. His research interests include federated learning.|WANG Jiabo received his BE degree in software engineering from the School of Information Science and Technology, Dalian Maritime University, China in 2021. He is currently pursuing his ME degree with the School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China. His research interests include federated learning.|LUO Wenjian received his BS and PhD degrees from the Department of Computer Science and Technology, University of Science and Technology of China, in 1998 and 2003, respectively. He is currently a professor with the School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China. His current research interests include computational intelligence and applications, network security and data privacy, machine learning, and data mining. Dr. LUO is also a senior member of the Association for Computing Machinery (ACM) and the China Computer Federation (CCF). He has been a member of the organizational team of more than ten academic conferences, in various functions, such as the program chair, the symposium chair and the publicity chair. He also serves as the chair of the IEEE CIS ECTC Task Force on Artificial Immune Systems. He also serves as an associate editor or an editorial board member for several journals, including Information Sciences, Swarm and Evolutionary Computation, Journal of Information Security and Applications, Applied Soft Computing, and Complex & Intelligent Systems.
  • Supported by:
    his work is supported by Shenzhen Basic Research (General Project)(JCYJ20190806142601687);Shenzhen Stable Supporting Program (General Project)(GXWD20201230155427003-20200821160539001);Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(2022B1212010005);Shenzhen Basic Research (Key Project)(JCYJ20200109113405927)

Abstract:

Recent years have witnessed a spurt of progress in federated learning, which can coordinate multi-participation model training while protecting the data privacy of participants. However, low communication efficiency is a bottleneck when deploying federated learning to edge computing and IoT devices due to the need to transmit a huge number of parameters during co-training. In this paper, we verify that the outputs of the last hidden layer can record the characteristics of training data. Accordingly, we propose a communication-efficient strategy based on model split and representation aggregate. Specifically, we make the client upload the outputs of the last hidden layer instead of all model parameters when participating in the aggregation, and the server distributes gradients according to the global information to revise local models. Empirical evidence from experiments verifies that our method can complete training by uploading less than one-tenth of model parameters, while preserving the usability of the model.

Key words: federated learning, communication load, efficient communication, privacy protection