Secure Federated Learning over Wireless Communication Networks with Model Compression

doi:10.12142/ZTECOM.202301006

ZTE Communications ›› 2023, Vol. 21 ›› Issue (1): 46-54.DOI: 10.12142/ZTECOM.202301006

• Special Topic • Previous Articles Next Articles

Secure Federated Learning over Wireless Communication Networks with Model Compression

DING Yahao¹, SHIKH‑BAHAEI Mohammad¹, YANG Zhaohui², HUANG Chongwen²(), YUAN Weijie³

^1.King’s College London, London WC2R 2LS, U. K.
^2.Zhejiang University, Hangzhou 310058, China
^3.Southern University of Science and Technology, Shenzhen 518055, China

Received:2023-02-11 Online:2023-03-25 Published:2024-03-15
About author:DING Yahao received her master's degree in communications and signal processing from Imperial College London, U.K. in 2020. She is currently pursuing her PhD degree in information and communication engineering with King's College London, U.K. Her current research interests include federated learning, security, and UAV swarms.
Mohammad SHIKH-BAHAEI received his BSc degree from the University of Tehran, Iran in 1992, MSc degree from the Sharif University of Technology, Iran in 1994, and PhD degree from King's College London, U.K. in 2000. He has worked for two start-up companies and for National Semiconductor Corporation, USA (now part of Texas Instruments Inc.). In 2002, he joined King's College London, where he is currently a full professor. Since then, he has authored numerous journals and conference papers and worked as an expert consultant to a number of international high-tech companies and legal firms. His research interests are secure communications and connected intelligence, full-duplex and cognitive dense networks, visual data communications over the IoT, applications of wireless communications in healthcare, and communication protocols for autonomous vehicle/drone networks. He has been the founder and the chair of the Wireless Advanced (formerly SPWC) Annual International Conference from 2003 to 2018.
YANG Zhaohui received his PhD degree from Southeast University, China in 2018. From 2018 to 2020, he was a postdoctoral research associate with the Center for Telecommunications Research, Department of Informatics, King's College London, U.K. From 2020 to 2022, he was a research fellow with the Department of Electronic and Electrical Engineering, University College London, U.K. He is currently a young professor with the College of Information Science and Electronic Engineering, Zhejiang Key Laboratory of Information Processing Communication and Networking, Zhejiang University, China, and also a research scientist with Zhejiang Laboratory. His research interests include joint communication, sensing and computation, federated learning, and semantic communications. He is an associate editor for IEEE Communications Letters, IET Communications, and EURASIP Journal on Wireless Communications and Networking. He was the guest editor of several journals, including JSAC, WCM and CM. He was the co-chair for international workshops with more than ten times, including ICC, GLOBECOM, WCNC, PIMRC and INFOCOM.
HUANG Chongwen (chongwenhuang@zju.edu.cn) received his BSc degree from the Binhai College, Nankai University, China in 2010, and MSc degree from the University of Electronic Science and Technology of China (UESTC), China in 2013. He has been joining the Institute of Electronics, Chinese Academy of Sciences (IECAS) as a research engineer, since July 2013. Since September 2015, he has been starting his PhD journey with the Singapore University of Technology and Design (SUTD), Singapore and CentraleSupélec University, Paris, France under the supervision of Prof. Chau YUEN and Prof. Mérouane DEBBAH. From October 2019 to September 2020, he was a post-doctoral researcher at SUTD. Since September 2020, he has been joining Zhejiang University as a Tenure-Track Young Professor. His main research interests include holographic MIMO surface/reconfigurable intelligent surface, B5G/6G wireless communications, mmWave/THz communications, and deep learning technologies for wireless communications. He was a recipient of the IEEE Marconi Prize Paper Award in Wireless Communications in 2021. He was also a recipient of the Singapore Government PhD Scholarship and received PHC Merlion PhD Grant (2016–2019) for studying in CentraleSupélec, France. He has been serving as an editor of IEEE Communications Letter, Signal Processing (Elsevier), EURASIP Journal on Wireless Communications and Networking, and Physical Communication since 2021. In addition, he has served as the chair of several wireless communications flagship conferences, including the session chair of 2021 IEEE WCNC, 2021 IEEE VTC-Fall, and the symposium chair of IEEE WCSP 2021.
YUAN Weijie received his BE degree from the Beijing Institute of Technology, China in 2013, and PhD degree from the University of Technology Sydney, Australia in 2019. In 2016, he was a visiting PhD student with the Institute of Telecommunications, Vienna University of Technology, Austria. He was a research assistant with the University of Sydney, Australia, a visiting associate fellow with the University of Wollongong, Australia and a visiting fellow with the University of Southampton, U.K. from 2017 to 2019. From 2019 to 2021, he was a research associate with the University of New South Wales, Australia. He is currently an assistant professor with the Department of Electrical and Electronic Engineering, Southern University of Science and Technology, China. He was a recipient of the Best PhD Thesis Award from the Chinese Institute of Electronics and an Exemplary Reviewer from IEEE TCOM/WCL. He currently serves as an associate editor of IEEE Communications Letters, an associate editor and an award committee member of EURASIP Journal on Advances in Signal Processing. He has led the guest editorial teams for three special issues in IEEE Communications Magazine, IEEE Transactions on Green Communications and Networking, and China Communications. He was an organizer/the chair of several workshops and special sessions on orthogonal time frequency space and integrated sensing and communication in flagship IEEE and ACM conferences, including IEEE ICC, IEEE/CIC ICCC, IEEE SPAWC, IEEE VTC, IEEE WCNC, IEEE ICASSP, and ACM MobiCom. He is the founding chair of the IEEE ComSoc Special Interest Group on Orthogonal Time Frequency Space.

Abstract

Abstract:

Although federated learning (FL) has become very popular recently, it is vulnerable to gradient leakage attacks. Recent studies have shown that attackers can reconstruct clients’ private data from shared models or gradients. Many existing works focus on adding privacy protection mechanisms to prevent user privacy leakages, such as differential privacy (DP) and homomorphic encryption. These defenses may cause an increase in computation and communication costs or degrade the performance of FL. Besides, they do not consider the impact of wireless network resources on the FL training process. Herein, we propose weight compression, a defense method to prevent gradient leakage attacks for FL over wireless networks. The gradient compression matrix is determined by the user’s location and channel conditions. We also add Gaussian noise to the compressed gradients to strengthen the defense. This joint learning of wireless resource allocation and weight compression matrix is formulated as an optimization problem with the objective of minimizing the FL loss function. To find the solution, we first analyze the convergence rate of FL and quantify the effect of the weight matrix on FL convergence. Then, we seek the optimal resource block (RB) allocation by exhaustive search or ant colony optimization (ACO) and then use the CVX toolbox to obtain the optimal weight matrix to minimize the optimization function. The simulation results show that the optimized RB can accelerate the convergence of FL.

Key words: federated learning (FL), data leakage from gradient, resource block (RB) allocation

DING Yahao, SHIKH‑BAHAEI Mohammad, YANG Zhaohui, HUANG Chongwen, YUAN Weijie. Secure Federated Learning over Wireless Communication Networks with Model Compression[J]. ZTE Communications, 2023, 21(1): 46-54.

Figures/Tables 5

Figure 1 Architecture of FL algorithm with one eavesdropper in wireless networks

Figure 2 Illustration of the differential privacy (DP) method to protect the privacy of federated learning (FL)

Figure 3 An example of proposed weight compression

Table 1 Simulation Parameters

Description	Parameter	Value
Total bandwidth of uplink	B	20 MHz
Bandwidth of each RB	B₀	3.33 MHz
Noise power spectral density	N₀	-174 dBm/MHz
Total number of training samples for user	K_i	[10, 20, 15, 25, 10]
Gradient compression ratio of user	α_i	$39, 69, 49, 69, 59$
Number of gradients for each user	M	9
Delay requirement of uplink	τ	2 s
Distance between user and BS	d	30 m
Number of RBs	Q	6
Transmit power of user	P_i	0.001–0.012 W

Table 1 Simulation Parameters

Description	Parameter	Value
Total bandwidth of uplink	B	20 MHz
Bandwidth of each RB	B₀	3.33 MHz
Noise power spectral density	N₀	-174 dBm/MHz
Total number of training samples for user	K_i	[10, 20, 15, 25, 10]
Gradient compression ratio of user	α_i	$39, 69, 49, 69, 59$
Number of gradients for each user	M	9
Delay requirement of uplink	τ	2 s
Distance between user and BS	d	30 m
Number of RBs	Q	6
Transmit power of user	P_i	0.001–0.012 W

Figure 4 Objective function as user power and resource block (RB) allocation varies

References 28

1	YANG Q, LIU Y, CHEN T J, et al. Federated machine learning: concept and applications [J]. ACM transactions on intelligent systems and technology, 2019, 10(2): 1–19. DOI: 10.1145/3298981
2	JOCHEMS A, DEIST T M, VAN SOEST J, et al. Distributed learning: developing a predictive model based on data from multiple hospitals without data leaving the hospital—a real life proof of concept [J]. Radiotherapy and oncology, 2016, 121(3): 459–467. DOI: 10.1016/j.radonc.2016.10.002
3	BONAWITZ K, EICHNER H, GRIESKAMP W, et al. Towards federated learning at scale: system design [C]//Conference on machine learning and systems. MLSys, 2019: 374–388
4	SHOKRI R, STRONATI M, SONG C Z, et al. Membership inference attacks against machine learning models [C]//IEEE Symposium on Security and Privacy (SP). IEEE, 2017: 3–18. DOI: 10.1109/SP.2017.41
5	HITAJ B, ATENIESE G, PEREZ-CRUZ F. Deep models under the GAN: information leakage from collaborative deep learning [C]//ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017: 603–618. DOI: 10.1145/3133956.3134012
6	WANG Z B, SONG M K, ZHANG Z F, et al. Beyond inferring class representatives: user-level privacy leakage from federated learning [C]//IEEE Conference on Computer Communications. IEEE, 2019: 2512–2520. DOI: 10.1109/INFOCOM.2019.8737416
7	PHONG L T, AONO Y, HAYASHI T, et al. Privacy-preserving deep learning via additively homomorphic encryption [J]. IEEE transactions on information forensics and security, 2017, 13(5): 1333–1345. DOI: 10.1109/TIFS.2017.2787987
8	ZHU L G, LIU Z J, HAN S. Deep leakage from gradients [C]//33rd Conference on Neural Information Processing Systems. NeurIPS, 2019: 8389
9	ZHAO B, MOPURI K R, BILEN H. iDLG: improved deep leakage from gradients [EB/OL]. [2020-01-08].
10	YIN H X, MALLYA A, VAHDAT A, et al. See through gradients: image batch recovery via GradInversion [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021: 16332–16341. DOI: 10.1109/CVPR46437.2021.01607
11	CHEN S, JIA R X, QI G J. Improved techniques for model inversion attack [EB/OL]. [2021-08-19].
12	JERE M S, FARNAN T, KOUSHANFAR F. A taxonomy of attacks on federated learning [J]. IEEE security privacy, 2021, 19(2): 20–28. DOI: 10.1109/MSEC.2020.3039941
13	GEIPING J, BAUERMEISTER H, DRÖGE H, et al. Inverting gradients: how easy is it to break privacy in federated learning? [C]//34th International Conference on Neural Information Processing Systems. NeurIPS, 2020: 16937–16947
14	ZHANG C L, LI S Y, XIA J Z, et al. BatchCrypt: efficient homomorphic encryption for cross-silo federated learning [C]//USENIX Conference on Usenix Annual Technical Conference. ACM, 2020: 493–506. DOI: 10.5555/3489146.3489179
15	CHENG K W, FAN T, JIN Y L, et al. SecureBoost: a lossless federated learning framework [J]. IEEE intelligent systems, 2021, 36(6): 87–98. DOI: 10.1109/MIS.2021.3082561
16	MOHASSEL P, ZHANG Y P. Secureml: a system for scalable privacy-preserving machine learning [C]//IEEE Symposium on Security and Privacy (SP). IEEE, 2017: 19–38. DOI: 10.1109/SP.2017.12
17	BONAWITZ K, IVANOV V, KREUTER B, et al. Practical secure aggregation for privacy-preserving machine learning [C]//ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017: 1175–1191. DOI: 10.1145/3133956.3133982
18	GEYER R C, KLEIN T, NABI M. Differentially private federated learning: a client level perspective [EB/OL]. [2022-03-01].
19	MCMAHAN H B, RAMAGE D, TALWAR K, et al. Learning differentially private recurrent language models [EB/OL]. [2022-02-24].
20	WEI K, LI J, DING M, et al. Federated learning with differential privacy: algorithms and performance analysis [J]. IEEE transactions on information forensics and security, 2020, 15: 3454–3469. DOI: 10.1109/TIFS.2020.2988575
21	WEI W Q, LIU L, LOPER M, et al. A framework for evaluating gradient leakage attacks in federated learning [EB/OL]. [2021-04-23].
22	SUN J W, LI A, WANG B H, et al. Soteria: provable defense against privacy leakage in federated learning from representation perspective [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021: 9307–9315. DOI: 10.1109/CVPR46437.2021.00919
23	SCHELIGA D, MÄDER P, SEELAND M. Precode—a generic model extension to prevent deep gradient leakage [C]//IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022: 3605–3614. DOI: 10.1109/WACV51458.2022.00366
24	WANG J X, GUO S, XIE X, et al. Protect privacy from gradient leakage attack in federated learning [C]//IEEE Conference on Computer Communications. IEEE, 2022: 580–589. DOI: 10.1109/INFOCOM48880.2022.9796841
25	KONEČNÝ J, MCMAHAN H B, RAMAGE D, et al. Federated optimization: distributed machine learning for on-device intelligence [EB/OL]. [2021-10-08].
26	HATAMIZADEH A, YIN H X, MOLCHANOV P, et al. Do gradient inversion attacks make federated learning unsafe? [EB/OL]. [2022-02-14].
27	TAVANGARAN N, CHEN M Z, YANG Z H, et al. On differential privacy for federated learning in wireless systems with multiple base stations [EB/OL]. [2022-08-25].
28	CHEN M Z, YANG Z H, SAAD W, et al. A joint learning and communications framework for federated learning over wireless networks [J]. IEEE transactions on wireless communications, 2021, 20(1): 269–283. DOI: 10.1109/TWC.2020.3024629

Secure Federated Learning over Wireless Communication Networks with Model Compression

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 5

References 28

Related Articles 1

Recommended Articles

Metrics