Neursafe-FL: A Reliable, Efficient, Easy-to- Use Federated Learning Framework

doi:10.12142/ZTECOM.202203006

ZTE Communications ›› 2022, Vol. 20 ›› Issue (3): 43-53.DOI: 10.12142/ZTECOM.202203006

• Special Topic • Previous Articles Next Articles

Neursafe-FL: A Reliable, Efficient, Easy-to- Use Federated Learning Framework

TANG Bo¹, ZHANG Chengming¹, WANG Kewen¹, GAO Zhengguang², HAN Bingtao²()

^1.ZTE Corporation, Shenzhen 518057, China
^2.The State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen 518055, China

Received:2022-06-18 Online:2022-09-13 Published:2022-09-14
About author:TANG Bo received his master's degree from Northwestern Polytechnical University, China in 2005. Currently, as the system architect of ZTE Corporation, he is mainly responsible for federated learning and AI security solutions. His current research interests include container and container cloud, heterogeneous resource scheduling, and AI security and trustworthy AI. He holds several patents in the above research areas.|ZHANG Chengming received his BS degree from Yangzhou University, China in 2011, and ME degree from Nanjing University of Posts and Telecommunications, China in 2015. He is a senior R&D engineer of ZTE Corporation. His current research interests include AI platform, container cloud, resource scheduling, federated learning, and AI security.|WANG Kewen received his BS degree from China University of Mining and Technology, China in 2015, and the ME degree from Beijing Institute of Technology, China in 2017. He is an AI senior algorithm engineer of ZTE Corporation. His current research interests include AI Systems and AI security, such as federated learning, adversarial attack, and defense in deep learning.|GAO Zhengguang received his PhD degree from Beijing University of Posts and Telecommunications, China in 2020. He was a visiting PhD student in High Performance Networks Group, University of Bristol, UK from 2018 to 2019. After graduation, he was selected for “LAN JIAN” program of ZTE Corporation as an algorithm researcher. His current research interests include 5G/6G communication technologies, mobile networks, and machine learning for future communications.|HAN Bingtao (han.bingtao@zte.com.cn) received his BS degree from Tianjin University, China in 2001, and MS degree from Nankai University, China in 2004. He is the deputy director of the State Key Laboratory of Mobile Network and Mobile Multimedia Technology, and the leader for “Adlik” project of the LF AI & Data Foundation. Currently, he is the AI system architect of Central R&D Institute, ZTE Corporation. His current research interests include deep learning algorithms, AI systems, and network intelligence. He is the author and co-author for numbers of patents and related monographs.

Abstract

Abstract:

Federated learning (FL) has developed rapidly in recent years as a privacy-preserving machine learning method, and it has been gradually applied to key areas involving privacy and security such as finance, medical care, and government affairs. However, the current solutions to FL rarely consider the problem of migration from centralized learning to federated learning, resulting in a high practical threshold for federated learning and low usability. Therefore, we introduce a reliable, efficient, and easy-to-use federated learning framework named Neursafe-FL. Based on the unified application program interface (API), the framework is not only compatible with mainstream machine learning frameworks, such as Tensorflow and Pytorch, but also supports further extensions, which can preserve the programming style of the original framework to lower the threshold of FL. At the same time, the design of componentization, modularization, and standardized interface makes the framework highly extensible, which meets the needs of customized requirements and FL evolution in the future. Neursafe-FL is already on Github as an open-source project¹.

Key words: federated learning, privacy-preserving, Neursafe-FL

TANG Bo, ZHANG Chengming, WANG Kewen, GAO Zhengguang, HAN Bingtao. Neursafe-FL: A Reliable, Efficient, Easy-to- Use Federated Learning Framework[J]. ZTE Communications, 2022, 20(3): 43-53.

Figures/Tables 15

Figure 1 Federated learning scenarios

Table 1 Comparison of open source frameworks

Concerns	Features	TFF	PySyft	FedML	FATE	PaddleFL	Neursafe-FL
Supported running mode	Standalone	√	√	√	√	√	√
	Cross-device	×	×	√	×	×	√
	Cross-silo	×	×	√	√	√	√
Aggregation algorithms	IID (FedAvg, etc.)	√	√	√	√	√	√
Aggregation algorithms	Non-IID (FedProx, etc.)	×	×	√	√	-	√
Supported underlying framework	Tensorflow	√	√	×	√	×	√
Supported underlying framework	Pytorch	×	√	√	√	×	√
Privacy protection methods	DP	√	√	√	×	√	√
	MPC	×	√	×	√	√	√
	HE	×	√	×	√	×	×
Flexibility	Device management	×	×	×	×	×	√
Flexibility	Customization	×	×	√	×	×	√

Figure 2 Example of model migration for MNIST

Figure 3 Multi-running mode

Figure 4 Architecture of Neursafe-FL

Figure 5 Interaction between Neursafe-FL core components

Figure 6 Comparison of federated training and centralized training

Figure 7 Convergence comparison of federated training with different client numbers

Figure 8 Results of different aggregation algorithms under non-IID data

Figure 9 A comparative test of introducing DP and not introducing differential privacy

Table 2 Results of different parameters for differential privacy

Type	Round Number	Noise_multiplier	Time Cost/s	Accuracy
Without DP	100	-	2 873.22	0.914 3
Without DP	50	-	1 396.56	0.898 4
With DP	100	0.01	2 879.36	0.912 2
	100	0.005	2 882.86	0.891 0
	50	0.01	1 390.42	0.876 3

Figure 10 A comparative test of federated training with SSA and without SSA

Figure 11 Impacts of SSA in the scenario where the clients are disconnected

Figure 12 Impacts of client’s number on the performance of SSA

Figure 13 Impacts of different dropout rates on the performance of federated training

References 61

1	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data [C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2017: 1273–1282
2	YANG Q, LIU Y, CHEN T J, et al. Federated machine learning: concept and applications [EB/OL]. [2022-04-01].
3	BONAWITZ K, EICHNER H, GRIESKAMP W, et al. Towards federated learning at scale: system design [EB/OL]. [2022-04-01].
4	KAIROUZ E B P, MCMAHAN H B. Advances and open problems in federated learning [J]. Foundations and trends in machine learning, 2021, 14(1): 1–21. DOI: 10.1561/2200000083 DOI
5	LI T, SAHU A K, TALWALKAR A, et al. Federated learning: challenges, methods, and future directions [J]. IEEE signal processing magazine, 2020, 37(3): 50–60. DOI: 10.1109/MSP.2020.2975749 DOI
6	VOIGT P, VON DEM BUSSCHE A. The EU general data protection regulation (GDPR): a practical guide [M]. Cham: Springer International Publishing, 2017
7	LONG G D, TAN Y, JIANG J, et al. Federated learning for open banking federated learning [J]. Federated Learning 2020: 240–254. DOI: 10.1007/978-3-030-63076-8_17 DOI
8	XU J, GLICKSBERG B S, SU C, et al. Federated learning for healthcare informatics [J]. Journal of healthcare informatics research, 2021, 5(1): 1–19. DOI: 10.1007/s41666-020-00082-4 DOI
9	JIANG J C, KANTARCI B, OKTUG S, et al. Federated learning in smart city sensing: challenges and opportunities [J]. Sensors, 2020, 20(21): 6230. DOI: 10.3390/s20216230 DOI
10	LYU L, YU H, YANG Q. Threats to federated learning: a survey [EB/OL]. [2022-04-01].
11	BAGDASARYAN E, VEIT A, HUA Y, et al. How to backdoor federated learning [C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2020: 2938–2948
12	WANG Z B, SONG M K, ZHANG Z F, et al. Beyond inferring class representatives: user-level privacy leakage from federated learning [C]//IEEE Conference on Computer Communications. IEEE, 2019: 2512–2520. DOI: 10.1109/INFOCOM.2019.8737416 DOI
13	NASR M, SHOKRI R, HOUMANSADR A. Comprehensive privacy analysis of deep learning: passive and active white-box inference attacks against centralized and federated learning [C]//IEEE Symposium on Security and Privacy. IEEE, 2019: 739–753. DOI: 10.1109/SP.2019.00065 DOI
14	ZHAO Y, LI M, LAI L, et al. Federated learning with non-IID [EB/OL]. (2018-06-02) [2022-04-01].
15	LI X, HUANG K, YANG W, et al. On the convergence of FedAvg on non-IID data [EB/OL]. (2020-06-25) [2022-04-01].
16	ZHANG J L, CHEN J J, WU D, et al. Poisoning attack in federated learning using generative adversarial nets [C]//18th IEEE International Conference on Trust, Security and Privacy in Computing and Communications. IEEE, 2019: 374–380. DOI: 10.1109/TrustCom/BigDataSE.2019.00057 DOI
17	FANG M, CAO X, JIA J, et al. Local model poisoning attacks to byzantine-robust federated learning [C]//29th USENIX Security Symposium. USENIX, 2020: 1605–1622
18	REISIZADEH A, TZIOTIS I, HASSANI H, et al. Straggler-resilient federated learning: leveraging the interplay between statistical accuracy and system heterogeneity [EB/OL]. [2022-04-01].
19	DWORK C. Differential privacy: a survey of results [C]//International Conference on Theory and Applications of Models of Computation. TAMC, 2008: 1–19
20	MCMAHAN H B, ANDREW G, ERLINGSSON U, et al. A general approach to adding differential privacy to iterative training procedures [EB/OL]. [2022-04-01].
21	ABADI M, CHU A, GOODFELLOW I, et al. Deep learning with differential privacy [C]//ACM SIGSAC Conference on Computer and Communications Security. ACM, 2016: 308–318
22	DWORK C, ROTH A. The algorithmic foundations of differential privacy [J]. Foundations and trends in theoretical computer science, 2014, 9(3–4): 211–407
23	BU Z, DONG J, LONG Q, et al. Deep learning with gaussian differential privacy [J]. Harvard data science review, 2020, 23. DOI: 10.1162/99608f92.cfc5dd25 DOI
24	GOLDREICH O. Secure multi-party computation [EB/OL]. [2022-04-01].
25	DU W, ATALLAH M J. Secure multi-party computation problems and their applications: a review and open problems [C]//Proceedings of the 2001 Workshop on New Security Paradigms. ACM, 2001: 13–22
26	MOHASSEL P, RINDAL P. ABY3: a mixed protocol framework for machine learning [C]//Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM: 2018: 35–52. DOI: 10.1145/3243734.3243760 DOI
27	CHEN V, PASTRO V, RAYKOVA M. Secure computation for machine learning with SPDZ [EB/OL]. [2022-04-01].
28	BONAWITZ K, IVANOV V, KREUTER B, et al. Practical secure aggregation for privacy-preserving machine learning [C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 2017. DOI: 10.1145/3133956.3133982 DOI
29	GENTRY C. Fully homomorphic encryption using ideal lattices [C]//Proceedings of the 41st Annual ACM Symposium on Theory of Computing. ACM, 2009: 169–178. DOI: 10.1145/1536414.1536440 DOI
30	GENTRY C. A fully homomorphic encryption scheme [M]. Stanford, USA: Stanford University, 2009
31	ACAR A, AKSU H, ULUAGAC A S, et al. A survey on homomorphic encryption schemes [J]. ACM computing surveys, 2019, 51(4): 1–35. DOI: 10.1145/3214303 DOI
32	ZHANG C, LI S, XIA J, et al. BatchCrypt: efficient homomorphic encryption for cross-silo federated learning [C]//2020 USENIX Annual Technical Conference. USENIX, 2020: 493–506
33	FANG H K, QIAN Q. Privacy preserving machine learning with homomorphic encryption and federated learning [J]. Future Internet, 2021, 13(4): 94. DOI: 10.3390/fi13040094 DOI
34	WANG J, LIU Q, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization [J]. Advances in neural information processing systems, 2020, 33: 7611–7623
35	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks [EB/OL]. [2022-04-01].
36	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for on-device federated learning [EB/OL]. [2022-04-01].
37	HE C Y, ANNAVARAM M, AVESTIMEHR S. Towards non-IID and invisible data with FedNAS: federated deep learning via neural architecture search [EB/OL]. [2022-04-01].
38	BLANCHARD P, MHAMDI E M EL, GUERRAOUI R, et al. Machine learning with adversaries: Byzantine tolerant gradient descent [C]//The 31st International Conference on Neural Information Processing Systems. ACM, 2017
39	REISIZADEH A, FARNIA F, PEDARSANI R, et al. Robust federated learning: the case of affine distribution shifts [J]. Advances in neural information processing systems, 2020, 33: 21554–21565
40	PARK J, HAN D J, CHOI M, et al. Sageflow: robust federated learning against both stragglers and adversaries [J]. Advances in neural information processing systems, 2021, 34: 840–851
41	PILLUTLA K, KAKADE S M, HARCHAOUI Z. Robust aggregation for federated learning [J]. IEEE transactions on signal processing, 2022, 70: 1142–1154. DOI: 10.1109/TSP.2022.3153135 DOI
42	LI S, CHENG Y, WANG W, et al. Learning to detect malicious clients for robust federated learning [EB/OL]. [2022-04-01].
43	SATTLER F, WIEDEMANN S, MULLER K R, et al. Robust and communication-efficient federated learning from non-IID data [J]. IEEE transactions on neural networks and learning systems, 2020, 31(9): 3400–3413. DOI: 10.1109/TNNLS.2019.2944481 DOI
44	NISHIO T, YONETANI R. Client selection for federated learning with heterogeneous resources in mobile edge [C]//2019 IEEE International Conference on Communications. IEEE, 2019 : 1–7. DOI: 10.1109/ICC.2019.8761315 DOI
45	GEYER R C, KLEIN T, NABI M. Differentially private federated learning: A client level perspective [EB/OL]. [2022-04-01].
46	KANG J W, XIONG Z H, NIYATO D, et al. Incentive design for efficient federated learning in mobile networks: a contract theory approach [C]//2019 IEEE VTS Asia Pacific Wireless Communications Symposium. IEEE, 2019: 1–5. DOI: 10.1109/VTS-APWCS.2019.8851649 DOI
47	CHEN W, HORVATH S, RICHTARIK P. Optimal client sampling for federated learning [EB/OL]. [2022-04-01].
48	KONEČNÝ J, MCMAHAN H B, YU F X, et al. Federated learning: strategies for improving communication efficiency [EB/OL]. [2022-04-01].
49	TRAN N H, BAO W, ZOMAYA A, et al. Federated learning over wireless networks: optimization model design and analysis [C]//IEEE Conference on Computer Communications. IEEE, 2019: 1387–1395. DOI: 10.1109/INFOCOM.2019.8737464 DOI
50	GUHA N, TALWALKAR A, SMITH V. One-shot federated learning [EB/OL]. [2022-04-01].
51	CHOI B, SOHN J Y, HAN D J, et al. Communication-computation efficient secure aggregation for federated learning [EB/OL]. [2022-04-01].
52	DU W, ZENG X, YAN M, et al. Efficient federated learning via variational dropout [EB/OL]. [2022-04-01].
53	REN J J, WANG H C, HOU T T, et al. Federated learning-based computation offloading optimization in edge computing-supported Internet of Things [J]. IEEE access, 7: 69194–69201. DOI: 10.1109/ACCESS.2019.2919736 DOI
54	YU H, LIU Z L, LIU Y, et al. A fairness-aware incentive scheme for federated learning [C]//Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. ACM, 2020: 393–399. DOI: 10.1145/3375627.3375840 DOI
55	LI T, SANJABI M, SMITH V. Fair resource allocation in federated learning [EB/OL]. [2022-04-01].
56	LYU L, XU X, WANG Q, et al. Collaborative fairness in federated learning [M]//Federated Learning. Cham: Springer, 2020: 189–204. DOI: 10.1007/978-3-030-63076-8_14 DOI
57	INGERMAN A, OSTROWSKI K. TensorFlow federated [EB/OL]. [2022-04-01].
58	LIU Y, FAN T, CHEN T, et al. FATE: an industrial grade platform for collaborative learning with data protection [J]. Journal of machine learning research, 2021, 22(226): 1–6
59	ZILLER A, TRASK A, LOPARDO A, et al. Pysyft: a library for easy federated learning [M]//Federated learning systems. Cham: Springer, 2021: 111–139
60	HE C Y, LI S Z, SO J, et al. FedML: a research library and benchmark for federated machine learning [EB/OL]. [2022-04-01].
61	MA Y, YU D, WU T, et al. PaddlePaddle: an open-source deep learning platform from industrial practice [J]. Frontiers of data and computing, 2019, 1(1): 105–115

[1]	YAN Yuna, LIU Ying, NI Tao, LIN Wensheng, LI Lixin. Content Popularity Prediction via Federated Learning in Cache-Enabled Wireless Networks [J]. ZTE Communications, 2023, 21(2): 18-24.
[2]	ZHAO Moke, HUANG Yansong, LI Xuan. Federated Learning for 6G: A Survey From Perspective of Integrated Sensing, Communication and Computation [J]. ZTE Communications, 2023, 21(2): 25-33.
[3]	ZHANG Weiting, LIANG Haotian, XU Yuhua, ZHANG Chuan. Reliable and Privacy-Preserving Federated Learning with Anomalous Users [J]. ZTE Communications, 2023, 21(1): 15-24.
[4]	WANG Yiji, WEN Dingzhu, MAO Yijie, SHI Yuanming. RIS-Assisted Federated Learning in Multi-Cell Wireless Networks [J]. ZTE Communications, 2023, 21(1): 25-37.
[5]	TONG Ze, DENG Bowen, ZHENG Lele, ZHANG Tao. Utility-Improved Key-Value Data Collection with Local Differential Privacy for Mobile Devices [J]. ZTE Communications, 2022, 20(4): 15-21.
[6]	WANG Pengfei, SONG Wei, SUN Geng, WEI Zongzheng, ZHANG Qiang. Air-Ground Integrated Low-Energy Federated Learning for Secure 6G Communications [J]. ZTE Communications, 2022, 20(4): 32-40.
[7]	NAN Yucen, FANG Minghao, ZOU Xiaojing, DOU Yutao, Albert Y. ZOMAYA. A Collaborative Medical Diagnosis System Without Sharing Patient Data [J]. ZTE Communications, 2022, 20(3): 3-16.
[8]	HAN Xuming, GAO Minghan, WANG Limin, HE Zaobo, WANG Yanze. A Survey of Federated Learning on Non-IID Data [J]. ZTE Communications, 2022, 20(3): 17-26.
[9]	LIU Qinbo, JIN Zhihao, WANG Jiabo, LIU Yang, LUO Wenjian. MSRA-Fed: A Communication-Efficient Federated Learning Method Based on Model Split and Representation Aggregate [J]. ZTE Communications, 2022, 20(3): 35-42.
[10]	SHI Wenqi, SUN Yuxuan, HUANG Xiufeng, ZHOU Sheng, NIU Zhisheng. Scheduling Policies for Federated Learning in Wireless Networks: An Overview [J]. ZTE Communications, 2020, 18(2): 11-19.
[11]	YANG Howard H., ZHAO Zhongyuan, QUEK Tony Q. S.. Enabling Intelligence at Network Edge:An Overview of Federated Learning [J]. ZTE Communications, 2020, 18(2): 2-10.
[12]	Xinhua Dong, Ruixuan Li, Wanwan Zhou, Dongjie Liao, and Shuoyi Zhao. Data Security and Privacy in Cloud Storage [J]. ZTE Communications, 2013, 11(2): 18-23.

Neursafe-FL: A Reliable, Efficient, Easy-to- Use Federated Learning Framework

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 61

Related Articles 12

Recommended Articles

Metrics