Hierarchical Federated Learning Architectures for the Metaverse

doi:10.12142/ZTECOM.202402006

ZTE Communications ›› 2024, Vol. 22 ›› Issue (2): 39-48.DOI: 10.12142/ZTECOM.202402006

• Special Topic • Previous Articles Next Articles

Hierarchical Federated Learning Architectures for the Metaverse

GU Cheng, LI Baochun

Department of Electrical and Computer Engineering, University of Toronto, Toronto M5S2E8, Canada

Received:2024-05-31 Online:2024-06-25 Published:2024-06-25
About author:GU Cheng received his BASc Degree of Honours in Computer Engineering Cooperative Program with Distinction in 2022, and his MASc degree from the Department of Electrical and Computer Engineering in May 2024, both from University of Waterloo, Canada. His research interests focus on building next-generation AI assisted distributed systems.
LI Baochun (bli@ece. toronto. edu) received his BE degree from Tsinghua University, China in 1995 and his MS and PhD degrees from the University of Illinois at Urbana-Champaign, USA in 1997 and 2000, respectively. Since 2000, he has been with the Department of Electrical and Computer Engineering, the University of Toronto, Canada, where he is currently a Professor. Since August 2005, he has been holding the Bell Canada Endowed Chair in computer engineering. He was the recipient of IEEE Communications Society Leonard G. Abraham Award in the field of communications systems in 2000, the Multimedia Communications Best Paper Award from the IEEE Communications Society in 2009, the University of Toronto McLean Award in 2009, the Best Paper Award from IEEE INFOCOM in 2023, and the IEEE INFOCOM Achievement Award in 2024. He is a Fellow of the Canadian Academy of Engineering, the Engineering Institute of Canada, and IEEE. His current research interests include cloud computing, security and privacy, distributed machine learning, federated learning, and networking.

Abstract

Abstract:

In the context of edge computing environments in general and the metaverse in particular, federated learning (FL) has emerged as a distributed machine learning paradigm that allows multiple users to collaborate on training a shared machine learning model locally, eliminating the need for uploading raw data to a central server. It is perhaps the only training paradigm that preserves the privacy of user data, which is essential for computing environments as personal as the metaverse. However, the original FL architecture proposed is not scalable to a large number of user devices in the metaverse community. To mitigate this problem, hierarchical federated learning (HFL) has been introduced as a general distributed learning paradigm, inspiring a number of research works. In this paper, we present several types of HFL architectures, with a special focus on the three-layer client-edge-cloud HFL architecture, which is most pertinent to the metaverse due to its delay-sensitive nature. We also examine works that take advantage of the natural layered organization of three-layer client-edge-cloud HFL to tackle some of the most challenging problems in FL within the metaverse. Finally, we outline some future research directions of HFL in the metaverse.

Key words: federated learning, hierarchical federated learning, metaverse

GU Cheng, LI Baochun. Hierarchical Federated Learning Architectures for the Metaverse[J]. ZTE Communications, 2024, 22(2): 39-48.

Figures/Tables 7

References 22

1	MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data [EB/OL]. (2016-02-17) [2024-04-16].
2	ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data [EB/OL]. (2018-06-02) [2024-04-16].
3	MAO Y Y, YOU C S, ZHANG J, et al. A survey on mobile edge computing: The communication perspective [J]. IEEE communications surveys & tutorials, 2017, 19(4): 2322–2358. DOI: 10.1109/COMST.2017.2745201
4	KREUTZ D, RAMOS F M V, ESTEVES VERISSIMO P, et al. Software-defined networking: A comprehensive survey [J]. Proceedings of the IEEE, 2015, 103(1): 14–76. DOI: 10.1109/jproc.2014.2371999
5	LIU L M, ZHANG J, SONG S H, et al. Client-edge-cloud hierarchical federated learning [C]//IEEE International Conference on Communications (ICC). IEEE, 2020: 1–6. DOI: 10.1109/ICC40277.2020.9148862
6	ABAD M S H, OZFATURA E, GUNDUZ D, et al. Hierarchical federated learning ACROSS heterogeneous cellular networks [C]//Proceedings of ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8866–8870. DOI: 10.1109/ICASSP40776.2020.9054634
7	LIU L M, ZHANG J, SONG S H, et al. Hierarchical federated learning with quantization: Convergence analysis and system design [EB/OL]. (2021-03-26) [2024-04-16].
8	LIU C, CHUA T J, ZHAO J. Time minimization in hierarchical federated learning [EB/OL]. (2022-10-07) [2024-04-16].
9	LIU X F, WANG Q, SHAO Y F, et al. Sparse federated learning with hierarchical personalized models [EB/OL]. (2023-09-25) [2024-04-16].
10	NGUYEN M D, PHAM Q V, HOANG D T, et al. Label driven Knowledge Distillation for Federated Learning with non-IID Data [EB/OL]. (2022-09-30) [2024-04-16].
11	Wang X, Wang Y J. Asynchronous Hierarchical Federated Learning [EB/OL]. (2022-05-31) [2024-04-16].
12	WANG J Y, WANG S Q, CHEN R R, et al. Demystifying why local aggregation helps: convergence analysis of hierarchical SGD [J]. Proceedings of the AAAI conference on artificial intelligence, 2022, 36(8): 8548–8556. DOI: 10.1609/aaai.v36i8.20832
13	WU W T, HE L G, LIN W W, et al. Accelerating federated learning over reliability-agnostic clients in mobile edge computing systems [J]. IEEE transactions on parallel and distributed systems, 2021, 32(7): 1539–1551. DOI: 10.1109/TPDS.2020.3040867
14	RIZK E, SAYED A H. A graph federated architecture with privacy preserving learning [C]//The 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). IEEE, 2021: 131–135. DOI: 10.1109/SPAWC51858.2021.9593148
15	SUN Y C, SHAO J W, MAO Y Y, et al. Semi-decentralized federated edge learning for fast convergence on non-IID data [C]//Proceedings of IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2022: 1898–1903. DOI: 10.1109/WCNC51071.2022.9771904
16	ZHONG Z C, ZHOU Y P, WU D, et al. P-FedAvg: Parallelizing federated learning with theoretical guarantees [C]//IEEE Conference on Computer Communications. IEEE, 2021: 1–10. DOI: 10.1109/INFOCOM42981.2021.9488877
17	DAS A, PATTERSON S. Multi-tier federated learning for vertically partitioned data [C]//IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 3100–3104. DOI: 10.1109/ICASSP39728.2021.9415026
18	HOSSEINALIPOUR S, BRINTON C G, AGGARWAL V, et al. From federated to fog learning: distributed machine learning over heterogeneous wireless networks [J]. IEEE communications magazine, 58(12): 41–47, 2020. DOI: 10.1109/MCOM.001.2000410
19	LIN T, STICH S U, PATEL K K, et al. Don’t use large mini-batches, use local SGD [EB/OL]. (2018-08-22) [2024-04-16].
20	ZHOU F, CONG G J. A distributed hierarchical SGD algorithm with sparse global reduction [EB/OL]. (2022-02-17) [2024-04-16].
21	ZHU G, LIU D, DU Y, et al. Toward an intelligent edge: wireless communication meets machine learning [J]. IEEE communications magazine, 58(1): 19–25, 2020. DOI: 10.1109/MCOM.001.1900103
22	KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and Open Problems in Federated Learning [J]. Foundations and trends^® in machine learning, 14(1–2): 1–210, 2021

Architecture	Number of Layers	Client-Edge-Cloud	Scalability
HLSGD	3	Unspecified (but can be)	Unlimited
Hier-AVG	3	Unspecified (but can be)	Unlimited
HFAVG	3	Yes	Unlimited
Cross-HCN FEEL	3	No (client-edge-edge)	Limited by the coverage of the macro-cell base station
Graph-FL	2	No (client-edge)	Unlimited, but communication costs between servers exhibit quadratic growth
SD-FEEL	2	No (client-edge)	Unlimited, but may be limited by max network span in spare edge networks
Federated fog learning	N≥ 2	No (client-edge*N-cloud)	Unlimited
Multi-level HSGD	N≥ 2	No (client-edge * N-cloud)	Unlimited
F2L-LDK	3	Yes	Unlimited (supports dynamic edge participation)

Architecture	Number of Layers	Client-Edge-Cloud	Scalability
HLSGD	3	Unspecified (but can be)	Unlimited
Hier-AVG	3	Unspecified (but can be)	Unlimited
HFAVG	3	Yes	Unlimited
Cross-HCN FEEL	3	No (client-edge-edge)	Limited by the coverage of the macro-cell base station
Graph-FL	2	No (client-edge)	Unlimited, but communication costs between servers exhibit quadratic growth
SD-FEEL	2	No (client-edge)	Unlimited, but may be limited by max network span in spare edge networks
Federated fog learning	N≥ 2	No (client-edge*N-cloud)	Unlimited
Multi-level HSGD	N≥ 2	No (client-edge * N-cloud)	Unlimited
F2L-LDK	3	Yes	Unlimited (supports dynamic edge participation)

[1]	CHEN Rui, LI Hui, LI Wuyang, BAI He, WANG Han, WU Naixing, FAN Ping, KANG Jian, DENG Selwyn, ZHU Xiang. MetaOracle: A High-Throughput Decentralized Oracle for Web 3.0-Empowered Metaverse [J]. ZTE Communications, 2024, 22(2): 11-18.
[2]	YAN Yuna, LIU Ying, NI Tao, LIN Wensheng, LI Lixin. Content Popularity Prediction via Federated Learning in Cache-Enabled Wireless Networks [J]. ZTE Communications, 2023, 21(2): 18-24.
[3]	ZHAO Moke, HUANG Yansong, LI Xuan. Federated Learning for 6G: A Survey From Perspective of Integrated Sensing, Communication and Computation [J]. ZTE Communications, 2023, 21(2): 25-33.
[4]	YAN Jintao, CHEN Tan, XIE Bowen, SUN Yuxuan, ZHOU Sheng, NIU Zhisheng. Hierarchical Federated Learning: Architecture, Challenges, and Its Implementation in Vehicular Networks [J]. ZTE Communications, 2023, 21(1): 38-45.
[5]	DING Yahao, SHIKH‑BAHAEI Mohammad, YANG Zhaohui, HUANG Chongwen, YUAN Weijie. Secure Federated Learning over Wireless Communication Networks with Model Compression [J]. ZTE Communications, 2023, 21(1): 46-54.
[6]	ZHANG Weiting, LIANG Haotian, XU Yuhua, ZHANG Chuan. Reliable and Privacy-Preserving Federated Learning with Anomalous Users [J]. ZTE Communications, 2023, 21(1): 15-24.
[7]	WANG Yiji, WEN Dingzhu, MAO Yijie, SHI Yuanming. RIS-Assisted Federated Learning in Multi-Cell Wireless Networks [J]. ZTE Communications, 2023, 21(1): 25-37.
[8]	WANG Pengfei, SONG Wei, SUN Geng, WEI Zongzheng, ZHANG Qiang. Air-Ground Integrated Low-Energy Federated Learning for Secure 6G Communications [J]. ZTE Communications, 2022, 20(4): 32-40.
[9]	NAN Yucen, FANG Minghao, ZOU Xiaojing, DOU Yutao, Albert Y. ZOMAYA. A Collaborative Medical Diagnosis System Without Sharing Patient Data [J]. ZTE Communications, 2022, 20(3): 3-16.
[10]	HAN Xuming, GAO Minghan, WANG Limin, HE Zaobo, WANG Yanze. A Survey of Federated Learning on Non-IID Data [J]. ZTE Communications, 2022, 20(3): 17-26.
[11]	LIU Qinbo, JIN Zhihao, WANG Jiabo, LIU Yang, LUO Wenjian. MSRA-Fed: A Communication-Efficient Federated Learning Method Based on Model Split and Representation Aggregate [J]. ZTE Communications, 2022, 20(3): 35-42.
[12]	TANG Bo, ZHANG Chengming, WANG Kewen, GAO Zhengguang, HAN Bingtao. Neursafe-FL: A Reliable, Efficient, Easy-to- Use Federated Learning Framework [J]. ZTE Communications, 2022, 20(3): 43-53.
[13]	SHI Wenqi, SUN Yuxuan, HUANG Xiufeng, ZHOU Sheng, NIU Zhisheng. Scheduling Policies for Federated Learning in Wireless Networks: An Overview [J]. ZTE Communications, 2020, 18(2): 11-19.
[14]	YANG Howard H., ZHAO Zhongyuan, QUEK Tony Q. S.. Enabling Intelligence at Network Edge:An Overview of Federated Learning [J]. ZTE Communications, 2020, 18(2): 2-10.

Hierarchical Federated Learning Architectures for the Metaverse

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 22

Related Articles 14

Recommended Articles

Metrics