Multi-User MmWave Beam Tracking via Multi-Agent Deep Q-Learning

doi:10.12142/ZTECOM.202302008

ZTE Communications ›› 2023, Vol. 21 ›› Issue (2): 53-60.DOI: 10.12142/ZTECOM.202302008

• Special Topic • Previous Articles Next Articles

Multi-User MmWave Beam Tracking via Multi-Agent Deep Q-Learning

MENG Fan¹(), HUANG Yongming², LU Zhaohua³, XIAO Huahua³

^1.Purple Mountain Laboratories, Nanjing 211111, China
^2.School of Information Science and Engineering, Southeast University, Nanjing 210096, China
^3.State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE Corporation, Shenzhen 518057, China

Received:2022-11-08 Online:2023-06-13 Published:2023-06-13
About author:MENG Fan (mengfan@pmlabs.com.cn) received his BS degree from the School of Electronic Engineering, University of Electronic Science and Technology of China in 2015 and the PhD degree with the School of Information Science and Engineering, Southeast University, China in 2020. He is now working in the Purple Mountain Laboratories. His research mainly focuses on applying machine learning techniques in the wireless communication systems. His research interests include machine learning in physical layer, model- and data-driven design, beamforming, beam alignment and tracking, and AI-enhanced positioning.|HUANG Yongming received his BS and MS degrees from Nanjing University, China in 2000 and 2003, respectively, and PhD degree in electrical engineering from Southeast University, China in 2007. Since March 2007, he has been with the School of Information Science and Engineering, Southeast University, where he is currently a full professor. During 2008–2009, he visited the Signal Processing Lab, Royal Institute of Technology, Sweden. He has authored or coauthored more than 200 peer-reviewed papers, and holds more than 80 invention patents. He has submitted 20 technical contributions to IEEE standards. His research interests include intelligent 5G/6G mobile communications and millimeter wave wireless communications.|LU Zhaohua received his PhD degree from Tianjin University, China in 2006. He is currently a senior wireless communication system research expert at ZTE Corporation and has long been engaged in the field of wireless communication system design and the key technologies of the physical layer. He has many technical contributions, papers, and patents in interference mitigation in the MIMO field.|XIAO Huahua received his MS degree in computer software and theories from Sun Yat-Sen University, China. He is currently a senior engineer in the field of antenna algorithm pre-research with ZTE Corporation. He has applied for more than 150 patents in the multi-antenna field home and abroad.

Abstract

Abstract:

Beamforming is significant for millimeter wave multi-user massive multi-input multi-output systems. In the meanwhile, the overhead cost of channel state information and beam training is considerable， especially in dynamic environments. To reduce the overhead cost, we propose a multi-user beam tracking algorithm using a distributed deep Q-learning method. With online learning of users’ moving trajectories, the proposed algorithm learns to scan a beam subspace to maximize the average effective sum rate. Considering practical implementation, we model the continuous beam tracking problem as a non-Markov decision process and thus develop a simplified training scheme of deep Q-learning to reduce the training complexity. Furthermore, we propose a scalable state-action-reward design for scenarios with different users and antenna numbers. Simulation results verify the effectiveness of the designed method.

Key words: multi-agent deep Q-learning, centralized training and distributed execution, mmWave communication, beam tracking, scalability

MENG Fan, HUANG Yongming, LU Zhaohua, XIAO Huahua. Multi-User MmWave Beam Tracking via Multi-Agent Deep Q-Learning[J]. ZTE Communications, 2023, 21(2): 53-60.

Figures/Tables 8

Figure 1 Three phases of a time slot

Figure 2 State of user u after masking

Table 1 Deep Q-network (DQN) setting

DQL type	Centralized DQL	Distributed DQL
DQL type	Centralized DQL	(proposed)
Output layer	linear, $L U$	linear, $L$
Hidden layer	ReLU, $32$	ReLU, $32$
Input layer	linear, $M × U$	linear, $2 B$

Table 1 Deep Q-network (DQN) setting

DQL type	Centralized DQL	Distributed DQL
DQL type	Centralized DQL	(proposed)
Output layer	linear, $L U$	linear, $L$
Hidden layer	ReLU, $32$	ReLU, $32$
Input layer	linear, $M × U$	linear, $2 B$

Figure 3 PSB performance versus training epochs

Figure 4 PSB performance versus user numbers

Table 2 Training time cost

User Number	Centralized DQL/s	Centralized DQL	Distributed DQL
User Number	Centralized DQL/s	(simplified)/s	(proposed)/s
$U = 1$	10.61	8.82	10.09
$U = 2$	11.91	10.18	11.51
$U = 3$	16.32	14.10	12.82
$U = 4$	87.55	61.97	14.40

Table 2 Training time cost

User Number	Centralized DQL/s	Centralized DQL	Distributed DQL
User Number	Centralized DQL/s	(simplified)/s	(proposed)/s
$U = 1$	10.61	8.82	10.09
$U = 2$	11.91	10.18	11.51
$U = 3$	16.32	14.10	12.82
$U = 4$	87.55	61.97	14.40

Figure 5 AESR performance versus BS antenna numbers

Figure 6 Scalability of proposed distributed DQL

References 13

1	KIM Y J, CHO Y S. Beam-tracking technique for millimeter-wave cellular systems using subarray structures [J]. IEEE transactions on vehicular technology, 2018, 67(8): 7806–7810. DOI: 10.1109/TVT.2018.2834346 DOI
2	DUAN Q Y, KIM T, HUANG H, et al. AoD and AoA tracking with directional sounding beam design for millimeter wave MIMO systems [C]//The 26th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC). IEEE, 2015: 2271–2276. DOI: 10.1109/PIMRC.2015.7343676 DOI
3	ZHANG D Y, LI A, SHIRVANIMOGHADDAM M, et al. Codebook-based training beam sequence design for millimeter-wave tracking systems [J]. IEEE transactions on wireless communications, 2019, 18(11): 5333–5349. DOI: 10.1109/twc.2019.2935731 DOI
4	HUANG H, PENG Y, YANG J, et al. Fast beamforming design via deep learning [J]. IEEE transactions on vehicular technology, 2020, 69(1): 1065–1069. DOI: 10.1109/tvt.2019.2949122 DOI
5	HE H T, JIN S, WEN C K, et al. Model-driven deep learning for physical layer communications [J]. IEEE wireless communications, 2019, 26(5): 77–83. DOI: 10.1109/mwc.2019.1800447 DOI
6	HE W L, ZHANG C, HUANG Y M, et al. Intelligent optimization of base station array orientations via scenario-specific modeling [J]. IEEE transactions on communications, 2022, 70(3): 2117–2130. DOI: 10.1109/TCOMM.2021.3135532 DOI
7	SU J Y, MENG F, LIU S H, et al. Learning to predict and optimize imperfect MIMO system performance: Framework and application [C]//The IEEE Global Communications Conference. IEEE, 2023: 335–340. DOI: 10.1109/GLOBECOM48099.2022.10001369 DOI
8	VA V, CHOI J, SHIMIZU T, et al. Inverse multipath fingerprinting for millimeter wave V2I beam alignment [J]. IEEE transactions on vehicular technology, 2018, 67(5): 4042–4058. DOI: 10.1109/TVT.2017.2787627 DOI
9	MENG F, LIU S H, HUANG Y M, et al. Learning-aided beam prediction in mmWave MU-MIMO systems for high-speed railway [J]. IEEE transactions on communications, 2022, 70(1): 693–706. DOI: 10.1109/TCOMM.2021.3124963 DOI
10	ZHANG J J, HUANG Y M, ZHOU Y, et al. Beam alignment and tracking for millimeter wave communications via bandit learning [J]. IEEE transactions on communications, 2020, 68(9): 5519–5533. DOI: 10.1109/TCOMM.2020.2988256 DOI
11	ZHANG J J, HUANG Y M, WANG J H, et al. Intelligent interactive beam training for millimeter wave communications [J]. IEEE transactions on wireless communications, 2021, 20(3): 2034–2048. DOI: 10.1109/TWC.2020.3038787 DOI
12	SUTTON R S, BARTO A G. Reinforcement learning: an introduction [M]. Cambridge, USA: MIT Press. 2018
13	MENG F, CHEN P, WU L N, et al. Power allocation in multi-user cellular networks: Deep reinforcement learning approaches [J]. IEEE transactions on wireless communications, 2020, 19(10): 6255–6267. DOI: 10.1109/TWC.2020.3001736 DOI

[1]	HE Miao, LI Xiangman, NI Jianbing. Physical Layer Security for MmWave Communications: Challenges and Solutions [J]. ZTE Communications, 2022, 20(4): 41-51.
[2]	Yongsheng Hu, Tian Tian, and Jun Wang. D-ZENIC: A Scalable Distributed SDN Controller Architecture [J]. ZTE Communications, 2014, 12(2): 23-27.

Multi-User MmWave Beam Tracking via Multi-Agent Deep Q-Learning

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 13

Related Articles 2

Recommended Articles

Metrics