UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

doi:10.12142/ZTECOM.202302011

ZTE Communications ›› 2023, Vol. 21 ›› Issue (2): 80-87.DOI: 10.12142/ZTECOM.202302011

• Special Topic • Previous Articles Next Articles

UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

LI Yuting^1,, DING Yi², GAO Jiangchuan¹, LIU Yusha¹(), HU Jie¹, YANG Kun³

^1.School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
^2.China Mobile Communications Group Jilin Co. , Ltd. , Changchun 130061, China
^3.School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, United Kingdom
the University of Electronic Science and Technology of China. Her current research interests include data and energy integrated communication networks and machine learning

Received:2023-02-11 Online:2023-06-13 Published:2023-06-13
About author:LI Yuting is with the University of Electronic Science and Technology of China. Her current research interests include data and energy integrated communication networks and machine learning.|LIU Yusha (yusha.liu@uestc.edu.cn) received her PhD degree from the University of Southampton, UK, and is currently with the University of Electronic Science and Technology of China. Her current research interests include wireless communications, signal processing and deep learning.|HU Jie received his BE and MS degrees from Beijing University of Posts and Telecommunications, China in 2008 and 2011, respectively, and received his PhD degree from the School of Electronics and Computer Science, University of Southampton, UK in 2015. Since 2016, he has been working with the School of Information and Communication Engineering, University of Electronic Science and Technology of China (UESTC). He is now a research professor and PhD supervisor. He won UESTC’s Academic Young Talent Award in 2019. Now he is supported by the “100 Talents” program of UESTC. He is an editor for IEEE Wireless Communications Letters, IEEE/CIC China Communications and IET Smart Cities. He serves for IEEE Communications Magazine, Frontiers in Communications and Networks as well as ZTE communications as a guest editor. He is a program vice-chair for IEEE TrustCom 2020, a technical program committee (TPC) chair for IEEE UCET 2021 and a program vice-chair for UbiSec 2022. He also serves as a TPC member for several prestigious IEEE conferences. He has won the best paper award of IEEE SustainCom 2020 and the best paper award of IEEE MMTC 2021. His current research focuses on wireless communications and resource management for B5G/6G, wireless information and power transfer as well as integrated communication, computing and sensing.|YANG Kun received his PhD from the Department of Electronic & Electrical Engineering of University College London (UCL), UK. He is a Chair Professor in the School of Computer Science & Electronic Engineering, University of Essex, UK, and is leading the Network Convergence Laboratory (NCL), UK. He is also an affiliated professor at UESTC, China. Before joining in the University of Essex at 2003, he had worked at UCL on several European Union (EU) research projects for several years. His main research interests include wireless networks and communications, IoT networking, data and energy integrated networks and mobile computing. He manages research projects funded by various sources such as UK EPSRC, EU FP7/H2020, etc. He has published more than 400 journal papers and filed 20 patents. He is an IEEE ComSoC Distinguished Lecturer (2020–2021) and a member of Academia Europaea (MAE). Professor YANG is an IEEE Fellow.

Abstract

Abstract:

In a rechargeable wireless sensor network, utilizing the unmanned aerial vehicle (UAV) as a mobile base station (BS) to charge sensors and collect data effectively prolongs the network’s lifetime. In this paper, we jointly optimize the UAV’s flight trajectory and the sensor selection and operation modes to maximize the average data traffic of all sensors within a wireless sensor network (WSN) during finite UAV’s flight time, while ensuring the energy required for each sensor by wireless power transfer (WPT). We consider a practical scenario, where the UAV has no prior knowledge of sensor locations. The UAV performs autonomous navigation based on the status information obtained within the coverage area, which is modeled as a Markov decision process (MDP). The deep Q-network (DQN) is employed to execute the navigation based on the UAV position, the battery level state, channel conditions and current data traffic of sensors within the UAV’s coverage area. Our simulation results demonstrate that the DQN algorithm significantly improves the network performance in terms of the average data traffic and trajectory design.

Key words: unmanned aerial vehicle, wireless power transfer, deep Q-network, autonomous navigation

LI Yuting, DING Yi, GAO Jiangchuan, LIU Yusha, HU Jie, YANG Kun. UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network[J]. ZTE Communications, 2023, 21(2): 80-87.

Figures/Tables 9

Figure 1 A UAV-assisted rechargeable wireless sensor network (WSN)

Table 1 Notation list

Notation	Definition
$k,$ $? K$	Sensor index, number of ground sensors
$t,$ $? T$	Time slot index, total UAV flight time slots
$τ,$ $? T$	Time slot duration, total UAV flight time
$w k$	Coordinate of the k-th sensor
$v U A V t$	Velocity of the UAV at the t-th time slot
$q t$	Position of the UAV at the t-th time slot
$d k t$	Distance between sensor $k$ and the UAV
$h k t$	Channel gain between sensor $k$ and the UAV
$R$	The coverage range of the UAV
$P ˉ k t$	Battery level of the k-th sensor
$ρ k t$	Operation mode factor of sensor $k$
$P U A V,$ $? P s$	Transmit power of the UAV and ground sensors

Table 1 Notation list

Notation	Definition
$k,$ $? K$	Sensor index, number of ground sensors
$t,$ $? T$	Time slot index, total UAV flight time slots
$τ,$ $? T$	Time slot duration, total UAV flight time
$w k$	Coordinate of the k-th sensor
$v U A V t$	Velocity of the UAV at the t-th time slot
$q t$	Position of the UAV at the t-th time slot
$d k t$	Distance between sensor $k$ and the UAV
$h k t$	Channel gain between sensor $k$ and the UAV
$R$	The coverage range of the UAV
$P ˉ k t$	Battery level of the k-th sensor
$ρ k t$	Operation mode factor of sensor $k$
$P U A V,$ $? P s$	Transmit power of the UAV and ground sensors

Figure 2 Communication protocol for a UAV-assisted wireless sensor network

Figure 3 Schematics of the proposed algorithm

Table 2 Simulation parameters: DNN

DNN Parameters	Value
Learning rate	0.000 1
Discount factor	0.9
Replay memory size	10 000
Batch size	32
ReLu hidden neurons	20
Number of neural network layers	2

Table 3 System parameters

System Parameters	Value
Bandwidth	1 MHz
Energy conversion efficiency	0.9
Noise power	-60 dBm
Flying height	10 m
Coverage area	70 m²
Steering angle $α t$	$k 4 π k = 0, ?, 7$
Flying velocity $v U A V t$	$0,5, 10$ m/s

Table 3 System parameters

System Parameters	Value
Bandwidth	1 MHz
Energy conversion efficiency	0.9
Noise power	-60 dBm
Flying height	10 m
Coverage area	70 m²
Steering angle $α t$	$k 4 π k = 0, ?, 7$
Flying velocity $v U A V t$	$0,5, 10$ m/s

Figure 4 UAV’s trajectory under different flight time: (a) 2 000 s and (b) 4 000 s

Figure 5 Energy and data traffic of k=1.5 and 10 ground sensors: (a) battery and (b) data traffic

Figure 6 Comparison of average data traffic per sensor by DQN and three circular trajectories

References 15

1	HU J, WANG Q, YANG K. Energy self-sustainability in full-spectrum 6G [J]. IEEE wireless communications, 2021, 28(1): 104–111. DOI: 10.1109/MWC.001.2000156 DOI
2	WU D P, HE J, WANG H G, et al. A hierarchical packet forwarding mechanism for energy harvesting wireless sensor networks [J]. IEEE communications magazine, 2015, 53(8): 92–98. DOI: 10.1109/MCOM.2015.7180514 DOI
3	GHARIBI M, BOUTABA R, WASLANDER S L. Internet of drones [J]. IEEE access, 2016, 4: 1148–1162. DOI: 10.1109/ACCESS.2016.2537208 DOI
4	ZHU S C, GUI L, CHENG N, et al. Joint design of access point selection and path planning for UAV-assisted cellular networks [J]. IEEE Internet of Things journal, 2020, 7(1): 220–233. DOI: 10.1109/JIOT.2019.2947718 DOI
5	GHORBEL M BEN, RODRÍGUEZ-DUARTE D, GHAZZAI H, et al. Joint position and travel path optimization for energy efficient wireless data gathering using unmanned aerial vehicles [J]. IEEE transactions on vehicular technology, 2019, 68(3): 2165–2175. DOI: 10.1109/TVT.2019.2893374 DOI
6	HU J, CAI X P, YANG K. Joint trajectory and scheduling design for UAV aided secure backscatter communications [J]. IEEE wireless communications letters, 2020, 9(12): 2168–2172. DOI: 10.1109/LWC.2020.3016174 DOI
7	XIE L F, XU J, ZHANG R. Throughput maximization for UAV-enabled wireless powered communication networks [J]. IEEE Internet of Things journal, 2019, 6(2): 1690–1703. DOI: 10.1109/JIOT.2018.2875446 DOI
8	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning [EB/OL]. [2023-03-10].
9	LI K, NI W, TOVAR E, et al. On-board deep Q-network for UAV-assisted online power transfer and data collection [J]. IEEE transactions on vehicular technology, 2019, 68(12): 12215–12226. DOI: 10.1109/TVT.2019.2945037 DOI
10	TANG J, SONG J R, OU J H, et al. Minimum throughput maximization for multi-UAV enabled WPCN: A deep reinforcement learning method [J]. IEEE access, 2020, 8: 9124–9132. DOI: 10.1109/ACCESS.2020.2964042 DOI
11	LIU C H, CHEN Z Y, TANG J, et al. Energy-efficient UAV control for effective and fair communication coverage: a deep reinforcement learning approach [J]. IEEE journal on selected areas in communications, 2018, 36(9): 2059–2070. DOI: 10.1109/JSAC.2018.2864373 DOI
12	LU Y P, XIONG G, ZHANG X, et al. Uplink throughput maximization in UAV-aided mobile networks: a DQN-based trajectory planning method [J]. Drones, 2022, 6(12): 378. DOI: 10.3390/drones6120378 DOI
13	ABEDIN S F, MUNIR M S, TRAN N H, et al. Data freshness and energy-efficient UAV navigation optimization: a deep reinforcement learning approach [J]. IEEE transactions on intelligent transportation systems, 2021, 22(9): 5994–6006. DOI: 10.1109/TITS.2020.3039617 DOI
14	ZHANG T K, LEI J Y, LIU Y W, et al. Trajectory optimization for UAV emergency communication with limited user equipment energy: a safe-DQN approach [J]. IEEE transactions on green communications and networking, 2021, 5(3): 1236–1247. DOI: 10.1109/TGCN.2021.3068333 DOI
15	LI K, NI W, TOVAR E, et al. Joint flight cruise control and data collection in UAV-aided Internet of Things: an onboard deep reinforcement learning approach [J]. IEEE Internet of Things journal, 2021, 8(12): 9787–9799. DOI: 10.1109/JIOT.2020.3019186 DOI

[1]	WEI Zhiqing, ZHANG Yongji, JI Danna, LI Chenfei. Sensing and Communication Integrated Fast Neighbor Discovery for UAV Networks [J]. ZTE Communications, 2024, 22(3): 69-82.
[2]	ZHAO Yaqiong, KE Hongqin, XU Wei, YE Xinquan, CHEN Yijian. RIS-Assisted Cell-Free MIMO: A Survey [J]. ZTE Communications, 2024, 22(1): 77-86.
[3]	YANG Bo, MITANI Tomohiko, SHINOHARA Naoki, ZHANG Huaiqing. High-Power Simultaneous Wireless Information and Power Transfer: Injection-Locked Magnetron Technology [J]. ZTE Communications, 2022, 20(2): 3-12.
[4]	CHANG Mingyang, HAN Jiaqi, MA Xiangjin, XUE Hao, WU Xiaonan, LI Long, CUI Tiejun. Programmable Metasurface for Simultaneously Wireless Information and Power Transfer System [J]. ZTE Communications, 2022, 20(2): 48-62.
[5]	LI Xiuxian, LI Zhetao, OUYANG Yan, DUAN Haohua, XIANG Liyao. Using UAV to Detect Truth for Clean Data Collection in Sensor‑Cloud Systems [J]. ZTE Communications, 2021, 19(3): 30-45.
[6]	LIN Xinhua, ZHANG Jing, LI Qiang. Cluster Head Selection Algorithm for UAV Assisted Clustered IoT Network Utilizing Blockchain [J]. ZTE Communications, 2021, 19(1): 30-38.
[7]	Mohammed SEID, Stephen ANOKYE, SUN Guolin. Machine Learning Based Unmanned Aerial Vehicle Enabled Fog-Radio Aerial Vehicle Enabled Fog-Radio Access Network and Edge Computing [J]. ZTE Communications, 2019, 17(4): 33-45.
[8]	Stephen ANOKYE, Mohammed SEID, SUN Guolin. A Survey on Machine Learning Based Proactive Caching [J]. ZTE Communications, 2019, 17(4): 46-55.
[9]	LI Tongxin, SHENG Min, LYU Ruiling, LIU Junyu, LI Jiandong. UAV Assisted Heterogeneous Wireless Networks: Potentials and Challenges [J]. ZTE Communications, 2018, 16(2): 3-8.

UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 15

Related Articles 9

Recommended Articles

Metrics