Indoor Environment and Human Sensing via Millimeter Wave Radio: A Review

doi:10.12142/ZTECOM.202103004

ZTE Communications ›› 2021, Vol. 19 ›› Issue (3): 22-29.DOI: 10.12142/ZTECOM.202103004

• Special Topic • Previous Articles Next Articles

Indoor Environment and Human Sensing via Millimeter Wave Radio: A Review

LIU Haipeng(), ZHANG Xingyue, ZHOU Anfu, LIU Liang, MA Huadong

Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2021-06-10 Online:2021-09-25 Published:2021-10-11
About author:LIU Haipeng (fengxudi@qq.com) is a Ph.D. student of Beijing University of Posts and Telecommunications, China. He is directly recommended for a doctorate degree from undergraduate. His research interest lies in wireless networks and mobile sensing, with the emphasis on ubiquitous mobile sensing for enabling new Internet-of-Things applications.|ZHANG Xingyue is a second-grade student at the International School of Beijing University of Posts and Telecommunications, China. She is currently majoring in telecommunications engineering and management, and is about to enter the direction of multimedia learning.|ZHOU Anfu received his Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Science, China in 2012. Before that, he received B.S. degree from the Renmin University of China. He is now a professor at the School of Computer Science, Beijing University of Posts and Telecommunications, China. His research interest lies in mobile computing and wireless networking and IoT systems.|LIU Liang obtained his Ph.D. degree from Beijing University of Posts and Telecommunications, China in 2009, and conducted visiting research in TAMU in the US. His main research directions are multimedia and sensor networks. He has published nearly 40 papers in international authoritative journals and conferences such as IEEE Transactions and INFOCOM.|MA Huadong is an IEEE Fellow. He received his Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Science, China in 1995. He is now a Chang Jiang Scholar Professor and Director of Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia. He is also the Executive Dean of School of Computer Science, Beijing University of Posts and Telecommunications, China. His research interests include multimedia systems and networking, sensor networks, and the Internet of Things. He has authored over 200 papers and four books.
Supported by:
the National Key R&D Program of China(2019YFB2102202);the A3 Foresight Program of NSFC(62061146002);the NSFC(61772084);the Funds for Creative Research Groups of China(61921003);the Youth Top Talent Support Program, the 111 Project(B18008);the Fundamental Research Funds for the Central Universities(2019XD-A13)

Abstract

Abstract:

With the rapid development of 5G technology, more and more attention has been attracted to mmWave sensing. As an emerging sensing medium, mmWave has the advantages of both high sensitivity and precision. Different from its networking applications, the core method of mmWave sensing is to analyze the reflected signal changes containing the relevant information of different surrounding environments. In this paper, we conduct a systemic review for mmWave sensing. We first summarize the prior works on environmental sensing with different signal analysis methods. Then, we classify and discuss the work of sensing humans, including their behavior and gestures. Finally, we discuss and put forward more possibilities of mmWave human perception.

Key words: millimeter wave radio, review, environment construction, human sensing, 5G

LIU Haipeng, ZHANG Xingyue, ZHOU Anfu, LIU Liang, MA Huadong. Indoor Environment and Human Sensing via Millimeter Wave Radio: A Review[J]. ZTE Communications, 2021, 19(3): 22-29.

Figures/Tables 7

References 23

1	ZHU Y Z, ZHU Y B, ZHAO B Y, et al. Reusing 60 GHz radios for mobile radar imaging [C]//The 21st Annual International Conference on Mobile Computing and Networking. Paris, France: ACM, 2015: 103–116. DOI: 10.1145/2789168.2790112 DOI
2	PALACIOS J, CASARI P, WIDMER J. JADE: Zero‑knowledge device localization and environment mapping for millimeter wave systems [C]//IEEE Conference on Computer Communications. Atlanta, USA: IEEE, 2017: 1–9. DOI: 10.1109/INFOCOM.2017.8057183 DOI
3	MENG Z, FU S, YAN J, et al. Gait recognition for co‑existing multiple people using millimeter wave sensing [C]//The AAAI Conference on Artificial Intelligence. New York, USA: AAAI, 2020, 34(01): 849–856. DOI: index.php/AAAI/article/view/5430 DOI
4	WANG S W, SONG J, LIEN J, et al. Interacting with soli: exploring fine‑grained dynamic gesture recognition in the radio‑frequency spectrum [C]//The 29th Annual Symposium on User Interface Software and Technology. Tokyo, Japan: ACM, 2016: 851–860. DOI: 10.1145/2984511.2984565 DOI
5	LIEN J, GILLIAN N, KARAGOZLER M E, et al. Soli: ubiquitous gesture sensing with millimeter wave radar [J]. ACM transactions on graphics, 2016, 35(4): 1–19. DOI: 10.1145/2897824.2925953 DOI
6	WEI T, ZHANG X Y. MTrack: high‑precision passive tracking using millimeter wave radios [C]//The 21st Annual International Conference on Mobile Computing and Networking. Paris, France: ACM, 2015: 117–129. DOI: 10.1145/2789168.2790113 DOI
7	PALACIOS J, BIELSA G, CASARI P, et al. Communication‑driven localization and mapping for millimeter wave networks [C]//IEEE Conference on Computer Communications. Honolulu, USA: IEEE, 2018: 2402–2410. DOI: 10.1109/INFOCOM.2018.8485819 DOI
8	WEI T, ZHOU A, ZHANG X. Facilitating robust 60 GHz network deployment by sensing ambient reflectors [C]//14th USENIX Symposium on Networked Systems Design and Implementation. Boston, USA: IEEE, 2017: 213–226
9	ZHOU A F, ZHANG X Y, MA H D. Beam‑forecast: facilitating mobile 60 GHz networks via model‑driven beam steering [C]//IEEE Conference on Computer Communications. Atlanta, USA: IEEE, 2017: 1–9. DOI:10.1109/INFOCOM.2017.8057188 DOI
10	ZHU Y, YAO Y, ZHAO B Y, et al. Object recognition and navigation using a single networking device [C]//Annual International Conference on Mobile Systems, Applications, and Services. Niagara Falls, USA: ACM, 2017: 265–277. DOI: 10.1145/3081333.3081339 DOI
11	YEO H S, FLAMICH G, SCHREMPF P, et al. RadarCat: radar categorization for input & interaction [C]//The 29th Annual Symposium on User Interface Software and Technology. Tokyo, Japan: ACM, 2016: 833–841.DOI: 10.1145/2984511.2984515 DOI
12	ZHOU A F, YANG S Y, YANG Y, et al. Autonomous environment mapping using commodity millimeter‑wave network device [C]//IEEE Conference on Computer Communications. Paris, France: IEEE, 2019: 1126–1134. DOI:10.1109/INFOCOM.2019.8737624 DOI
13	ZHOU A F, XU S Q, WANG S, et al. Robot navigation in radio beam space: Leveraging robotic intelligence for seamless mmWave network coverage [C]//The 20th ACM International Symposium on Mobile Ad Hoc Networking and Computing. Catania, Italy: ACM, 2019: 161–170. DOI: 10.1145/3323679.3326514 DOI
14	YANG Z C, PATHAK P H, ZENG Y Z, et al. Monitoring vital signs using millimeter wave [C]//Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing. Paderborn, Germany: ACM, 2016: 211–220. DOI: 10.1145/2942358.2942381 DOI
15	GU T B, FANG Z, YANG Z C, et al. MmSense: multi‑person detection and identification via mmWave sensing [C]//The 3rd ACM Workshop on Millimeter‑wave Networks and Sensing Systems. Los Cabos, Mexico: ACM, 2019: 45–50. DOI: 10.1145/3349624.3356765 DOI
16	LIU H P, BAI X W, GAO H Y, et al. MID: accurate and robust user identification and authentication through hand‑gesture sensing with mmwave radar [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2021.
17	SMITH K A, CSECH C, MURDOCH D, et al. Gesture recognition using mm‑wave sensor for human‑car interface [J]. IEEE sensors letters, 2018, 2(2): 1–4. DOI: 10.1109/LSENS.2018.2810093 DOI
18	PATRA A, GEUER P, MUNARI A, et al. Mm‑wave radar based gesture recognition: development and evaluation of a low‑power, low‑complexity system [C]//The 2nd ACM Workshop on Millimeter Wave Networks and Sensing Systems. New Delhi, India: ACM, 2018: 51–56. DOI: 10.1145/3264492.3264501 DOI
19	LIU H P, WANG Y H, ZHOU A F, et al. Real‑time arm gesture recognition in smart home scenarios via millimeter wave sensing [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2020, 4(4): 140. DOI: 10.1145/3432235 DOI
20	PALIPANA S, SALAMI D, LEIVA L A, et al. Pantomime: mid‑air gesture recognition with sparse millimeter‑wave radar point clouds [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2021, 5(1): 1–27. DOI:10.1145/3448110 DOI
21	SANTHALINGAM P S, HOSAIN A A, ZHANG D, et al. MmASL: environment‑independent asl gesture recognition using 60 GHz millimeter‑wave signals [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2020, 4(1): 1–30. DOI: 10.1145/3381010 DOI
22	LIU H P, CUI K N, HU K Y, et al. Environment‑independent mmWave sensing based gesture recognition via transfer learning [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2021
23	XUE H, JU Y, MIAO C, et al. MmMesh: towards 3D real‑time dynamic human mesh construction using millimeter‑wave [C]//Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 2021: 269–282. DOI: 10.1145/3458864.3467679 DOI

Citation	Usage	Technology	Method
JADE	Position the client	An iterative algorithm	Both phase and RSS
CLAM	Map the environment	A distributed localization algorithm	Both phase and RSS
E-Mi	Boost the indoor network	A multi-path analysis framework	Both phase and RSS
Beam-Forcast	Improve the mobile links	Reverse engineering of E-Mi	Both phase and RSS
RSA	Position object & identify materials	Move Rx along trajectory while collecting	Only RSS
Ulysses	Image environment	Bind Tx & Rx together to collect reflection signals	Only RSS
RadarCat	Identify plenty of materials	Classification by learning on the radar signals	Only RSS
mmRanger	Sense environment	Automatically collect signals	RSS with a robot
miDroid	Improve mobile links	Set a network relay piggybacked on the robot	RSS with a robot

Citation	Usage	Technology	Method
JADE	Position the client	An iterative algorithm	Both phase and RSS
CLAM	Map the environment	A distributed localization algorithm	Both phase and RSS
E-Mi	Boost the indoor network	A multi-path analysis framework	Both phase and RSS
Beam-Forcast	Improve the mobile links	Reverse engineering of E-Mi	Both phase and RSS
RSA	Position object & identify materials	Move Rx along trajectory while collecting	Only RSS
Ulysses	Image environment	Bind Tx & Rx together to collect reflection signals	Only RSS
RadarCat	Identify plenty of materials	Classification by learning on the radar signals	Only RSS
mmRanger	Sense environment	Automatically collect signals	RSS with a robot
miDroid	Improve mobile links	Set a network relay piggybacked on the robot	RSS with a robot

Citation	Usage	Equipment	Technology	Whether AI
Deep-soli	Gesture recognition	Soli	Range-Doppler images	Customized CNN
Ubi-soli	Gesture recognition	Soli	Multiple abstract representations	Random forest
HCI	Vehicular gesture recognition	Soli	Several physical features	Random forest
MengZhenGait	Identify different users	TI-IWR1443&6843	Three spatial features	MmGaitNet
MmVital	Monitor vital signs	Horn antennas	Extract periodic changes	No
MTrack	Track a vertical finger	Horn antennas	Combine target’s angle and phase change	No
MmSense	Multi-human detection	Free	Features of 60 GHz signal	No
MID	Gesture recognition	Free	MmWave sensing	No
LowCostGes	Detect gesture	RIC60a	Extract power profile and AoA	No
MmASL	Home-assistant system	Free	Features of 60 GHz signal	No
MHomeGes	Smart home-usage	TI-IWR1443	MmWave sensing	No
MTranseSee	User recognition	Free	MmWave radar	No
Pantomime	Gesture recognition	Pantomime	MmWave sensing	No
MmMesh	Human mesh construction	Free	Deep learning framework	No

Citation	Usage	Equipment	Technology	Whether AI
Deep-soli	Gesture recognition	Soli	Range-Doppler images	Customized CNN
Ubi-soli	Gesture recognition	Soli	Multiple abstract representations	Random forest
HCI	Vehicular gesture recognition	Soli	Several physical features	Random forest
MengZhenGait	Identify different users	TI-IWR1443&6843	Three spatial features	MmGaitNet
MmVital	Monitor vital signs	Horn antennas	Extract periodic changes	No
MTrack	Track a vertical finger	Horn antennas	Combine target’s angle and phase change	No
MmSense	Multi-human detection	Free	Features of 60 GHz signal	No
MID	Gesture recognition	Free	MmWave sensing	No
LowCostGes	Detect gesture	RIC60a	Extract power profile and AoA	No
MmASL	Home-assistant system	Free	Features of 60 GHz signal	No
MHomeGes	Smart home-usage	TI-IWR1443	MmWave sensing	No
MTranseSee	User recognition	Free	MmWave radar	No
Pantomime	Gesture recognition	Pantomime	MmWave sensing	No
MmMesh	Human mesh construction	Free	Deep learning framework	No

	AI	Non-AI
Human dynamics	2,3,5,6	1,7,16,20,21,22,23
Indoor environment	8	9,11,12,13,14,15,17,18,19

Indoor Environment and Human Sensing via Millimeter Wave Radio: A Review

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 23

Related Articles 15

Recommended Articles

Metrics

[1]	YU Xiaohui, YU Shucheng, LIU Xiqing, PENG Mugen. On Normalized Least Mean Square Based Interference Cancellation Algorithm for Integrated Sensing and Communication Systems [J]. ZTE Communications, 2024, 22(3): 21-28.
[2]	FENG Jianxin, PAN Yi, WU Xiao. Building a Stronger Foundation for Web3: Advantages of 5G Infrastructure [J]. ZTE Communications, 2024, 22(2): 3-10.
[3]	YANG Yibing, LIU Ming, XU Rongtao, WANG Gongpu, GONG Wei. Link Budget and Enhanced Communication Distance for Ambient Internet of Things [J]. ZTE Communications, 2024, 22(1): 16-23.
[4]	ZHAO Yaqiong, KE Hongqin, XU Wei, YE Xinquan, CHEN Yijian. RIS-Assisted Cell-Free MIMO: A Survey [J]. ZTE Communications, 2024, 22(1): 77-86.
[5]	LI Hanwen, BI Ningjing, SHA Jin. Design of Raptor-Like LDPC Codes and High Throughput Decoder Towards 100 Gbit/s Throughput [J]. ZTE Communications, 2023, 21(3): 86-92.
[6]	DING Jianwen, LIU Yao, LIAO Hongjian, SUN Bin, WANG Wei. Statistical Model of Path Loss for Railway 5G Marshalling Yard Scenario [J]. ZTE Communications, 2023, 21(3): 117-122.
[7]	SHI Xiangyi, HAN Tongzhou, TIAN Hai, ZHAO Danfeng. Design of Raptor-Like Rate Compatible SC-LDPC Codes [J]. ZTE Communications, 2022, 20(S1): 16-21.
[8]	ZHANG Jintao, HE Zhenqing, RUI Hua, XU Xiaojing. Spectrum Sensing for OFDMA Using Multicarrier Covariance Matrix Aware CNN [J]. ZTE Communications, 2022, 20(3): 61-69.
[9]	HOU Xiaolin, LI Xiang, WANG Xin, CHEN Lan, SUYAMA Satoshi. Some Observations and Thoughts about Reconfigurable Intelligent Surface Application for 5G Evolution and 6G [J]. ZTE Communications, 2022, 20(1): 14-20.
[10]	YAN Xincheng, TENG Huiyun, PING Li, JIANG Zhihong, ZHOU Na. Study on Security of 5G and Satellite Converged Communication Network [J]. ZTE Communications, 2021, 19(4): 79-89.
[11]	XIAO Kai, LIU Xing, HAN Xianghui, HAO Peng, ZHANG Junfeng, ZHOU Dong, WEI Xingguang. Flexible Multiplexing Mechanism for Coexistence of URLLC and EMBB Services in 5G Networks [J]. ZTE Communications, 2021, 19(2): 82-90.
[12]	LIU Zhuang, GAO Yin, LI Dapeng, CHEN Jiajun, HAN Jiren. Enabling Energy Efficiency in 5G Network [J]. ZTE Communications, 2021, 19(1): 20-29.
[13]	ZHANG Jing, WEI Xiao, CHENG Junfeng, FENG Xu. Satellite E2E Network Slicing Based on 5G Technology [J]. ZTE Communications, 2020, 18(4): 26-33.
[14]	LI Yezhen, REN Yongli, YANG Fan, XU Shenheng, ZHANG Jiannian. A Novel 28 GHz Phased Array Antenna for 5G Mobile Communications [J]. ZTE Communications, 2020, 18(3): 20-25.
[15]	CHANG Su-Wei, LIN Chueh-Jen, TSAI Wen-Tsai, HUNG Tzu-Chieh, HUANG Po-Chia. Design of Millimeter-Wave Antenna-in-Package (AiP) for 5G NR [J]. ZTE Communications, 2020, 18(3): 26-32.