Device-Free In-Air Gesture Recognition Based on RFID Tag Array

doi:10.12142/ZTECOM.202103003

ZTE Communications ›› 2021, Vol. 19 ›› Issue (3): 13-21.DOI: 10.12142/ZTECOM.202103003

收稿日期:2021-06-10 出版日期:2021-09-25 发布日期:2021-10-11

Device-Free In-Air Gesture Recognition Based on RFID Tag Array

WU Jiaying, WANG Chuyu(), XIE Lei

State Key Laboratory for Novel Software Technology, Nanjing 210023, China

Received:2021-06-10 Online:2021-09-25 Published:2021-10-11
About author:WU Jiaying is a Ph.D. student in the Department of Computer Science and Technology, Nanjing University, China, supervised by Prof. XIE Lei and WANG Chuyu. Her research interests include smart sensing and RFID.|WANG Chuyu (chuyu@nju.edu.cn) received his Ph.D. degree in computer science from Nanjing University, China in 2018. He is an assistant professor in the Department of Computer Science and Technology, Nanjing University, China. His research interests include RFID systems, software-defined radio, activity sensing, indoor localization, etc. WANG Chuyu is the corresponding author.|XIE Lei received his B.S. and Ph.D. degrees from Nanjing University, China in 2004 and 2010, respectively, all in computer science. He is a full professor in the Department of Computer Science and Technology, Nanjing University, China. He has published over 100 papers in IEEE Transactions on Mobile Computing, IEEE Transactions on Parallel and Distributed Systems, ACM Transactions on Sensor Networks, ACM MOBICOM, ACM UbiComp, ACM MobiHoc, IEEE INFOCOM, IEEE ICNP, IEEE ICDCS, etc.
Supported by:
National Natural Science Foundation of China(61902175);Natural Science Foundation of China(BK20190293)

摘要/Abstract

Abstract:

Due to the function of gestures to convey information, gesture recognition plays a more and more important part in human-computer interaction. Traditional methods to recognize gestures are mostly device-based, which means users need to contact the devices. To overcome the inconvenience of the device-based methods, studies on device-free gesture recognition have been conducted. However, computer vision methods bring privacy issues and light interference problems. Therefore, we turn to wireless technology. In this paper, we propose a device-free in-air gesture recognition method based on radio frequency identification (RFID) tag array. By capturing the signals reflected by gestures, we can extract the gesture features. For dynamic gestures, both temporal and spatial features need to be considered. For static gestures, spatial feature is the key, for which a neural network is adopted to recognize the gestures. Experiments show that the accuracy of dynamic gesture recognition on the test set is 92.17%, while the accuracy of static ones is 91.67%.

Key words: gesture recognition, RFID tag array, neural network

. [J]. ZTE Communications, 2021, 19(3): 13-21.

WU Jiaying, WANG Chuyu, XIE Lei. Device-Free In-Air Gesture Recognition Based on RFID Tag Array[J]. ZTE Communications, 2021, 19(3): 13-21.

图/表 11

Figure 1 Feature image

Table 1 CNN-LSTM structure

	Layer	Description
CNN part	Input layer	Input: feature image sequence Length of sequence: 5 Image size: 15×21
	Convolution layer-1	Extract $n_c o n v 1$ features from the image Kernel size: 3×3× $n_c o n v 1$ Step size: 1 Activation function: ReLU
	Pooling layer-1	Downsample the extracted features Pooling type: max pooling Template size: 2×2 Step size: 2
	Convolution layer-2	Extract $n_c o n v 2$ features from the image Kernel size: 3×3× $n_c o n v 2$ Step size: 1 Activation function: ReLU
	Pooling layer-2	Downsample the extracted features Pooling type: max pooling Template size: 2×2 Step size: 2
	Fully connected layer	Fully connect the features to $n_f c$ -dimensional summary vector
LSTM part	LSTM layer	Extract features from summary vector sequence Time steps: 5 Number of hidden units: $n_h d$
LSTM part	Fully connected layer	Fully connect the features to six-dimensional prediction vector

Table 1 CNN-LSTM structure

	Layer	Description
CNN part	Input layer	Input: feature image sequence Length of sequence: 5 Image size: 15×21
	Convolution layer-1	Extract $n_c o n v 1$ features from the image Kernel size: 3×3× $n_c o n v 1$ Step size: 1 Activation function: ReLU
	Pooling layer-1	Downsample the extracted features Pooling type: max pooling Template size: 2×2 Step size: 2
	Convolution layer-2	Extract $n_c o n v 2$ features from the image Kernel size: 3×3× $n_c o n v 2$ Step size: 1 Activation function: ReLU
	Pooling layer-2	Downsample the extracted features Pooling type: max pooling Template size: 2×2 Step size: 2
	Fully connected layer	Fully connect the features to $n_f c$ -dimensional summary vector
LSTM part	LSTM layer	Extract features from summary vector sequence Time steps: 5 Number of hidden units: $n_h d$
LSTM part	Fully connected layer	Fully connect the features to six-dimensional prediction vector

Figure 2 CNN-LSTM structure

Figure 3 Loss curves for dynamic gesture recognition with and without dropout

Figure 4 Static gesture recognition by the distance to noise

Figure 5 Experiment deployment

Figure 6 Dynamic and static gestures used in our experiments

Figure 7 Cross validation of hyperparameters on (a) convolution kernel depth, (b) dimensionality of the fully connected layer, and (c) the number of hidden units

Figure 8 Cross validation on (a) model selection for dynamic gesture and (b) feature decision for static gesture

Figure 9 Evaluation of dynamic gesture recognition: (a) Confusion matrix, (b) accuracy of different users, and (c) accuracy at different speeds

Figure 10 Evaluation of static gesture recognition: (a) Confusion matrix and (b) accuracy at different speeds

参考文献 18

1	JIANG S, GAO Q, LIU H, et al. A novel, co‑located EMG‑FMG‑sensing wearable armband for hand gesture recognition [J]. Sensors and actuators a: physical, 2020, 301: 111738. DOI: 10.1016/j.sna.2019.111738 DOI
2	XIE R, CAO J. Accelerometer‑based hand gesture recognition by neural network and similarity matching [J]. IEEE sensors journal, 2016, 16(11): 4537–4545. DOI: 10.1109/JSEN.2016.2546942 DOI
3	SUN Y, WENG Y, LUO B, et al. Gesture recognition algorithm based on multi‑scale feature fusion in RGB‑D images [J]. IET image processing, 2020. DOI: 10.1049/iet-ipr.2020.0148 DOI
4	WANG C, LIU Z, CHAN S C. Superpixel‑based hand gesture recognition with kinect depth camera [J]. IEEE transactions on multimedia, 2014, 17(1): 29–39. DOI: 10.1109/TMM.2014.2374357 DOI
5	MITRA S, ACHARYA T. Gesture recognition: A survey [J]. IEEE transactions on systems, man, and cybernetics, part C (applications and reviews), 2007, 37(3): 311–324. DOI: 10.1109/TSMCC.2007.893280 DOI
6	RAUTARAY S S, AGRAWAL A. Vision based hand gesture recognition for human computer interaction: a survey [J]. Artificial intelligence review, 2015, 43(1): 1–54. DOI: 10.1007/s10462-012-9356-9 DOI
7	BHASKARAN K A, NAIR A G, RAM K D, et al. Smart gloves for hand gesture recognition: sign language to speech conversion system [C]//International Conference on Robotics and Automation for Humanitarian Applications (RAHA). Coimbatore, India: IEEE, 2016: 1–6. DOI: 10.1109/RAHA.2016.7931887 DOI
8	CHEN B, ZHANG Q, ZHAO R, et al. SGRS: a sequential gesture recognition system using COTS RFID [C]//IEEE Wireless Communications and Networking Conference (WCNC). Barcelona, Spain: IEEE, 2018: 1–6. DOI: 10.1109/WCNC.2018.8376998 DOI
9	CHENG K, YE N, MALEKIAN R, et al. In‑air gesture interaction: real time hand posture recognition using passive RFID tags [J]. IEEE access, 2019, 7: 94460–94472. DOI: 10.1109/ACCESS.2019.2928318 DOI
10	HONGO H, OHYA M, YASUMOTO M, et al. Focus of attention for face and hand gesture recognition using multiple cameras [C]//Fourth IEEE International Conference on Automatic Face and Gesture Recognition Cat. No. PR00580. Grenoble, France: IEEE, 2000: 156–161. DOI: 10.1109/AFGR.2000.840627 DOI
11	WANG W, LIU A X, SUN K. Device‑free gesture tracking using acoustic signals [C]//22nd Annual International Conference on Mobile Computing and Networking. New York, USA: ACM, 2016: 82–94. DOI: 10.1145/2973750.2973764 DOI
12	ABDELNASSER H, YOUSSEF M, HARRAS K A. Wigest: a ubiquitous wifi‑based gesture recognition system [C]//IEEE Conference on Computer Communications (INFOCOM). Hong Kong, China: IEEE, 2015: 1472–1480. DOI: 10.1109/INFOCOM.2015.7218525 DOI
13	LIU H, WANG Y, ZHOU A, et al. Real‑time arm gesture recognition in smart home scenarios via millimeter wave sensing [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2020, 4(4): 1–28. DOI: 10.1145/3432235 DOI
14	DING H, QIAN C, HAN J, et al. Rfipad: enabling cost‑efficient and device‑free in‑air handwriting using passive tags [C]//37th International Conference on Distributed Computing Systems (ICDCS). Atlanta, USA: IEEE, 2017: 447–457. DOI: 10.1109/ICDCS.2017.141 DOI
15	LIU J, CHEN X, CHEN S, et al. TagSheet: sleeping posture recognition with an unobtrusive passive tag matrix [C]//5th IEEE Conference on Computer Communications. Chengdu, China: IEEE, 2019: 874–882. DOI: 10.1109/INFOCOM.2019.8737599 DOI
16	DOBKIN D M. The RF in RFID: passive UHF RFID in practice [M]. Oxford, UK: Newnes, 2007: 1–493. DOI: 10.1016/B978-0-7506-8209-1.X5001-3 DOI
17	WANG C, LIU J, CHEN Y, et al. Multi‑touch in the air: device‑free finger tracking and gesture recognition via cots RFID [C]//IEEE Conference on Computer Communications. Honolulu, USA: IEEE, 2018: 1691–1699. DOI: 10.1109/INFOCOM.2018.8486346 DOI
18	PASCANU R, MIKOLOV T, BENGIO Y. On the difficulty of training recurrent neural networks [C]//International Conference on Machine Learning. Atlanta, USA: PMLR, 2013: 1310–1318

Device-Free In-Air Gesture Recognition Based on RFID Tag Array

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 18

相关文章 0

编辑推荐

Metrics