Scene Visual Perception and AR Navigation Applications

doi:10.12142/ZTECOM.202301010

ZTE Communications ›› 2023, Vol. 21 ›› Issue (1): 81-88.DOI: 10.12142/ZTECOM.202301010

• Research Paper • Previous Articles

Scene Visual Perception and AR Navigation Applications

LU Ping¹^,², SHENG Bin²(), SHI Wenzhe¹^,²

^1.State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen 518055, China
^2.ZTE Corporation, Shenzhen 518057, China
^3.Shanghai Jiao Tong University, Shanghai 200240, China

Received:2022-11-01 Online:2023-03-25 Published:2024-03-15
About author:LU Ping is the Vice President and general manager of the Industrial Digitalization Solution Department of ZTE Corporation, and Executive Deputy Director of the National Key Laboratory of Mobile Network and Mobile Multimedia Technology. His research directions include cloud computing, big data, augmented reality, and multimedia service-based technologies. He has supported and participated in major national science and technology projects and national science and technology support projects. He has published multiple papers, and authored two books.
SHENG Bin ( shengbin@cs.sjtu.edu.cn) is a professor of computer science and engineering from Shanghai Jiao Tong University, China. His research directions include virtual reality and computer graphics. He has presided over two projects on the National Natural Science Foundation of China, one youth project of the National Natural Science Foundation of China, and participates in one high-technology research and development plan (the “863” plan) and one key project of the National Natural Science Foundation of China. He has published 121 papers in different journals.
SHI Wenzhe is a strategy planning engineer with ZTE Corporation, a member of the National Key Laboratory for Mobile Network and Mobile Multimedia Technology, and an engineer of XRExplore Platform Product Planning. His research interests include indoor visual AR navigation, SFM 3D reconstruction, visual SLAM, real-time cloud rendering, VR, and spatial perception.
Supported by:
ZTE Industry‐University‐Institute Cooperation Funds;HC‐CN‐20210707004

Abstract

Abstract:

With the rapid popularization of mobile devices and the wide application of various sensors, scene perception methods applied to mobile devices occupy an important position in location-based services such as navigation and augmented reality (AR). The development of deep learning technologies has greatly improved the visual perception ability of machines to scenes. The basic framework of scene visual perception, related technologies and the specific process applied to AR navigation are introduced, and future technology development is proposed. An application (APP) is designed to improve the application effect of AR navigation. The APP includes three modules: navigation map generation, cloud navigation algorithm, and client design. The navigation map generation tool works offline. The cloud saves the navigation map and provides navigation algorithms for the terminal. The terminal realizes local real-time positioning and AR path rendering.

Key words: 3D reconstruction, image matching, visual localization, AR navigation, deep learning

LU Ping, SHENG Bin, SHI Wenzhe. Scene Visual Perception and AR Navigation Applications[J]. ZTE Communications, 2023, 21(1): 81-88.

Figures/Tables 4

Figure 1 Overall framework of an AR navigation application (APP)

Figure 2 Result of dense reconstruction: (a) photometric depth map, (b) photometric normal map, (c) geometric depth map, (d) geometric normal map, and (e) dense reconstruction effect

Figure 4 Augmented reality (AR) navigation application (APP) and AR rendering result

Figure 5 An example of a 2D point cloud map aligned with CAD map

References 19

1	LI H Q, LI L, LI Z. A review of point cloud compression [J]. ZTE technology journal, 2021, 27( 1): 5– 9. DOI: 10.12142/ZTETJ.202101003
2	CAO M, WEI J, LYU Z, et al. Fast and robust feature tracking for 3D reconstruction [J]. Optics & laser technology, 2019, 110: 120– 128. DOI: 10.1016/j.optlastec.2018.05.036
3	LINDENBERGER P, SARLIN P E, LARSSON V, et al. Pixel-perfect structure-from-motion with featuremetric refinement [C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. IEEE, 2022: 5967– 5977. DOI: 10.1109/ICCV48922.2021.00593
4	YI K M, TRULLS E, LEPETIT V, et al. LIFT: learned invariant feature transform [C]// European Conference on Computer Vision. ECCV, 2016: 467– 483. DOI: 10.1007/978-3-319-46466-4_28
5	ZHANG X, YU F X, KARAMAN S, et al. Learning discriminative and transformation covariant local feature detectors [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 4923– 4931. DOI: 10.1109/CVPR.2017.523
6	ZHANG L G, RUSINKIEWICZ S. Learning to detect features in texture images [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 6325– 6333. DOI: 10.1109/CVPR.2018.00662
7	DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description [C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2018: 224– 236. DOI: 10.1109/CVPRW.2018.00060
8	LAGUNA A B, RIBA E, PONSA D, et al. Key.Net: keypoint detection by handcrafted and learned CNN filters [C]// International Conference on Computer Vision (ICCV). IEEE, 2020: 5835– 5843. DOI: 10.1109/ICCV.2019.00593
9	ONO Y, TRULLS E, FUA P, et al. LF-Net: Learning local features from images [C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. ACM, 2018: 6237– 6247. DOI: 10.5555/3327345.3327521
10	SCHÖNBERGER J L, HARDMEIER H, SATTLER T, et al. Comparative evaluation of hand-crafted and learned local features [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 6959– 6968. DOI: 10.1109/CVPR.2017.736
11	WANG J, ZHOU F, WEN S L, et al. Deep metric learning with angular loss [C]// International Conference on Computer Vision. IEEE, 2017: 2612– 2620. DOI: 10.1109/ICCV.2017.283
12	ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2015: 4353– 4361. DOI: 10.1109/CVPR.2015.7299064
13	LUO Z X, SHEN T W, ZHOU L, et al. ContextDesc: local descriptor augmentation with cross-modality context [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 2522– 2531. DOI: 10.1109/CVPR.2019.00263
14	TIAN Y R, YU X, FAN B, et al. SOSNet: second order similarity regularization for local descriptor learning [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 11008– 11017. DOI: 10.1109/CVPR.2019.01127
15	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks [C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 4937– 4946. DOI: 10.1109/CVPR42600.2020.00499
16	SUN J M, SHEN Z H, WANG Y A, et al. LoFTR: detector-free local feature matching with transformers [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2021: 8918– 8927. DOI: 10.1109/CVPR46437.2021.00881
17	SARLIN P E, CADENA C, SIEGWART R, et al. From coarse to fine: robust hierarchical localization at large scale [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 12708– 12717. DOI: 10.1109/CVPR.2019.01300
18	LU P, SHENG B, ZHU F. Next-generation communications technology facilitates real-time distributed cloud rendering [J]. ZTE technology journal, 2021, 27( 1): 17– 20. DOI: 10.12142/ZTETJ.202101005
19	LU P, OUYANG X Z, GAO W W. Capacity improvement and practice of 5G industry virtual private network [J]. ZTE technology journal, 2022, 28( 2): 68– 74. DOI: 10.12142/ZTETJ.202202011

[1]	WANG Chongchong, LI Yao, WANG Beibei, CAO Hong, ZHANG Yanyong. Point Cloud Processing Methods for 3D Point Cloud Detection Tasks [J]. ZTE Communications, 2023, 21(4): 38-46.
[2]	GONG Panyin, ZHANG Guidong, ZHANG Zhigang, CHEN Xiao, DING Xuan. Research on Fall Detection System Based on Commercial Wi-Fi Devices [J]. ZTE Communications, 2023, 21(4): 60-68.
[3]	DENG Letian, ZHAO Yanru. Deep Learning-Based Semantic Feature Extraction: A Literature Review and Future Directions [J]. ZTE Communications, 2023, 21(2): 11-17.
[4]	FAN Guotian, WANG Zhibin. Intelligent Antenna Attitude Parameters Measurement Based on Deep Learning SSD Model [J]. ZTE Communications, 2022, 20(S1): 36-43.
[5]	GAO Zhengguang, LI Lun, WU Hao, TU Xuezhen, HAN Bingtao. A Unified Deep Learning Method for CSI Feedback in Massive MIMO Systems [J]. ZTE Communications, 2022, 20(4): 110-115.
[6]	ZHANG Jintao, HE Zhenqing, RUI Hua, XU Xiaojing. Spectrum Sensing for OFDMA Using Multicarrier Covariance Matrix Aware CNN [J]. ZTE Communications, 2022, 20(3): 61-69.
[7]	HE Hongye, YANG Zhiguo, CHEN Xiangning. Payload Encoding Representation from Transformer for Encrypted Traffic Classification [J]. ZTE Communications, 2021, 19(4): 90-97.
[8]	XUE Songyan, LI Ang, WANG Jinfei, YI Na, MA Yi, Rahim TAFAZOLLI, Terence DODGSON. To Learn or Not to Learn:Deep Learning Assisted Wireless Modem Design [J]. ZTE Communications, 2019, 17(4): 3-11.
[9]	ZHENG Xiaoqing, LU Yaping, PENG Haoyuan, FENG Jiangtao, ZHOU Yi, JIANG Min, MA Li, ZHANG Ji, JI Jie. Detecting Abnormal Start-Ups, Unusual Resource Consumptions of the Smart Phone: A Deep Learning Approach [J]. ZTE Communications, 2019, 17(2): 38-43.
[10]	ZHENG Xiaoqing, CHEN Jun, SHANG Guoqiang. Deep Neural Network-Based Chinese Semantic Role Labeling [J]. ZTE Communications, 2017, 15(S2): 58-64.

Scene Visual Perception and AR Navigation Applications

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 4

References 19

Related Articles 10

Recommended Articles

Metrics