Scene Visual Perception and AR Navigation Applications

doi:10.12142/ZTECOM.202301010

ZTE Communications ›› 2023, Vol. 21 ›› Issue (1): 81-88.DOI: 10.12142/ZTECOM.202301010

收稿日期:2022-11-01 出版日期:2023-03-25 发布日期:2024-03-15

Scene Visual Perception and AR Navigation Applications

LU Ping¹^,², SHENG Bin²(), SHI Wenzhe¹^,²

^1.State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen 518055, China
^2.ZTE Corporation, Shenzhen 518057, China
^3.Shanghai Jiao Tong University, Shanghai 200240, China

Received:2022-11-01 Online:2023-03-25 Published:2024-03-15
About author:LU Ping is the Vice President and general manager of the Industrial Digitalization Solution Department of ZTE Corporation, and Executive Deputy Director of the National Key Laboratory of Mobile Network and Mobile Multimedia Technology. His research directions include cloud computing, big data, augmented reality, and multimedia service-based technologies. He has supported and participated in major national science and technology projects and national science and technology support projects. He has published multiple papers, and authored two books.
SHENG Bin ( shengbin@cs.sjtu.edu.cn) is a professor of computer science and engineering from Shanghai Jiao Tong University, China. His research directions include virtual reality and computer graphics. He has presided over two projects on the National Natural Science Foundation of China, one youth project of the National Natural Science Foundation of China, and participates in one high-technology research and development plan (the “863” plan) and one key project of the National Natural Science Foundation of China. He has published 121 papers in different journals.
SHI Wenzhe is a strategy planning engineer with ZTE Corporation, a member of the National Key Laboratory for Mobile Network and Mobile Multimedia Technology, and an engineer of XRExplore Platform Product Planning. His research interests include indoor visual AR navigation, SFM 3D reconstruction, visual SLAM, real-time cloud rendering, VR, and spatial perception.
Supported by:
ZTE Industry‐University‐Institute Cooperation Funds;HC‐CN‐20210707004

摘要/Abstract

Abstract:

With the rapid popularization of mobile devices and the wide application of various sensors, scene perception methods applied to mobile devices occupy an important position in location-based services such as navigation and augmented reality (AR). The development of deep learning technologies has greatly improved the visual perception ability of machines to scenes. The basic framework of scene visual perception, related technologies and the specific process applied to AR navigation are introduced, and future technology development is proposed. An application (APP) is designed to improve the application effect of AR navigation. The APP includes three modules: navigation map generation, cloud navigation algorithm, and client design. The navigation map generation tool works offline. The cloud saves the navigation map and provides navigation algorithms for the terminal. The terminal realizes local real-time positioning and AR path rendering.

Key words: 3D reconstruction, image matching, visual localization, AR navigation, deep learning

. [J]. ZTE Communications, 2023, 21(1): 81-88.

LU Ping, SHENG Bin, SHI Wenzhe. Scene Visual Perception and AR Navigation Applications[J]. ZTE Communications, 2023, 21(1): 81-88.

图/表 4

Figure 1 Overall framework of an AR navigation application (APP)

Figure 2 Result of dense reconstruction: (a) photometric depth map, (b) photometric normal map, (c) geometric depth map, (d) geometric normal map, and (e) dense reconstruction effect

Figure 4 Augmented reality (AR) navigation application (APP) and AR rendering result

Figure 5 An example of a 2D point cloud map aligned with CAD map

参考文献 19

1	LI H Q, LI L, LI Z. A review of point cloud compression [J]. ZTE technology journal, 2021, 27( 1): 5– 9. DOI: 10.12142/ZTETJ.202101003
2	CAO M, WEI J, LYU Z, et al. Fast and robust feature tracking for 3D reconstruction [J]. Optics & laser technology, 2019, 110: 120– 128. DOI: 10.1016/j.optlastec.2018.05.036
3	LINDENBERGER P, SARLIN P E, LARSSON V, et al. Pixel-perfect structure-from-motion with featuremetric refinement [C]// Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. IEEE, 2022: 5967– 5977. DOI: 10.1109/ICCV48922.2021.00593
4	YI K M, TRULLS E, LEPETIT V, et al. LIFT: learned invariant feature transform [C]// European Conference on Computer Vision. ECCV, 2016: 467– 483. DOI: 10.1007/978-3-319-46466-4_28
5	ZHANG X, YU F X, KARAMAN S, et al. Learning discriminative and transformation covariant local feature detectors [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 4923– 4931. DOI: 10.1109/CVPR.2017.523
6	ZHANG L G, RUSINKIEWICZ S. Learning to detect features in texture images [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 6325– 6333. DOI: 10.1109/CVPR.2018.00662
7	DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description [C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2018: 224– 236. DOI: 10.1109/CVPRW.2018.00060
8	LAGUNA A B, RIBA E, PONSA D, et al. Key.Net: keypoint detection by handcrafted and learned CNN filters [C]// International Conference on Computer Vision (ICCV). IEEE, 2020: 5835– 5843. DOI: 10.1109/ICCV.2019.00593
9	ONO Y, TRULLS E, FUA P, et al. LF-Net: Learning local features from images [C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems. ACM, 2018: 6237– 6247. DOI: 10.5555/3327345.3327521
10	SCHÖNBERGER J L, HARDMEIER H, SATTLER T, et al. Comparative evaluation of hand-crafted and learned local features [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2017: 6959– 6968. DOI: 10.1109/CVPR.2017.736
11	WANG J, ZHOU F, WEN S L, et al. Deep metric learning with angular loss [C]// International Conference on Computer Vision. IEEE, 2017: 2612– 2620. DOI: 10.1109/ICCV.2017.283
12	ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2015: 4353– 4361. DOI: 10.1109/CVPR.2015.7299064
13	LUO Z X, SHEN T W, ZHOU L, et al. ContextDesc: local descriptor augmentation with cross-modality context [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 2522– 2531. DOI: 10.1109/CVPR.2019.00263
14	TIAN Y R, YU X, FAN B, et al. SOSNet: second order similarity regularization for local descriptor learning [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 11008– 11017. DOI: 10.1109/CVPR.2019.01127
15	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks [C]// Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 4937– 4946. DOI: 10.1109/CVPR42600.2020.00499
16	SUN J M, SHEN Z H, WANG Y A, et al. LoFTR: detector-free local feature matching with transformers [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2021: 8918– 8927. DOI: 10.1109/CVPR46437.2021.00881
17	SARLIN P E, CADENA C, SIEGWART R, et al. From coarse to fine: robust hierarchical localization at large scale [C]// Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 12708– 12717. DOI: 10.1109/CVPR.2019.01300
18	LU P, SHENG B, ZHU F. Next-generation communications technology facilitates real-time distributed cloud rendering [J]. ZTE technology journal, 2021, 27( 1): 17– 20. DOI: 10.12142/ZTETJ.202101005
19	LU P, OUYANG X Z, GAO W W. Capacity improvement and practice of 5G industry virtual private network [J]. ZTE technology journal, 2022, 28( 2): 68– 74. DOI: 10.12142/ZTETJ.202202011

Scene Visual Perception and AR Navigation Applications

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 19

相关文章 0

编辑推荐

Metrics