Local Scenario Perception and Web AR Navigation

doi:10.12142/ZTECOM.202304007

ZTE Communications ›› 2023, Vol. 21 ›› Issue (4): 54-59.DOI: 10.12142/ZTECOM.202304007

• Special Topic • Previous Articles Next Articles

Local Scenario Perception and Web AR Navigation

SHI Wenzhe¹^,²(), LIU Yanbin¹^,², ZHOU Qinfen¹

^1.State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen 518055, China
^2.ZTE Corporation, Shenzhen 518057, China

Received:2023-06-20 Online:2023-12-07 Published:2023-12-07
About author:SHI Wenzhe (shi.wenzhe@zte.com.cn) is a strategy planner and engineer for XRExplore Platform product planning at ZTE Corporation. He is also a member of the National Key Laboratory for Mobile Network and Mobile Multimedia Technology. His research interests include indoor visual AR navigation, SFM 3D reconstruction, visual SLAM, real-time cloud rendering, VR, and spatial perception.
LIU Yanbin is a strategy planner and product manager for XRExplore Platform product planning at ZTE Corporation. He is also a member of the National Key Laboratory for Mobile Network and Mobile Multimedia Technology. His research interests include real-time remote rendering, visual SLAM, and computer vision.
ZHOU Qinfen is the XR product leader director of new media industry and a senior architect of ZTE Corporation. She has more than 20 years of experience in the communication industry and media business. She has held the positions of product manager of short message center, product manager of cloud desktop, product line cost director, and XR product director at ZTE Corporation. She has a thorough understanding of products and related standards including Short Message Center, Cloud Desktop GPU Virtualization, XR, etc. As a member of the Shenzhen 8K UHD Video Industry Cooperation Alliance (SUCA) and Virtual Display Professional Committee of Jiangsu Communication Association, she leads a team responsible for the research on the latest video technology and related standards.
Supported by:
ZTE Industry‐University‐Institute Cooperation Funds

Abstract

Abstract:

This paper proposes a local point cloud map-based Web augmented reality (AR) indoor navigation system solution. By delivering the local point cloud map to the web front end for positioning, the real-time positioning can be implemented only with the help of the computing power of the web front end. In addition, with the characteristics of short time consumption and accurate positioning, an optimization solution to the local point cloud map is proposed, which includes specific measures such as descriptor de-duplicating and outlier removal, thus improving the quality of the point cloud. In this document, interpolation and smoothing effects are introduced for local map positioning, enhancing the anchoring effect and improving the smoothness and appearance of user experience. In small-scale indoor scenarios, the positioning frequency on an iPhone 13 can reach 30 fps, and the positioning precision is within 50 cm. Compared with an existing mainstream visual-based positioning manner for AR navigation, this specification does not rely on any additional sensor or cloud computing device, thereby greatly saving computing resources. It takes a very short time to meet the real-time requirements and provide users with a smooth positioning effect.

Key words: Web AR, three-dimensional reconstruction, navigation, positioning

SHI Wenzhe, LIU Yanbin, ZHOU Qinfen. Local Scenario Perception and Web AR Navigation[J]. ZTE Communications, 2023, 21(4): 54-59.

Figures/Tables 9

References 16

1	CAO M W, JIA W, LV Z H, et al. Fast and robust feature tracking for 3D reconstruction [J]. Optics & laser technology, 2019, 110: 120–128. DOI: 10.1016/j.optlastec.2018.05.036
2	LINDENBERGER P, SARLIN P E, LARSSON V, et al. Pixel-perfect structure-from-motion with featuremetric refinement [C]//Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2022: 5967–5977. DOI: 10.1109/ICCV48922.2021.00593
3	YI K M, TRULLS E, LEPETIT V, et al. LIFT: learned Invariant Feature Transform [C]//European Conference on Computer Vision. Springer, 2016: 467–483. DOI: 10.1007/978-3-319-46466-4_28
4	ZHANG X, YU F X, KARAMAN S, et al. Learning discriminative and transformation covariant local feature detectors [C]//Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 4923–4931. DOI: 10.1109/CVPR.2017.523
5	ZHANG L G, RUSINKIEWICZ S. Learning to detect features in texture images [C]//Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 6325–6333. DOI: 10.1109/CVPR.2018.00662
6	DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description [C]//Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2018: 337–33712. DOI: 10.1109/CVPRW.2018.00060
7	LAGUNA A B, RIBA E, PONSA D, et al. Key.Net: keypoint detection by handcrafted and learned CNN filters [C]//Proc. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2020: 5835–5843. DOI: 10.1109/ICCV.2019.00593
8	ONO Y, TRULLS E, FUA P, et al. LF-net: learning local features from images [C]//Proc. 32nd International Conference on Neural Information Processing Systems. NIPS, 2018: 6273–6247. DOI:10.5555/3327345.3327521
9	SCHÖNBERGER J L, HARDMEIER H, SATTLER T, et al. Comparative evaluation of hand-crafted and learned local features [C]//Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 6959–6968. DOI: 10.1109/CVPR.2017.736
10	WANG J, ZHOU F, WEN S L, et al. Deep metric learning with angular loss [C]//Proc. 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2612–2620. DOI: 10.1109/ICCV.2017.283
11	ZAGORUYKO S, KOMODAKIS N. Learning to compare image patches via convolutional neural networks [C]//Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015: 4353–4361. DOI: 10.1109/CVPR.2015.7299064
12	LUO Z X, SHEN T W, ZHOU L, et al. ContextDesc: local descriptor augmentation with cross-modality context [C]//Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 2522–2531. DOI: 10.1109/CVPR.2019.00263
13	TIAN Y R, YU X, FAN B, et al. SOSNet: second order similarity regularization for local descriptor learning [C]//Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 11008–11017. DOI: 10.1109/CVPR.2019.01127
14	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks [C]//Proc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 4937–4946. DOI: 10.1109/CVPR42600.2020.00499
15	SUN J M, SHEN Z H, WANG Y A, et al. LoFTR: detector-free local feature matching with transformers [C]//Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021: 8918–8927. DOI: 10.1109/CVPR46437.2021.00881
16	SARLIN P E, CADENA C, SIEGWART R, et al. From coarse to fine: Robust hierarchical localization at large scale [C]//Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 12708–12717. DOI: 10.1109/CVPR.2019.01300

Descriptor Size/MB	Mobile Phone Model	Extracting ORB Features	Feature Matching (KNN)	PNP	Total Calculated Time/ms	Frames per Second/fps
0.9	MEIZU 11	1.052	38	4	50	20
	OnePlus 6	0.876	37	4	46	21
	Xiaomi 11	0.557	19	2	26	38
	Iphone 13	0.098	8	2	11	90

Descriptor Size/MB	Mobile Phone Model	Extracting ORB Features	Feature Matching (KNN)	PNP	Total Calculated Time/ms	Frames per Second/fps
0.9	MEIZU 11	1.052	38	4	50	20
	OnePlus 6	0.876	37	4	46	21
	Xiaomi 11	0.557	19	2	26	38
	Iphone 13	0.098	8	2	11	90

[1]	LI Yuting, DING Yi, GAO Jiangchuan, LIU Yusha, HU Jie, YANG Kun. UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network [J]. ZTE Communications, 2023, 21(2): 80-87.
[2]	LU Ping, SHENG Bin, SHI Wenzhe. Scene Visual Perception and AR Navigation Applications [J]. ZTE Communications, 2023, 21(1): 81-88.
[3]	CHEN Liangqin, TIAN Liping, XU Zhimeng, CHEN Zhizhang. A Survey of Wi-Fi Sensing Techniques with Channel State Information [J]. ZTE Communications, 2020, 18(3): 57-63.
[4]	CHEN Xiaohong, QIAN Chen, WEI Wei. An Indoor Positioning Scheme for Visible Light Using Fingerprint Database with Multi-Parameters [J]. ZTE Communications, 2017, 15(1): 43-48.

Local Scenario Perception and Web AR Navigation

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 16

Related Articles 4

Recommended Articles

Metrics