A Content-Aware Bitrate Selection Method Using Multi-Step Prediction for 360-Degree Video Streaming

doi:10.12142/ZTECOM.202204012

ZTE Communications ›› 2022, Vol. 20 ›› Issue (4): 96-109.DOI: 10.12142/ZTECOM.202204012

• Research Paper • Previous Articles Next Articles

A Content-Aware Bitrate Selection Method Using Multi-Step Prediction for 360-Degree Video Streaming

GAO Nianzhen¹, YU Yifang², HUA Xinhai², FENG Fangzheng¹, JIANG Tao¹()

^1.Huazhong University of Science and Technology, Wuhan 430074, China
^2.ZTE Corporation, Shenzhen 518057, China

Received:2021-12-12 Online:2022-12-31 Published:2022-12-30
About author:GAO Nianzhen received her bachelors’ degree in computer science and technology from Sichuan University, China in 2020. She is currently working toward the PhD degree with the Research Center of 6G Mobile Communications, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, China. Her research interests include multimedia transmission and mobile edge computing.|YU Yifang received his MS degree in engineering from Xi’an Jiaotong University, China. He currently serves as the Senior Vice President of ZTE Corporation and President of the Cloud Video and Energy Product Operation Division. He has engaged in market planning and operations management in the telecommunications industry for over 20 years. His research interests include traditional telecom networks as well as the emerging fields such as cloud computing and the mobile Internet.|HUA Xinhai received his PhD degree from Nanjing University, China. He is currently the Vice President of ZTE Corporation and General Manager of Cloud Video Product Department. His research interests include cloud computing, IP-based video product technology and solutions, video business security solutions, content distribution network technology, product solutions, etc.|FENG Fangzheng received his bachelors’ degree in communication engineering from Hunan University, China in 2019. He is currently working toward the PhD degree with the Research Center of 6G Mobile Communications, School of Cyber Science and Engineering, Huazhong University of Science and Technology, Hubei, China. His research interests include wireless communication and mobile multimedia transmission.|JIANG Tao (taojiang@hust.edu.cn) received his PhD degree in information and communication engineering from Huazhong University of Science and Technology, China in 2004. He is currently a Distinguished Professor with the Research Center of 6G Mobile Communications and School of Cyber Science and Engineering, Huazhong University of Science and Technology. He has authored or coauthored more than 300 technical papers in main journals and conferences, and nine books or chapters in the areas of communications and networks. He has served or is serving as an associate editor of some technical journals in communications, including the IEEE Network, IEEE Transactions on Signal Processing, IEEE Communications Surveys and Tutorials, IEEE Transactions on Vehicular Technology, and IEEE Internet of Things Journal, and he is an associate editor-in-chief of China Communications. His main research directions include wireless communication, mobile multimedia transmission, etc.
Supported by:
ZTE Corporation(2021420118000065)

Abstract

Abstract:

A content-aware multi-step prediction control (CAMPC) algorithm is proposed to determine the bitrate of 360-degree videos, aiming to enhance the quality of experience (QoE) of users and reduce the cost of video content providers (VCP). The CAMPC algorithm first employs a neural network to generate the content richness and combines it with the current field of view (FOV) to accurately predict the probability distribution of tiles being viewed. Then, for the tiles in the predicted viewport which directly affect QoE, the CAMPC algorithm utilizes a multi-step prediction for future system states, and accordingly selects the bitrates of multiple subsequent steps, instead of an instantaneous state. Meanwhile, it controls the buffer occupancy to eliminate the impact of prediction errors. We implement CAMPC on players by building a 360-degree video streaming platform and evaluating other advanced adaptive bitrate (ABR) rules through the real network. Experimental results show that CAMPC can save 83.5% of bandwidth resources compared with the scheme that completely transmits the tiles outside the viewport with the Dynamic Adaptive Streaming over HTTP (DASH) protocol. Besides, the proposed method can improve the system utility by 62.7% and 27.6% compared with the DASH official and viewport-based rules, respectively.

Key words: DASH, content-aware FOV prediction, bitrate adaptation, multi-step prediction, generalized predictive control

GAO Nianzhen, YU Yifang, HUA Xinhai, FENG Fangzheng, JIANG Tao. A Content-Aware Bitrate Selection Method Using Multi-Step Prediction for 360-Degree Video Streaming[J]. ZTE Communications, 2022, 20(4): 96-109.

Figures/Tables 13

Figure 1 Streaming video system structure

Figure 2 (a) 360-degree video segmentation that the content-aware multi-step prediction control algrithm (CAMPC) uses to judge the importance of tiles; (b) example of FOV priority allocation according to FOV at a certain moment

Table 1 Spherical coordinates of tiles

	Latitude Range	Latitude Center	Longitude Range	Longitude Center
$V 1$	$[225 °, ? 315 °]$	$270 °$	$[45 °, ? 135 °]$	$90 °$
$V 2$	$[45 °, ? 135 °]$	$90 °$	$[45 °, ? 135 °]$	$90 °$
$V 3$	$[0 °, ? 360 °]$	$*$	$[0 °, ? 45 °]$	$0$
$V 4$	$[0 °, ? 360 °]$	$*$	$[135 °, ? 180 °]$	$180 °$
$V 5$	$[135 °, ? 225 °]$	$180 °$	$[45 °, ? 135 °]$	$90 °$
$V 6$	$[315 °, ? 45 °]$	$0 °$	$[45 °, ?? 135 °]$	$90 °$

Table 1 Spherical coordinates of tiles

	Latitude Range	Latitude Center	Longitude Range	Longitude Center
$V 1$	$[225 °, ? 315 °]$	$270 °$	$[45 °, ? 135 °]$	$90 °$
$V 2$	$[45 °, ? 135 °]$	$90 °$	$[45 °, ? 135 °]$	$90 °$
$V 3$	$[0 °, ? 360 °]$	$*$	$[0 °, ? 45 °]$	$0$
$V 4$	$[0 °, ? 360 °]$	$*$	$[135 °, ? 180 °]$	$180 °$
$V 5$	$[135 °, ? 225 °]$	$180 °$	$[45 °, ? 135 °]$	$90 °$
$V 6$	$[315 °, ? 45 °]$	$0 °$	$[45 °, ?? 135 °]$	$90 °$

Table 2 Tile priority and spherical coordinates mapping relations

Priority	100	75	50	25	0
$V 1$	$(90 °, ? 270 °)$	$(45 ° ~ 135 °, 225 ° ~ 315 °)$	$(10 ° ~ 170 °, 225 ° ~ 315 °)$ $(45 ° ~ 135 °, 180 ° ~ 360 °)$	$(10 ° ~ 170 °, 180 ° ~ 360 °)$	Others
$V 2$	$(90 °, ? 90 °)$	$(45 ° ~ 135 °, 45 ° ~ 135 °)$	$(10 ° ~ 170 °, 45 ° ~ 135 °)$ $(45 ° ~ 135 °, 0 ° ~ 180 °)$	$(10 ° ~ 170 °, 0 ° ~ 180 °)$	Others
$V 3$	$(0 ° ~ 5 °, *)$	$(5 ° ~ 10 °, *)$	$(0 ° ~ 45 °, *)$	$(0 ° ~ 90 °, *)$	$(90 ° ~ 180 °, *)$
$V 4$	$(175 ° ~ 180 °, *)$	$(170 ° ~ 175 °, *)$	$(135 ° ~ 180 °, *)$	$(90 ° ~ 180 °, *)$	$(0 ° ~ 90 °, *)$
$V 5$	$(90 °, ? 0 °)$	$(45 ° ~ 135 °, 135 ° ~ 225 °)$	$(10 ° ~ 170 °, 135 ° ~ 225 °)$ $(45 ° ~ 135 °, 270 ° ~ 90 °)$	$(10 ° ~ 170 °, 270 ° ~ 90 °)$	Others
$V 6$	$(90 °, ? 180 °)$	$(45 ° ~ 135 °, 315 ° ~ 45 °)$	$(10 ° ~ 170 °, 315 ° ~ 45 °)$ $(45 ° ~ 135 °, 90 ° ~ 270 °)$	$(10 ° ~ 170 °, 90 ° ~ 270 °)$	Others

Table 2 Tile priority and spherical coordinates mapping relations

Priority	100	75	50	25	0
$V 1$	$(90 °, ? 270 °)$	$(45 ° ~ 135 °, 225 ° ~ 315 °)$	$(10 ° ~ 170 °, 225 ° ~ 315 °)$ $(45 ° ~ 135 °, 180 ° ~ 360 °)$	$(10 ° ~ 170 °, 180 ° ~ 360 °)$	Others
$V 2$	$(90 °, ? 90 °)$	$(45 ° ~ 135 °, 45 ° ~ 135 °)$	$(10 ° ~ 170 °, 45 ° ~ 135 °)$ $(45 ° ~ 135 °, 0 ° ~ 180 °)$	$(10 ° ~ 170 °, 0 ° ~ 180 °)$	Others
$V 3$	$(0 ° ~ 5 °, *)$	$(5 ° ~ 10 °, *)$	$(0 ° ~ 45 °, *)$	$(0 ° ~ 90 °, *)$	$(90 ° ~ 180 °, *)$
$V 4$	$(175 ° ~ 180 °, *)$	$(170 ° ~ 175 °, *)$	$(135 ° ~ 180 °, *)$	$(90 ° ~ 180 °, *)$	$(0 ° ~ 90 °, *)$
$V 5$	$(90 °, ? 0 °)$	$(45 ° ~ 135 °, 135 ° ~ 225 °)$	$(10 ° ~ 170 °, 135 ° ~ 225 °)$ $(45 ° ~ 135 °, 270 ° ~ 90 °)$	$(10 ° ~ 170 °, 270 ° ~ 90 °)$	Others
$V 6$	$(90 °, ? 180 °)$	$(45 ° ~ 135 °, 315 ° ~ 45 °)$	$(10 ° ~ 170 °, 315 ° ~ 45 °)$ $(45 ° ~ 135 °, 90 ° ~ 270 °)$	$(10 ° ~ 170 °, 90 ° ~ 270 °)$	Others

Figure 3 Analyzing the priority of content perception under the semantic segmentation model

Figure 4 Bandwidth predictor using a Kalman filter

Figure 5 Continuous bandwidth predictor working process

Figure 6 Multi-step bandwidth predictor working process

Figure 7 Single simulator running status over Belgium 4G/Long Term Evolution (LTE) dataset

Figure 8 Time sequence of downloading process that may occur in multiple players

Figure 9 Changes in the system utility metrics of each advanced adaptive bitrate (ABR) rule over time

Table 3 Performance comparison of ABR algorithms in a real network environment

	CAMPC	DYNAMIC	LoL+	FOV	FOVContent
QoE	80.356	41.111	100.011	33.630	28.222
Saved-bandwidth	0.834 6	0.596	0	0.947	0.884
Utility	81.908	50.356	50.006	64.165	58.311

Table 4 Performance comparison of ABR algorithms in a weak network environment

	CAMPC	DYNAMIC	LoL+	FOV	FOVContent
QoE	12.051	4.690	6.565	23.526	6.904
Saved?bandwidth	0.975	0.954	0.941	0.926	0.790
Rebuffer?time	1.867	8.103	5.767	0	0.886 9
Utility	54.776	52.39	50.332	64.165	58.311

References 18

1	AFZAL S, CHEN J S, RAMAKRISHNAN K K. Characterization of 360-degree videos [C]//Proceedings of the Workshop on Virtual Reality and Augmented Reality Network. ACM, 2017: 1–6. DOI: 10.1145/3097895.3097896 DOI
2	XIE L, XU Z M, BAN Y X, et al. 360ProbDASH: improving QoE of 360 video streaming using tile-based HTTP adaptive streaming [C]//Proceedings of the 25th ACM international conference on Multimedia. ACM, 2017: 345–323. DOI: 10.1145/3123266.3123291 DOI
3	QIAN F, HAN B, XIAO Q Y, et al. Flare: practical viewport-adaptive 360-degree video streaming for mobile devices [C]//Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 2018: 99–114. DOI: 10.1145/3241539.3241565 DOI
4	SONG J R, YANG F Z, ZHANG W, et al. A fast FoV-switching DASH system based on tiling mechanism for practical omnidirectional video services [J]. IEEE transactions on multimedia, 2020, 22(9): 2366–2381. DOI: 10.1109/TMM.2019.2957976 DOI
5	SUN L Y, DUANMU F, LIU Y, et al. Multi-path multi-tier 360-degree video streaming in 5G networks [C]//Proceedings of the 9th ACM Multimedia Systems Conference. ACM, 2018: 162–173. DOI: 10.1145/3204949.3204978 DOI
6	ZHANG Y X, GUAN Y S, BIAN K G, et al. EPASS360: QoE-aware 360-degree video streaming over mobile devices [J]. IEEE transactions on mobile computing, 2021, 20(7): 2338–2353. DOI: 10.1109/tmc.2020.2978187 DOI
7	GUAN Y, ZHENG C Y, ZHANG X G, et al. Pano: optimizing 360° video streaming with a better understanding of quality perception [C]//Proceedings of the ACM Special Interest Group on Data Communication. ACM, 2019: 394–407. DOI: 10.1145/3341302.3342063 DOI
8	FENG X L, SWAMINATHAN V, WEI S. Viewport prediction for live 360-degree mobile video streaming using user-content hybrid motion tracking [J]. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2019, 3(2): 1–22. DOI: 10.1145/3328914 DOI
9	QIAO M L, XU M, WANG Z L, et al. Viewport-dependent saliency prediction in 360° video [J]. IEEE transactions on multimedia, 2021, 23: 748–760. DOI: 10.1109/ TMM.2020.2987682 DOI
10	YUAN H, ZHAO S Y, HOU J H, et al. Spatial and temporal consistency-aware dynamic adaptive streaming for 360-degree videos [J]. IEEE journal of selected topics in signal processing, 2020, 14(1): 177–193. DOI: 10.1109/JSTSP.2019.2957981 DOI
11	WEI X K, ZHOU M L, KWONG S, et al. A hybrid control scheme for 360-degree dynamic adaptive video streaming over mobile devices [J]. IEEE transactions on mobile computing, 2022, 21(10): 3428–3442. DOI: 10.1109/TMC.2021.3058099 DOI
12	SOBHANI A, YASSINE A, SHIRMOHAMMADI S. A video bitrate adaptation and prediction mechanism for HTTP adaptive streaming [J]. ACM transactions on multimedia computing, communications, and applications, 2017, 13(2): 1–25. DOI: 10.1145/3052822 DOI
13	ZHOU C, LIN C W, ZHANG X G, et al. Buffer-based smooth rate adaptation for dynamic HTTP streaming [C]//Proceedings of 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 2013: 1–9
14	YUAN H, HU X Q, HOU J H, et al. An ensemble rate adaptation framework for dynamic adaptive streaming over HTTP [J]. IEEE transactions on broadcasting, 2020, 66(2): 251–263. DOI: 10.1109/TBC.2019.2954074 DOI
15	SPITERI K, URGAONKAR R, SITARAMAN R K. BOLA: Near-optimal bitrate adaptation for online videos [C]//The 35th Annual IEEE International Conference on Computer Communications. IEEE, 2016: 1–9. DOI: 10.1109/INFOCOM.2016.7524428 DOI
16	YUAN H, FU H Y, LIU J, et al. Non-cooperative game theory based rate adaptation for dynamic video streaming over HTTP [J]. IEEE transactions on mobile computing, 2018, 17(10): 2334–2348. DOI: 10.1109/TMC.2018.2800749 DOI
17	SUN Y, YIN X Q, JIANG J C, et al. CS2P: improving video bitrate selection and adaptation with data-driven throughput prediction [C]//Proceedings of the 2016 ACM SIGCOMM Conference. ACM, 2016: 272–285. DOI: 10.1145/2934872.2934898 DOI
18	AKHTAR Z, NAM Y S, GOVINDAN R, et al. Oboe: auto-tuning video ABR algorithms to network conditions [C]//Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. ACM, 2018: 44–58. DOI: 10.1145/3230543.3230558 DOI

[1]	Christian Timmerer, Anatoliy Zabrovskiy. Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems [J]. ZTE Communications, 2019, 17(1): 18-24.
[2]	WU Kesong, CAO Xianbin, CHEN Zhifeng, WU Dapeng. Adaptive Mobile Video Delivery Based on Fountain Codes and DASH: A Survey [J]. ZTE Communications, 2018, 16(3): 9-14.
[3]	XIE Lan, ZHANG Xinggong, HUANG Cheng, DONG Zhenjiang. Markov Based Rate Adaption Approach for Live Streaming over HTTP/2 [J]. ZTE Communications, 2018, 16(2): 37-41.
[4]	Yiling Xu, Shaowei Xie, Hao Chen, Le Yang, Jun Sun. DASH and MMT and Their Applications in ATSC 3.0 [J]. ZTE Communications, 2016, 14(1): 39-49.

A Content-Aware Bitrate Selection Method Using Multi-Step Prediction for 360-Degree Video Streaming

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 18

Related Articles 4

Recommended Articles

Metrics