Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

doi:10.12142/ZTECOM.201901004

ZTE Communications ›› 2019, Vol. 17 ›› Issue (1): 18-24.DOI: 10.12142/ZTECOM.201901004

• Special Topic • Previous Articles Next Articles

Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

Christian Timmerer¹, Anatoliy Zabrovskiy²

1. Alpen-Adria-Universit?t Klagenfurt/Bitmovin Inc., Klagenfurt 9020, Austria
2. Petrozavodsk State University & Alpen-Adria-Universit?t Klagenfurt Petrozavodsk, Petrozavodsk 185910, Russia

Received:2018-08-16 Online:2019-02-20 Published:2019-11-14
About author:Christian Timmerer (christian.timmerer@itec.aau.at) is an associate professor with Alpen-Adria-Universita?t Klagenfurt, Austria. He is a Co-Founder of Bitmovin Inc., San Francsico, USA, as well as the CIO and the Head of Research and Standardization. He has co-authored seven patents and over 200 publications in workshops, conferences, journals, and book chapters. He participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, ICoSOLE, and the COST Action IC1003 QUALINET. He also participated in ISO/MPEG work for several years, notably in the areas of MPEG-21, MPEG-M, MPEG-V, and MPEG- DASH. His research interests include immersive multimedia communications, streaming, adaptation, and quality of experience. He was the General Chair of WIAMIS 2008, QoMEX 2013, ACM MMSys 2016, and Packet Video 2018. Further information can be found at http://blog.timmerer.com.|Anatoliy Zabrovskiy received his B.S. and M.S. degrees in information and computer technology from Petrozavodsk State University, Russia in 2006 and 2008 respectively, and a Ph.D. degree in engineering from the same university in 2013. He has been working in the field of network and multimedia communication technologies for over ten years. He was a Cisco certified academy instructor for CCNA. He was award winner of two international programs: Scholarships of the Scholarship Foundation of the Republic of Austria for Postdocs and Erasmus Mundus External Cooperation Window program for doctorate students. He was a prize winner of Sun Microsystems contest “Idea2Project”. He is currently a postdoctoral researcher at the Department of Information Technology (ITEC), Alpen-Adria-Universit?t Klagenfurt, Austria. He is a member of the Technical Program Committee of ACM MMSys 2019. His research interests include video streaming, network technologies, quality of experience, and machine learning.
Supported by:
This work was supported in part by the Austrian Research Promotion Agency (FFG) under the next generation video streaming project “PROMETHEUS”.

Abstract

Abstract:

Streaming audio and video content currently accounts for the majority of the Internet traffic and is typically deployed over the top of the existing infrastructure. We are facing the challenge of a plethora of media players and adaptation algorithms showing different behavior but lacking a common framework for both objective and subjective evaluation of such systems. This paper aims to close this gap by proposing such a framework, describing its architecture, providing an example evaluation, and discussing open issues.

Key words: HTTP adaptive streaming, DASH, QoE, performance evaluation

Christian Timmerer, Anatoliy Zabrovskiy. Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems[J]. ZTE Communications, 2019, 17(1): 18-24.

Figures/Tables 6

Figure 1. General framework architecture: AdViSE and WESP framework for the automated testing of web-based media players and adaptation algorithms.

Figure 2. Example screenshots of AdViSE and WESP to demonstrate easy setup of HTTP Adaptive Streaming (HAS) evaluations.

Figure 3. Architecture of adaptive video streaming evaluation framework for the automated testing of media players and adaptation algorithms.

Figure 4. A Web-Based Subjective Evaluation Platform (WESP).

Figure 5. Download video bitrate (top) and video buffer length (bottom) for the selected industry players (left) and adaptation algorithms proposed in the research literature (right).

Table 1 Overview of example results

Metrics	Bitmovin		dash.js		Flowplayer		Festive		Instant		Thang
Metrics	μ	σ	μ	σ	μ	σ	μ	σ	μ	σ	μ	σ
a. Startup time/s	1.8	0.2	3.5	0.3	3.2	0.1	3.2	0.2	9.0	1.4	9.7	0.8
b. Stalls [#]	0	0	4	1.6	7	1.7	1	0.8	0	0	0	0
c. Stall duration/s	0	0	5.4	3.4	14.2	3.5	1.0	1.0	0	0	0	0
d. Quality switches [#]	17	1	29	4	23	1	65	5	45	3	43	9
e. Bitrate/(kbit/s)	982	22	981	9	908	10	664	10	916	19	617	14
f. QoE/MOS [M?ki] [21]	4.56	0.0	4.38	0.08	4.2	0.09	4.53	0.04	4.56	0.0	4.56	0.0
g. QoE/MOS [Mok] [22]	4.1	0.0	3.84	0.05	3.79	0.02	3.93	0.1	3.62	0.09	3.58	0.06

References 23

[1]	Cisco Systems, Inc. Cisco Visual Networking Index: Forecast and Methodology, 2016-2021 (White Paper). [R/OL]. ( 2017- 09- 15)[2018-07-28].
[2]	NIELSEN J . Nielsen’s Law of Internet Bandwidth (updated 2018) [EB/OL]. (1998-04)[2018-03-03].
[3]	Sodagar I . The MPEG-DASH Standard for Multimedia Streaming Over the Internet[J]. IEEE Multimedia, 2011,18(4):62-67. DOI: 10.1109/MMUL.2011.71
[4]	PANTOS R, MAY W . HTTP Live Streaming [EB/OL]. ( 2017)[2018-07-28]. https://www.ietf.org/rfc/rfc8216.txt.
[5]	ISO/IEC. Information Technology—Multimedia Application Format (MPEG-A)—Part 19: Common Media Application Format (CMAF) for Segmented Media: ISO/IEC 23000-19[S]. 2017.
[6]	SEUFERT M, EGGER S, SLANINA M , et al. A Survey on Quality of Experience of HTTP Adaptive Streaming[J]. IEEE Communications Surveys & Tutorials, 2015,17(1):469-492. DOI: 10.1109/comst.2014.2360940
[7]	BENTALEB A, TAANI B, BEGEN A C , et al. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP[J]. IEEE Communications Surveys Tutorials, 2019,21(1):562-585. DOI: 10.1109/COMST.2018.2862938
[8]	MÜLLER C, LEDERER S, TIMMERER . An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments [C]//Proceedings of the 4th Workshop on Mobile Video, ser. MoVid’12, New York, USA: ACM, 2012: 37-42. DOI: 10.1145/2151677.2151686
[9]	CICCO De L, CALDARALO V, PALMISANO V , et al. TAPAS: A Tool for rApid Prototyping of Adaptive Streaming Algorithms [C]//Proceedings of the 2014 Workshop on Design, Quality and Deployment of Adaptive Video Streaming, ser. VideoNext’14, New York, USA: ACM: 2014: 1-6. DOI: 10.1145/2676652. 2676654
[10]	ZABROVSKIY A, PETROV E, KUZMIN E , et al. Evaluation of the Performance of Adaptive HTTP Streaming Systems [EB/OL]. CoRR, vol. abs/1710.02459 [2017].
[11]	TIMMERER C, ZABROVSKIY A, KUZMIN E , et al. Quality of Experience of Commercially Deployed Adaptive Media Players [C]//21st Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2017: 330-335
[12]	STOHR D, FRÖMMGEN A, RIZK A , et al. Where are the Sweet Spots? A Systematic Approach to Reproducible DASH Player Comparisons [C]//Proceedings of the 2017 ACM on Multimedia Conference, ser. MM’17, New York, USA: ACM, 2017: 1113-1121. DOI: 10.1145/3123266.3123426
[13]	ZABROVSKIY A, KUZMIN E, PETROV E , et al. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players [C]//Proceedings of the 8th ACM on Multimedia Systems Conference, ser. MMSys’17, New York, USA: ACM, 2017: pp. 217-220. DOI: 10.1145/3083187.3083221
[14]	RAINER B, WALTL M, TIMMERER C . A Web Based Subjective Evaluation Platform [C]//Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wörthersee, Austria, 2013: 24-25. DOI: 10.1109/QoMEX.2013.6603196
[15]	HOSSFELD T, KEIMEL C, HIRTH M , et al. Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing[J]. IEEE Transactions on Multimedia, 2014,16(2):541-558. DOI: 10.1109/tmm.2013.2291663 DOI URL
[16]	HOßFELD T, HIRTH M, KORSHUNOV P , et al. Survey of Web-Based Crowdsourcing Frameworks for Subjective Quality Assessment [C]//IEEE 16th International Workshop on Multimedia Signal Processing (MMSP), Jakarta, Indonesia, 2014: 1-6. DOI: 10.1109/MMSP.2014.6958831
[17]	JIANG J, SEKAR V, ZHANG H . Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with FESTIVE [C]//Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’12, New York, USA: ACM, 2012: 97-108. DOI: 10.1145/2413176.2413189
[18]	ROMERO L R . A Dynamic Adaptive HTTP Streaming Video Service for Google Android [D]. Master of Science Thesis, Stockholm, Sweden: Royal Institute of Technology (KTH) Stockholm, 2011.
[19]	THANG T, HO Q D, KANG J , et al. Adaptive Streaming of Audiovisual Content Using MPEG DASH[J]. IEEE Transactions on Consumer Electronics, 2012,58(1):78-85. DOI: 10.1109/tce.2012.6170058
[20]	TIMMERER C, MAIERO M, RAINER B. Which Adaptation Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media Streaming Systems [EB/OL]. arXiv:1606.00341 (2016)[ 2018- 07- 28]. http://arxiv.org/abs/1606.00341
[21]	MÄKI T, VARELA M, AMMAR D . A Layered Model for Quality Estimation of HTTP Video from QoS Measurements [C]//11th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Bangkok, Thailand, 2015: 591-598. DOI: 10.1109/SITIS.2015.41
[22]	MOK R K P, CHAN E W W, CHANG R K C . Measuring the Quality of Experience of HTTP Video Streaming [C]//12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops, Dublin, Ireland, 2011: 485-492. DOI: 10.1109/INM.2011.5990550
[23]	BENTALEB A, BEGEN A C, ZIMMERMANN R , et al. SDNHAS: An SDN-Enabled Architecture to Optimize QoE in HTTP Adaptive Streaming[J]. IEEE Transactions on Multimedia, 2017,19(10):2136-2151. DOI: 10.1109/tmm.2017.2733344

Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 23

Related Articles 12

Recommended Articles

Metrics

[1]	GAO Nianzhen, YU Yifang, HUA Xinhai, FENG Fangzheng, JIANG Tao. A Content-Aware Bitrate Selection Method Using Multi-Step Prediction for 360-Degree Video Streaming [J]. ZTE Communications, 2022, 20(4): 96-109.
[2]	ZHANG Man, LI Dapeng, LIU Zhuang, GAO Yin. QoE Management for 5G New Radio [J]. ZTE Communications, 2021, 19(3): 64-72.
[3]	ZHANG Xinggong, XIE Lan, GUO Zongming. Quality Assessment and Measurement for Internet Video Streaming [J]. ZTE Communications, 2019, 17(1): 12-17.
[4]	CHEN Jinling, XU Yiwen, LIU Yisang, HUANG Huiwen, ZHUANG Zhongwen. Quality of Experience Effects in Video Delivery [J]. ZTE Communications, 2019, 17(1): 25-30.
[5]	LIU Qian, ZHAO Tiesong. Quality-of-Experience in Human-in-the-Loop Haptic Communications [J]. ZTE Communications, 2019, 17(1): 48-55.
[6]	Amulya Karaadi, Is-Haka Mkwawa, Lingfen Sun. How to Manage Multimedia Traffic: Based on QoE or QoT? [J]. ZTE Communications, 2018, 16(3): 23-29.
[7]	WU Kesong, CAO Xianbin, CHEN Zhifeng, WU Dapeng. Adaptive Mobile Video Delivery Based on Fountain Codes and DASH: A Survey [J]. ZTE Communications, 2018, 16(3): 9-14.
[8]	XIE Lan, ZHANG Xinggong, HUANG Cheng, DONG Zhenjiang. Markov Based Rate Adaption Approach for Live Streaming over HTTP/2 [J]. ZTE Communications, 2018, 16(2): 37-41.
[9]	LIU Yongsheng, GU Yu, WEN Xiangjiang, WANG Xiaoyan, FU Yufei. Measuring QoE of Web Service with Mining DNS Resolution Data [J]. ZTE Communications, 2017, 15(S2): 38-42.
[10]	Yiling Xu, Shaowei Xie, Hao Chen, Le Yang, Jun Sun. DASH and MMT and Their Applications in ATSC 3.0 [J]. ZTE Communications, 2016, 14(1): 39-49.
[11]	Yingxiao Xu, Prasad Calyam, David Welling, Saravanan Mohan, Alex Berryman, and Rajiv Ramnath. Human-Centric Composite-Quality Modeling and Assessment for Virtual Desktop Clouds [J]. ZTE Communications, 2013, 11(1): 27-36.
[12]	Is-Haka Mkwawa and Lingfen Sun. Battery Voltage Discharge Rate Prediction and Video Content Adaptation in Mobile Devices on 3G Access Networks [J]. ZTE Communications, 2013, 11(1): 44-50.