Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

doi:10.12142/ZTECOM.201901004

ZTE Communications ›› 2019, Vol. 17 ›› Issue (1): 18-24.DOI: 10.12142/ZTECOM.201901004

收稿日期:2018-08-16 出版日期:2019-02-20 发布日期:2019-11-14

Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

Christian Timmerer¹, Anatoliy Zabrovskiy²

1. Alpen-Adria-Universit?t Klagenfurt/Bitmovin Inc., Klagenfurt 9020, Austria
2. Petrozavodsk State University & Alpen-Adria-Universit?t Klagenfurt Petrozavodsk, Petrozavodsk 185910, Russia

Received:2018-08-16 Online:2019-02-20 Published:2019-11-14
About author:Christian Timmerer (christian.timmerer@itec.aau.at) is an associate professor with Alpen-Adria-Universita?t Klagenfurt, Austria. He is a Co-Founder of Bitmovin Inc., San Francsico, USA, as well as the CIO and the Head of Research and Standardization. He has co-authored seven patents and over 200 publications in workshops, conferences, journals, and book chapters. He participated in several EC-funded projects, notably DANAE, ENTHRONE, P2P-Next, ALICANTE, SocialSensor, ICoSOLE, and the COST Action IC1003 QUALINET. He also participated in ISO/MPEG work for several years, notably in the areas of MPEG-21, MPEG-M, MPEG-V, and MPEG- DASH. His research interests include immersive multimedia communications, streaming, adaptation, and quality of experience. He was the General Chair of WIAMIS 2008, QoMEX 2013, ACM MMSys 2016, and Packet Video 2018. Further information can be found at http://blog.timmerer.com.|Anatoliy Zabrovskiy received his B.S. and M.S. degrees in information and computer technology from Petrozavodsk State University, Russia in 2006 and 2008 respectively, and a Ph.D. degree in engineering from the same university in 2013. He has been working in the field of network and multimedia communication technologies for over ten years. He was a Cisco certified academy instructor for CCNA. He was award winner of two international programs: Scholarships of the Scholarship Foundation of the Republic of Austria for Postdocs and Erasmus Mundus External Cooperation Window program for doctorate students. He was a prize winner of Sun Microsystems contest “Idea2Project”. He is currently a postdoctoral researcher at the Department of Information Technology (ITEC), Alpen-Adria-Universit?t Klagenfurt, Austria. He is a member of the Technical Program Committee of ACM MMSys 2019. His research interests include video streaming, network technologies, quality of experience, and machine learning.
Supported by:
This work was supported in part by the Austrian Research Promotion Agency (FFG) under the next generation video streaming project “PROMETHEUS”.

摘要/Abstract

Abstract:

Streaming audio and video content currently accounts for the majority of the Internet traffic and is typically deployed over the top of the existing infrastructure. We are facing the challenge of a plethora of media players and adaptation algorithms showing different behavior but lacking a common framework for both objective and subjective evaluation of such systems. This paper aims to close this gap by proposing such a framework, describing its architecture, providing an example evaluation, and discussing open issues.

Key words: HTTP adaptive streaming, DASH, QoE, performance evaluation

. [J]. ZTE Communications, 2019, 17(1): 18-24.

Christian Timmerer, Anatoliy Zabrovskiy. Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems[J]. ZTE Communications, 2019, 17(1): 18-24.

图/表 6

Figure 1. General framework architecture: AdViSE and WESP framework for the automated testing of web-based media players and adaptation algorithms.

Figure 2. Example screenshots of AdViSE and WESP to demonstrate easy setup of HTTP Adaptive Streaming (HAS) evaluations.

Figure 3. Architecture of adaptive video streaming evaluation framework for the automated testing of media players and adaptation algorithms.

Figure 4. A Web-Based Subjective Evaluation Platform (WESP).

Figure 5. Download video bitrate (top) and video buffer length (bottom) for the selected industry players (left) and adaptation algorithms proposed in the research literature (right).

Table 1 Overview of example results

Metrics	Bitmovin		dash.js		Flowplayer		Festive		Instant		Thang
Metrics	μ	σ	μ	σ	μ	σ	μ	σ	μ	σ	μ	σ
a. Startup time/s	1.8	0.2	3.5	0.3	3.2	0.1	3.2	0.2	9.0	1.4	9.7	0.8
b. Stalls [#]	0	0	4	1.6	7	1.7	1	0.8	0	0	0	0
c. Stall duration/s	0	0	5.4	3.4	14.2	3.5	1.0	1.0	0	0	0	0
d. Quality switches [#]	17	1	29	4	23	1	65	5	45	3	43	9
e. Bitrate/(kbit/s)	982	22	981	9	908	10	664	10	916	19	617	14
f. QoE/MOS [M?ki] [21]	4.56	0.0	4.38	0.08	4.2	0.09	4.53	0.04	4.56	0.0	4.56	0.0
g. QoE/MOS [Mok] [22]	4.1	0.0	3.84	0.05	3.79	0.02	3.93	0.1	3.62	0.09	3.58	0.06

参考文献 23

[1]	Cisco Systems, Inc. Cisco Visual Networking Index: Forecast and Methodology, 2016-2021 (White Paper). [R/OL]. ( 2017- 09- 15)[2018-07-28].
[2]	NIELSEN J . Nielsen’s Law of Internet Bandwidth (updated 2018) [EB/OL]. (1998-04)[2018-03-03].
[3]	Sodagar I . The MPEG-DASH Standard for Multimedia Streaming Over the Internet[J]. IEEE Multimedia, 2011,18(4):62-67. DOI: 10.1109/MMUL.2011.71
[4]	PANTOS R, MAY W . HTTP Live Streaming [EB/OL]. ( 2017)[2018-07-28]. https://www.ietf.org/rfc/rfc8216.txt.
[5]	ISO/IEC. Information Technology—Multimedia Application Format (MPEG-A)—Part 19: Common Media Application Format (CMAF) for Segmented Media: ISO/IEC 23000-19[S]. 2017.
[6]	SEUFERT M, EGGER S, SLANINA M , et al. A Survey on Quality of Experience of HTTP Adaptive Streaming[J]. IEEE Communications Surveys & Tutorials, 2015,17(1):469-492. DOI: 10.1109/comst.2014.2360940
[7]	BENTALEB A, TAANI B, BEGEN A C , et al. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP[J]. IEEE Communications Surveys Tutorials, 2019,21(1):562-585. DOI: 10.1109/COMST.2018.2862938
[8]	MÜLLER C, LEDERER S, TIMMERER . An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments [C]//Proceedings of the 4th Workshop on Mobile Video, ser. MoVid’12, New York, USA: ACM, 2012: 37-42. DOI: 10.1145/2151677.2151686
[9]	CICCO De L, CALDARALO V, PALMISANO V , et al. TAPAS: A Tool for rApid Prototyping of Adaptive Streaming Algorithms [C]//Proceedings of the 2014 Workshop on Design, Quality and Deployment of Adaptive Video Streaming, ser. VideoNext’14, New York, USA: ACM: 2014: 1-6. DOI: 10.1145/2676652. 2676654
[10]	ZABROVSKIY A, PETROV E, KUZMIN E , et al. Evaluation of the Performance of Adaptive HTTP Streaming Systems [EB/OL]. CoRR, vol. abs/1710.02459 [2017].
[11]	TIMMERER C, ZABROVSKIY A, KUZMIN E , et al. Quality of Experience of Commercially Deployed Adaptive Media Players [C]//21st Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 2017: 330-335
[12]	STOHR D, FRÖMMGEN A, RIZK A , et al. Where are the Sweet Spots? A Systematic Approach to Reproducible DASH Player Comparisons [C]//Proceedings of the 2017 ACM on Multimedia Conference, ser. MM’17, New York, USA: ACM, 2017: 1113-1121. DOI: 10.1145/3123266.3123426
[13]	ZABROVSKIY A, KUZMIN E, PETROV E , et al. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players [C]//Proceedings of the 8th ACM on Multimedia Systems Conference, ser. MMSys’17, New York, USA: ACM, 2017: pp. 217-220. DOI: 10.1145/3083187.3083221
[14]	RAINER B, WALTL M, TIMMERER C . A Web Based Subjective Evaluation Platform [C]//Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am Wörthersee, Austria, 2013: 24-25. DOI: 10.1109/QoMEX.2013.6603196
[15]	HOSSFELD T, KEIMEL C, HIRTH M , et al. Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing[J]. IEEE Transactions on Multimedia, 2014,16(2):541-558. DOI: 10.1109/tmm.2013.2291663 DOI URL
[16]	HOßFELD T, HIRTH M, KORSHUNOV P , et al. Survey of Web-Based Crowdsourcing Frameworks for Subjective Quality Assessment [C]//IEEE 16th International Workshop on Multimedia Signal Processing (MMSP), Jakarta, Indonesia, 2014: 1-6. DOI: 10.1109/MMSP.2014.6958831
[17]	JIANG J, SEKAR V, ZHANG H . Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with FESTIVE [C]//Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’12, New York, USA: ACM, 2012: 97-108. DOI: 10.1145/2413176.2413189
[18]	ROMERO L R . A Dynamic Adaptive HTTP Streaming Video Service for Google Android [D]. Master of Science Thesis, Stockholm, Sweden: Royal Institute of Technology (KTH) Stockholm, 2011.
[19]	THANG T, HO Q D, KANG J , et al. Adaptive Streaming of Audiovisual Content Using MPEG DASH[J]. IEEE Transactions on Consumer Electronics, 2012,58(1):78-85. DOI: 10.1109/tce.2012.6170058
[20]	TIMMERER C, MAIERO M, RAINER B. Which Adaptation Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media Streaming Systems [EB/OL]. arXiv:1606.00341 (2016)[ 2018- 07- 28]. http://arxiv.org/abs/1606.00341
[21]	MÄKI T, VARELA M, AMMAR D . A Layered Model for Quality Estimation of HTTP Video from QoS Measurements [C]//11th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Bangkok, Thailand, 2015: 591-598. DOI: 10.1109/SITIS.2015.41
[22]	MOK R K P, CHAN E W W, CHANG R K C . Measuring the Quality of Experience of HTTP Video Streaming [C]//12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops, Dublin, Ireland, 2011: 485-492. DOI: 10.1109/INM.2011.5990550
[23]	BENTALEB A, BEGEN A C, ZIMMERMANN R , et al. SDNHAS: An SDN-Enabled Architecture to Optimize QoE in HTTP Adaptive Streaming[J]. IEEE Transactions on Multimedia, 2017,19(10):2136-2151. DOI: 10.1109/tmm.2017.2733344

Automating QoS and QoE Evaluation of HTTP Adaptive Streaming Systems

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 23

相关文章 0

编辑推荐

Metrics