Recent Advances and Challenges in Video Quality Assessment

doi:10.12142/ZTECOM.201901002

Abstract

Abstract:

Video quality assessment (VQA) plays a vital role in the field of video processing, including areas of video acquisition, video filtering in retrieval, video compression, video restoration, and video enhancement. Since VQA has gained much attention in recent years, this paper gives an up-to-date review of VQA research and highlights current challenges in this filed. The subjective study and common VQA databases are first reviewed. Then, a survey on the objective VQA methods, including full-reference, reduced-reference, and no-reference VQA, is reported. Last but most importantly, the key limitations of current research and several challenges in the field of VQA are discussed, which include the impact of video content, memory effects, computational efficiency, personalized video quality prediction, and quality assessment of newly emerged videos.

Key words: databases, perceptual optimization, personalization, video content, VQA

LI Dingquan, JIANG Tingting, JIANG Ming. Recent Advances and Challenges in Video Quality Assessment[J]. ZTE Communications, 2019, 17(1): 3-11.

Figures/Tables 2

Table 1 VQA databases with the subjective study methods, numbers of (#) reference/distorted videos and score types

VQA Database	Subjective Study Method	#Reference/Distorted Videos	Score Type^**
VQEG FR-TV Phase I [9]	DSCQS	22/352	DMOS+σ
VQEG HDTV [10]	ACR-HR	49/740	Raw
EPFL-PoliMI [11]	SS/ACR	12/156	Raw
LIVE [12]	ACR-HR	10/150	DMOS+σ
LIVE Mobile [13]	SSCQE-HR	10/200	DMOS+σ
CSIQ [14]	SAMVIQ	12/216	DMOS+σ
CVD2014 [15]	SS/ACR	None/234	Raw
LIVE-Qualcomm [16]	SS/ACR	None/208	MOS+σ
KoNViD-1k^* [17]	SS/ACR	None/1 200	Raw
LIVE-VQC^* [18]	SS/ACR	None/585	MOS+σ

Figure 1. The six images are the first frame of six diferent distorted videos in LIVE-Qualcomm [16]. The distortions in these videos are at similar levels: a) NightSence, mean opinion socre (MOS)=35.7252; b) NightSence, MOS=32.2755; c) DogsOnBeach, MOS=65.9423; d) DogsOnBeach, MOS=61.2497; e) ManUnderTree, MOS=52.5666; f) ManUnderTree, MOS=57.2888. It can be seen that the perceived video quality does strongly depend on video content.

References 76

[1]	SESHADRINATHAN K, BOVIK A C . Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos[J]. IEEE Transactions on Image Processing, 2010,19(2):335-350. DOI: 10.1109/tip.2009.2034992
[2]	SOUNDARARAJAN R, BOVIK A C . Video Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2013,23(4):684-694. DOI: 10.1109/tcsvt.2012.2214933 DOI URL
[3]	MITTAL A, SAAD M A, BOVIK A C . A Completely Blind Video Integrity Oracle[J]. IEEE Transactions on Image Processing, 2016,25(1):289-300. DOI: 10.1109/tip.2015.2502725
[4]	CHIKKERUR S, SUNDARAM V, REISSLEIN M , et al. Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison[J]. IEEE Transactions on Broadcasting, 2011,57(2):165-182. DOI: 10.1109/tbc.2011.2104671 DOI URL
[5]	SHAHID M, ROSSHOLM A, LÖVSTRÖM B , et al. No-Reference Image and Video Quality Assessment: A Classification and Review of Recent Approaches[J]. EURASIP Journal on Image and Video Processing, 2014. DOI: 10.1186/1687-5281-2014-40
[6]	ITU. Radiocommunication Sector & Telecommunication Standardization Sector [EB/OL]. [2018-05-27]. https://www.itu.int/en/Pages/default.aspx
[7]	XU Q Q, HUANG Q M, JIANG T T , et al. HodgeRank on Random Graphs for Subjective Video Quality Assessment[J]. IEEE Transactions on Multimedia, 2012,14(3):844-857. DOI: 10.1109/tmm.2012.2190924 DOI URL
[8]	FAN Z W, JIANG T T, HUANG T J . Active Sampling Exploiting Reliable Informativeness for Subjective Image Quality Assessment Based on Pairwise Comparison[J]. IEEE Transactions on Multimedia, 2017,19(12):2720-2735. DOI: 10.1109/tmm.2017.2711860
[9]	VQEG. VQEG FR-TV Phase I Database [EB/OL]. (2000)[ 2018- 05- 27]. http://www.its.bldrdoc.gov/vqeg/projects/frtv-phase-i/frtv-phase-i.aspx
[10]	VQEG HDTV Group. VQEG HDTV Database [EB/OL]. (2009)[ 2018- 05- 27]. http://www.its.bldrdoc.gov/vqeg/projects/hdtv/hdtv.aspx
[11]	DE SIMONE F, NACCARI M, TAGLIASACCHI M , et al. Subjective Assessment of H.264/AVC Video Sequences Transmitted over a Noisy Channel [C]//International Workshop on Quality of Multimedia Experience. San Diego, USA, 2009: 204-209. DOI: 10.1109/QOMEX.2009.5246952
[12]	SESHADRINATHAN K, SOUNDARARAJAN R, BOVIK A C , et al. Study of Subjective and Objective Quality Assessment of Video[J]. IEEE Transactions on Image Processing, 2010,19(6):1427-1441. DOI: 10.1109/tip.2010. 2042111
[13]	MOORTHY A K, CHOI L K, BOVIK A C , et al. Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies[J]. IEEE Journal of Selected Topics in Signal Processing, 2012,6(6):652-671. DOI: 10.1109/jstsp.2012.2212417 DOI URL
[14]	VU P V, CHANDLER D M . ViS3: An Algorithm for Video Quality Assessment via Analysis of Spatial and Spatiotemporal Slices[J]. Journal of Electronic Imaging, 2014,23(1):013016. DOI: 10.1117/1.jei.23.1.013016
[15]	NUUTINEN M, VIRTANEN T, VAAHTERANOKSA M , et al. CVD2014: A Database for Evaluating No-Reference Video Quality Assessment Algorithms[J]. IEEE Transactions on Image Processing, 2016,25(7):3073-3086. DOI: 10.1109/tip.2016.2562513
[16]	GHADIYARAM D, PAN J, BOVIK A C , et al. In-Capture Mobile Video Distortions: A Study of Subjective Behavior and Objective Algorithms[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018,28(9):2061-2077. DOI: 10.1109/tcsvt.2017.2707479
[17]	HOSU V, HAHN F, JENADELEH M , et al. The Konstanz Natural Video Database (KoNViD-1k) [C]//Ninth International Conference on Quality of Multimedia Experience (QoMEX). Erfurt, Germany, 2017: 1-6. DOI: 10.1109/QoMEX.2017.7965673
[18]	SINNO Z, BOVIK A C . Large Scale Study of Perceptual Video Quality [EB/OL]. ( 2018-03-05)[2018-05-27]. https://arxiv.org/abs/1803.01761
[19]	WINKLER S . Image and Video Quality Resources [EB/OL]. [ 2018- 05- 27].
[20]	WINKLER S . Analysis of Public Image and Video Databases for Quality Assessment[J]. IEEE Journal of Selected Topics in Signal Processing, 2012,6(6):616-625. DOI: 10.1109/jstsp.2012.2215007 DOI URL
[21]	WANG Z, BOVIK A C, SHEIKH H R , et al. Image Quality Assessment: From Error Visibility to Structural Similarity[J]. IEEE Transactions on Image Processing, 2004,13(4):600-612. DOI: 10.1109/tip.2003.819861
[22]	WANG Z, LU L G, BOVIK A C . Video Quality Assessment Based on Structural Distortion Measurement[J]. Signal Processing: Image Communication, 2004,19(2):121-132. DOI: 10.1016/s0923-5965(03)00076-6
[23]	WANG Z, LI Q . Video Quality Assessment Using a Statistical Model of Human Visual Speed Perception[J]. Journal of the Optical Society of America A, 2007,24(12):B61. DOI: 10.1364/josaa.24.000b61
[24]	MOORTHY A K, BOVIK A C . Efficient Video Quality Assessment along Temporal Trajectories[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2010,20(11):1653-1658. DOI: 10.1109/tcsvt.2010.2087470
[25]	SESHADRINATHAN K, BOVIK A C . Temporal Hysteresis Model of Time Varying Subjective Video Quality [C]//IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Prague, Czech Republic, 2011: 1153-1156. DOI: 10.1109/ICASSP.2011.5946613
[26]	WANG Y, JIANG T T, MA S W , et al. Novel Spatio-Temporal Structural Information Based Video Quality Metric[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(7):989-998. DOI: 10.1109/tcsvt.2012. 2186745 DOI URL
[27]	XU Q Q, WU Z P, SU L , et al. Bridging the Gap between Objective Score and Subjective Preference in Video Quality Assessment [C]//IEEE International Conference on Multimedia and Expo. Suntec City, Singapore, 2010: 908-913. DOI: 10.1109/ICME.2010.5583853
[28]	PARK J, SESHADRINATHAN K, LEE S , et al. Video Quality Pooling Adaptive to Perceptual Distortion Severity[J]. IEEE Transactions on Image Processing, 2013,22(2):610-620. DOI: 10.1109/tip.2012.2219551 DOI URL
[29]	CHANDLER D M . Most Apparent Distortion: Full-Reference Image Quality Assessment and the Role of Strategy[J]. Journal of Electronic Imaging, 2010,19(1):011006. DOI: 10.1117/1.3267105
[30]	VU P V, VU C T, CHANDLER D M . A Spatiotemporal Most-Apparent-Distortion Model for Video Quality Assessment [C]//18th IEEE International Conference on Image Processing. Brussels, Belgium, 2011: 2505-2508. DOI: 10.1109/ICIP.2011.6116171
[31]	MANASA K, CHANNAPPAYYA S S . An Optical Flow-Based Full Reference Video Quality Assessment Algorithm[J]. IEEE Transactions on Image Processing, 2016,25(6):2480-2492. DOI: 10.1109/tip.2016.2548247
[32]	XUE W F, ZHANG L, MOU X Q , et al. Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index[J]. IEEE Transactions on Image Processing, 2014,23(2):684-695. DOI: 10.1109/tip.2013.2293423 DOI URL
[33]	YAN P, MOU X Q . Video Quality Assessment Based on Motion Structure Partition Similarity of Spatiotemporal Slice Images[J]. Journal of Electronic Imaging, 2018,27(3):1. DOI: 10.1117/1.jei.27.3.033019
[34]	AYDIN T O, CADÍK M, MYSZKOWSKI K , et al. Video Quality Assessment for Computer Graphics Applications[J]. ACM Transactions on Graphics, 2010,29(6):1. DOI: 10.1145/1882261.1866187
[35]	ZHANG F, BULL D R . A Perception-Based Hybrid Model for Video Quality Assessment[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016,26(6):1017-1028. DOI: 10.1109/tcsvt.2015.2428551
[36]	YOU J Y, EBRAHIMI T, PERKIS A . Attention Driven Foveated Video Quality Assessment[J]. IEEE Transactions on Image Processing, 2014,23(1):200-213. DOI: 10.1109/tip.2013.2287611 DOI URL
[37]	PENG P, LIAO D P, LI Z N . An Efficient Temporal Distortion Measure of Videos Based on Spacetime Texture[J]. Pattern Recognition, 2017,70:1-11. DOI: 10.1016/j.patcog.2017.04.031
[38]	ZHANG W, LIU H T . Study of Saliency in Objective Video Quality Assessment[J]. IEEE Transactions on Image Processing, 2017,26(3):1275-1288. DOI: 10.1109/tip.2017.2651410
[39]	HE L H, LU W, JIA C C , et al. Video Quality Assessment by Compact Representation of Energy in 3D-DCT Domain[J]. Neurocomputing, 2017,269:108-116. DOI: 10.1016/j.neucom.2016.08.143
[40]	FREITAS P G, AKAMINE W Y L, FARIAS M C Q . Using Multiple Spatio-Temporal Features to Estimate Video Quality[J]. Signal Processing: Image Communication, 2018,64:1-10. DOI: 10.1016/j.image.2018.02.010
[41]	LI Z, AARON A, KATSAVOUNIDIS I, MOORTHY A, MANOHARA M . Toward A Practical Perceptual Video Quality Metric [EB/OL]. ( 2016-06) [2018-05-27]. https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652
[42]	BAMPIS C G, LI Z, BOVIK A C . SpatioTemporal Feature Integration and Model Fusion for Full Reference Video Quality Assessment[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018: 1. DOI: 10.1109/tcsvt.2018.2868262
[43]	PINSON M H, WOLF S . A New Standardized Method for Objectively Measuring Video Quality[J]. IEEE Transactions on Broadcasting, 2004,50(3):312-322. DOI: 10.1109/tbc.2004.834028
[44]	MASRY M, HEMAMI S S, SERMADEVI Y . A Scalable Wavelet-Based Video Distortion Metric and Applications[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2006,16(2):260-273. DOI: 10.1109/tcsvt.2005. 861946
[45]	GUNAWAN I P, GHANBARI M . Reduced-Reference Video Quality Assessment Using Discriminative Local Harmonic Strength with Motion Consideration[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2008,18(1):71-83. DOI: 10.1109/tcsvt.2007.913755
[46]	ZENG K, WANG Z . Temporal Motion Smoothness Measurement for Reduced-Reference Video Quality Assessment [C]//IEEE International Conference on Acoustics, Speech and Signal Processing. Dallas, USA, 2010: 1010-1013. DOI: 10.1109/ICASSP.2010.5495316
[47]	WANG M M, ZHANG F, AGRAFIOTIS D . A very Low Complexity Reduced Reference Video Quality Metric Based on Spatio-Temporal Information Selection [C]//IEEE International Conference on Image Processing (ICIP). Quebec City, Canada, 2015: 571-575. DOI: 10.1109/ICIP.2015.7350863
[48]	LE CALLET P, VIARD-GAUDIN C, BARBA D . A Convolutional Neural Network Approach for Objective Video Quality Assessment[J]. IEEE Transactions on Neural Networks, 2006,17(5):1316-1327. DOI: 10.1109/tnn.2006.879766
[49]	ZHU K F, BARKOWSKY M, SHEN M M , et al. Optimizing Feature Pooling and Prediction Models of VQA Algorithms [C]//IEEE International Conference on Image Processing (ICIP). Paris, France, 2014: 541-545. DOI: 10.1109/ICIP.2014.7025108
[50]	MITTAL A, SOUNDARARAJAN R, BOVIK A C . Making a “Completely Blind” Image Quality Analyzer[J]. IEEE Signal Processing Letters, 2013,20(3):209-212. DOI: 10.1109/lsp.2012.2227726 DOI URL
[51]	MA L, LI S N, NGAN K N . Reduced-Reference Video Quality Assessment of Compressed Video Sequences[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(10):1441-1456. DOI: 10.1109/tcsvt.2012. 2202049 DOI URL
[52]	BAMPIS C G, GUPTA P, SOUNDARARAJAN R , et al. SpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality[J]. IEEE Signal Processing Letters, 2017,24(9):1333-1337. DOI: 10.1109/lsp.2017.2726542
[53]	ZHU K F, LI C Q, ASARI V , et al. No-Reference Video Quality Assessment Based on Artifact Measurement and Statistical Analysis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015,25(4):533-546. DOI: 10.1109/tcsvt.2014.2363737
[54]	ZHANG F, LIN W S, CHEN Z B , et al. Additive Log-Logistic Model for Networked Video Quality Assessment[J]. IEEE Transactions on Image Processing, 2013,22(4):1536-1547. DOI: 10.1109/tip.2012.2233486
[55]	ZHAO T S, LIU Q, CHEN C W . QoE in Video Transmission: A User Experience-Driven Strategy[J]. IEEE Communications Surveys & Tutorials, 2017,19(1):285-302. DOI: 10.1109/comst.2016.2619982
[56]	ROMANIAK P, JANOWSKI L, LESZCZUK M, et al. A no Reference Metric for the Quality Assessment of Videos Affected by Exposure Distortion [C]//IEEE International Conference on Multimedia and Expo. Barcelona, Spain, 2011: 1-6. DOI: 10.1109/ICME.2011.6011903
[57]	VALENZISE G, MAGNI S, TAGLIASACCHI M , et al. No-Reference Pixel Video Quality Monitoring of Channel-Induced Distortion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012,22(4):605-618. DOI: 10.1109/tcsvt.2011.2171211 DOI URL
[58]	CUI Z X, JIANG T T. No-Reference Video Shakiness Quality Assessment [M] //CUI Z X, JIANG T T. eds. Computer Vision-ACCV 2016. Cham: Springer International Publishing, 2017: 396-411. DOI: 10.1007/978-3-319-54193-8_25
[59]	CHEN C, IZADI M, KOKARAM A . A Perceptual Quality Metric for Videos Distorted by Spatially Correlated Noise [C]//ACM on Multimedia Conference. Amsterdam, The Netherlands, 2016: 1277-1285. DOI: 10.1145/2964284.2964302
[60]	GHADIYARAM D, CHEN C, INGUVA S , et al. A No-Reference Video Quality Predictor for Compression and Scaling Artifacts [C]//IEEE International Conference on Image Processing (ICIP). Beijing, China, 2017: 3445-3449. DOI: 10.1109/ICIP.2017.8296922
[61]	XU J T, YE P, LIU Y , et al. No-Reference Video Quality Assessment via Feature Learning [C]//IEEE International Conference on Image Processing (ICIP). Paris, France, 2014: 491-495. DOI: 10.1109/ICIP.2014.7025098
[62]	YE P, KUMAR J, KANG L , et al. Unsupervised Feature Learning Framework for No-Reference Image Quality Assessment [C]//IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA, 2012: 1098-1105. DOI: 10.1109/CVPR.2012.6247789
[63]	MEN H, LIN H H, SAUPE D . Empirical Evaluation of No-Reference VQA Methods on a Natural Video Quality Database [C]//Ninth International Conference on Quality of Multimedia Experience (QoMEX). Erfurt, Germany, 2017: 1-3. DOI: 10.1109/QoMEX.2017.7965644
[64]	NARVEKAR N D, KARAM L J . A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD)[J]. IEEE Transactions on Image Processing, 2011,20(9):2678-2683. DOI: 10.1109/tip.2011.2131660 DOI URL
[65]	SAAD M A, BOVIK A C, CHARRIER C . Blind Prediction of Natural Video Quality[J]. IEEE Transactions on Image Processing, 2014,23(3):1352-1365. DOI: 10.1109/tip.2014.2299154 DOI URL
[66]	MANASA K, CHANNAPPAYYA S S . An Optical Flow-Based No-Reference Video Quality Assessment Algorithm [C]//IEEE International Conference on Image Processing (ICIP). Phoenix, USA, 2016: 2400-2404. DOI: 10.1109/ICIP.2016.7532789
[67]	ZHU Y, WANG Y F, SHUAI Y . Blind Video Quality Assessment Based on Spatio-Temporal Internal Generative Mechanism [C]//IEEE International Conference on Image Processing (ICIP). Beijing, China, 2017: 305-309. DOI: 10.1109/ICIP.2017.8296292
[68]	LI X L, GUO Q, LU X Q . Spatiotemporal Statistics for Video Quality Assessment[J]. IEEE Transactions on Image Processing, 2016,25(7):3329-3342. DOI: 10.1109/tip.2016.2568752
[69]	LI Y M, PO L M, CHEUNG C H , et al. No-Reference Video Quality Assessment with 3D Shearlet Transform and Convolutional Neural Networks[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016,26(6):1044-1057. DOI: 10.1109/tcsvt.2015.2430711
[70]	SHABEER P M, BHATI S, CHANNAPPAYYA S S . Modeling Sparse Spatio-Temporal Representations for No-Reference Video Quality Assessment [C]//IEEE Global Conference on Signal and Information Processing (GlobalSIP). Montreal, Canada, 2017: 1220-1224. DOI: 10.1109/GlobalSIP.2017.8309155
[71]	ZHU Y, GUNTUKU S C, LIN W, GHINEA G, REDI J A , Measuring Individual Video QoE: A Survey, and Proposal for Future Directions Using Social Media[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018,14(2s):30. DOI: 10.1145/3183512
[72]	CHEN Z B, ZHOU W, LI W P . Blind Stereoscopic Video Quality Assessment: From Depth Perception to Overall Experience[J]. IEEE Transactions on Image Processing, 2018,27(2):721-734. DOI: 10.1109/tip.2017.2766780
[73]	NARWARIA M, PERREIRA DA SILVA M, LE CALLET P . HDR-VQM: An Objective Quality Measure for High Dynamic Range Video[J]. Signal Processing: Image Communication, 2015,35:46-60. DOI: 10.1016/j.image.2015. 04.009
[74]	ZHANG Y X, WANG Y B, LIU F Y , et al. Subjective Panoramic Video Quality Assessment Database for Coding Applications[J]. IEEE Transactions on Broadcasting, 2018,64(2):461-473. DOI: 10.1109/tbc.2018.2811627
[75]	BAI C, REIBMAN A R . Image Quality Assessment in First-Person Videos[J]. Journal of Visual Communication and Image Representation, 2018,54:123-132. DOI: 10.1016/j.jvcir.2018.05.005
[76]	LING S Y, LE CALLET P . Image Quality Assessment for Free Viewpoint Video Based on Mid-Level Contours Feature [C]//IEEE International Conference on Multimedia and Expo (ICME). Hong Kong, China, 2017: 79-84. DOI: 10.1109/ICME.2017.8019431