Perceptual Quality Assessment of Omnidirectional Images: Subjective Experiment and Objective Model Evaluation

doi:10.12142/ZTECOM.201901007

ZTE Communications ›› 2019, Vol. 17 ›› Issue (1): 38-47.DOI: 10.12142/ZTECOM.201901007

• Special Topic • Previous Articles Next Articles

Perceptual Quality Assessment of Omnidirectional Images: Subjective Experiment and Objective Model Evaluation

DUAN Huiyu, ZHAI Guangtao, MIN Xiongkuo, ZHU Yucheng, FANG Yi, YANG Xiaokang

Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

Received:2018-06-09 Online:2019-02-20 Published:2019-11-14
About author:DUAN Huiyu (huiyuduan@sjtu.edu.cn) received the B.E. degree from the University of Electronic Science and Technology of China in 2017. He is currently pursuing the Ph.D. degree at the Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, China. His research interests include image quality assessment, visual attention modeling, and perceptual signal processing.|ZHAI Guangtao received the B.E. and M.E. degrees from Shandong University, China in 2001 and 2004, respectively, and the Ph.D. degree from Shanghai Jiao Tong University, China in 2009. From 2008 to 2009, he was a visiting student with the Department of Electrical and Computer Engineering, McMaster University, Hamilton, Canada, where he was a post-doctoral fellow from 2010 to 2012. From 2012 to 2013, he was a Humboldt Research Fellow with the Institute of Multimedia Communication and Signal Processing, Friedrich Alexander University of Erlangen-Nuremberg, Germany. He is currently a Research Professor with the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University. His research interests include multimedia signal processing and perceptual signal processing. He received the National Excellent Ph.D. Thesis Award from the Ministry of Education of China in 2012.|MIN Xiongkuo received the B.E. degree from Wuhan University, China in 2013, and the Ph.D. degree from Shanghai Jiao Tong University, China in 2018. From 2016 to 2017, he was a visiting student with the Department of Electrical and Computer Engineering, University of Waterloo, Canada. He is currently a post-doctoral fellow with Shanghai Jiao Tong University. His research interests include image quality assessment, visual attention modeling, and perceptual signal processing. He received the Best Student Paper Award from the IEEE ICME 2016.|ZHU Yucheng received the B.E. degree from the Shanghai Jiao Tong University, China in 2015. He is currently pursuing the Ph.D. degree at the Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University. His research interests include image quality assessment, visual attention modeling, and perceptual signal processing. He received Grand Challenge Best Performance Awards in ICME 2017 and 2018.|FANG Yi is an undergraduate student at Shanghai Jiao Tong University, China and will receive the B.E. degree in 2019. She will pursue the M.E. degree at the Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University. Her research interests include image quality assessment, visual attention modeling, and perceptual signal processing.|YANG Xiaokang received the B.S. degree from Xiamen University, China in 1994, the M.S. degree from the Chinese Academy of Sciences, China in 1997, and the Ph.D. degree from Shanghai Jiao Tong University, China in 2000. From 2000 to 2002, he was a Research Fellow with the Centre for Signal Processing, Nanyang Technological University, Singapore. From 2002 to 2004, he was a Research Scientist with the Institute for Infocomm Research, Singapore. From 2007 to 2008, he visited the Institute for Computer Science, University of Freiburg, Freiburg im Breisgau, Germany, as an Alexander von Humboldt Research Fellow. He is currently a Distinguished Professor with the School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, where he is also the Deputy Director of the Institute of Image Communication and Information Processing. His current research interests include image processing and communication, computer vision, and machine learning.

Abstract

Abstract:

Virtual reality (VR) environment can provide immersive experience to viewers. Under the VR environment, providing a good quality of experience is extremely important. Therefore, in this paper, we present an image quality assessment (IQA) study on omnidirectional images. We first build an omnidirectional IQA (OIQA) database, including 16 source images with their corresponding 320 distorted images. We add four commonly encountered distortions. These distortions are JPEG compression, JPEG2000 compression, Gaussian blur, and Gaussian noise. Then we conduct a subjective quality evaluation study in the VR environment based on the OIQA database. Considering that visual attention is more important in VR environment, head and eye movement data are also tracked and collected during the quality rating experiments. The 16 raw and their corresponding distorted images, subjective quality assessment scores, and the head-orientation data and eye-gaze data together constitute the OIQA database. Based on the OIQA database, we test some state-of-the-art full-reference IQA (FR-IQA) measures on equirectangular format or cubic format omnidirectional images. The results show that applying FR-IQA metrics on cubic format omnidirectional images could improve their performance. The performance of some FR-IQA metrics combining the saliency weight of three different types are also tested based on our database. Some new phenomena different from traditional IQA are observed.

Key words: perceptual quality assessment, omnidirectional images, subjective experiment, objective model evaluation, visual saliency

DUAN Huiyu, ZHAI Guangtao, MIN Xiongkuo, ZHU Yucheng, FANG Yi, YANG Xiaokang. Perceptual Quality Assessment of Omnidirectional Images: Subjective Experiment and Objective Model Evaluation[J]. ZTE Communications, 2019, 17(1): 38-47.

Figures/Tables 11

References 34

[1]	SHEIKH H R, WANG Z, CORMACK L , et al. Live Image Quality Assessment Database Release 2 [EB/OL]. (2005)[2018]. http://live.ece.utexas.edu/research/quality
[2]	PONOMARENKO N, LUKIN V, ZELENSKY A . Tid2008-A Database for Evaluation of Full-Reference Visual Quality Assessment Metrics[J]. Advances of Modern Radioelectronics, 2009,10(4):30-45
[3]	LARSON E C, CHANDLER D . Categorical Image Quality (CSIQ) Database [EB/OL]. (2010)[2018]. http://vision.okstate.edu/csiq
[4]	GU K, LIU M, ZHAI G T , et al. Quality Assessment Considering Viewing Distance and Image Resolution[J]. IEEE Transactions on Broadcasting, 2015,61(3):520-531. DOI: 10.1109/tbc.2015.2459851
[5]	RAI Y, LE CALLET P, GUILLOTEL P . Which Saliency Weighting for Omni Directional Image Quality Assessment? [C]//Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany, 2017: 1-6. DOI: 10.1109/QoMEX.2017.7965659
[6]	UPENIK E, RERÁBEK M, EBRAHIMI T . Testbed for Subjective Evaluation of Omnidirectional Visual Content [C]//2016 Picture Coding Symposium (PCS), Nuremberg, Germany, 2016: 1-5. DOI: 10.1109/PCS.2016.7906378
[7]	YU M, LAKSHMAN H, GIROD B . A Framework to Evaluate Omnidirectional Video Coding Schemes [C]//IEEE International Symposium on Mixed and Augmented Reality, Fukuoka, Japan, 2015: 31-36. DOI: 10.1109/ISMAR.2015.12
[8]	SUN W, GU K, ZHAI G T , et al. CVIQD: Subjective Quality Evaluation of Compressed Virtual Reality Images [C]//IEEE International Conference on Image Processing (ICIP), Beijing, China, 2017: 3450-3454. DOI: 10.1109/ICIP.2017. 8296923
[9]	DUAN H Y, ZHAI G T, YANG X K , et al. IVQAD 2017: An Immersive Video Quality Assessment Database [C]//International Conference on Systems, Signals and Image Processing (IWSSIP), Poznan, Poland, 2017: 1-5. DOI: 10.1109/IWSSIP.2017.7965610
[10]	DUAN H Y, ZHAI G T, MIN X K , et al. Perceptual Quality Assessment of Omnidirectional Images [C]//IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 2018: 1-5. DOI: 10.1109/ISCAS.2018.8351786
[11]	ZHAI G T, CAI J F, LIN W S , et al. Cross-Dimensional Perceptual Quality Assessment for Low Bit-Rate Videos[J]. IEEE Transactions on Multimedia, 2008,10(7):1316-1324. DOI: 10.1109/tmm.2008.2004910
[12]	ZHAI G T, WU X L, YANG X K , et al. A Psychovisual Quality Metric in Free-Energy Principle[J]. IEEE Transactions on Image Processing, 2012,21(1):41-52. DOI: 10.1109/tip.2011.2161092 DOI URL
[13]	GU K, ZHAI G T, YANG X K , et al. Using Free Energy Principle for Blind Image Quality Assessment[J]. IEEE Transactions on Multimedia, 2015,17(1):50-63. DOI: 10.1109/tmm.2014.2373812
[14]	GU K, ZHAI G T, LIN W S , et al. No-Reference Image Sharpness Assessment in Autoregressive Parameter Space[J]. IEEE Transactions on Image Processing, 2015,24(10):3218-3231. DOI: 10.1109/tip.2015.2439035
[15]	ZHU W, ZHAI G, SUN W , et al. On the Impact of Environmental Sound on Perceived Visual Quality [C]//Pacific Rim Conference on Multimedia (PCM), Harbin, China, 2017: 723-734
[16]	XU M, LI C, LIU Y F , et al. A Subjective Visual Quality Assessment Method of Panoramic Videos [C]//IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China, 2017: 517-522. DOI: 10.1109/ICME.2017.8019351
[17]	WALLACE G K . The JPEG still Picture Compression Standard [J]. IEEE Transactions on Consumer Electronics, 1992, 38(1): xviii-xxxiv. DOI: 10.1109/30.125072
[18]	SKODRAS A, CHRISTOPOULOS C, EBRAHIMI T . The JPEG 2000 still Image Compression Standard[J]. IEEE Signal Processing Magazine, 2001,18(5):36-58. DOI: 10.1109/79.952804
[19]	7invensun. Aglass [EB/OL]. (2017)[2018]. http://www.aglass.com
[20]	DUAN H Y, ZHAI G T, MIN X K , et al. Assessment of Visually Induced Motion Sickness in Immersive Videos [C]//18th Pacific-Rim Conference on Multimedia (PCM), Harbin, China, 2017, pp. 662-672. DOI: 10.1007/978-3-319-77380-3_63
[21]	RAI Y, CALLET P L . A dataset of head and eye movements for 360 degree images [C]//ACM on Multimedia Systems Conference, Taipei, China, 2017: 205-210. DOI: 10.1145/3083187.3083218
[22]	CURCIO C A, SLOAN K R, KALINA R E , et al. Human Photoreceptor Topography[J]. The Journal of Comparative Neurology, 1990,292(4):497-523. DOI: 10.1002/cne.902920402
[23]	ENGELKE U, ZHANG W, CALLET P L . Perceived Interest Versus Overt Visual Attention in Image Quality Assessment[J]. Proceedings of SPIE, vol. 9394, 2015. DOI: 10.1117/12.2086371
[24]	ZHANG L, ZHANG L, MOU X Q , et al. FSIM: A Feature Similarity Index for Image Quality Assessment[J]. IEEE Transactions on Image Processing, 2011,20(8):2378-2386. DOI: 10.1109/tip.2011.2109730 DOI URL
[25]	XUE W F, ZHANG L, MOU X Q , et al. Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index[J]. IEEE Transactions on Image Processing, 2014,23(2):684-695. DOI: 10.1109/tip.2013.2293423 DOI URL
[26]	LIU A M, LIN W S, NARWARIA M . Image Quality Assessment Based on Gradient Similarity[J]. IEEE Transactions on Image Processing, 2012,21(4):1500-1512. DOI: 10.1109/tip.2011.2175935 DOI URL
[27]	WANG Z, LI Q . Information Content Weighting for Perceptual Image Quality Assessment[J]. IEEE Transactions on Image Processing, 2011,20(5):1185-1198. DOI: 10.1109/tip.2010.2092435 DOI URL
[28]	WANG Z, SIMONCELLI E P, BOVIK A C . Multiscale Structural Similarity for Image Quality Assessment [C]//Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, USA, 2003: 1398-1402. DOI: 10.1109/ACSSC.2003.1292216
[29]	WANG Z, BOVIK A C, SHEIKH H R , et al. Image Quality Assessment: From Error Visibility to Structural Similarity[J]. IEEE Transactions on Image Processing, 2004,13(4):600-612. DOI: 10.1109/tip.2003.819861
[30]	SHEIKH H R, BOVIK A C . Image Information and Visual Quality[J]. IEEE Transactions on Image Processing, 2006,15(2):430-444. DOI: 10.1109/tip.2005.859378
[31]	ZHANG L, SHEN Y, LI H Y . VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment[J]. IEEE Transactions on Image Processing, 2014,23(10):4270-4281. DOI: 10.1109/tip.2014.2346028 DOI URL
[32]	MIN X K, GU K, ZHAI G T , et al. Blind Quality Assessment Based on Pseudo-Reference Image[J]. IEEE Transactions on Multimedia, 2018,20(8):2049-2062. DOI: 10.1109/tmm.2017.2788206
[33]	MIN X K, GU K, ZHAI G T , et al. Saliency-Induced Reduced-Reference Quality Index for Natural Scene and Screen Content Images[J]. Signal Processing, 2018,145:127-136. DOI: 10.1016/j.sigpro.2017.10.025
[34]	MIN X K, MA K D, GU K , et al. Unified Blind Quality Assessment of Compressed Natural, Graphic, and Screen Content Images[J]. IEEE Transactions on Image Processing, 2017,26(11):5462-5474. DOI: 10.1109/tip.2017. 2735192

Parameter	α₁	β₁	γ₁	α₂	β₂	γ₂	α₃	β₃	γ₃
Fitting value (head)	-0.2404	72.75	7.53	0.7957	81.19	12.53	0.1353	77.78	40.94
Fitting value (eye)	0.4943	92.75	6.05	0.2622	91.03	13.53	0.1288	81.26	48.26

Parameter	α₁	β₁	γ₁	α₂	β₂	γ₂	α₃	β₃	γ₃
Fitting value (head)	-0.2404	72.75	7.53	0.7957	81.19	12.53	0.1353	77.78	40.94
Fitting value (eye)	0.4943	92.75	6.05	0.2622	91.03	13.53	0.1288	81.26	48.26

a) Assessing the perceptual quality of omnidirectional images on equirectangular format images
Metrics	PLCC	SRCC	RMSE
FSIMc	0.9188	0.9140	5.6800
GMSD	0.7412	0.7378	9.6574
GMSM	0.6768	0.6642	10.590
GSI	0.9008	0.8924	6.2473
IW-MSE	0.6207	0.7328	11.280
IW-PSNR	0.7371	0.7328	9.7223
IW-SSIM	0.7805	0.7766	8.9934
MSE	0.3279	0.4971	13.590
MS-SSIM	0.6745	0.6653	10.621
PSNR	0.5060	0.4971	12.408
SSIM1	0.5271	0.3479	12.225
SSIM2	0.8888	0.8800	6.5917
VIF	0.7878	0.7867	8.8614
VIFp	0.7555	0.7501	9.4246
VSI	0.9087	0.9055	6.0059
b) Assessing the perceptual quality of omnidirectional images on cubic format images
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9316	0.9278		5.2273
GMSD	0.7120	0.7042		10.101
GMSM	0.6448	0.6393		10.996
GSI	0.9215	0.9162		5.5878
IW-MSE	0.6165	0.7110		11.327
IW-PSNR	0.7179	0.7054		10.014
IW-SSIM	0.7799	0.7755		9.0045
MSE	0.3919	0.5693		13.235
MS-SSIM	0.6699	0.6651		10.681
PSNR	0.5621	0.5603		11.898
SSIM1	0.4462	0.3870		12.875
SSIM2	0.8843	0.8740		6.7175
VIF	0.7725	0.7716		9.1351
VIFp	0.7761	0.7699		9.0723
VSI	0.9236	0.9192		5.5158
c) Assessing the perceptual quality of omnidirectional images combining head movement direction information (Fig. 6a)
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9118	0.9049		5.9061
GMSM	0.6530	0.7035		10.895
MSE	0.3420	0.5026		13.518
PSNR	0.4123	0.3958		13.106
SSIM1	0.4481	0.3663		12.861
SSIM2	0.8967	0.8844		6.3664
VSI	0.9009	0.8946		6.2451
d) Assessing the perceptual quality of omnidirectional images combining eye viewing direction information (Fig. 6b)
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9148	0.9078		5.8090
GMSM	0.7350	0.7283		9.7549
MSE	0.3407	0.5175		13.525
PSNR	0.4364	0.4154		12.943
SSIM1	0.4665	0.3849		12.725
SSIM2	0.8988	0.8849		6.3075
VSI	0.9064	0.9005		6.0783
e) Assessing the perceptual quality of omnidirectional images combining original saliency map from subjects
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9113	0.9015		5.9245
GMSM	0.7508	0.7449		9.5026
MSE	0.3475	0.5317		13.489
PSNR	0.4858	0.4534		12.574
SSIM1	0.5083	0.4077		12.388
SSIM2	0.8927	0.8779		6.4817
VSI	0.9086	0.9027		6.0071

a) Assessing the perceptual quality of omnidirectional images on equirectangular format images
Metrics	PLCC	SRCC	RMSE
FSIMc	0.9188	0.9140	5.6800
GMSD	0.7412	0.7378	9.6574
GMSM	0.6768	0.6642	10.590
GSI	0.9008	0.8924	6.2473
IW-MSE	0.6207	0.7328	11.280
IW-PSNR	0.7371	0.7328	9.7223
IW-SSIM	0.7805	0.7766	8.9934
MSE	0.3279	0.4971	13.590
MS-SSIM	0.6745	0.6653	10.621
PSNR	0.5060	0.4971	12.408
SSIM1	0.5271	0.3479	12.225
SSIM2	0.8888	0.8800	6.5917
VIF	0.7878	0.7867	8.8614
VIFp	0.7555	0.7501	9.4246
VSI	0.9087	0.9055	6.0059
b) Assessing the perceptual quality of omnidirectional images on cubic format images
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9316	0.9278		5.2273
GMSD	0.7120	0.7042		10.101
GMSM	0.6448	0.6393		10.996
GSI	0.9215	0.9162		5.5878
IW-MSE	0.6165	0.7110		11.327
IW-PSNR	0.7179	0.7054		10.014
IW-SSIM	0.7799	0.7755		9.0045
MSE	0.3919	0.5693		13.235
MS-SSIM	0.6699	0.6651		10.681
PSNR	0.5621	0.5603		11.898
SSIM1	0.4462	0.3870		12.875
SSIM2	0.8843	0.8740		6.7175
VIF	0.7725	0.7716		9.1351
VIFp	0.7761	0.7699		9.0723
VSI	0.9236	0.9192		5.5158
c) Assessing the perceptual quality of omnidirectional images combining head movement direction information (Fig. 6a)
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9118	0.9049		5.9061
GMSM	0.6530	0.7035		10.895
MSE	0.3420	0.5026		13.518
PSNR	0.4123	0.3958		13.106
SSIM1	0.4481	0.3663		12.861
SSIM2	0.8967	0.8844		6.3664
VSI	0.9009	0.8946		6.2451
d) Assessing the perceptual quality of omnidirectional images combining eye viewing direction information (Fig. 6b)
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9148	0.9078		5.8090
GMSM	0.7350	0.7283		9.7549
MSE	0.3407	0.5175		13.525
PSNR	0.4364	0.4154		12.943
SSIM1	0.4665	0.3849		12.725
SSIM2	0.8988	0.8849		6.3075
VSI	0.9064	0.9005		6.0783
e) Assessing the perceptual quality of omnidirectional images combining original saliency map from subjects
Metrics	PLCC	SRCC		RMSE
FSIMc	0.9113	0.9015		5.9245
GMSM	0.7508	0.7449		9.5026
MSE	0.3475	0.5317		13.489
PSNR	0.4858	0.4534		12.574
SSIM1	0.5083	0.4077		12.388
SSIM2	0.8927	0.8779		6.4817
VSI	0.9086	0.9027		6.0071

Perceptual Quality Assessment of Omnidirectional Images: Subjective Experiment and Objective Model Evaluation

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 34

Related Articles 0

Recommended Articles

Metrics