Video Enhancement Network Based on CNN and Transformer

doi:10.12142/ZTECOM.202404011

ZTE Communications ›› 2024, Vol. 22 ›› Issue (4): 78-88.DOI: 10.12142/ZTECOM.202404011

• Research Papers • Previous Articles Next Articles

Video Enhancement Network Based on CNN and Transformer

YUAN Lang¹(), HUI Chen¹, WU Yanfeng¹, LIAO Ronghua¹, JIANG Feng², GAO Ying³

^1.Harbin Institute of Technology, Harbin 150001, China
^2.Sichuan University of Science & Engineering, Zigong 643002, China
^3.ZTE Corporation, Shenzhen 518057, China

Received:2023-07-29 Online:2024-12-20 Published:2024-12-03
About author:YUAN Lang (1190201114@stu.hit.edu.cn) is now a senior student at Harbin Institute of Technology (HIT), China. He is working at the Research Center of Intelligent Interface and Human-Computer Interaction, HIT. His research interests include deep learning, image coding, and compressive sensing.
HUI Chen received his BS degree in software engineering from Yanshan University, China in 2017, MS degree in software engineering, and PhD in computer science and technology from Harbin Institute of Technology, China in 2020 and 2024, respectively. He has been a visiting scholar with the College of Computing and Data Science, Nanyang Technological University, Singapore since 2023. He is currently with the School of Future Technology, Nanjing University of Information Science and Technology, China. His research interests include image compression, quality assessment, and multimedia security.
WU Yanfeng is currently an undergraduate student at Harbin Institute of Technology (HIT), China. He has joined the Research Center of Intelligent Interface and Human-Computer Interaction, HIT. His research interests include machine learning and deep learning, with a current focus on video coding and compressive sensing.
LIAO Ronghua is pursuing a master’s degree at the Department of Computing, Harbin Institute of Technology, China. His research interests include compressed sensing and video quality assessment.
JIANG Feng received his BS, MS and PhD degrees from the Harbin Institute of Technology (HIT), China in 2001, 2003, and 2008, respectively, all in computer science. He is currently a professor with the Department of Computer Science, Sichuan University of Science & Engineering, China and a visiting scholar with the School of Electrical Engineering, Princeton University, USA. His research interests include computer vision, image and video processing, and pattern recognition.
GAO Ying received her master’s degree in mathematics from Hohai University, China in 2011. Since then, she has been engaged in the research on Internet of Things (IoT), video surveillance, video transmission, and video coding. She has applied for over 100 patents in these fields. She is currently a chief engineer of standard pre-research at ZTE Corporation and a member of the State Key Laboratory of Mobile Network and Multimedia Technology in China, where she mainly focuses on the research and standardization of video coding technology.
Supported by:
the Key R&D Program of China(2022YFC3301800);Sichuan Local Technological Development Program(24YRGZN0010);ZTE Industry?University?Institute Cooperation Funds(HC?CN?03?2019?12)

Abstract

Abstract:

To enhance the video quality after encoding and decoding in video compression, a video quality enhancement framework is proposed based on local and non-local priors in this paper. Low-level features are first extracted through a single convolution layer and then processed by several conv-tran blocks (CTB) to extract high-level features, which are ultimately transformed into a residual image. The final reconstructed video frame is obtained by performing an element-wise addition of the residual image and the original lossy video frame. Experiments show that the proposed Conv-Tran Network (CTN) model effectively recovers the quality loss caused by Versatile Video Coding (VVC) and further improves VVC's performance.

Key words: attention fusion mechanism, H.266/VVC, transformer, video coding, video quality enhancement

YUAN Lang, HUI Chen, WU Yanfeng, LIAO Ronghua, JIANG Feng, GAO Ying. Video Enhancement Network Based on CNN and Transformer[J]. ZTE Communications, 2024, 22(4): 78-88.

Figures/Tables 13

Table 1 Standard testing sequences in CTC

Class	Video Sequence	Frames	Resolution	FPS	Bit Depth
Class A1	Tango2	294	3 840 $×$ 2 160	60	10
	FoodMarket4	300	3 840 $×$ 2 160	60	10
	Campfire	300	3 840 $×$ 2 160	30	10
Class A2	CatRobot	300	3 840 $×$ 2 160	60	10
	DaylightRoad2	300	3 840 $×$ 2 160	60	10
	ParkRunning3	300	3 840 $×$ 2 160	50	10
Class B	MarketPlace	600	1 920 $×$ 1 080	60	10
	RitualDance	600	1 920 $×$ 1 080	60	10
	Cactus	500	1 920 $×$ 1 080	50	8
	BasketballDrive	500	1 920 $×$ 1 080	50	8
	BQTerrace	500	1 920 $×$ 1 080	60	8
Class C	RaceHorses	300	832 $×$ 480	30	8
	BQMall	600	832 $×$ 480	60	8
	PartyScene	500	832 $×$ 480	50	8
	BasketballDrill	500	832 $×$ 480	50	8
Class D	RaceHorses	300	410 $×$ 240	30	8
	BQSquare	600	410 $×$ 240	60	8
	BlowingBubbles	500	410 $×$ 240	50	8
	BasketballPass	500	410 $×$ 240	50	8

Table 1 Standard testing sequences in CTC

Class	Video Sequence	Frames	Resolution	FPS	Bit Depth
Class A1	Tango2	294	3 840 $×$ 2 160	60	10
	FoodMarket4	300	3 840 $×$ 2 160	60	10
	Campfire	300	3 840 $×$ 2 160	30	10
Class A2	CatRobot	300	3 840 $×$ 2 160	60	10
	DaylightRoad2	300	3 840 $×$ 2 160	60	10
	ParkRunning3	300	3 840 $×$ 2 160	50	10
Class B	MarketPlace	600	1 920 $×$ 1 080	60	10
	RitualDance	600	1 920 $×$ 1 080	60	10
	Cactus	500	1 920 $×$ 1 080	50	8
	BasketballDrive	500	1 920 $×$ 1 080	50	8
	BQTerrace	500	1 920 $×$ 1 080	60	8
Class C	RaceHorses	300	832 $×$ 480	30	8
	BQMall	600	832 $×$ 480	60	8
	PartyScene	500	832 $×$ 480	50	8
	BasketballDrill	500	832 $×$ 480	50	8
Class D	RaceHorses	300	410 $×$ 240	30	8
	BQSquare	600	410 $×$ 240	60	8
	BlowingBubbles	500	410 $×$ 240	50	8
	BasketballPass	500	410 $×$ 240	50	8

Table 2 PSNR under the random access mode

		VTM11.0-NNVC				VTM11.0-NNVC with CTN-C
Class	Video Sequence	Y-PSNR	U-PSNR	V-PSNR	Decoding	Y-PSNR	U-PSNR	V-PSNR	Decoding
		/dB	/dB	/dB	Time/s	/dB	/dB	/dB	Time/s
A1	Tango2	38.86	47.51	44.89	65.64	38.88	47.63	44.83	4 181.96
	FoodMarket4	41.21	46.00	46.16	70.30	41.23	46.02	46.19	4 234.27
	Campfire	36.52	35.97	39.81	80.85	36.54	36.03	39.86	4 267.28
A2	CatRobot	38.41	40.82	41.37	65.04	38.45	40.90	41.44	4 250.03
	DaylightRoad2	36.47	44.15	41.85	69.64	36.51	44.29	41.91	4 278.40
	ParkRunning3	36.50	32.89	34.59	106.77	36.53	33.00	34.63	4 285.42
B	MarketPlace	36.86	41.96	42.80	36.46	36.89	42.16	42.93	1 270.89
	RitualDance	38.73	44.30	44.25	35.87	38.80	44.44	44.45	1 269.88
	Cactus	35.60	38.87	41.22	27.34	35.61	38.92	41.30	1 059.97
	BasketballDrive	36.31	42.06	42.23	46.94	36.33	42.16	42.39	1 078.13
	BQTerrace	34.38	40.84	43.44	30.64	34.35	40.96	43.48	1 265.19
C	BasketbalDrill	35.79	39.87	40.10	6.83	35.85	40.05	40.32	204.68
	BQMall	35.82	41.07	42.03	10.80	35.91	41.34	42.26	247.90
	PartyScene	32.59	38.04	38.89	10.98	32.71	38.34	38.98	208.78
	RaceHorses	33.75	37.51	39.50	9.57	33.79	37.69	39.65	128.08
D	BasketballPass	34.18	39.82	38.13	3.20	34.33	40.12	38.42	58.09
	BQSquare	32.64	40.59	41.68	1.91	32.98	40.76	41.92	67.57
	BlowingBubbles	32.58	37.51	38.28	2.88	32.71	37.81	38.38	58.01
	RaceHorses	33.13	37.16	38.49	1.37	33.24	37.44	38.70	34.16
Overall		35.81	40.37	41.04	35.95	35.88	40.53	41.16	1 707.83

Table 3 Improvement of CTN-C compared with VTM11.0-NNVC under the random access mode

Class	Y-PSNR	U-PSNR	V-PSNR	Y-MSIM	U-MSIM	V-MSIM	DecT
Class A1	-0.80%	-2.87%	-0.12%	-1.02%	-5.32%	-1.28%	5 415%
Class A2	-1.54%	-6.75%	-2.47%	-1.45%	-5.30%	-1.35%	5 268%
Class B	-0.27%	-6.22%	-4.84%	-1.21%	-5.91%	-4.31%	3 137%
Class C	-1.91%	-8.13%	-5.53%	-1.40%	-6.47%	-3.67%	1 849%
Class D	-4.09%	-8.58%	-6.39%	-1.99%	-6.53%	-3.91%	2 220%
Overall	-1.70%	-6.68%	-4.19%	-1.42%	-5.97%	-3.15%	3 087%

Table 4 BD rate (PSNR) of CTN-C on VTM11.0-NNVC under the random access mode

Class	Video Sequence	Low QP (22–37)			High QP (27–42)
Class	Video Sequence	Y	U	V	Y	U	V
A1	Tango2	-0.78%	-6.16%	2.14%	-1.04%	-7.10%	2.35%
	FoodMarket4	-0.62%	-0.21%	-0.73%	-0.87%	-0.56%	-0.82%
	Campfire	-0.48%	-1.16%	-1.46%	-1.07%	-2.00%	-2.27%
A2	CatRobot	-1.75%	-6.47%	-3.11%	-1.98%	-7.69%	-3.81%
	DaylightRoad2	-2.41%	-7.77%	-2.61%	-2.13%	-9.33%	-1.95%
	ParkRunning3	-0.35%	-3.96%	-1.70%	-0.70%	-4.78%	-1.68%
B	MarketPlace	-1.05%	-9.68%	-6.17%	-1.23%	-10.11%	-7.17%
	RitualDance	-1.53%	-4.21%	-5.29%	-1.50%	-5.12%	-6.41%
	Cactus	-0.19%	-3.63%	-3.08%	-0.59%	-5.55%	-3.28%
	BasketballDrive	-0.54%	-4.98%	-5.63%	-0.90%	-4.91%	-5.62%
	BQTerrace	3.18%	-6.99%	-3.04%	1.66%	-5.73%	-2.75%
C	BasketbalDrill	-1.49%	-5.63%	-6.27%	-1.58%	-6.37%	-6.36%
	BQMall	-2.26%	-8.41%	-6.42%	-2.37%	-10.39%	-7.65%
	PartyScene	-3.22%	-9.02%	-2.67%	-2.88%	-11.47%	-3.38%
	RaceHorses	-0.57%	-6.47%	-5.76%	-1.05%	-8.04%	6.85%
D	BasketballPass	-2.97%	-8.83%	-7.96%	-3.24%	-10.70%	-8.35%
	BQSquare	-8.20%	-6.02%	-7.88%	-8.13%	-6.53%	-8.39%
	BlowingBubbles	-3.14%	-8.62%	-2.49%	-3.11%	-10.78%	-3.70%
	RaceHorses	-2.39%	-8.40%	-6.13%	-2.42v	-10.26%	-7.32%
Overall		-1.62%	-6.14%	-4.01%	-1.85%	-7.32%	-4.50%

Table 5 Gain effects of CTN-C compared with CTN-E

Class	VTM with CTN-C			VTM with CTN-E
Class	Y-PSNR	U-PSNR	V-PSNR	Y- PSNR	U- PSNR	V- PSNR
Class A1	-0.80%	-2.87%	-0.12%	-0.20%	-1.91%	-0.41%
Class A2	-1.54%	-6.75%	-2.47%	-0.74%	-3.04%	-0.40%
Class B	-0.27%	-6.22%	-4.84%	-0.11%	-2.82%	-0.36%
Class C	-1.91%	-8.13%	-5.53%	-0.59%	-3.38%	-0.96%
Class D	-4.09%	-8.58%	-6.39%	-1.88%	-3.98%	-0.99%
Overall	-1.70%	-6.68%	-4.19%	-0.64%	-3.07%	-0.63%

Table 6 Comparison of CTN-G and CTN-L with CTN-C on VTM

Class	VTM with CTN-C			VTM with CTN-G			VTM with CTN-L
Class	Y-PSNR	U-PSNR	V-PSNR	Y-PSNR	U-PSNR	V-PSNR	Y-PSNR	U-PSNR	V-PSNR
Class A1	-0.80%	-2.87%	-0.12%	-0.42%		-0.37%	-0.33%	-1.74%	-0.15%
Class A2	-1.54%	-6.75%	-2.47%	-1.13%	-3.77%	-1.75%	-0.69%	-3.58%	-0.95%
Class B	-0.27%	-6.22%	-4.84%	-0.33%	-2.79%	-2.06%	-0.18%	-2.23%	-1.36%
Class C	-1.91%	-8.13%	-5.53%	-0.92%	-3.19%	-2.24%	-0.74%	-2.93%	-1.58%
Class D	-4.09%	-8.58%	-6.39%	-1.64%	-3.53%	-3.35%	-1.13%	-2.46%	-1.87%
Overall	-1.70%	-6.68%	-4.19%	-0.88%	-2.83%	-2.25%	-0.49%	-1.17%	-1.07%

References 22

1	QuestMobile. QuestMobile's 2023 annual report on china internet core trends (condensed version) [R/OL]. (2023-12-19)[2024-03-20].
2	WIEGAND T, SULLIVAN G J, BJONTEGAARD G, et al. Overview of the H.264/AVC video coding standard [J]. IEEE transactions on circuits and systems for video technology, 2003, 13(7): 560–576. DOI: 10.1109/TCSVT.2003.815165
3	SULLIVAN G J, OHM J R, HAN W J, et al. Overview of the high efficiency video coding (HEVC) standard [J]. IEEE transactions on circuits and systems for video technology, 2012, 22(12): 1649–1668. DOI: 10.1109/TCSVT.2012.2221191
4	DONG C, DENG Y B, LOY C C, et al. Compression artifacts reduction by a deep convolutional network [C]//Proc. IEEE International Conference on Computer Vision (ICCV). IEEE, 2015: 576–584. DOI: 10.1109/ICCV.2015.73
5	WANG Z Y, LIU D, CHANG S Y, et al . D 3: deep dual-domain based fast restoration of JPEG-compressed images [C]//Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 2764–2772. DOI: 10.1109/CVPR.2016.302
6	ORORBIA A G, MALI A, WU J, et al. Learned neural iterative decoding for lossy image compression systems [C]//Proc. Data Compression Conference (DCC). IEEE, 2019: 3–12. DOI: 10.1109/DCC.2019.00008
7	DAI Y Y, LIU D, WU F. A convolutional neural network approach for post-processing in HEVC intra coding [M]//Lecture notes in computer science. Cham: Springer International Publishing, 2016: 28–39. DOI: 10.1007/978-3-319-51811-4_3
8	YANG R, XU M, WANG Z L. Decoder-side HEVC quality enhancement with scalable convolutional neural network [C]//Proc. IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2017: 817–822. DOI: 10.1109/ICME.2017.8019299
9	QI Z Y, JUNG C, LIU Y, et al. CNN-based post-processing filter for video compression with multi-scale feature representation [C]//Proc. IEEE International Conference on Visual Communications and Image Processing (VCIP). IEEE, 2022: 1–5. DOI: 10.1109/VCIP56404.2022.10008797
10	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]//Proc. 31st International Conference on Neural Information Processing Systems. NIPS, 2017: 6000–6010. DOI: 10.48550/arXiv.1706.03762
11	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale [EB/OL]. (2020-10-22)[2024-03-20].
12	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows [C]//Proc. IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021: 9992–10002. DOI: 10.1109/ICCV48922.2021.00986
13	CHEN H T, WANG Y H, GUO T Y, et al. Pre-trained image processing transformer [C]//Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021: 12294–12305. DOI: 10.1109/CVPR46437.2021.01212
14	LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer [C]//2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE, 2021. DOI: 10.1109/ICCVW54120.2021.00210
15	ZHANG K, LI Y W, LIANG J Y, et al. Practical blind image denoising via swin-conv-UNet and data synthesis [J]. Machine Intelligence Research, 2023, 20: 822–836. DOI: 10.1007/s11633-023-1466-0
16	LI X, WANG W H, HU X L, et al. Selective kernel networks [C]//Proce. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019: 510–519. DOI: 10.1109/CVPR.2019.00060
17	MA D, ZHANG F, BULL D R. BVI-DVC: a training database for deep video compression [J]. IEEE transactions on multimedia, 2022, 24: 3847–3858. DOI: 10.1109/TMM.2021.3108943
18	LIU S, SEGALL A, ALSHINA E, et al. Common test conditions and evaluation procedures for neural network-based video coding technology [Z]. JVET-W2016, 2021
19	JVET. VTM-11.0-NNVC, VVC test model [EB/OL]. [2024-03-20].
20	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity [J]. IEEE transactions on image processing, 2004, 13(4): 600–612. DOI: 10.1109/tip.2003.819861
21	CHEN L H, BAMPIS C G, LI Z, et al. Perceptual video quality prediction emphasizing chroma distortions [J]. IEEE transactions on image processing, 2021, 30: 1408–1422. DOI: 10.1109/TIP.2020.3043127
22	HEINDEL A, PRESTELE B, GEHLERT A, et al. Enhancement layer coding for chroma sub-sampled screen content video [J]. IEEE transactions on circuits and systems for video technology, 2022, 32(2): 788–801. DOI: 10.1109/TCSVT.2021.3061944

[1]	ZHANG Qiang, MEI Junjun, GUAN Tao, SUN Zhewen, ZHANG Zixiang, YU Li. Recent Advances in Video Coding for Machines Standard and Technologies [J]. ZTE Communications, 2024, 22(1): 62-76.
[2]	LI Wei, JI Jingkang, LIU Yuanlong, SUN Jiawei, LIN Subin. Simulation and Modeling of Common Mode EMI Noise in Planar Transformers [J]. ZTE Communications, 2023, 21(3): 105-116.
[3]	ZHONG Min, ZHOU Yimin, LUO Minke, ZUO Wen. Variable Bit Rate Fuzzy Control for Low Delay Video Coding [J]. ZTE Communications, 2017, 15(2): 55-65.
[4]	LIN Tao, ZHAO Liping, ZHOU Kailun. Screen Content Coding in HEVC and Beyond [J]. ZTE Communications, 2016, 14(S0): 51-58.
[5]	ZHANG Tao, ZHANG Caixia, ZHAO Xin. Review of AVS Audio Coding Standard [J]. ZTE Communications, 2016, 14(2): 56-62.
[6]	Shanshe Wang, Falei Luo, Siwei Ma. Overview of the Second Generation AVS Video Coding Standard (AVS2) [J]. ZTE Communications, 2016, 14(1): 3-11.
[7]	Bin Li, Jizheng Xu. An Introduction to High Efficiency Video Coding Range Extensions [J]. ZTE Communications, 2016, 14(1): 12-18.
[8]	Tao Lin. Screen Content Coding with Primary and Secondary Reference Buffers for String Matching and Copying [J]. ZTE Communications, 2015, 13(4): 53-60.
[9]	Siwei Ma, Shiqi Wang, and Wen Gao. AVS 3D Video Coding Technology and System [J]. ZTE Communications, 2012, 10(2): 13-18.

Video Enhancement Network Based on CNN and Transformer

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 22

Related Articles 9

Recommended Articles

Metrics