Post on 29-May-2020
Super Resolution для сжатого видео
Моисейцев Алексей
Video GroupCS MSU Graphics & Media Lab
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
222
Содержание
Введение
Example-based SR
SRME
HMRF SR
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
3
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
4
source
Bi-cubic
Super-res
Lossless 2Mbps 1Mbps 256Kbps
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
Алгоритмы SR:
Iterative Backprojection (IBP)
Projection Onto Convex Sets (POCS)
Probabilistic Methods
Maximum a posteriori (MAP)
Model-based approach (MBSR)
Example-based
6
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
IBP
7
C2
CC1
C3
g
P1g
P3g
P2g
f=Pg
POCS
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
8Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
Типичная скорость работы: 0.15-0.4 fps
9
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
101010
Содержание
Введение
Example-based SR
SRME
HMRF SR
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Example-based SR
В SR только по LR-кадрам существует теоретический предел качества
Иногда есть доступ и к отдельным кадрам в высоком разрешении
11Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Registration
Для компенсации движения используется алгоритм Lucas-Kanade:
12Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Registration
13
В 99% случаев ошибка не превышает 0.15 LR-пикселя и 0.45 HR-пикселя
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Построение пар HR-LR
14
Используются LR-блоки 8x8
Рассматриваются повороты блоков
Ключ —10 AC-коэффициентов LR-блока и 68 граничных пикселей HR-блока
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Восстановление
15Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Восстановление
16Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Результат
MJPEG, Q=50
Обучение на первом кадре
Восстановление двадцатого кадра
17
source, frame 0 compressed frame 20 SR, frame 20
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Результат
18
compressed frame 20 SR, frame 20 Spatial example-based SR, frame 20
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Key-frame based SR
19Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
Идея: использовать HR ключевые кадры
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Восстановление
20Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
21Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
22Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
23Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Example-based SR
Слабая обоснованность метода
Требуется наличие специфичного видеопотока
Зависимость от обучающей выборки
24
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
252525
Содержание
Введение
Example-based SR
SRME
HMRF SR
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
26
SRME
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
27Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
28
— интересующее
решениеSimultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
— апостериорная
вероятность
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (1)
29Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (2)
30Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (3)
31Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Решение (1)
32Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Решение (2)
Две итерации:
Компенсация движения
Построение улучшенного кадра
33Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
34Y-PSNR = 20.44dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
source
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
35Y-PSNR = 21.98dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
bilinear
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
36Y-PSNR = 25.64dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
proposed SR
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
373737
Содержание
Введение
Example-based SR
SRME
HMRF SR
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
38Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
Standard decompression Smoothed image Sharpened image
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
39Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум квантования (1)
40Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум квантования (2)
41Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум компенсации (1)
42Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум компенсации (2)
43
— общее выражение
— независимость шума
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Условие гладкости
44Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
45
— гладкость
— квантование
— компенсация
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Решение
46Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
Градиент:
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Решение
Упрощение:
Градиентный спуск:
47Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
48Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
49Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
50Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Заключение
Рассмотрены Example-based и MAP методы
Многие методы используют информацию из потока или требуют дополнительные данные для работы
Прямые реализации алгоритмов сложны, но тем не менее многие из них возможно распараллелить (например, с использованием CUDA)
51
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
52
Литература
Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
DCT Quantization Noise in Compressed Images, Mark A. Robertson and Robert L. Stevenson, 2004
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
Z. Lin and H-Y. Shum. Fundamental limits of reconstruction-based superresolution algorithms under local translation. PAMI, 2004
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks. Performance of optimal registration estimators, SPIE 2005
Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005
Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
5353
Вопросы
?
53