1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008.
-
Upload
alan-shelton -
Category
Documents
-
view
212 -
download
0
Transcript of 1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008.
1
Efficient Reference Frame Selector for Efficient Reference Frame Selector for H.264H.264
Tien-Ying Kuo, Hsin-Ju LuTien-Ying Kuo, Hsin-Ju Lu
IEEE CSVT 2008
2
OutlineOutline
IntroductionIntroductionH.264 Inter-CodingH.264 Inter-CodingProposed MethodProposed MethodExperimental ResultsExperimental ResultsConclusionsConclusions
3
IntroductionIntroductionBackground Knowledge (1/4)Background Knowledge (1/4)
H.264 features for inter-frame codingH.264 features for inter-frame codingVariable block size motion compensationVariable block size motion compensationSubpixel motion estimationSubpixel motion estimationMultiple reference frame motion compensationMultiple reference frame motion compensation
N - 5 N - 2 N - 1 Current frame N
4
IntroductionIntroductionBackground Knowledge (2/4)Background Knowledge (2/4)
Why do Multiple reference frame help predictions ?Why do Multiple reference frame help predictions ?Uncovered backgroundUncovered backgroundNon-integer pixel displacementNon-integer pixel displacementLighting changeLighting changeCamera shakingCamera shakingNoises in the source signalNoises in the source signal
Noise effect
5
IntroductionIntroductionBackground Knowledge (3/4)Background Knowledge (3/4)
Sampling/noise effectSampling/noise effect of MobileCalendar of MobileCalendar
Original block (N-1) Ref block (N-2) Ref block
(N-1) Residual block (N-2) Residual block
MSE = 203.8 MSE = 32.4
6
IntroductionIntroductionBackground Knowledge (4/4)Background Knowledge (4/4)
Drawback of MRF-MEDrawback of MRF-MEHigh computational complexityHigh computational complexity
In order to reduce complexityIn order to reduce complexityReduce search pointsReduce search pointsContinuous tracking techniqueContinuous tracking techniqueEarly stop criteriaEarly stop criteria
7
IntroductionIntroductionMain Purpose (1/1)Main Purpose (1/1)
A simple and effective method of selecting proper A simple and effective method of selecting proper reference frames in MRF-ME.reference frames in MRF-ME.
It enables working with any existing ME It enables working with any existing ME algorithms.algorithms.
Experimental results demonstrate the effectiveness Experimental results demonstrate the effectiveness of proposed algorithm.of proposed algorithm.
8
H.264 Inter-CodingH.264 Inter-CodingOverview (1/3)Overview (1/3)
Variable block sizeVariable block sizeMacroblock partition : 16*16, 16*8, 8*16, 8*8Macroblock partition : 16*16, 16*8, 8*16, 8*8Submacroblock partition : 8*8, 8*4, 4*8, 4*4Submacroblock partition : 8*8, 8*4, 4*8, 4*4
16*16 16*8 8*16 8*8
8*8 8*4 4*8 4*4
9
H.264 Inter-CodingH.264 Inter-CodingOverview (2/3)Overview (2/3)
Reference parameter Reference parameter REFREFSignal to which frame referredSignal to which frame referredCoded only once for each submacroblock partitionCoded only once for each submacroblock partition
All subblock smaller than 8*8 in the same subpartition must refer to the same frame
10
H.264 Inter-CodingH.264 Inter-CodingOverview (3/3)Overview (3/3)
Rate-distortion cost of each possible partitionRate-distortion cost of each possible partition
))())()(((
)))(,(,()(),(
REFRREFpREFmR
REFmREFcsDTSAREFJ
REF : Reference parameter
m( REF ) : The motion vector in the reference frame
p ( REF ) : The predicted motion vector from the neighbors
s : Original video signal
c : Coded video signal
R : Rate function of motion vectors
SA(T)D : SAD or SATD
Reference frame selectorReference frame selectorTreating 8*8 block as a minimal unitTreating 8*8 block as a minimal unitMaking the selection using a mode 8*8 motion searchMaking the selection using a mode 8*8 motion search
11
Proposed MethodProposed MethodMultiple Reference Frame Selection Multiple Reference Frame Selection
(1/9)(1/9)
Flow chartFlow chart
12
Proposed MethodProposed MethodMultiple Reference Frame Selection Multiple Reference Frame Selection
(2/9)(2/9)
Input a MB and perform four 8*8 block motion searchon reference frame t-1
Variance of four MVs of 8*8 blocks TH?≦
Perform motion estimation on reference frame t-1
Enter 2nd stage
YesNo
Output results
First Stage
Flow chartFlow chart
13
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(3/9)(3/9)Perform four 8*8 block motion search of the MB on
the reference frame t-i
Yes
No
i = 2
Is i N?≧i + 1
Is ref-frame t-k reffered by one of the four 8*8 blocks?
Drop the non-referred ref frame t-k Drop the non-referred ref frame t-k
Is k N?≧
Perform motion estimation on each valid reference frame
Output results
Second Stage
Yes
YesNo
No
First stageFirst stageDisjoint 16*16 macroblock into four 8*8 blocksDisjoint 16*16 macroblock into four 8*8 blocksMotion search on ref-frame t-1 and check thresholdMotion search on ref-frame t-1 and check threshold
14
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(4/9)(4/9)
Current Frame tReference frame t - 1
4 MVs and corresponding minimal R-D cost
Motion vectorsMotion vectors
R-D costR-D cost
Check x and y components of Check x and y components of V V t-1t-1
Less than a small threshold ?Less than a small threshold ? Encoder terminate early by designating only the Encoder terminate early by designating only the previous frame as reference frameprevious frame as reference frame
15
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(5/9)(5/9)
3,2,1,0|11 iVV ti
t
3,2,1,0|)( 11 iVCC ti
t
Early stop criteria
The design of threshold depends upon the The design of threshold depends upon the computational capacity of the encodercomputational capacity of the encoderLarger threshold, lower complexity, but worse coding Larger threshold, lower complexity, but worse coding
efficiencyefficiency
16
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(6/9)(6/9)
Determine upper bound of the coding efficiency and the worst case of complexity
Second stageSecond stageMotion search of mode 8*8 should be tested on all of Motion search of mode 8*8 should be tested on all of
the remaining reference framesthe remaining reference frames
17
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(7/9)(7/9)
Nti
ti
ti
ti VVVV ,...,,, 432
NkiVC kti ,...,3,2;3,2,1,0|
N : The maximal number of reference frames
For a given block index i, we let k be set asFor a given block index i, we let k be set as
The frame selector sets the ref-frame n-k as the The frame selector sets the ref-frame n-k as the valid, qualified reference framevalid, qualified reference frame
18
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(8/9)(8/9)
)(minarg,...,2,1
kni
NkVCk
Block i has the best motion vector with the lowest cost,
by referring to reference frame n-k
Drop unqualified framesDrop unqualified frames
19
Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection
(9/9)(9/9)
Current frame tFrame t-2Frame t-3Frame t-4Frame t-5
N = 5
Verifying the effectiveness of frame selectorVerifying the effectiveness of frame selectorMeasuring the Measuring the hit ratehit rate by comparing retained frames by comparing retained frames
with the actual frames used via exhaustive search with the actual frames used via exhaustive search method (hit rate = 88% ~ 95%)method (hit rate = 88% ~ 95%)
20
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(1/6)usage(1/6)
Even with the high hit rate, we expect that frame Even with the high hit rate, we expect that frame selector can drop as many frames as possible.selector can drop as many frames as possible.The The false alarmfalse alarm ranges from 13%~32% ranges from 13%~32%
21
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(2/6)usage(2/6)
Reference frame usage of the proposed frame Reference frame usage of the proposed frame selector for various video sequenceselector for various video sequence
22
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(3/6)usage(3/6)
Why using mode 8*8 but not other modes in Why using mode 8*8 but not other modes in frame selection?frame selection?Hit rateHit rateFalse alarmFalse alarmMotion estimation time spendingMotion estimation time spending
23
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(4/6)usage(4/6)
Analysis of hit rate and false alarm using different Analysis of hit rate and false alarm using different modes on various sequencemodes on various sequence
24
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(5/6)usage(5/6)
The motion estimation time spending on a MBThe motion estimation time spending on a MB
Judging from the Judging from the hit ratehit rate, , falsefalse alarmalarm and and complexitycomplexity8*8 is the best choice8*8 is the best choice
25
Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame
usage(6/6)usage(6/6)
ModeMode 16*1616*16 16*816*8 8*168*16 8*88*8 8*48*4 4*84*8 4*44*4
TimeTime 0.980.98 0.980.98 0.990.99 11 1.031.03 1.031.03 1.081.08
Reference software : JM 9.2Reference software : JM 9.2 Intel Pentium 4 3.0 GHz with 512MB RAMIntel Pentium 4 3.0 GHz with 512MB RAM Encoder parametersEncoder parameters
26
Experimental ResultsExperimental Results Overview (1/2)Overview (1/2)
Using Fast Full Search (FFS) and Fast Motion Using Fast Full Search (FFS) and Fast Motion Estimation (FME) in JM 9.2Estimation (FME) in JM 9.2
Comparing the results with a frame selection Comparing the results with a frame selection method of Li.[1]method of Li.[1]
27
Experimental ResultsExperimental Results Overview (2/2)Overview (2/2)
[1] X. Li, E. Q. Li, and Y. K. Chen, “Fast multi-frame motion estimation algorithm with adaptive search strategies in H.264 ,”in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., May 2004, vol.3, pp.369–372.
R-D curve comparison (Foreman)R-D curve comparison (Foreman)
28
Experimental ResultsExperimental Results
R-D curve comparison (Mobile)R-D curve comparison (Mobile)
29
Experimental ResultsExperimental Results
Using BDPSNR and BDBR to measure the Using BDPSNR and BDBR to measure the performance difference between the methodsperformance difference between the methodsCalculates average PSNR and bitrate distance between Calculates average PSNR and bitrate distance between
two RD curves of two method, respectively.two RD curves of two method, respectively.
30
Experimental ResultsExperimental Results
Discussing the computational complexityDiscussing the computational complexity
31
Experimental ResultsExperimental Results
Speed up of SAD and SATD calculationsSpeed up of SAD and SATD calculations
32
Experimental ResultsExperimental Results
Number of MBs references in each reference Number of MBs references in each reference frame (Mobile)frame (Mobile)
33
Experimental ResultsExperimental Results
Number of MBs references in each reference Number of MBs references in each reference frame (Foreman)frame (Foreman)
34
Experimental ResultsExperimental Results
An efficient reference frame selector is An efficient reference frame selector is proposed for the h.264 encoder to deal with proposed for the h.264 encoder to deal with the complexity issue pertaining to MRFME.the complexity issue pertaining to MRFME.
The experimental results demonstrate that the The experimental results demonstrate that the proposed algorithm can reduce significantly proposed algorithm can reduce significantly the complexity of ME at the encoder end, the complexity of ME at the encoder end, while keeping almost the same R-D while keeping almost the same R-D performance as FFS.performance as FFS.
35
ConclusionsConclusions