Spatiotemporal Saliency Detection and Its Applications in Static and Dynamic Scenes

IEEE TCSVT 2011Wonjun Kim

Chanho JungChangick Kim

OutlineIntroductionProposed MethodExperiment ResultApplicationConclusion

IntroductionProblem occurs when background is highly textured

Proposed Method

Feature RepresentationEdge orientation histogram (EOH)Color orientation histogram (COH)Temporal Feature

Self-ordinal MeasureSaliency MapScale-invariant Saliency Map

Edge Orientation Histogram (EOH)

1. Compute the edge orientation of every pixel in the local region center at the pixel

2. Quantized into K angle in the range of [,]3. Compute the histogram of edge orientation

m(x,y,n):edge magnitude(x,y,n):quantized orientation

𝑖

local region

Color Orientation Histogram (COH)

1. Quantize the angle in HSV color space in the range of [,] into H angles

2. Compute the histogram of color orientation

s(x,y,n):saturation value(x,y,n):quantized hue value

Temporal FeatureCompute the intensity differences between frames

Feature at the pixel of frame

P :total number of pixels in local regionj :index of those pixels in P :user-defined latency

Self-ordinal MeasureDefine a 1(K+1) rank matrix by ordering the

elements of EOH(COH) ex:

Self-ordinal Measure

Saliency Map of Edge and ColorCompute the distance from the rank matrix of

center region to surrounding regions

Saliency Map of Edge Saliency Map of Color

N :total number of local regions in a center-surround window

, :maximum distance between two rank matrices

Spatial Saliency MapCombine the edge and color saliency

Combining with Temporal SaliencyCompute the SAD of temporal gradients between

center and the surrounding regions

Combine the spatial and temporal saliency

Scale-invariant Saliency MapCombine 3 different scales of saliency Map

(3232, 6464, 128128)

3232 1281286464

Algorithm

Experiment Result

Static ImagesVideo Sequences

Experiment ResultStatic Image

Local region = 55center-surround window = 77K = 8, H= 6 = 40, = 24

Video Sequence = 49Speed: 23ms per frame (43 fps)

Static Images

Static Images

Video Sequences

Video Sequences

Application

Image RetargetingMoving Object Extraction

Image Retargeting

Image Retargeting

Moving Object DetectionG:the set of salient pixels in the ground truth imageP:salient pixels in the binarized object mapCard(A):the size of the set A

Moving Object Detection

ConclusionOrdinal signature can tolerate more local feature

distribution than sample values.The proposed scheme performs in real-time and

can be extended in both static and dynamic scenes.