A Generic Virtual Content Insertion System Based on Visual Attention Analysis H. Liu 1, 2, S. Jiang...

A Generic Virtual Content Insertion SystemBased on Visual Attention Analysis

H. Liu1, 2, S. Jiang1, Q. Huang1, 2, C. Xu2, 3

1Institute of Computing Technology, Chinese Academy of Sciences2China-Singapore Institute of Digital Media

3National Lab of Pattern Recognition, Institute of Automation

Outline

Motivation Related work The proposed Virtual Content

Insertion (VCI) system Experimental results Conclusions

23/4/19 http://www.jdl.ac.cn 2

Virtual Content Insertion


Convenient Changeable Cost less

To construct a generic VCI system

Challenge

Advertisement insertion VS. Augmentation

Software based VS. Hardware based Challenge

Insertion time Insertion place Insertion method Insertion content

Related work– Insertion Time

Insert advertisements into video prologue Be neglected Insert the ad into

interesting segments Our method

Temporal attention Higher Attentive

Shots• K. Wan, C. Xu, “Automatic Content Placement in Sports Highlights”,

ICME, 2006.23/4/19 http://www.jdl.ac.cn 5

Related work– Insertion Place

Static region Color consistent

region Visual relevance

measure Lower informative

region Our method

Spatial attention Lower Attentive Region

• C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004.

• K. Wan, etc., “Automatic Content Placement in Sports Highlights”, ICME, 2006.

• Y. Li, etc., “Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect”, ACM Multimedia, 2005.


Related work – Insertion Method

Challenge Camera parameters

unknown Existing methods

Structure of the scene Predefined landmarks

Our method Affine transformation Global Motion Estimation

• X. Yu, etc., “Inserting 3D Projected Virtual Content into Broadcast Tennis Video”, ACM Multimedia 2006.

• C. Xu, etc., “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004.


Related work – Insertion Content

Improve the ad effect Decrease intrusion VideoSense

Textual relevance Local visual and aural relevance

T. Mei, X-S. Hua, L. Yang, S. Li, “VideoSense-Towards Effective Online Video Advertising”, 16th ACM International Conference on Multimedia, pp: 1075-1084, 2007


Outline

Motivation Related work The proposed VCI system Experimental results Conclusions



T em poral A ttention A nalysis

Temporal Attention

Basic idea The more different a frame/shot/video

clip is to the preceding ones, the more probable for it to be attended

Measure Novelty



, 1 min ,i t i tbin

diff F F H H

1

, ,t

t t i t ii t l

Nol F diff F F w F F

HAS Detection

Shot novelty

The longer a shot is, the more it is probable to be attended


1

, ,t

t i t i ti t l

Nol S diff S S w S S

t t tAV S L Nol S


Spatial Attention Analysis

Static attention Spatio-temporal attention

Motion saliency Static novelty

Static saliency calculation

Motion saliency

calculation

Static novelty calculation

Static saliency maps

Motion saliency maps

Static novelty maps

+Video Frames

Attention Maps

Static Saliency (1)

Psychological basis Contrast Information theory

Our method Contrast and information theory

Calculation Property of receptive field


20, 0,DoG N N

Static Saliency (2)

Perceptive unit Pixel/block Region Object

Color quantization



Static Saliency (3)

Contrast

Information density

Saliency

1

, ,K

ki

Con k d f k f i G i k

logID k p f k

Sal k Con k ID k

23/4/19http://www.jdl.ac.cn 19

Motion Saliency

Motion Vector Space HSV color space Angle H

Magnitude S

Texture V

Static Novelty (1)



Static Novelty (2)

Static novelty: An event’s importance along temporal axis

Distance: KL 1X

log tt

t

M xNol t M x dx

M x

Static LAR Detection


Dynamic LAR Detection

1

L

M t tt

MAM P w t AM P

, 1

1

1k k

t

t Mk

P H P


Affine transformation


1

1 1

x

V y

f

2

2 2

x

V y

f

1 1 1r V V

2 2 2r V V

3 1 2r r r

1 2 3A r r r

1 10 0AP P A P A P


Global Motion Estimation

1

, 1 0 00

t

t k k Ak

P H AP P AP

Outline

Motivation Related work The proposed VCI system Experimental results Conclusions


Experiment Data Set – Test Video


No. Video Genre Shot Time

1 Friendssituation comedy

200 11:25

2Children at

Housesituation comedy

200 14:48

3A Date with

LuYuInterview 200 20:49

4Adventure to the

westOutdoor teleplay

200 25:48

Sum ---- ---- 800 72:50


Experiment Data Set -- Virtual Content


Temporal attention & HAS (1)

0 10 20 30 40 500

0.5

1

Shot

Attention curve Notice Rate



Noticing rate:

Consistency: the similarity between attention curve and noticing curve

in N

min ,k

cos AC k NC k

No. Consistency

1 0.75

2 0.79

3 0.84

4 0.82



Relationship between noticing rate and attention value


Spatial attention & LAR

Invited the users to evaluate the brands he/she has noticed in the video.

rate of GOOD 1 1

n n

i ii i

Rg Ng N

Video GOOD NEUTRAL BAD

1 72.25 19.13 8.62

2 70.87 23.13 6.00

3 66.25 25.00 8.75

4 70.38 25.62 4.00

Mean 69.94 23.22 6.84

Variance 6.67 8.55 5.20

Static Insertion Demo


Dynamic Insertion Evaluation

Subjective evaluation Criteria

1. Is the result’s deformation consistent with the scene?

2. Does the inserted VC follow the camera motion?

3. To what degree the user is satisfied with the result?

Scores: 15

1 2 3 4 5 60

2

4

VCI Result

Eva

luat

ion

Mean Variance


Dynamic Insertion Result


Conclusion

Main contribution A generic virtual content insertion system. A new method of temporal attention and HAS

detection A new method of spatial attention and LAR

detection A dynamic insertion method

Future work The attention change caused by content insertion The interaction between insertion time and place

A Generic Virtual Content Insertion System Based on Visual Attention Analysis H. Liu 1, 2, S. Jiang...

Documents

Transcript of A Generic Virtual Content Insertion System Based on Visual Attention Analysis H. Liu 1, 2, S. Jiang...