E VENT D ETECTION U SING AN A TTENTION -B ASED T RACKER 22 Outubro 2007. Universidade Federal do...

EVENT DETECTION USING AN ATTENTION-BASED TRACKER

22 Outubro 2007. Universidade Federal do Paraná.Gerald Dalley, Xiaogang Wang, and W. Eric L. Grimson

Glasgow Airport

4 cameras

9 video clips 1 for training 8 for testing

Dataset Description

2

Loitering Staying in the scene for > 1 minute

Theft Taking someone else’s baggage

Left luggage Leaving the scene without your luggage

Evaluation Correct event times Correct 3D locations Without false detections

Challenge Problem

3

Basic tracking first

Detect interesting tracks

Improve interesting tracks to detect events

Our Approach

4

Processing Pipeline

5

Tracking & Event

Detection

Input Data

Background Subtraction

BackgroundModelingIllumination

Normalization

Illumination Changes

6

BA

CK

GR

OU

ND

126 999

2999

S08

Illumination Changes by Clip

7

0

20

40

60

80

100

120

Intra-Clip Time

Mea

n In

tens

ity

Background Modeling

8

Tracking & Event

Detection

Input Data


Robust Gaussian Fit

Ground-Plane Registration

Background Model Search

NormalizedBACKGROUND

NormalizedS01-S08

BackgroundModeling

Background Models

IlluminationNormalization

Fundamental assumption foreground is rare at every pixel

Reality for PETS 2007… background is rare for the pixels we care about

most Some pixels: foreground as much as 90% of the time

Breaking Adaptive Background Subtraction

9

0

2000

3000

4000

Robust Gaussian Fit (per pixel)

10

BACKGROUND clip Foreground is rare

everywhere

Fit a Gaussian Refit to inliers

Need for Model Adaptation

11

Another clip (S02) BACKGROUND’s

model: Suboptimal fit

BACKGROUND’s Gaussian modelBackground samplesForeground samples

BACKGROUND’s Gaussian modelBackground samplesForeground samples

Model Adaptation

12

Until convergence Find inliers Shift Gaussian center

Improvement

13

Adaptation Is robust Improves FG/BG

classification rates

Before adaptationAfter adaptationBefore adaptationAfter adaptation


14

Tracking & Event

Detection

Input Data



Normalization

MRF blobsBlob

Extraction

Mahalanobis Distance

Background Models

NormalizedS01-S08


15

Frame 2000 With Foreground Mask Foreground Mask

Mahalanobis DistancesAdapted ModelNormalized

- =

Tracking & Event Detection

16

Tracking & Event

Detection

Input Data



Normalization

alerts

blobsKalman Tracking

Meanshift

Tracking

Event Detection

Idea: Focus on tracking what we care about. Loitering humans Dropped luggage that becomes dissociated from

its owner

Kalman tracking Constant velocity Low false positive rate

Blob Tracking

17

Loitering humans Remain in the scene for a long time Likely to create isolated tracks

Dispossessed luggage Likely to create at least an isolated blob detection

Detecting Humans and Luggage

18

Object Type

Min. Blob Area (% of

frame)

Max. Blob Area (% of

frame)

Min. Blob Track Length

humans 1.5% 3.0% 16s

luggage 0.2% 1.0% 1 frame

Blob tracking Yields high-quality tracks (good) Requires isolated blobs (bad)

Meanshift tracker Learn a model (color histogram) from the good blob

tracks Tracks through occlusions

Humans Find scene entry/exit times

Luggage Find drop/pickup times Associate with human owners

Mean-Shift Tracking

19

Results

20

No events occur

None detected

S00 – No Defined Behavior

21

Staged loitering 5.1s late

S01 – General Loitering 1 (Easy)


S02 – General Loitering 2 (Hard)

23


Dropped luggage Should not trigger an alarm No alarm triggered

Staged loitering man near purple-outlined woman Missed

Unscripted loitering Detected

S03 – Bag Swap 1 (Easy)

24

S04 – Bag Swap 2 (Hard)

25

Stay close to each other the whole time

S05 – Theft 1 (Easy)

26

Victim enters 19.2s late

Luggage stolen 0.08s late

Thief exits 0.08s late

S06 – Theft 2 (Hard)

27

Thief

Luggage (never isolated)

Victims

Assistant thief

S07 – Left Luggage 1 (Easy)

28

Luggage dropped 0.12s late

Owner tracked Luggage taken

0.08s late By owner

S08 – Left Luggage 2 (Hard)

29

Owner drops: 0.2s late

Owner picks up: 0.12s late

PETS 2007 Problem description, datasets, etc. http://www.pets2007.net

My Website The paper http://people.csail.mit.edu/dalleyg

Websites

30

Perguntas

31

Illumination Normalization

32

~ci ;t = § ¡ 12

t (ci ;t ¡ ¹ct);~ci ;t = § ¡ 12

t (ci ;t ¡ ¹ct);

where

ci ;t = the color of pixel i in frame t;

¹ct =1N

NX

i=1

ci ;t; and

§ t =1

N ¡ 1

NX

i=1

(ci ;t ¡ ¹ct)2:

ci ;t = the color of pixel i in frame t;

¹ct =1N

NX

i=1

ci ;t; and

§ t =1

N ¡ 1

NX

i=1

(ci ;t ¡ ¹ct)2:

Region-of-Interest Masks

33

BG-S04

S06

S05

S07-S08


34

Tracking & Event

Detection

Input Data



Normalization

BG-Biased MRF

FG-Biased MRF

Blob Extractio

n

fragmented blobs

merged blobs

Blob Extractio

n

Mahalanobis Distance

Background Models

NormalizedS01-S08

Dual Background Subtraction

35

Low fragmentation But blobs merged Good for human

tracking

Mahalanobis Distance Map

Background-Biased Blobs

Foreground-Biased Blobs

Sharp boundaries But fragmented

blobs Good for

dropped luggage detection

Tracking & Event Detection

36

Tracking & Event

Detection

Input Data



Normalization

alerts

fragmented blobs

mergedblobs

Kalman Tracking

Kalman Tracking

Meanshift

Tracking

Event Detection

Luggage: it often doesn’t travel far from the owner, so we need BG-biased

to avoid merging the dropped luggage blob with the owner The floor is more boring (except for specularities), so

camouflaging doesn’t occur much there, relative to the vertical surfaces

Easy to tell people fragments from luggage: small people fragments move They move when they’re isolated They move before and after isolation

Humans With the busyness of the scene, a BG-biased MRF produces a lot

of fragments and many-to-many blob matching quickly becomes impractical

A FG-biased MRF avoids the fragmentation issue but merges lots of blobs We only care about loiterers

Loiterers are in the scene for a long time They’re likely to be isolated from other people at least at some point in time

Why Dual Trackers/Motion Blobs

37

Oriented Ellipsoids

38

E VENT D ETECTION U SING AN A TTENTION -B ASED T RACKER 22 Outubro 2007. Universidade Federal do...

Documents

Transcript of E VENT D ETECTION U SING AN A TTENTION -B ASED T RACKER 22 Outubro 2007. Universidade Federal do...