Video Object Tracking and Replacement for Post TV Production

LYU0303 Final Year Project

Spring 2004

Outline

Brief project overview and targets achieved New components added to the proposed

system Working principles of individual parts Future work Q & A

Overview

Post-TV production processes and changes the content of a video clip

Difficult for computers to process video in a global view

Step-by-step programming enables the computer to “think” more thoroughly

Overview

We have introduced the following parts in the last semester: Bitmap I/O RGB <-> HSV converter Edge detector Edge equation finder Translation detector Texture mapper

New parts have been added in order to increase the functionality of the system

Additional Basic Components

The following parts have been added Graphical User Interface (GUI) Corner point finder Video file reader and writer Video player processor (frame-based) Improved texture mapper

The following part has been modified Bitmap I/O Edge finder Texture mapper Removed Equation processor

Graphical User Interface

Although UI is not a main project part, a good GUI can significantly decrease the time needed for processing and program maintenance

Since C++ language is being used, the Microsoft Foundation Classes are adopted to develop the user interface

Provide most of the basic functions needed for information input and output

Graphical User Interface

Corner Point Finder

As we know that processing dots are much faster than processing lines, we may wish to try to find the corner points directly before finding the edge equations

Stick some brightly coloured labels at the corners of the rectangular surface

Can apply on cylindrical objects as well to indicated the curve control points

A simple K-Means method is used to group the points together.

Corner Point Finder In order to know the orientation of the mapped

texture, we need to know which is the “first” point For simplicity we just make the first point larger so

that it contains more corner points than the others

First point

Corner Point Finder

We repeat the process until all the corner points has been processed

Points which are too far away from the main groups are discarded

Find out the centroids of the groups and move them out to give an estimation of corner positions

Group No. of Member

1 1

2 1

3 1

1 2

4 1

1 5

2 5

3 5

4 5

Corner Point Finder

Video File I/O

As we are talking about video processing, it is inevitable to handle video reading and writing

DirectShow interfaces are being used in the project

Able to process any type of AVI video as long as the Window Media Player can play them

Video Player

We need to choose a video renderer that is suitable for our application

To reserve the function of real-time processing and video texture in the future, the Video Mixing Renderer 9 (VMR9) is used

Allows the rendering of more than one video or bitmap on the screen at the same time

Video Processor

Video are sequence of bitmap frames We capture each video frame and process t

hem in the same way as processing the bitmaps

After a video frame is being processed, we add them to an AVI file

Default video compressor is DivX Mpeg4 encoder (FourCC code “divx”)

Video RendererVideo System Structure

Texture Mapper

A graphics design technique used to wrap a surface of a 3-D object with a texture map

The 3-D object acquires a surface texture similar to the texture map.

Colors, brightness values or altitudes

Texture Mapper

Mapping

function

(u,v)(r,c)

Texture coordinates

Image coordinates

line by line process each pixel on every line last semester, we mapped cubes. How about this

semester?

Scan-line conversion

Scanline yk

Scanline yk+1

scanning order

for every line

Cylindrical mapping

How to map this cylinder with different textures?

Cylindrical mapping

Mapping cylinders

3 parts: Surface detection Line scanning Texture lookup

+ Orientation determination

Surface detection

1. Hough algorithm

to detect the two straight edges

four points are detected

Surface detection

2. Scanning from top to bottom

the scanning slope is orthogonal

yellow points are detected

by color transition

Surface detection

3. Model the curved edges by ellipses

x2 / a2 + y2 / b2= 1 2 ellipses are

needed rotational factor

x

y

z

Scan-line mapping4. Scan-line from left t

o right

Problem:

How to determine the corresponding texture for each pixel, say p=(x,y) ?

x

y

z

(x,y)

Texture lookup5. Rotate the cylinder t

o the upright position along z-axis

6. Translate the cylinder to (0,0) and determine the z coordinate p=(x’,y’,z’)

7. Rotate the along x-axis upright from viewpoint

x

y

z

(0,0)h

Texture lookup8. Detect the marker w

hich indicates where the texture starts

9. Look up the texture coordinate (u,v) by

(y’+h/2)/h = v

tan-1(x’/z’) / 2 = u

x

y

z

(u,v)

Cylindrical mappingAfter mapping the top we have :

TEXTURES

MAPPING

MAPPED CYLINDER

Spherical mapping

How to map this sphere with different textures?

Spherical mapping

Mapping spheres

Also 3 parts: Surface detection Line scanning Texture lookup

+ Orientation determination

Surface detection

1. Scan from 4 directions and get the boundary points

2. Determine center

Line scanning

3. Scan from left to right.

4. For each pixel p=(x,y), look up the texture by transforming it to the upright position

x

y

z

(0,0)

Upright

Texture lookup

4. Translate to origin

5. Look up z-coordinate

6. Rotate the ball along z-axis

7. Rotate the ball along x-axis (x’,y’,z’)

x

y

z

(0,0)

(5)(6)

Texture lookup8. Lookup the texture by

v = cos-1(y’/R) / u = (cos-1(x’/(R sin(v))) ) / (2)

Spherical mapping After mapping:

Mapped Sphere

Texture

Shadow mapping

Mapping of surface brightness Method:

Covert the image from RGB HSV Modify the V (intensity) value

5 types of shadow mapping

Shadow mapping

1. Mapping with texture brightness

often unrealistic

2. Mapping with image brightness

Shadow mapping

3. Mapping with average brightness of the image

more realistic

4. Mapping with other shadow maps (e.g. wood)

Shadow mapping

5. Transparency mapping

(background information has to be obtained)

Problem with sphere mapping

The texture is “squeezed” at the pole

Solution

Modify the texture to this:

Mapped sphere becomes:

No distortion at

pole

Q & A