Introduction MM Processing S08

8/17/2019 Introduction MM Processing S08

1/70

CS 4763 Fundamentals of Multimedia Systems

- Introduction to Image Processing

Qi Tian

Computer Science Department

University of Texas at San Antonio

[email protected]://www.cs.utsa.edu/~qitian/


2/70

Image Processing

Manipulation of multidimensional signals− image (photo)

− video

− CT, MRI

− Fluid flow

),( y x f

),,( t y x f

),,,( t z y x f

),,,( t z y xv


3/70

A Typical Image Processing System

object observe digitize store process Refresh

/storeRecord

output

DisplayImaging

systems

Sample and

quantize

Digital

storage

(disk)

Digital

computer

On-line

buffer

X-ary, radar imaging, infrared

imaging, ultrasound imaging,

medical imaging, geophysicalimaging

A/D


4/70

Fundamentals of Image Processing

Representation – acquisition, digitization, and display to mathematical

characterization of images for subsequent processing

– a prerequisite for an efficient processing techniques such as

enhancement, filtering, and restoration.

Processing Techniques

– Image compression, image restoration, and image reconstruction

– Statistical image processing techniques

Communications


5/70

Multimedia Processing Techniques

– Coding/compressionStorage and communications JPEG, JPEG2000

MPEG-1 (CD, mp3), MPEG-2 (HDTV, DVD)

H.261, H.263

– Enhancement, restoration, reconstruction feature extraction for image analysis and visual information display

removal of degradation in an image, LS, ML, Max entropy, MAP

2D -> 3D image MRI, CT, Radon transform

– Analysis, detection, recognition, understandingquantitative measurements from an image to produce a description on it

– Visualization


6/70

Advanced Processing Techniques

Statistical processing techniques – Hidden Markov model (HMM)

– Probabilistic graphical models

– Bayesian networks

– Markov random field

Many applications to speech recognition, pattern classification, data

compression, and channel coding, etc.


7/70

History of Image/Video Coding

1950

1960

1970

1980

1990

2000+

Math PR, CV, CG

Fractal 3-D Model based

coding

Signal ProcessingBased

PCM

DPCM

Transform Coding

VQ

Subband CodingWavelets


8/70

Reference:

– F. Nebeker, Signal Processing: The Emergency of a Discipline,

1948-1998

– IEEE History Center, 1998

Broadband TV (NTSC)

500 × 500 × 8 × 3

× 30 bits/sec

≈

100 Mb/sec (compression is necessary!)Modem: 56Kb/sec

Picture Element

– Pixel West coast people in USC – Pel East people in MIT


9/70

Image/Video Compression

Signal-Processing Based:Encoder

),( y x f

H),( y xg

Signal

Proc.

Representation ),( y xg

Decoder 1H−),(ˆ y x f ),( y xg


10/70


3D Model-Based:Encoder

Representation P

Decoder

),( y x f

H

Analysis Model

Parameter P

Model

),(ˆ y x f P 3D

Model


11/70


Fractal-Based:Encoder

Representation S

Decoder

),( y x f System S

),( y x f

Find S for which is an Attractor.),( y x f

SAny

signal),(ˆ y x f

Iteration


12/70

Image/Video CompressionStandard

Facsimile: Fax Group 1, 2, 3, 4

JBIG (Joint Bi-level Image Expert Group)

Images: JPEG (http://www.jpeg.org/)

JPEG2000

Video: H.261, H.263 P × 64 Kb/s (P =1 ~ 30)MPEG 1 1.2 Mb/s Video, CD, MP3

MPEG 2 1.2 – 20 Mb/s, sports, HDTV, DVD

MPEG 4 1 kb/s → 1Mb/s, very low speed video

coding, MultimediaMPEG 7 Multimedia description, audio/video

MPEG 21 Multimedia framework

Based on Wavelet

Transform


13/70

A de facto image for the past three decade for its rich texture

Lena


14/70


15/70

What are Challenging Problems inMultimedia Processing?

Multimedia Processing is taken in a broad sense,

including:

Image/Video compression, enhancement, restoration,

reconstruction, analysis, recognition, understanding,

visualization, and synthesis/animation.


16/70

Examples

Face modeling, detection, and recognition

Emotion recognition

Gesture recognition

Gender/age/ethnicity recognition

Audio-visual speech recognition

Image/video superresolution

Image/video browsing, indexing, and retrieval

Biometrics


17/70

Face Related Research

Face modeling Face detection

Face recognition

Facial expression recognition


18/70

Generic Face Model

Texture

mapping

Face model morphing


19/70

Generic Face Model

The generic face model is generated from a MRI data set


20/70

Customize A Genetic Face on AnIndividual

Polygon Mesh: 2240 Vertices + 3946 Triangles.Polygon Mesh: 2240 Vertices + 3946 Triangles.

Non Non--Uniform Rational BUniform Rational B--Splines (NURBS): 63 control points.Splines (NURBS): 63 control points.


21/70

The iFACE system in a distributed collaborative environment. (a)

Avatar in the head mounted display, (b) avatar in the desk screen of

MIC3E, (c) avatar in the main screen of MIC3E

Avatar – talking head

University of Illinois at Urbana-Champaign


22/70

Text-Driven Face Animation

“We strive to make the meter onanimation production, and are

always looking for new technology

that will enable faster, more

appealing character creation,”

said Joel Kransove, Digital Director of

Nickelodeon. (Source: Digital

Producer)


23/70

Speech-Driven Face Animation

“Game characters have becomesynthetic actors and dialogue is anessential element of the effect wecreate. The quality of the lip-

synching can make or break thesense of reality,”

said Scott Cronce, vice president and CTO at

Electronic Art (Source: Gamepro)


24/70

Video-Driven Face Animation


25/70

Emotion Recognition


26/70

Emotion Recognition


27/70

Emotion Recognition


28/70

Face Detection Techniques


29/70

Face Detection Techniques


30/70

Face Recognition: Why it is easy?


31/70

Face Recognition: Why it is hard?


32/70

Beauty Check

What Are the Causes and Consequences ofHuman Facial Attractiveness?

Babyfaceness

Symmetry

Social perception

Universities of Regensburg, Germany


33/70

Which is more attractive?

Universities of Regensburg, Germany


34/70

Babyfaceness Large head

Large curved forehead

Facial elements (eyes,

nose, mouth) located

relatively low

Large, round eyes

Small, short nose

Round cheeks

Small chinKate Moss4-year old girl

Include mature female features: high, prominent cheek bones and

concave cheeks


35/70

Which one is cuter?


36/70

Miss Germany (2002)


37/70

A selection of the 22 contestants of the final round of

the contest


38/70


39/70

Image Analysis

Texture synthesis and transfer

Image Super-resolution

Image Repairs

Illumination/Lighting changes and transfer


40/70

Texture Synthesis and Transfer

+

SIGGRAPH’01 Effros & Freeman, MIT, 2001

synthesis

transfer


41/70

Texture Synthesis and Transfer


42/70

Image Superresolution

True Sub-sampled

Intelligent guess about details of texture


43/70


Gaussian filter Bicubic interpolation


44/70


Median filter Wiener filter


45/70


Dynamic resolutionenhancement

Amos Storkey

True


46/70

Image Repairs


47/70

Image Repairs

Original Image

Result

Segmentation

Image synthesis

based on Tensor

Voting

Curve connection


48/70

Image Repairs


49/70

Illumination Effects on Images


50/70

Relighting – Basic Algorithm

Step 2: Approximate radiance environment map

Step 3: Synthesize novel appearance by adjustingthe 9 spherical harmonic coefficients

Step 1: Align image with generic 3D face model


51/70

Lighting Transfer

input target results


52/70

Image/Video Retrieval

Image database

CBIR b d l


53/70

CBIR based on color, texture,shape/structure

MARS: Multimedia Analysis and Retrieval System

metadata

User

Interface

Similarity

ranking

memory

Featureweighting

Visual

C++

Feature

Extraction

C/C++ Color

Texture

structure


54/70

State-of-the-art CBIR Systems

QBIC (IBM), PhotoBook (Media Lab), Netra (UCSB),VisualSeek (Columbia), PicHunter (NEC-NJ), Amore (NEC-

CA), EI Niňo (Praja), MARS (UIUC), Virage (Virage Inc.),

CORE, PictoSeek, Piction, InfoScope …

Research Communities

Computer Vision, Image/Video Processing, Library and

Information Science, Database and Management Systems

Leading Journals & Standard

PAMI, ACM Multimedia, IJCV, CVIU

MPEG-7


55/70

MARS using global features


56/70

Biometrics

Security Threats:

We now live in a global society of increasing desperate and dangerous

people whom we can no longer trust based on identification documents

which may have been compromised.

A challenging Pattern Recognition Problem

Enabling technology to make our society safer,

reduce fraud and offer user convenience.


57/70

Too many passwords to remember


58/70

Identification Problems

Identity Theft: Identity

thieves steal PIN (e.g., dateof birth) to open credit card

account, withdraw money

from accounts and take out

loans

3.3 million identity thefts in

U.S. in 2002; 6.7 million

victims of credit card fraud

Surrogate representations of identity such as password

and ID cards no longer suffice


59/70

Biometrics

Automatic recognition of people on their

distinctive anatomical (e.g., face, fingerprint, iris,

retina, hand geometry) and behavioral (e.g.,

signature, gait) characteristics.

Person identification is now an integral part of the

infrastructure needed for diverse business sectors

such as banking, border control, law

enforcement…


60/70

Biometric Applications


61/70

Biometric Applications

There are ~500 million border

crossing/year (each way) in the US


62/70

Want to charge it?


63/70

Biometric Characteristics


64/70

Biometric Market Growth

International Biometric Group


65/70

“State-of-the-art” Error Rate

False accept rate

(FAR):

Proportion of

imposters

accepted

False reject rate

(FRR):

Proportions of

genuine users

rejected


66/70

Multibiometrics


67/70

Soft Biometrics


68/70

Privacy Concerns


69/70

Tracking


70/70

Introduction MM Processing S08

Documents

Transcript of Introduction MM Processing S08