Pattern Recognition

CSE 802

Michigan State University

Spring 2008

Pattern Recognition

“The real power of human thinking is based on recognizing patterns. The better computers get at pattern recognition, the more humanlike they will become”. Ray Kurzweil, NY Times, Nov 24, 2003

What is a Pattern?“A pattern is the opposite of a chaos; it is an entity vaguely defined, that could be given a name.” (Watanabe)

RecognitionIdentification of a pattern as a member of a category we already know, or we are familiar with

Classification (known categories)Clustering (creation of new categories)

Category “A”

Category “B”

ClassificationClustering

Pattern Recognition

• Given an input pattern, make a decision about the “category” or “class” of the pattern

• Pattern recognition is a very broad subject with many applications

• In this course we will study a variety of techniques to solve P.R. problems and discuss their relative strengths and weaknesses

Pattern Class

A collection of “similar” (not necessarily identical) objects

A class is defined by class samples (paradigms, exemplars, prototypes, training/learning samples)

Inter-class variability

Intra-class variability

Pattern Class Model

Different descriptions, which are typically mathematical/statistical in form for each class/population

Given a pattern, choose the best-fitting model for it and then assign it to class associated with the model

Intra-class and Inter-class Variability

The letter “T” in different typefaces

Same face under different expression, pose….

Interclass SimilarityInterclass SimilarityInterclass Similarity

Identical twins

Characters that look similar

Pattern RecognitionHaving been shown a few positive examples (and perhaps a few negative examples) of a pattern class, the system “learns” to tell whether or not a new object belongs in this class (Watanabe)

Inferring a generality from a few exemplars

COGNITION = Formation of new classesRECOGNITION = known classes

Pattern Recognition Applications

Alphanumeric charactersOptical scanned imageCharacter recognition (page readers, zip code, license plate)

Tanks, airfieldsVisual, infrared, radar imagesAerial reconnaissance

Terrain forms, vegetation cover

Multispectral imagesNatural resource identification

Types of cardiac conditions, classes of brain conditions

EKG, EEG waveformsDetection and diagnosis of disease

Presence/absence of flaw, type of flaw

Ultrasound, eddy current, acoustic emission waveforms

Non-destructive testing

Spoken words, speaker identity

Speech waveformsSpeech recognition

OutputInputProblem

Pattern Recognition Applications

Occurrence of the word in the database

Query word written by a userOnline handwriting retrieval

Owner of the fingerprint, fingerprint classes

Input image from fingerprint sensors

Fingerprint identification

Text relevant to the userKey words specified by a userWeb search

Identify objects, pose, assembly

3-D images (structured light, laser, stereo)

Manufacturing

Acceptable/unacceptableScanned image (visible, infrared)

Inspection (PC boards, IC masks, textiles)

Type of cellsSlides of blood samples, micro-sections of tissues

Identification and counting of cells

OutputInputProblem

Pattern Recognition System

Challenges

RepresentationMatching

A pattern recognition system involves

TrainingTesting

Difficulties of Representation

John P. Frisby, Seeing. Illusion, Brian and Mind, Oxford University Press, 1980

How should we model a face to account for the large intra-class variability?

Difficulties of Representation

“How do you instruct someone (or some computer) to recognize caricatures in a magazine, let alone find a human figure in a misshapen piece of work?”“A program that could distinguish between male and female faces in a random snapshot would probably earn its author a Ph.D. in computer science.” (Penzias 1989)A representatin could consist of a vector of real-valued numbers, ordered list of attributes, parts and their relations….

Good Representation!

Should have some invariant properties (e.g., w.r.t. rotation, translation, scale…)Account for intra-class variationsAbility to discriminate pattern classes of interestRobustness to noise/occlusionLead to simple decision making (e.g., linear decision boundary)Low cost (affordable)

Pattern Recognition SystemDomain-specific knowledge

Acquisition, representationData acquisition

camera, ultrasound, MRI,….Preprocessing

Image enhancement/restoration, segmentationRepresentation

Features: color, shape, textureDecision making

Statistical/geometric pattern recognitionsyntactic/structural pattern recognitionArtificial neural networks

Post-processing/Context

Pattern Recognition System Performance

Error rate (Prob. of misclassification) on independent test samplesSpeedCostRobustnessReject optionReturn on investment

Fingerprint ClassificationFingerprint Classification

Assign fingerprints into one of pre-specified types

Plain Arch Tented Arch Right Loop Left Loop

Accidental Pocket Whorl Plain Whorl Double Loop

Fingerprint EnhancementFingerprint Enhancement

Noisy image Enhanced image

• To address the problem of poor quality fingerprints

Segmentation: Face DetectionSegmentation: Face Detection

*Theo Pavlidis, http://home.att.net/~t.pavlidis/comphumans/comphuman.htm

Games Magazine, September 2001

Segmentation: Face DetectionSegmentation: Face Detection

Fish Classification

Preprocessing will involve image enhancement, separating touching/occluding fishes and finding the boundary of the fish

Length FeatureTraining (design or learning) Samples

Lightness Feature

Overlap in the histograms is small compared to length feature

Two-dimensional Feature Space (Representation)

Two features together are better than individual features

Cost of misclassification?

Complex Decision Boundary

Issue of generalization

Boundary With Good Generalization

Simplify the decision boundary!

Feature Selection/extraction

• How many features and which ones to use in constructing the decision boundary?

• Some features may be redundant!

• Curse of dimensionality—problems with too many features especially when we have a small number of training samples

Fruit Sorter

cherries

apples

lemons grapefruits

Decision boundaries

diameter

redness

Castleman, Digital Image Processing, Prentice-Hall, 1979

General Purpose P.R. System

• Humans have the ability to switch rapidly and seamlessly between different pattern recognition tasks

• It is very difficult to design a device that is capable of performing a variety of different classification tasks

Cat vs. Dog

Access control for water in areas with water shortage (e.g. Australian outback); wildlife vs. livestockInstall a gate that opens only when livestock enters

Identify livestock using a PR systemRugged outdoor camera captures the imageEdge detection and outline tracingMatch to a library of existing shape templatesOpen the gate when there is a match

Prototype system by Dunn et al., U. South Queensland, Australia. Claim that Sheep & goats can be separated with ~100% accuracy

Sheep Vs. Goat

Vision Systems Design, November 2007 (www.vision-systems.com)

AllowDeny

Supervised Classification

Training samples are labeled

Unsupervised Classification

Training samples are unlabeled

Models for Pattern Recognition

Template matching

Statistical (geometric)

Syntactic (structural)

Artificial neural networks (biologically motivated?)

Hybrid approach

Template Matching

Template

Input scene

Deformable Template: Corpus Callosum SegmentationShape training set Prototype and

variation learning

Prototype registration to the low-level segmented image

Prototype warping

Statistical Pattern Recognition

Preprocessing Feature extraction Classification

LearningFeature selection

Recognition

Training

pattern

Patterns+

Class labels

Preprocessing

Each pattern is represented as a point in the d-dimensional feature spaceFeatures and their desired invariance properties are domain-specific

Good representation leads to small intraclass variation, large interclass separation & simple decision rule

Representation

x1

x2

x1

x2

Invariant Representation

Invariance to

• Translation

• Rotation

• Scale

• Skew

• Deformation

• Color

Structural Patten Recognition

Decision-making when features are non-numeric or structuralDescribe complicated objects in terms of simple primitives and structural relationship

Y

N

M

LT

X

Z

Scene

Object Background

D E

L T X Y Z

M ND E

Syntactic Pattern Recognition

Preprocessing

Primitive, relation

extraction

Syntax, structural analysis

Grammatical, structural inference

Primitive selection

Recognition

Training

pattern

Patterns+

Class labels

Preprocessing

Chromosome Grammars

Terminals:VT={∩,⏐,∪,⎨, }Non-terminals: VN={A,B,C,D,E,F}Pattern Classes:

Median Submedian

Acrocentric Telocentric

Chromosome Grammars

Image of human chromosomes

Hierarchical-structure description of a submedium chromosome

Artificial Neural Networks

Massive parallelism is essential for complex pattern recognition tasks (e.g., speech and image recognition)

Humans take only a few hundred milliseconds for most cognitive tasks; this suggests parallel computation in human brain

Biological networks achieve excellent recognition performance via dense interconnection of simple computational elements (neurons)

Number of neurons ≈ 1010 – 1012

Number of interconnections/neuron ≈ 103 – 104

Total number of interconnections ≈ 1014

Artificial Neural Networks

Nodes in neural networks are nonlinear, typically analog

where is internal threshold or offset

x1x2

xd

Y (output)

w1

wd

Feed-forward nets with one or more layers (hidden) between the input and output nodesA three-layer net can generate arbitrary complex decision regions

These nets can be trained by back-propagation training algorithm

Multilayer Perceptron

.

.

.

.

.

.

.

..

d inputs First hidden layerNH1 input units

Second hidden layerNH2 input units

c outputs

How m chinfo mation are

y u mi sing

Utilizing Context

Qvest

Constraining the Problem

GRAFFITI’S MODIFIED alphabet is largely based on single pen strokes, starting at the dots. As soon as the pen is lifted from the screen, the letter is immediately translated into normal text. The letter “X” is the exception

Graffiti alphabet

Comparing Pattern Recognition Models

Template MatchingAssumes very small intra-class variabilityLearning is difficult for deformable templates

SyntacticPrimitive extraction is sensitive to noiseDescribing a pattern in terms of primitives is difficult

StatisticalAssumption of density model for each class

Neural NetworkParameter tuning and local minima in learning

In practice, statistical and neural network approaches work well

“Super Classifier”

Pool the evidence from component recognizers (classifier combination, mixture of experts, evidence accumulation)

Statistical Pattern Recognition• Patterns represented in a feature space• Statistical model for pattern generation in feature space

• Given training patterns from each class, goal is to partition the feature space.

Approaches to Statistical Pattern Recognition

Bayes DecisionTheory

COMPLETE

"Optimal"Rules

Plug-in Rules

ParametricApproach

Density Estimation

GeometricRules

(K-NN,MLP)

NonparametricApproach

SupervisedLearning

MixtureResolving

ParametricApproach

ClusterAnalysis

(Hard, Fuzzy)

Non-parametric Approach

UnsupervisedLearning

INCOMPLETE

Prior Information

Summary

Pattern recognition is extremely useful forAutomatic decision makingAssisting human decision makers

Pattern recognition is a very difficult problemSuccessful systems have been built in well-constrained domainsNo single technique/model is suited for all pattern recognition problemsUse of object models, constraints, and context is necessary for identifying complex patternsCareful sensor design and feature extraction can lead to simple classifiers

Key Concepts

Pattern classRepresentationFeature extractionFeature selection Invariance (rotation, translation, scale, deformation)PreprocessingSegmentationTraining samplesTest samplesError rateReject rateCurse of dimensionality

Key Concepts

Supervised classificationDecision boundary unsupervised classification (clustering)Density Estimation Cost of misclassification/RiskFeature space partitioningGeneralization/overfittingContextual informationMultiple classifiersPrior knowledge