1

Manifold Alignment for Multitemporal Hyperspectral Image Classification

H. Lexie Yang1, Dr. Melba M. Crawford2

School of Civil Engineering, Purdue Universityand

Laboratory for Applications of Remote Sensing

Email: {hhyang1, mcrawford2}@purdue.eduJuly 29, 2011

IEEE International Geoscience and Remote Sensing Symposium

2

Outline

• Introduction

• Research Motivation

− Effective exploitation of information for multitemporal classification in nonstationary environments

− Goal: Learn “representative” data manifold

• Proposed Approach

− Manifold alignment via given features

− Manifold alignment via correspondences

− Manifold alignment with spectral and spatial information

• Experimental Results

• Summary and Future Directions

3

Introduction

• Challenges for classification of hyperspectral data

− temporally nonstationary spectra

− high dimensionality

2001 2003 2004 2005 200620022001

June July May May May May June

N n

arro

w spe

ctra

l ban

ds

123

N>>30

4

• Nonstationarities in sequence of images

− Spectra of same class

may evolve or drift

over time

• Potential approaches

− Semi-supervised methods

− Adaptive schemes

− Exploit similar data geometries Explore data manifolds

Research Motivation

Good initial conditions required

5

Manifold Learning for Hyperspectral Data

• Characterize data geometry with manifold learning

− To capture nonlinear structures

− To recover intrinsic space (preserve spectral neighbors)

− To reduce data dimensionality

• Classification performed in low dimensional space

Spect

ral b

ands

Spatial dimension

Spa

tial d

imen

sion

1234

2nd dim 1st dim

3rd dim

n

56

Original space Manifold space

6

Challenges: Modeling Multitemporal Data

• Unfaithful joint manifold

due to spectra shift

• Often difficult to model the inter-image correspondences

Data manifold at T1 Data manifold at T2 Data manifolds at T1 and T2

7

Proposed Approach: Exploit Local Structure

Assumption: local geometric structures are similar Approach: Extract and optimally align local geometry

to minimize overall differences

Locality

Spectral space at T1 Spectral space at T2

8

Proposed Approach: Conceptual Idea

(Ham, 2005)

9

Proposed Approach: Manifold Alignment

• Exploit labeled data for classification of multitemporal data sets

Samples with no class labels

Joint manifold

Samples with class labels

10

Manifold Alignment: Introduction

• and are 2 multitemporal hyperspectral images

− Predict labels of using labeled

• Explore local geometries using graph Laplacian and some form of prior information

• Define Graph Laplacian

− Two potential forms of prior information: given features and pairwise correspondences [Ham et al. 2005]

1I 2I

L

where

0 , otherwise

1 , neighbors of iji j

ii ijj

L D W

Wx x

D W

2I 1I

11

Manifold Alignment via Given Features

Given Features is Joint Manifold*F

Minimize ( )C F

1

1 2

{ ,..., }: Given features of labeled samples

: Graph Laplacian of and

: Relative weighting coefficient

i ns s s n

L I I

1 1 1 21 2

2 1 2 2

, ,,

, ,

I I I II I

I I I I

L LL

L L

n

i

IITii

FFFLFsfFCF 21 ,* minarg)(minarg

12

Manifold Alignment via Pairwise Correspondences

Correspondences between and 1I 2I

Minimize ( , )C F G1IL

2IL

Joint Manifold * *[ ; ]F G

1

2

1 1 1 2

1 2

1

{ ,..., } ;{ ,..., }

( , ) : Pairwise correspondences in [ ; ]

where index corresponds to pair ( , ) extracted from and

: Graph Laplacian of

: Graph Laplacian

N M

i i

i i

I

I

x x I y y I

f g F G

i x y I I

L I

L

2of

: Relative weighting coefficent

I

k

iI

TI

Tii

GFGFGLGFLFgfGFCGF

21,,

** minarg),(minarg];[

13

MA with spectral and spatial information

• Combine spatial locations with spectral signatures

− To improve local geometries (spectral) quality

− Idea: Increase similarity measure when two samples are close together

Weight matrix for graph Laplacian:

where spatial location of each pixel is represented as

2 2spa spe

Spatial Distance ( , ) Spectral Distance ( , )exp expi j i j

ij

z z x xW a

ix2Riz

14

Experimental Results: Data

Three Hyperion images collected in May, June and July 2001 May, June pair: Adjacent

geographical area June, July pair: Targeted the same

area

May June July

15

Experimental Results: Framework

Joint manifoldGraph

LaplacianPrior information

Given features

Correspondences

Develop Data Manifold of

Pooled Data

GF

CF

PF

Data sets Labels

Pair 1 Pair 2

May June Training data For KNN classifier

June July Testing data For overall accuracy evaluation

Classificationwith KNN

I1, I2L

I1L I2L

16

Manifold Learning for Feature Extraction

• Global methods consider geodesic distance

− Isometric feature mapping (ISOMAP)

• Local methods consider pairwise Euclidian distance

− Locally Linear Embedding (LLE): (Saul and Roweis, 2000)

− Local Tangent Space Alignment (LTSA): (Zhang and Zha, 2004)

− Laplacian Eigenmaps (LE): (Belkin and Niyogi, 2004)

(Tenenbaum, 2000)

17

MA with Given Features

ISOMAP

LTSA

LLE

LE

75.69

70.18

47.65

62.38

0.620000000000005

7.69999999999999

29.64

16.63

Pooled DataAccuracy increment (Δ) with MA using extracted features

Overall Accuracy

• Baseline: Joint manifold developed by pooled data

(May, June pair)

79.21

77.29

77.88

76.31

18

MA Results – Classification Accuracy

• Evaluate results by overall accuracies

MethodsOverall Accuracy

May, June June, July

Manifold learning from pooled data 62.38% 83.00%

Manifold alignment(MA)

Given features (LE) 79.21% 86.16%

Correspondences 81.22% 84.27%

MethodsOverall Accuracy

May , June June, July

Given features (LE)

Spectral 79.21% 86.16%

Spectral + spatial 84.21% 90.30%

CorrespondencesSpectral 81.22% 84.27%

Spectral + spatial 84.74% 90.11%

19

Results – Class Accuracy

May, June pair

Typical class(Island Interior) Critical class

(Woodlands)Critical class

(Riparian)

20

Summary and Future Directions

• Multitemporal spectral changes result in failure to provide a faithful data manifold

• Manifold alignment framework demonstrates potential for nonstationary environment by utilizing similar local geometries and prior information

• Spatial proximity contributes to stabilization of local geometries for manifold alignment approaches

• Future directions

− Investigate alternative spatial and spectral integration strategy

− Address issue of longer sequences of images

21

Thank you.

Questions?

22

References

• J. Ham, D. D. Lee, and L. K. Saul, “Semisupervised alignment of manifolds,” in International Workshop on Artificial Intelligence and Statistics, August 2005.

23

Backup Slides

24

• Local geometry preserved via various strategies for embedding

• Popular local manifold learning methods

− Locally Linear Embedding (LLE): (Saul and Roweis, 2000)

− Local Tangent Space Alignment (LTSA): (Zhang and Zha, 2004)

− Laplacian Eigenmaps (LE): (Belkin and Niyogi, 2004) Pairwise distance between neighbors computed using Gaussian

kernel function - O(pN2) method

Embedding computed to minimize the total distance between neighbors

Local Manifold Learning for Feature Extraction (s,f)

2

,

min ( ) i j iji j

Y y y W

25

• Parameter values for local embedding

− s obtained via grid search

− k, p obtained empirically

LE: Impact of Parameter Values

BOT Class 3, 6 BOT Classes 1-9

26

• Island Interior

Alignment Results: Typical Class

Pooled Data MA: Given Features MA: Correspondences

Class Accuracy

24.9% 67.8% 96.25%

27

• Critical class: Riparian

Alignment Results: Critical Class

Pooled Data MA: Given Features MA: Correspondences

Class Accuracy

56.1% 71.6% 59.4%

28

Alignment Results: Critical Class

• Critical class: Woodlands

Pooled Data MA: Given Features MA: Correspondences

Class Accuracy

45.1% 35.8% 60.5%

29

MA Results – Classification Accuracy

• Evaluate results by overall accuracies

MethodsOverall Accuracy

May, June June, July

Manifold learning from pooled data 62.38% 83%

Manifold alignment(MA)

Given features 77.46% 86.16%

Correspondences 81.22% 84.27%

Labeled Class(Subset Data)

Classified viaGiven Features

Classified via Correspondences

Classified via Pooled Data

May, June pair

30

MA Results – Classification Accuracy

MethodsOverall Accuracy

May , June June, July

Given features using LE

Spectral 79.21% 86.16%

Spectral + spatial 84.21% 90.3%

CorrespondencesSpectral 81.22% 84.27%

Spectral + spatial 84. 74% 90.11%

Labeled Class(Subset Data)

Classified via Given Features

(Spectral)

Classified via Correspondences

(Spectral)

Classified viaGiven Features

(Spectral + spatial)

Classified via Correspondences(Spectral + spatial)

May, June pair