HARDWARE IMPLEMENTATIONS OF HYPERESPECTRAL IMAGE ...

Instituto Universitario de Microelectrónica Aplicada

(Tecnología de información y las comunicaciones)

HARDWARE IMPLEMENTATIONS

OF HYPERESPECTRAL IMAGE

PROCESSING ALGORITHMS

Developments at IUMA 23/03/2012

An introduction to IUMA and Integrated System Design (DSI) Division

Research Institute for Applied Microelectronics (IUMA)

23/03/2012

A

Hyperspectral Image Processing

IUMA started it activities in the early 90s

The first out of 6 research institutes at ULPGC

35 PhD, 3 technicians, 10 PhD students, 26 MsC students

Areas of expertise in microelectronics, signal processing, computer science, MEMs, mathematics, etc.

Contributions to the creation of 8 spin-offs

23/03/2012 Hyperspectral Image Processing

3

IUMA

Director Antonio Núñez

Integrated Systems Design

IPs, NoCs, H.264, HEVC, HyperSpect.

Information Technology

IT, Workflows, Security, FT,

Microelectronic Technology

RF, Wireless com., HFETs

Communi-cations

Biosensors, Wireless

sensor net.

Industrial Systems and CAD Tools

Embedded systems,

codesign, ESL

Microelectro-mechanical

Systems

Microsensors, microtrans-

ducers, micro motors …

Executive Director

José F. López

Secretary

Pedro. P. Carballo

People:

6 PhD, 3 PhD students, 20 MsC students

Research lines:

Design of IP (Intellectual Property) modules and integrated systems (SoCs) for multimedia applications, especially in video compression standards H.264 and SVC (Scalable Video Coding).

Development of algorithms for images and video improvement through Super-resolution and its implementation in hardware.

System design for hyperspectral image processing and implementation of these algorithms for real-time applications.

Hyperspectral image compression. Implementation over FPGA and GPUs.

Reconfigurable hardware (both dynamic and static) for the realization of high performance multimedia systems.

Networks on a Chip. Development of architectures for advanced networks on chip, and solutions to problems dominated by interconnections.


4

Integrated System Design

Hardware implementation of the Modified Vertex Component Algorithm

Hyperspectral Image processing

23/03/2012

B


1. Hyperspectral images • Introduction to hyperspectral images

• Hyperspectral sensors

• Hyperspectral images applications

2. Hyperspectral analysis • Linear unmixing

3. Hardware implementations • Modified Vertex Component Analysis (MVCA)

implementations

4. Partners and publications

Talk overview


A vision about the hyperspectral images, hyperspectral sensors and its applications

Hyperspectral images

23/03/2012

1


What a hyperspectral image is?


8


Visible spectrum

Spectral Reflectance

Spectral libraries: ASTER Spectral library (NASA): http://speclib.jpl.nasa.gov

USGS Spectral library: http://speclab.cr.usgs.gov/spectral.lib04/spectral-lib04.html.


9


Short Infrared

Typical hyperspectral bands:


10


0.4 µm

0.7 µm

1.4 µm

3.0 µm

8.0 µm

15 µm

VIS = Visible

NIR = Near Infrared

SWIR = Short Wavelength

Infrared

MIR = Mid Infrared

LWIR = Large Wavelength

Infrared

Satellite sensors

NASA Hyperion:

224 bands

0.4 to 2.5 µm

Airborne sensors

NASA AVIRIS:

Whiskbroom sensor

224 bands

4 ports

From 0.4 µm to 2.5 µm

12 bits per pixel

Spectral response: 10 nm

INTA AHS:

Whiskbroom sensor

80 bands

From 0.43 µm to 12.7 µm

5 ports

Handheld sensors


11

Hyperspectral sensors

Hyperspectral cube

23/03/2012

12


Hyperspectral cube: hypercube

Panchromatic

Hyperspectral

(Hundreds of bands)

Multispectral

(Tens of bands)

Ultraspectral

(Thousands of bands)

λ x

y

Wide bands

widely separate

Narrow bands

and contiguous

Hyperspectral images are used in a wide array of applications:

Agriculture

Mineralogy

Surveillance

Physics

Chemical imaging

Environment

Global warming

Oil spills

Fires

Food

Medicine


13

Hyperspectral images: Applications

Spectral matching and linear unmixing procedures. Endmember extraction

algorithms

Hyperspectral Analysis

23/03/2012

2


Hyperspectral images in spectral space


15

x1,y1,λ1 Xm,yn,λ1

x1,y1,λr xm,yn,λr

Mathematical representation of

a pixel in the spectral space

Same material exhibits some variability in its spectral space representation

Different material in the spectral space Spectral matrix Y = [r1, r2, … rR]

Y =

Suelo

Árbol

Pasto

Mezcla macroscópica:

15% suelo, 25% árbol, 60% pasto

12 metros

12 m

etro

s16 m

ete

rs grass

soil

tree

Mixture 15% soil 25% tree 60% grass

16 meters


16

Hyperspectral images mixing

• Depending on the spatial resolution of the sensor, most of the pixels (vector) of a Hyperspectral

image correspond to a mixture of pure components at a macro level

• These pixels can be expressed as a combination of elements spectrally pure (called endmembers

in literature)

Endmember

B

Endmember

A

Mix pixel

C

abundances

p

i

a1

ii er

Assumes linear mixture model

Three main steps:

Determination of number of endmembers: p

Determination of the endmembers: e1 … ep

Abundances calculation: a1 … ap

Some endmembers extraction algorithms require a previous

dimensionality reduction using PCA, MNF, ICA or others


17

Linear unmixing

Number of endmembers

• HySIME

• VD (Virtual Dimensionality)

Endmembers extraction

• PPI

• NFINDR

• VCA (Vertex Component Analysis)

Abundances computation

• LSU (Least Square)

• FCLSU (Fully Constrained Least Square)

Endmembers will be the most different pixels

In its spectral representation will be the most extreme pixels

The N-FINDR method grows a simplex data until you get the bulk


18

Endmembers extraction: N-FINDR

Band i

Ban

d j

1e

2e

3e

3

1

ii

i

a er

Vertex Component Analysis (VCA)


19

Band i

Ban

d j

e0

Simplex Sp

rT·u =1

Convex cone Cp (γ≠1)

1e

f1

f2

<f1,e1>

2e<f2,e2>

INPUTS: p, Y = [r1, r2, …, rR]

% Y is composed by R hyperspectral pixels of

p frequency bands each

1: E = [e0 | 0 | … | 0 |];

% E is a p×p endmember matrix

2: for i = 1 to p do

3: f = ((I - EE+)w)/ (||(I - EE+)w||); % f is a random vector orthonormal to the

subspace spanned by the columns of E

4: v = fTY; % Y is projected onto the direction

indicated by f

5: index = arg maxindex = 1, …, R

|v[:,index]|; % the projection extreme is found

6: E[:,i] = Y[:,index]; % endmembers are updated

7: end for

OUTPUT: E = [e1, e2, …, ep]

Basically consists in finding the p more

extreme pixels of the scene

Modified Vertex Component Analysis implementation

Hardware implementation

23/03/2012

3


VCA has a high computational cost: f computation: orthonormal operation

Requires the computation of pseudoinverse of E (E+) and the product EE+

Pseudoinverse is usually performed by singular value decomposition (SVD)

Projection of Y over a vector Correspond to a vector by matrix floating point multiplication (v= fTY)

Modified Vertex Component Analysis Gram-Schmidt orthogonalization (f computation)

An alternative low computation cost orthogonalization process

Use fixed point instead of floating point for fTY computation

But this modification does not change the endmembers

The MVCA was tested using: Synthetic images of “p” endmembers based on demo_vca software tool

Cuprite image from AVIRIS


21

Modified Vertex Component Analysis (MVCA)

floating point computation

integer computation

IMAGE

PROJECTION

INPUT

MEMORYF_GENERATOR

U_GENERATOR

index

E [:,i]

f

Yint

U [:,i]

U[:,1:i-1]

PROJECTIONS

E[:, i]

shift_exp

int2fp

round2int(f)

Yint

MVCA Architecture


22

IMAGE

PROJECTION

INPUT

MEMORYF_GENERATOR

U_GENERATOR

index

E [:,i]

f

Yint

U [:,i]

U[:,1:i-1]

PROJECTIONS

E[:, i]

shift_exp

int2fp

round2int(f)

Yint

MVCA Architecture


23

U[1,4] U[2,4] ... U[p,4] U[1,3] U[2,3] ... U[p,3] U[1,2] U[2,2] ... U[p,2]

E[1,5] E[2,5] ... E[p,5]

x +

x +

/coeff_proj

(i, j-1)enable

‘0’

i = 5

j = 3

iter = 1

acc_U

acc_EU

0 U[1,4] ... U[p-1,4] U[p,4] U[1,3] ... U[p-1,3] U[p,3] U[1,2] ... U[p-1,2]

E[p,5] E[1,5] ... E[p-1,5]

x +

x +

/coeff_proj

(i, j-1)enable

‘0’

acc_U

acc_EU

0 0 ... U[1,4] U[2,4] U[3,4] ... U[1,3] U[2,3] U[3,3] ... U[1,2]

E[2,5] E[3,5] ... E[1,5]

x +

x +

acc_EU

/coeff_proj

(i, j-1)enable

‘1’

acc_U

0 0 ... 0 0 0 ... 0 0 0 ... U[1,4]

E[2,5] E[3,5] ... E[1,5]

x +

x +

/coeff_proj

(i, j-1)enable

‘1’

acc_U

acc_EU

i = 5

j = 3

iter = 2

i = 5

j = 3

iter = p

i = 5

j = 5

iter = p

(a)

(b)

(c)

(d)

𝒑𝒓𝒐𝒋 𝒆𝒌, 𝒖𝒋 =< 𝒆𝒌, 𝒖𝒋 >

< 𝒖𝒋, 𝒖𝒋 >𝒖𝒋; 𝒘𝒉𝒆𝒓𝒆 < 𝑥, 𝑦 > = 𝒙𝒛 ∙ 𝒚𝒛

𝒑

𝒛=𝟏

(3)

IMAGE

PROJECTION

INPUT

MEMORYF_GENERATOR

U_GENERATOR

index

E [:,i]

f

Yint

U [:,i]

U[:,1:i-1]

PROJECTIONS

E[:, i]

shift_exp

int2fp

round2int(f)

Yint

MVCA Architecture


24

U[1,i] U[2,i] ... U[p,i]

+

x +

/coeff_proj

(w,i)

acc_U

acc_wU

x

proj_acc

[1]

proj_acc

[2]...

proj_acc

[p]

+

- f[p] f[p-1] ... f[1]1

𝒇 = 𝒘 − 𝒑𝒓𝒐𝒋 𝒘, 𝒖𝒍

𝒊

𝒍=𝟏

(4)

𝒑𝒓𝒐𝒋 𝐰, 𝒖𝒋 =< 𝐰,𝒖𝒋 >

< 𝒖𝒋, 𝒖𝒋 >𝒖𝒋; 𝒘𝒉𝒆𝒓𝒆 < 𝑥, 𝑦 > = 𝒙𝒛 ∙ 𝒚𝒛

𝒑

𝒛=𝟏

IMAGE

PROJECTION

INPUT

MEMORYF_GENERATOR

U_GENERATOR

index

E [:,i]

f

Yint

U [:,i]

U[:,1:i-1]

PROJECTIONS

E[:, i]

shift_exp

int2fp

round2int(f)

Yint

MVCA Architecture


25

x x x x ...

+ +

+

x x x x

+ +

+

+

...

Comparatorindex

f[1] f[2] f[3] f[4] f[p-3]Yint[1,i] Yint[2,i] Yint[3,i] Yint[4,i] Yint[p-3,i] f[p-2] Yint[p-2,i] f[p-1] Yint[p-1,i] f[p] Yint[p,i]

...

...

...

...

AD

DIN

G

STAG

E

#1

AD

DIN

G

STAG

E

# (

NAD

D -

1)

AD

DIN

G

STAG

E

NAD

D

40

540

1040

1540

2040

3 4 5 6 7 8 9 10

nfl

op

s(V

CA

)/n

flo

ps(

MV

CA

)


R = 100

R = 1000

R = 10000

MVCA is at least 6

times faster than

VCA (standard PC)


26

Results: MVCA vs. VCA computational cost

VCA

(nflops)

MVCA (nflops)

Computation

of the vector f 23p3 2p×R

Projection of

Y onto f

7p+1 if i ≤ 2

7p+1+(i-2)(6p+1) if i > 2 0

MVCA requires at least 40 time

less floating point operations.

More appropriate for hardware

implementations

R = Number of pixels

in the image

Average difference (in degrees) between the RMS values of spectral angles obtained with the VCA and the MVCA algorithms


27

MVCA vs VCA performance (synthetic Images)

p SNR (noise)

10dB 20dB 30dB 40dB 50dB

3 -0.993 0.013 -0.012 -0.012 0

4 0.018 -0.525 -0.052 -0.009 -0.010

5 -0.146 0.874 0.042 -0.135 0.006

6 1.147 -0.496 -0.064 0.424 0.012

7 0.441 -1.190 0.396 -0.047 0.119

8 -0.710 -0.602 -0.128 0.059 0.015

9 -0.491 -1.083 -0.079 0.110 -0.003

10 2.271 0.022 0.073 -0.010 -0.056

3

45

67

89

10

10dB20dB

30dB40dB

50dB

Differences are negligible and noise

independent

0

2

4

6

8

10

12

14

16

18

20

Spe

ctra

l an

gle

(d

egr

ee

s)

max_VCA (PC)

max_MVCA (FPGA)

min_VCA (PC)

min_MVCA (FPGA)

MVCA Vs. VCA performance (Cuprite)


28

Cuprite, from AVIRIS, is a widely used image with

224 bands and includes 14 pure materials

(endmembers)

10000

15000

20000

25000

30000

35000

40000

45000

50000

5 6 7 8 9 10 11 12 13 14 15

Nu

mb

er o

f FP

GA

res

ou

rce

s


Slice registers (fp_MVCA)

Slice registers(int_MVCA)

Slice LUTs (fp_MVCA)

Slice LUTs (int_MVCA)

Two implementations:

fp_MVCA: with floating point

int_MVCA: with fixed point


29

Implementation results

XC5VSX95T

• Int_MVCA requires 16%

less resources

• Endmember extraction for

47,750 hyperspectral

pixels of Cuprite

• int_MVCA = 2.8 ms

• fp_MVCA = 3.6 ms.

• Int_MVCA is 1.3 times

faster

Partners and publications

23/03/2012

4


European project (FP7-SPACE): Flexible on-board HYPERspectral imaging systems for earth observation satellites.

Universidad de Extremadura (Antonio Plaza)

Politecnico di Torino, Italia (Enrico Magli)

Virginia Polytechnic Institute and State University, USA (Peter Athanas)

Surrey Space Center, UK (Craig Underwood)

Deimos Engenharia, Portugal

Taitus Software Italia

Helmholtz-Zentrum Potsdam Deutsches GeoForschungsZentrum, Alemania.

Hyperspectral image compression

Raffaele Vitulli, ESTEC-ESA

Enrico Magli, Politecnico di Torino


31

Collaborations

Hyperspectral image Analysis

Low-Computational-Complexity Algorithm for Hyperspectral Endmember Extraction: Modified Vertex Component Analysis. IEEE Geoscience and Remote Sensing Letters, vol. 9, no. 3, 2012

Parallel Implementation of a Hyperspectral Unmixing Chain: Graphics Processing Units versus Multi-Core Processors. Submitted to IEEE Geoscience and Remote Sensing Symposium (IGARSS'12), Munich, Germany, 2012

High-Level FPGA-Based Implementation of a Hyperspectral Endmember Extraction Algorithm. Submitted to 4rd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS 2012), Shanghai, China, 2012

GPU Implementation of an Automatic Target Detection and Classification Algorithm for Hyperspectral Image Analysis. IEEE Geoscience and Remote Sensing Letters (submitted)

A Novel Architecture for Hyperspectral Endmember Extraction by means of the Modified Vertex Component Analysis (MVCA) Algorithm. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (submitted)

FPGA Design of an Automatic Target Generation Process for Hyperspectral Image Analysis. 2011 IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS)

Novel architectures for real-time matching in hyperspectral images. 3rd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS 2011),

Hyperspectral image compression

Performance Evaluation of the H. 264/AVC Video Coding Standard for Lossy Hyperspectral Image Compression. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Lossy hyperspectral image compression with state-of-the-art video encoder. High-Performance Computing in Remote Sensing (SPIE Remote Sensing 2011)


32

Publications

Instituto Universitario de Microelectrónica Aplicada

(Tecnología de información y las comunicaciones)

HARDWARE IMPLEMENTATIONS

OF HYPERESPECTRAL IMAGE

PROCESSING ALGORITHMS

Developments at IUMA 23/03/2012

HARDWARE IMPLEMENTATIONS OF HYPERESPECTRAL IMAGE ...

Documents

Transcript of HARDWARE IMPLEMENTATIONS OF HYPERESPECTRAL IMAGE ...