Carnegie Mellon Ambient Intelligence...

Spatiotemporal Data Mining for Monitoring Ocean Objects

Carnegie MellonCarnegie MellonCarnegie MellonCarnegie MellonCarnegie Mellon Ambient Intelligence Lab

Yang Cai, Ph.D.Ambient Intelligence Lab

Carnegie Mellon [email protected]

Collaborators


Karl Fu, Carnegie MellonXavier Boutonnier, Carnegie MellonDaniel Chung, Carnegie MellonRichard Stumpf, NOAATimothy Wynne, NOAAMitchell Tomlison, NOAAJames Acker, GSFCCynthia Heil, FWRI Y. Hu, LaRC


Spatiotemporal dynamics of surface objects is a part of our everyday-life, from “red tide”, river plume, vessels, flood, urban growth, to urban traffic congestion.

Scientific Questions

Tracking:

Given an object in an image sequence (t=1,..,n), how to find the object at t=n+1 and beyond?

Prediction:

Given databases of historical data and current physical and biochemical conditions, how to predict the occurrence of the interested object at a particular time and location?


Why do we need data mining here?


1. Lots of historical data (8-year SeaWiFS and40-year cell count and wind databases)

2. Current models haven’t been successful.

3. Data mining appears to be inexpensive.

4. Most of domain experts still spend 80% time to do ‘manual’ data mining.

Challenges to Data Mining Technologies


• visual or hyper-spectrum content

• space and time

• deformation + transport

• missing data (80% clouds in images)

• multiple databases

• multi-physics

Objective

Key Milestones

• Motion tracking model 6 mo/1 yr• Case studies with plume/HAB 6 mo/1 yr• Spatiotemporal motion model 6 mo/2 yr• Spatial frequency pattern model 6 mo/2 yr

• To track the movement of ocean objects that have been identified

• To predict the movement of identified objects.

Sponsored by NASA ESTO-AIST-QRS-04-3031

Co-I/Partner

Co-I, Richard Stumpf, NOAA

Approach

spatiotemporal object motion tracking

TRLin = 4

Spatiotemporal Data Mining System for Tracking and Modeling Ocean Object Movement

• Computer vision and visualization• Statistical spatiotemporal data minining.• Case studies with SeaWiFS datasets.


V.M.S.V. Methodology

Vision:

• Spatial Density Filter

• Correlation Filter and Particle Filter

• Mutual Information

Mining:

• Spatiotemporal Neural Network

• Spatiotemporal Bayesian Model

• Periodicity Transform

Simulation:

• Cellular Automata

Visualization:

• Pseudo Color, Mapping, Animation...

Images above show a harmful algae bloom (HAB), highlighted as chlorophyll anomaly, drifting along the southwest Florida coast in December 2001.

Case Study:

Harmful Algal Blooms


Correlation Study

Use chlorophyll as a surrogate for Karenia Brevis blooms (NOAA)

References:

Tomlinson, M.C., R.P. Stumpf, V. Ransibrahmanakul, E.W. Truby, G.J. Kirkpatrick, B.A. Pederson, G.A. Vargo, C. A. Heil., 2004. Evaluation of the use of SeaWiFS imagery for detecting Karenia brevis harmful algal blooms in the eastern Gulf of Mexico. Remote Sensing of Environment, v. 91, pp. 293-303.


Chlorophyll channel Anomaly channel


Image Data Preprocessing

• Remove noises in the satellite images

• Grouping objects

• Recover the missing data


Spatial Density Clustering

1. Set all the neighboring dots within a distance (Dis) as one test set 2. If number of dots > Min, then the point is a core point 3. Remove the non-core points 4. Go to step 1

Dis


Sample of SDC vs. Binary Morphology

noisy image SDC output Morphology output


Missing Data Recovery

1. Which is which?

2. Concavity of objects.


Interpolation of a convex object

We take all the points of the contours of the marginal objects and by linear interpolation calculate the position of the interpolated point.

The Hull Convex of the interpolated points gives the contour of the interpolated convex object.


Work around concavity

1. First we extract the concavity.2. Then we interpolate the object and the

concavities.3. Then we remove the part

corresponding to the interpolated concavity from the interpolated object


Results


Shape Grouping Results


Correctly Marked Surface Objects (HAB)


Object Tracking with Correlation Filter

where,

a is the test image

b is the reference object in the previous image to be tracked.

FFT(x) represent Discrete Fast Fourier Transform

IFFT(x) is Inverse Discrete Fast Fourier Transform.

Shape Correlation = IFFT(FFT(a).* conj(FFT(b)))


Tracking Results


Tracking HAB with Correlation Filter within 4-day interval


Tracking of a bloom which has split into 2 pieces sampled in an interval of 4 days using particle filter


River plume tracking with Mutual Information


Neural network spatiotemporal prediction model

Radial Basis Function

Kohonen Network

raw image sequence

clustered points **

predicted image for t+1

** from Cartesian to polar coordinates

r

θ

point in frame t-1

point in frame t


Neural Network Prediction Model Example

• The first 3 images are input shapes (zero wind speed)

• The last image is the predicted shape overlaid on top of the ground truth.

• The blue dots are the clustered data points that represent the shape.

** Reference: Mitchell, T. Machine learning, McGraw-Hill, 1997


Prediction model results


Prediction model results

Cluster Resolution vs. Time

0.00

100.00

200.00

300.00

400.00

500.00

0 500 1000 1500 2000 2500

Cluster Resolution (clusters)

Tim

e (C

PU ti

me)

time


Periodicity Transform

26.4526.5

26.5526.6

26.65

-82.4

-82.3

-82.2

-82.17.2

7.25

7.3

7.35

x 105

latitude

Space time coordinates of the measurements

longitude

date

0 2000 4000 6000 8000 10000 12000 140000

1

2

3

4

5

6x 105

0 100 200 300 400 500 600 7000

0.05

0.1

0.15

0.2

0.25

Periodicity

Pow

er

Power of the components of periodicity transform of a medium set of data

2 12 2

20

1 ( )p

iEnergy x x i

p

−

== = ∑


Experiments with cell count data


Zoomed result


Super-zoomed results


Cellular Automata

CA is a two-dimensional simulation of surface physics, chemistry or biology. It’s simple however, it could be computationally expensive for large problems.


Visualization of the prediction model


3-D Stereo Projection Lab



V.M.S.V. Methodology

Vision:

• Spatial Density Filter

• Correlation Filter and Particle Filter

• Mutual Information

Mining:

• Spatiotemporal Neural Network

• Spatiotemporal Bayesian Model

• Periodicity Transform

Simulation:

• Cellular Automata

Visualization:

• Pseudo Color, Mapping, Animation...

Conclusions

1. Data mining without domain expertise is like fishing-in-the-dark. Multiple experts are needed.

2. Vision-Mining-Simulation-Visualization (VMSV) method puts human experts in the loop, which increase the chance of success.

3. Vision algorithm enables automated image-based data mining.

4. Neural network model shows promising in compressing the shape information in orders of magnitude. However, it has its limitations in long-term prediction.

5. Bayesian prediction shows promising in long-term prediction and efficient computational speed.

6. How to couple the multi-physics models with data mining models is a big challenge.


Publications

1. Y. Cai, R. Stumpf, etc. Spatiotemporal Data Mining for Prediction of Harmful Algal Blooms, International Harmful Algae Conference, Copenhagen, September 8-12, 2006

2. Y. Cai, Y. Hu, Onboard Inverse Physics from Sensor Web, Proceedings of Space Missions and Challenges, SMC-IT, JPL, 2006

3. Y. Cai and K. Fu, Spatiotemporal Data Mining with Cellular Automata, Proceedings of International Conference of Computational Science, ICCS 2006, May 30, UK

4. Y. Cai, D. Chung, K. Fu, R. Stumpf, T. Wynne, M. Tomlinson, Spatiotemporal Data Mining with Micro Visual Interaction, submitted to Journal of Knowledge and Information Systems

5. Y. Cai, K. Fu, R. Stumpf, T. Wynne, M. Tomlinson, Spatiotemporal Data Mining for Monitoring Ocean Objects, submitted to NASA Data Mining Workshop, JPL, 2006

6. Y. Cai, Y. Hu, Sensory Stream Data Mining on Chip, submitted to NASA Data Mining Workshop, JPL, 2006

7. Y. Cai, (editor), Special Issue of Visual data Mining, Journal of Information Visualization, to be published by Elsevier, 2006

8. Y. Cai and J. Abascal, (editors), Ambient Intelligence in Everyday Life, Lecture Notes in Artificial Intelligence, LNAI 3864, to be published by Springer, April, 2006


Acknowledgement

The study is supported by NASA ESTO grant AIST-QRS-04-3031. We are indebted to our collaborators in NOAA, FWRI, GSFC. The authors appreciate the comments and suggestions from Karen Meo, Kai-Dee Chu, Steven Smith, Gene Carl Feldman and James Acker from NASA. Also, many thanks to Christos Faloutous and Mel Siegel from CMU and Andrew Moore from Google Research Center in Pittsburgh.


Carnegie Mellon Ambient Intelligence...

Documents

Transcript of Carnegie Mellon Ambient Intelligence...