[email protected] 1 Automated analysis of Spatial Grids [email protected] Part of the...

[email protected] 1

Automated analysis of Spatial Grids

[email protected] of the short course on Artificial Intelligence at the American Meteorological Society Annual Meeting, Jan. 2006, Atlanta.

[email protected] 2

Information from grids

Observed data are often on gridsSatellite imageryGround-based radarOther data can be gridded

You may need to extract information from these grids Is there a boundary?Where are the troughs?

[email protected] 3

Spatial grids as images

Our spatial grids are often two dimensional

Automated extraction of information from images is a long-established field in machine intelligenceWe can treat spatial grids as imagesAnd apply image processing techniques to

our data.

[email protected] 4

Image processing terminology

What you think of it as

What they call it Where is it used?

Grid Image

Grid point Pixel

Finding signatures

Pattern recognition Identifying machine parts on an assembly line

Finding storms Segmentation (ditto)

Tracking storms Optical flow MPEG compression

Smoothing, finding changes

Filtering Photo-editing software

Combining gridded data

Stereography Robotics

[email protected] 5

Other terms for this field

There are subtle differences between image processing andData mining

Relationships between identified signaturesKnowledge discovery

A human draws the inferencesComputer Vision

Pattern recognition when it doesn’t work

[email protected] 6

Limitations of image processing

Scenarios where pattern recognition succeeds: Highly controlled environments

Assembly lines, robotics Photography

Red, green, blue channels can be assumed independent IR channels can not!

Objects have sharp boundaries Storms, etc. do not

The most mature sub-fields Filters (smoothing, edge finding) Segmentation

[email protected] 7

Workflow of AI applications

Artificial Intelligence applications that operate on spatial data usually follow these steps:

Gridding: take spatial data and make it a locally uniform resolution grid Filtering: smoothing, pattern matching, etc. Edge finding: find areas of drastic spatial change Segmentation: find contiguous areas of constant value Feature extraction: find features based on edges/regions Classification: classify features using a rule base, neural network, support vector

machine, etc. Normally done with all the features from all your data fields This will be covered by the other speakers

Gridding Filtering

Edge finding

Segmentation

Featureextraction

Classification(NN, etc.)

[email protected] 8

A note on convention

Image processing uses matrix notation (0,0) is the top-left corner of the image (0,1) is the first row, second column.

As long as you don’t think of the first index as “x” and the second index as “y”, you will be okay In what direction does the first index

increase?Answer: South or downwards

[email protected] 9

Image processing assumptions

Traditionally image processing assumes a Markov random process Two pixels are correlated if they are adjacent to

each other If you have pixels: a b c a and b are correlated as are b and c But the correlation between a and c is captured solely by

the correlation between a-b and b-c No “second-order” correlation term.

A simplifying assumption that usually causes no problems.

If your field is random (no correlation at all) or if the correlation is second-order image processing techniques may not work

[email protected] 10

Image processing assumptions

An assumption that could cause problems: The field has uniform resolution Projected spatial grids are not uniform resolution.

Locally, most projected grids can be assumed uniform. Pixels 10 km apart don’t vary that much in size. Be careful with global operators on projected grids

Segmentation is a global operation

Filtering, edge finding, etc. are fine. Filters are local operations.


Radar data

Radar data are NOT uniform resolution Size of a “pixel” varies with range Dramatic changes

Two options: Use varying-size windows for local operators

Hard to keep track! Convert to a Cartesian grid before applying image

processing techniques Will cause problems for local operators at far ranges

where data are “repeated” This is an issue with most projected data as well


Missing Data

Some times, you will have spatial data where some pixels are missingCan you treat this data as some value?

Say zero?Will simplify your task considerablyCan dramatically speed up operations

If not, you can vary the formulae to use only those pixels for which you have data

Filtering results may have higher qualityOperations will be much slower


Summary: Spatial grids

To treat spatial grids as images, Operate on the raw, un-projected data

Less chance of repeated values For “gridded” data, operate on grids at the resolution

closest to half the spacing between observations.

Change “missing” data to background value if available.

Use local operators (e.g: filters) freely Be careful with global operations (e.g: segmentation)

Convert computed features (such as area) to projected systems


Gridding

From insitu observations to grids


In-situ observations

Some environmental data are measured by in-situ instruments Not remotely sensed These measurements

are at geographically dispersed points

Need to be converted into grids


Gridding observations

In-situ observations have to be interpolated on to grids to process as imagesWhat is the pixel resolution?What interpolation techniques are

available?


Pixel resolution

If the chosen pixel resolution is too coarse, lose some of the observations

If the chosen pixel resolution is too fine, strong gradients may not be apparent

A good rule of thumb:Choose pixel resolution to be half the mean

distance between observation points


Interpolation methods

Interpolation methods:Cressman analysisBarnes analysisKriging I_xy

I_i


Cressman analysis

The value at each pixel of grid:

Every observation gets weighted based on its distance from grid point R is the “radius of influence” Higher the R, the more distant points are

considered

i iyyixxR

iyyixxR

iiyyixxR

iyyixxR

iI

xyI2)(2)(2

2)(2)(2

2)(2)(2

2)(2)(2


Barnes analysis

The problem with a Cressman analysis:Even if a pixel is collocated with an

observation, the final value of pixel will be different

Due to (small) effect of pixels further awayBarnes analysis tries to account for this


Barnes analysis equation

Barnes also changed the Cressman interpolation function to be exponential Smoother than

simple polynomials

i

iyyixx

i

iyyixx

i

e

eI

xyI22

2)(2)(

22

2)(2)(


Barnes analysis: the technique

Perform Cressman analysis on obs Compute errors at observation points Perform Cressman analysis on errors

Give the error field a weight Add the two fields One pass of Barnes analysis

Can repeat Barnes analysis N-pass Barnes analysis Closer and closer to the value of the observations

at the observation points.


Kriging

Kriging is a principled method of interpolation Compute correlation between observations Use variogram to determine scaling parameter Works only if the observations are very close to

each other M. A. Oliver and R. Webster "Kriging: a method of

interpolation for geographical information system", Int’l. J. Geographical Information Systems, 1990, VOL. 4, No. 3, 313-332


Convolution Filters

A surprisingly powerful local operation


Smoothing: an intuitive look

How would you smooth the image on the top right? So that the edges

aren’t quite so ragged?

To get an image like bottom right?


Averaging is a smoothing operation

The most obvious way to smooth is to replace each pixel by a local average:

I used k=3 in the WDSS-II filter “average” http://www.wdssii.org/ The dataset I’m using for

demonstration is available online. See:

http://cimms.ou.edu/~lakshman/aitut/

kxi

kxi

kyj

kyj

ijkkxy II )12).(12(1


How it works

What does this equation mean?

For every pixel x,y, look in a 2D neighborhood, add up all the values and divide by the number of pixels.

(2k+1) is the neighborhood size k is the half-size Because the neighborhood size

needs to be odd, most software will ask you for the half-size, k

kxi

kxi

kyj

kyj

ijkkxy II )12).(12(1


Convolution

Another way to think about smoothing: At each pixel, you

assign weights to the neighboring pixels.

And compute a weighted average.

W is a (2k+1)x(2k+1) matrix shown on right. Called the convolution

matrix or kernel Technically, this is cross-correlation, but if you

are using symmetric kernels (as we almost always will be), cross-correlation and convolution are the same thing.

kxi

kxi

kyj

kyj

ijijxy IWI

111

111

111

9

1W


Smoothing kernels

The “boxcar” kernel Poor characteristics in

transition areas. More spatial relocation of

maxima Smudges more

A better convolution kernel is where the kernel function drops off with distance.

A Gaussian convolution kernel offers the best space/frequency trade off

Frequency = raggedness Space = location

You can smooth a lot more effectively because the features don’t move or smudge as much.

Gauss:9

Average:3


Gaussian Kernel

2

2)(2)(

22

1x

yx

eWxy

x and y range from –k to k. Since the Gaussian has infinite range, you need to choose k

appropriately. Choosing k to be 3 * largest sigma is a reasonable choice.

The sigmas are the scale Divide by total weight (The W above doesn’t add up to 1)


Convolution kernels

By changing the weights in the convolution kernel, you can extract many features.

101

101

101

3

1W


Matching filters

In general, make your convolution kernel be the feature you are trying to extract.

The above kernel returns large magnitudes for thin vertical lines But not just thin vertical lines Need to normalize by smoothed value also

Can use the matched filter idea to find objects

101

101

101

3

1W


Scale and rotation

Convolution filters are: Scale dependent

What if the gradient is 5 pixels apart? Need a matching filter with columns 5 apart!

Orientation dependent What if the boundary is horizontal? Need a matching filter with that orientation.

Remember typical usage? Do you know the size of a machine part before

hand? How about a squall line?


Scale and orientation

)(

22

22

2)sin(2)cos(

1 yx

yx

eWyx

xy

If you specify your convolution filter as Gaussian functions, it is easy to specify rotation and scale. x and y range from –k to k.

Since the Gaussian has infinite range, you need to choose k appropriately.

Choosing k to be 3 * largest sigma is a reasonable choice. Theta is the orientation from x-axis (anti-clockwise from south) The sigmas are the scale (use same sigma for circular kernels;

different sigmas for elliptical ones) Divide by total weight (The W above doesn’t add up to 1)


Speed

The larger the scale of interest, the larger the half-size of the kernel. Will take a long time to compute. Since we need to compute weighted average at

every pixel.

Two speed ups possible Convolution can be performed in Fourier Transform

space. Convolution can be performed using separable

filters.


Convolution with Fourier Transforms

To convolve with Fourier Transforms: Compute 2D FFT of original image

Can not have missing data Fill with background value somehow

Compute 2D FFT of convolution kernel The convolution kernel has to be same size as image (pad with zeros)

Multiply the FFT values pixel-by-pixel Take inverse 2D FFT of result.

The larger your kernel, the more dramatic the speed up. Can be 100-200 times faster. Seconds instead of hours!

Resources fftw is a free, nice Fourier Transform library. Lakshmanan, V: 2000: Speeding up a large scale filter. J. of Oc. and

Atm. Tech., 17, 468-473


Convolution with separable filters

What if you have missing data? Can not use FFT method Problem with radar data in particular.

If the convolution filter is “separable”: Compute weighted average row-wise on original

image. Compute weighted average on the row-wise result

column-by-column. If your filter was 129x129, 60 times faster than

doing the standard way.


What is a separable filter

A filter is separable if

For example, if the convolution filter is a simple Gaussian:

If you want to vary orientation, see: Lakshmanan, V 2004:

A separable filter for directional smoothing. IEEE Geosc. and Remote Sensing Letters, 1, 192-195

2

2

2

2

2

22

1112

yxyx

eeeW

WWW

xy

yxxy


Edge finding

The “line-finding” filter is not a robust way to find edges

Similar problems to using boxcar for smoothing

Use Laplacian of a Gaussian (Canny)

Differentiate the smoothing kernel twice.

Convolve image with that kernel

Look for zero crossings in the convolution result.

J. Canny, ``A computational approach to edge detection", IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(6),pp679-698 (1986)

101

101

101

3

1W

2

22

)2

1(

2

22

yx

eyx

Wxy


Multi scale or orientation analysis Use filter banks:

To perform multi scale (multi resolution) or orientation analysis A bunch of filters at different scales and orientations Combine their results by a weighted average or maximum

Drawbacks: Repeated convolutions on original image. Results of the filters are not related. Can get very expensive

Simplifications possible: We can steer the results based on a few filters The resulting features are related

The squall line contains storm cells So far, we have only considered image-to-image

The resulting images are related. Use wavelets

Image

Filter 1

Filter 2 Pixel operation

Filter N

Result


Wavelets

Wavelet analysis breaks up images smoother and smoother results on the right

(s0,s1) The lost “detail” information on the left

3 images of the size of the corresponding s Can not use any old filter

So that s1 is half the size of s0 Convolution filter has to be a wavelet function

Haar function (boxcar) Daubechies p-functions Many others

Fast Wavelet transformations exist Use wavelets only if you need this

decomposition Otherwise, use simple multi scale analysis

with lots of Gaussian filters. Gaussian filters have lots of nice properties. Plus everybody knows what they are!

S0 (NxN)

D1 (N/2 x N/2)3

S1 (N/2 x N/2)

D2 (N/4 x N/4)3

S2 (N/4 x N/4)

D3 (N/8 x N/8)3

S3 (N/8 x N/8)


Texture

In convolution, we compute weighted average of the original image’s pixels

Can also do other operations inside neighborhoodTexture operatorsOperators based on relative ordering


Texture operators

Texture is a good way to distinguish different objects Not just on data value, but on distribution of data values Lakshmanan, V., V. DeBrunner, and R. Rabin, 2000: Texture-

based segmentation of satellite weather imagery. Int'l Conference on Image Processing, Vancouver, 732-735.

2nd and 3rd order statistics Variance, standard deviation Homogeneity

Entropy Shannon’s measure of information content Bin the data values and form a histogram P is the fraction of pixels in each bin

i ii pp )log(


Percent filters

Operators based on sorting Take the median value

Excellent smoothing filter Optimal when you have speck noise

Take the maximum value Effect of dilating the field spatially Minimum erodes the field spatially

Use percentiles 50% is the median (maximum noise reduction) 0% is the min (heavily noise dependent) 100% is the max Intermediate values trade off on noise reduction


Dilation and Erosion filters

Using the second-highest or second-lowest value in the neighborhood is an excellent dilation/erosion filter.

Implemented this way in WDSSII.

origerode

dilate


Filter Banks and Neural Networks

Classification using NN Use filter banks Convolution filters Texture operators Percentile operators Each pixel provides a

pattern May have to select pixels –

not use all of them Since a 1000x1000

image has 1 million patterns.

But is only data case

Image

Filter 1

Filter 2 Neural Network

Filter N

Result


QCNN

Features computed in local neighborhoods of radar gates

Classified by NN Returns [0,1] Is it perfect?

If we use this directly as a mask, there will be holes in the result!

Lakshmanan, V., A. Fritz, T. Smith, K. Hondl, and G. J. Stumpf, 2005: An automated technique to quality control radar reflectivity data. J. Applied Meteorology, subm.

Effect ofPre-processing(don’t care)

NN randomerror

Effect of postprocessing

QC’ed

Original


Actual QC technique

Effect ofPre-processing(don’t care)

NN randomerror

Effect of postprocessing

QC’ed

Original


Images to Objects

So far, we have looked at ways of processing imagesResult is still image

Bunch of pixelsNeed to group pixels together into objects

Classify, track, etc. these objects if possible.


Segmentation

From images to objects


What is segmentation?

Finding groups of pixels that together comprise an object.

This is how we (humans) make sense of most images.

Very hard for a computer algorithm.Simple region-growing approachComplex vector segmentation approach


Binary images

Region growing can be applied only to binary images.Apply to images that can be thresholdedPixels below (above) a threshold are not of

interest.Your image now has only 0’s (not

interested) and 1’s (interested)Threshold dependent on application


Hysterisis

Alternative is to employ “hysterisis” You are interested in areas > 30 dBZ (T_i –

threshold of interest) But if you see a 30 dBZ pixel, you lose interest only

if pixel value < 20 dBZ (T_d – threshold of drop off) Less chance of too many disconnected regions. Employed in QCNN

Lakshmanan, V., A. Fritz, T. Smith, K. Hondl, and G. J. Stumpf, 2005: An automated technique to quality control radar reflectivity data. J. Applied Meteorology, subm.

Choose T_i and T_d by experiment.


Region growing algorithm

Initialize: label, an image with all pixels = 0. curr_label = 0

Walk through image pixel-by-pixel. If label(x,y) = 0 and image(x,y) > T_i

Increment curr_label Perform region growing at x,y with curr_label

Region growing steps: Set label(x,y) = curr_label Check all 8 neighbors (i,j) of (x,y)

If image(i,j) > T_d, perform region growing at i,j Two pixels with the same label are part of the same region.

Can compute region properties based on all the pixels within a region.


Vector segmentation

Region growing:Can lead to disconnected regions

Even with hysterisisNot hierarchical

Can not always use hierarchy of thresholds

Need a segmentation approach that:Can avoid disconnected regions Is hierarchical


Watershed Segmentation

Watershed segmentation Sort pixels in ascending order

of magnitude; then, flood the plain:

Vincent, L., and Soille, P., Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations,PAMI(13), No. 6, June 1991, pp. 583-598.

Use saliency to form hierarchy

Watershed segmentation works very poorly in images with statistical noise (most meteorological images).


Contiguity-enhanced segmentation

Explicitly trade-off contiguity of region with the idea that a region’s pixels ought to be similar Similarity of pixels based on a number of texture or

wavelet features Pixels are dissimilar if vector distance between

features is large Lakshmanan V. 2001: A Heirarchical, Multiscale

Texture Segmentation Algorithm for Real-World Scenes. Ph.D. thesis, U. Oklahoma, Norman, OK

The WDSS-II algorithm w2segmotionll implements this segmentation approach http://www.wdssii.org/


K-Means Segmentation

Lakshmanan, V., R. Rabin, and V. DeBrunner 2003: Multiscale storm identification and forecast. J. Atm. Res., 367-380


Hierarchical Segmentation


Incorporating time


Frames and sequences

A spatial grid at a point time is called a “frame”

A bunch of frames form a sequenceTerminology from video

Focus of most image processing researchUsed for security or monitoring applications


Detecting change

To detect change between framesSimply difference the images pixel-by-pixel

Captures edges of moving objectsWorks well on video

Truck enters scene

Doesn’t work well when the objects themselves are changing

Such as storms


Object association

One alternative is to associate objects Segment the frames Try to associate objects in one frame to next based

on: object similarity (use texture features) object overlap (fraction of pixels in common) Closeness of centroids

A linear programming problem Dixon, M. and Wiener, G., 1993: TITAN: Thunderstorm

Identification, Tracking, Analysis and Nowcasting: A Radar-Based Methodology. J. Atmos. Oceanic. Tech, 10, 6


Object association (contd.)

Advantages of object association:Can extract underlying trends.Can advect each storm individually.

Disadvantages of object association:Segmentation results can change from

frame to frame.Splits/merges can not be handled easily.Accurate only for small stormsWorks only at one scale (not hierarchical)


Optical flow

Another way is to treat the images are 2D fluid fields Find movement that minimizes some cost measure

to move from one frame to another. Cross-correlation

Implemented as w2imgmotion in WDSS-II Lagrangian motion

Turner B., Zawadzki, I. and Germann, U. 2004: Predictability of Precipitation from Continental Radar Images. Part III: Operational Nowcasting Implementation (MAPLE), J. Applied Meteorology, 43, 2, pp.231-248

Typically more accurate than object association. Can extrapolate motion vector to get forecasts. No way to extract trends for individual storms


Hybrid technique

Object association gives us tracks/trends Optical flow gives us prediction accuracy A hybrid technique

Identify objects at current frame Match to image (not objects) in previous frame using cost

measure Extract trends from objects at current frame and images from

previous frames based on movement of object Lakshmanan, V., R. Rabin, and V. DeBrunner 2003:

Multiscale storm identification and forecast. J. Atm. Res., 367-380 Implemented as w2segmotion in WDSS-II


Hybrid technique (contd.)

Advantage of hybrid techniqueCan use shape of identified storms to track

other fields Identify storms on radar imagesTrack satellite cloud-top temperature within

stormPossible because the exact shape of the

storm is knownThough it changes from frame to frame.So item of interest should not be at edges.


GIS systems

Trends can include geographic propertiesJust grid the GIS data into the same

resolution at your spatial gridsThen, you can find the total population

affected by a storm, for example.Treat the GIS data as just another spatial

grid.


Polygons

Gridding the GIS data is not always practical.County boundaries are polygons.Want to find things that happen within that

polygonMaximum hail within that polygon

How can we extract spatial properties within a earth-based polygon? Implemented as w2alarm in WDSS-II


Polygon in/out tests

Put the polygon in the gridded reference

The polygon corners have to be converted to (x,y) locations.

We’ll assume the polygon is completely inside the grid.

Walk row-wise from the grid boundary to the

pixel of interest. count the number of times you

cross the sides of the polygon Odd: the pixel is inside the

polygon Even: the pixel is outside the

polygon Repeat column-wise

Has to be inside in both directions.

1

2

1

2

0

3


Summary

We looked at how image processing plays an important role in most AI applications on spatial grids.

Filtering Finding edges Finding objects

Also at purely image-driven applications Tracking Extraction of spatial properties.

Gridding Filtering

Edge finding

Segmentation

Featureextraction

Classification(NN, etc.)


Thank you!

http://www.cimms.ou.edu/~lakshman/

[email protected]

http://www.wdssii.org/

[email protected] 1 Automated analysis of Spatial Grids [email protected] Part of the...

Documents

Transcript of [email protected] 1 Automated analysis of Spatial Grids [email protected] Part of the...