14603782 Spatial SQL Who Needs a Traditional GIS

8/3/2019 14603782 Spatial SQL Who Needs a Traditional GIS

1/23

Spatial SQL: who

needs a traditionalGIS?

John Powell

[email protected]


2/23

What is GIS?

Traditionally narrow/specialist area involving expensiveand proprietary software, even though 80% concerned

with just drawing a mapLocation based services, google earth, virtual earth

changing traditional narrow focus

Spatial databases, open source geometry libraries and

powerful graphics packages making GIS much moreaccessible and inexpensive


3/23

The problem with geographic data

GML (geographic markup langauge, an xml dialect) isvendor independent and standards compliant

but staggeringly verbose geographic data tends to bevoluminous anyway

and cumbersome to parse


4/23

Example gml

10096

3

2003-11-27

Land

405.458376

1994-09-21

Reclassified

2002-12-01

Attributes

2003-07-31

Modified

Landform

Slope

Manmade

50


5/23

Gml example continued

607647.760,219141.590 607647.700,219141.310 607647.680,219141.170 607647.670,219141.020 607647.670,219140.880 607647.690,219140.600 607647.750,219140.320607647.790,219140.180 607647.800,219140.140 607647.850,219140.010 607647.910,219139.880 607647.980,219139.760 607648.060,219139.640 607648.140,219139.530607648.230,219139.420 607648.330,219139.320 607648.430,219139.230 607648.540,219139.140 607648.650,219139.060 607648.770,219138.980 607648.890,219138.910607649.150,219138.790 607649.280,219138.740 607649.420,219138.690 607649.550,219138.660 607649.690,219138.620 607649.830,219138.600 607649.980,219138.580607650.120,219138.570 607650.400,219138.570 607650.540,219138.580 607650.680,219138.600 607650.960,219138.660 607651.090,219138.700 607651.220,219138.750607651.340,219138.810 607651.470,219138.870 607651.590,219138.940 607651.700,219139.020 607651.810,219139.110 607651.910,219139.210 607654.220,219139.160607660.620,219137.030 607670.180,219134.320 607679.340,219131.950 607689.710,219128.850 607700.490,219125.660 607712.300,219123.450 607716.870,219122.940607717.040,219122.920 607717.210,219122.920 607717.380,219122.930 607717.540,219122.940 607717.710,219122.970 607717.870,219123.000 607718.190,219123.100607718.350,219123.160 607718.500,219123.230 607718.800,219123.390 607718.940,219123.480 607719.080,219123.580 607719.210,219123.680 607719.340,219123.790607719.580,219124.030 607719.690,219124.150 607719.790,219124.290 607719.890,219124.420 607719.980,219124.560 607720.060,219124.710 607720.200,219125.010607720.300,219125.330 607720.340,219125.490 607720.370,219125.660 607720.370,219125.700 607720.390,219125.870 607720.390,219126.200 607720.370,219126.360607720.350,219126.530 607720.270,219126.850 607720.220,219127.000 607720.160,219127.160 607720.090,219127.310 607720.010,219127.460 607719.930,219127.600607719.840,219127.740 607719.740,219127.880 607719.640,219128.010 607719.530,219128.140 607719.290,219128.380 607719.160,219128.490 607719.030,219128.590607718.750,219128.790 607718.610,219128.870 607718.460,219128.950 607718.310,219129.020 607718.160,219129.080 607718.000,219129.140 607717.680,219129.220607713.970,219130.040 607703.900,219132.940 607692.070,219136.110 607681.520,219137.900 607670.920,219140.090 607650.440,219142.840 607648.390,219142.690607648.290,219142.590 607648.200,219142.480 607648.110,219142.360 607647.970,219142.120 607647.900,219141.990 607647.850,219141.860 607647.800,219141.720

607647.760,219141.590

607653.360,219140.320 607653.390,219140.390 607653.480,219140.550 607653.520,219140.610 607653.630,219140.720 607653.690,219140.770 607653.900,219140.890607653.970,219140.910 607654.050,219140.940 607654.130,219140.950 607654.200,219140.970 607654.360,219140.970 607654.510,219140.950 607654.590,219140.930607654.730,219140.870 607656.500,219140.250 607659.760,219139.920 607664.890,219138.720 607669.360,219138.010 607674.240,219136.800 607677.740,219135.730

607708.090,219128.820 607712.470,219127.550 607712.540,219127.620607712.590,219127.660 607712.650,219127.690 607712.720,219127.720 607712.780,219127.740607712.920,219127.760 607713.130,219127.750 607713.340,219127.690 607713.410,219127.660 607713.470,219127.620 607713.540,219127.580 607713.650,219127.490607713.710,219127.440 607713.810,219127.340 607713.850,219127.280 607713.910,219127.160 607713.940,219127.090 607713.970,219126.960 607713.960,219126.750607713.940,219126.680 607713.920,219126.640 607713.890,219126.570 607713.760,219126.400 607713.650,219126.300 607713.580,219126.270 607713.520,219126.230607713.450,219126.200 607713.240,219126.140 607713.160,219126.130 607713.020,219126.130 607712.870,219126.150 607712.730,219126.190 607712.670,219126.220607712.550,219126.300 607712.490,219126.350 607704.730,219129.000 607680.450,219134.840 607675.310,219135.770 607665.360,219137.480 607655.200,219139.780607655.160,219139.650 607655.120,219139.590 607655.040,219139.490 607654.990,219139.450 607654.930,219139.410 607654.810,219139.350 607654.740,219139.330607654.600,219139.310 607654.300,219139.320 607654.140,219139.360 607654.070,219139.380 607653.990,219139.410 607653.850,219139.470 607653.780,219139.510607653.660,219139.590 607653.600,219139.640 607653.500,219139.740 607653.420,219139.860 607653.390,219139.920 607653.350,219140.040 607653.340,219140.180607653.360,219140.320


6/23

The solution: spatial databases

R-tree indexing massively speeds retrieval

Geometric representation of real-world objects

Numerous standards compliant functions that test spatialrelations or operate on single geometries

Allows analysis to be moved to the db


7/23

R-tree index


8/23

Loading data

Load data infile statement common for point data as x,y thenupdate table using set

geom=geomfromtext(concat('POINT(',x,' ',y,')'));Can use blackhole table with trigger to create geom column,

saving the unnecessary storage of the intermediate x, ycolumns

Or load string representation of geometry into blob/largetext and

update geom column in table with geomfromtextLoad directly using programming language

If data in GIS/CAD format, can load using 3rd-party softwaresuch as FME


9/23

Blackhole loading examplecreate table black_insert (x real, y real, z real) engine = BLACKHOLE;

create table points (geom POINT not null, z real, spatial index (geom))engine=MYISAM;

Delimiter //

create trigger trg_geom BEFORE INSERT on black_insert

for each row

begin

insert into points set geom = geomfromtext(concat('POINT(',NEW.x,' ', NEW.y, ')')),z=NEW.z;

end

//

Delimiter ;

load data infile 'points.txt' into table black_insert fields terminated by ',';

Query OK, 4000000 rows affected, 1 warning (6 min 25.22 sec)

Records: 4000000 Deleted: 0 Skipped: 0 Warnings: 0


10/23

New MySQL spatial functions

Open geospatial consortium defines spatial standards

Until last year, MySQL spatial functionality limited to

minimum bounding rectangle calculations

Thanks to MySQL developer Holyfoot, we now haveproper polygonal functions

See http://forge.mysql.com/wiki/GIS_Functions
http://forge.mysql.com/wiki/GIS_Functionshttp://forge.mysql.com/wiki/GIS_Functions


11/23

GIS example contours

Index on 2.6 billion points with height as attribute

select z from heights where mbrintersects(geom,@geom) order

by X(geom), Y(geom)Produces an ordered array of z values that can be sent to a

contouring algorithm to produce contour overlay in real time


12/23

Importance of indexes

As R-tree index lookup is based on MBRs, for point data the indexwill be bigger than the data

However, without it, on any non-trivial dataset, performance will bevery poor

With polygonal data, index size to data size becomes a (small)fraction rather than a multiple of data size

Only MyISAM supports spatial index

Could use Innodb master (insert) and MyISAM slave (select), ifconcurrency less important than data integrity


13/23

Spatial data types

Spatial data types: point, line, polygon, multipoint, multilinestring,multipolygon, geometry, geometrycollection

Geometry column can contain any of the other typesGeometrycollection can hold a combination, eg, set

@g='GEOMETRYCOLLECTION(POINT(0 0),POLYGON((0 0,10 0, 10 10, 0 10, 0 0)))';

Polygon cannot have any intersecting lines, must be closed, may

have any number of inner rings, which can only touch outerring at a single point


14/23

Spatial functions for plotting

java.awt.geom.Area which implements java.awt.Shape interfaceallows the subtraction of one geometry from another

Can be used for plotting polygons with inner ringsOr can cheat, by sorting on area of outer ring, ensuring polygons

drawn in correct order. Quicker

Set @bbox=geomfromtext('POLYGON((tlx tly, brx tly, brx bry, tlx bry, tlxtly))');

select intersection(@bbox, g.geom) from geomtable g whereintersects(@bbox, g.geom) and geometrytype(g.geom)='POLYGON' order byarea(polyfromwkb(polygon(asbinary(exteriorring(g.geom)))));


15/23

Spatial functions houses with flood risk

select geom into @pcode from codePoint.polygons wherePOSTCODE='CO5 7QL';

select `os address toid` from addresslayer.addresslayer_geomal,(select intersection(fl.geom,@pcode) as geom fromea.floodzone_combined fl where intersects(fl.geom,@pcode)) as flwhere intersects(al.geom,fl.geom) andintersects(@pcode,al.geom);


16/23

Indexes

Always check index use with explain

Spatial joins different to the more usual B-tree join

Beware of derived tables as index will not be used,resulting in horrific performance


17/23

Spatial functions land coverage

Show percentage of different land coverage types withina buffer zone

Set @buff=buffer(geomfromtext('POINT(520000, 185000),500);

select round(sum(area(intersection(a.geom,@buff)))/1000000,3) askm_sq, a.descriptivegroup from attributes a whereintersects(a.geom,@buff) and geometrytype(a.geom) ='POLYGON'

group by a.descriptivegroup order by km_sq desc;


18/23

Introducing time

Increasingly spatial databases have a temporal component epidemiology, crime analysis, climate change, location based

services, land use change, etcPartitioning can improve performance considerably

But... can't partition on table with geometry columns


19/23

Boats servicing windfarms

Boats use a system called automatic identificationsystem (AIS)

Uses a form of base 64 to encode info, eg,

!AIVDM,1,1,,A,18Hv1j002EP4QntMMN@C32M828Dv,0*13

This contains the boats position, course, speed and otherdynamic variables

A second message contains static information, such asship name, destination, length, call sign

!AIVDM,1,1,,A,18Hv1j002EP4QntMMN@C32M828Dv,0*13
mailto:18Hv1j002EP4QntMMN@C32M828Dvmailto:18Hv1j002EP4QntMMN@C32M828Dv


20/23

Boats tails

Get boat trails position of all boats going back two hours, butonly those seen in last 30 seconds to avoid orphaned trails

select a.mmsi, a.lat, a.lng,cast(a.time as decimal(14)) from ais aright join (select mmsi from ais where timedate_sub(@dt, interval 30 second) group by mmsi) as aston a.mmsi=ast.mmsi where a.time>date_sub(@dt, interval 2

hour) and a.time


21/23

Boats contd.

Current boat position plus static information involves joinon derived table and spatial position. Can't partition

with spatial dataSolution partition using to_days function on month with

datetime, shipID and ID (auto_inc) as keys

Create 2nd table with lat/lng as point. Add spatial index

with ID (auto_inc) as keyThen can do spatio-temporal query with efficient index

usage


22/23

Distributed GIS

Web or web-initiated front end (eg, java web start)

+ Graphics libraries and threading

+ MySQL spatial backend with partioning, replication,clustering

+ Open source math/geometry libaries

= Low cost, powerful distributed GIS


23/23

Future direction

Non-projected coordinate systems currently distance function

only accurate for projected dataExtra geometry functions, eg, translate, rotate, generalize

3D types/indexing

Partitioning on time but with spatial datatypes

Output to KML/GMLPresenter finishes 10-year plan and finally learns C++

14603782 Spatial SQL Who Needs a Traditional GIS

Documents

Transcript of 14603782 Spatial SQL Who Needs a Traditional GIS