Assuring spatial and temporal data

integrity via constraints and triggers in

PostgreSQL

Adrian C. Prelipceanacpr@kth.se

adrianprelipceanc@gmail.com

Ego page

Adrian C. Prelipcean

1. PhD student at KTH– Transportation systems and geoinformatics– Data collection and analysis methods for travel behaviour– Lab responsible for multiple courses

I GIS Algorithms (Java)I Spatial Databases (PostgreSQL, PostGIS, pgRouting)I Web and Mobile GIS (NodeJS, ExpressJS, Apache

Cordova, PostgreSQL)

2. Tech responsible at Airmee– Airmee - a smart platform for urban logistics– Currently working on MVPs (from the backend to the

frontend)– PostgreSQL as the backend

2

Background

Travel behaviour - understand how people travel and makedecisionsWe used to ask people how they are travelling - fill in paperforms, answer telephone interviews, fill in web forms.

I Found out how to adapt the infrastructure of our cities tohandle a growing population.

I However, also found out that:

– Physical mail is not that popular.– Filling up forms is not really a hobby.– Method doesn’t scale well.

3

Background

Travel behaviour - understand how people travel and makedecisionsWe used to ask people how they are travelling - fill in paperforms, answer telephone interviews, fill in web forms.

I Found out how to adapt the infrastructure of our cities tohandle a growing population.

I However, also found out that:– Physical mail is not that popular.

– Filling up forms is not really a hobby.– Method doesn’t scale well.

3

Background

Travel behaviour - understand how people travel and makedecisionsWe used to ask people how they are travelling - fill in paperforms, answer telephone interviews, fill in web forms.

I Found out how to adapt the infrastructure of our cities tohandle a growing population.

I However, also found out that:– Physical mail is not that popular.– Filling up forms is not really a hobby.

– Method doesn’t scale well.

3

Background

Travel behaviour - understand how people travel and makedecisionsWe used to ask people how they are travelling - fill in paperforms, answer telephone interviews, fill in web forms.

I Found out how to adapt the infrastructure of our cities tohandle a growing population.

I However, also found out that:– Physical mail is not that popular.– Filling up forms is not really a hobby.– Method doesn’t scale well.

3

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way

– Ask people to fill up forms

I New way

– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0

– Use [buzzwords] machine learning / AI to avoid asking peopleto fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way

– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0

– Use [buzzwords] machine learning / AI to avoid asking peopleto fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way

– Ask people to look at their GPS points and fill up forms(?)I New way 2.0

– Use [buzzwords] machine learning / AI to avoid asking peopleto fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way

– Ask people to look at their GPS points and fill up forms(?)I New way 2.0

– Use [buzzwords] machine learning / AI to avoid asking peopleto fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0

– Use [buzzwords] machine learning / AI to avoid asking peopleto fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0– Use [buzzwords] machine learning / AI to avoid asking people

to fill up forms

– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0– Use [buzzwords] machine learning / AI to avoid asking people

to fill up forms– But we need data to train the algorithms

– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0– Use [buzzwords] machine learning / AI to avoid asking people

to fill up forms– But we need data to train the algorithms– Also, algorithms will give wrong predictions

– How about a continuous feedback loop between users and MLalgorithms

– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0– Use [buzzwords] machine learning / AI to avoid asking people

to fill up forms– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms

– What can go wrong?

4

New ways to collect same data

Use end user devices (mostly smartphones), to (legally) collecthow people move (GPS points).

I Old way– Ask people to fill up forms

I New way– Ask people to look at their GPS points and fill up forms(?)

I New way 2.0– Use [buzzwords] machine learning / AI to avoid asking people

to fill up forms– But we need data to train the algorithms– Also, algorithms will give wrong predictions– How about a continuous feedback loop between users and ML

algorithms– What can go wrong?

4

Data

5

Data

5

Data

5

Dealing with predictionsI Machine learning algorithms infer

– Time periods for trips (when people travel betweendestinations)

– Time periods for triplegs (when people travel with differenttravel modes [between destinations])

– Destination of trips (where people travel)– Purpose of trips (why people travel)– Travel modes (how people travel)

I Users can look at the output– And confirm that it is correct– Or modify it to bring it to the correct state

I AloneI Using a weird UII ...

6

Dealing with predictionsI Machine learning algorithms infer

– Time periods for trips (when people travel betweendestinations)

– Time periods for triplegs (when people travel with differenttravel modes [between destinations])

– Destination of trips (where people travel)– Purpose of trips (why people travel)– Travel modes (how people travel)

I Users can look at the output– And confirm that it is correct– Or modify it to bring it to the correct state

I AloneI Using a weird UII ...

6

What can go wrong?I Invalid periods

– e.g., traveled to work between 08:00 AM and 07:00 AM(same day)

I Overlapping periods– e.g., traveled to work between 08:00 AM and 09:00 AM, and

traveled to the supermarket between 08:30 AM and 09:30

I Discontinuous periods– e.g., traveled to work between 08:00 AM to 09:00 AM,

worked between 09:00 AM and 11:30 AM worked, andtraveled to a restaurant between 13:00 AM to 13:20

I Improper nests– e.g., traveled to work between 08:00 AM to 09:00 AM, during

which I walked between 08:20 and 09:00

7

How to prevent things from going wrong?

8

How to prevent things from going wrong?

8

How to prevent things from going wrong?

8

Avoiding headaches

Constraints

I Good for enforcing conditions on the table’s attributes

I Specified at table declaration or afterward, via ALTERI Types:

– Check constraints (row-level), e.g., age >0– Not null constraints (row-level), e.g., name not null– Unique constraints (table-level), e.g., two people cannot have

the same ID– Primary key (table-level) - unique + not null– Foreign key (between tables) - referential integrity to other

tables– Exclude constraints (table level) - unique + table-level check

9

Avoiding headaches

Triggers

I Useful for enforcing constraints across multiple tables(works fine for same table constraints, as well)

I Can be used to implement complex logicI Types

– Event triggerI Global, for the databaseI Captures DDL events (CREATE, ALTER, DROP,

COMMENT, GRANT, REVOKE)– Data change trigger

I Local, for the tableI Captures DML events (INSERT, UPDATE, TRUNCATE,

DELETE)I Can be run BEFORE or AFTER the data change eventI Can be conditioned by a WHEN clause

I Triggers can be written in any procedural language withevent trigger support, or in C

10

Generate test data - Tables

11

Generate test data - Tables

11

Generate test data - Tables

11

Generate test data - Locations

12

Generate test data - Locations

12

Generate test data - Trips and triplegs

13

Generate test data - Trips and triplegs

13

Check Constraints

ERROR: new row for relation "trips" violates check

constraint "valid_periods"

DETAIL: Failing row contains

(5, 2016-10-25 07:37:28.6, 2016-10-25 06:37:28.6)

14

Not Null Constraints

ERROR: null value in column "from_time" violates

not-null constraint

DETAIL: Failing row contains (6, null, null).

15

Primary Key Constraints

ERROR: duplicate key value violates unique constraint

"trips_pkey"

DETAIL: Key (id)=(1) already exists.

16

Foreign Key Constraints

ERROR: insert or update on table "triplegs" violates

foreign key constraint "triplegs_trip_id_fkey"

DETAIL: Key (trip_id)=(100) is not present in

table "trips".

17

Exclusion Constraints

ERROR: conflicting key value violates exclusion

constraint "non_overlapping_trip_periods"

DETAIL: Key (tsrange(from_time, to_time))

= (["2016-10-25 12:17:00","2016-10-25 12:20:00"))

conflicts with existing key

(tsrange(from_time, to_time))

= (["2016-10-25 12:00:00","2016-10-25 13:00:00")).

18

Triggers - definition example

19

Triggers - minimum two locations per tripleg

ERROR: proposed period modification of tripleg 1

does not contain enough locations

20

Triggers - time update of trip affects neighbors

21

Triggers - time update of trip affects neighbors

21

Triggers - time update of trip affects neighbors

21

Triggers - time update of trip affects (nested)

triplegs

ERROR: proposed period modification of tripleg 4

does not contain enough locations

I Trigger attached to the Triplegs invalidated anoperation on the Trips table and rolled back thetransaction

I As long as the rules are properly defined andimplemented, data integrity is maintained

22

Triggers - time update of trip affects (nested)

triplegs

ERROR: proposed period modification of tripleg 4

does not contain enough locations

I Trigger attached to the Triplegs invalidated anoperation on the Trips table and rolled back thetransaction

I As long as the rules are properly defined andimplemented, data integrity is maintained

22

Triggers - time update of trip affects (nested)

triplegs

ERROR: proposed period modification of tripleg 4

does not contain enough locations

I Trigger attached to the Triplegs invalidated anoperation on the Trips table and rolled back thetransaction

I As long as the rules are properly defined andimplemented, data integrity is maintained

22

Wrapping up

Triggers and constraintsI Advantages

– Can be used to implement your unified custom logic (writeonce, fix once)

– Constraints are great for enforcing checks on a row-level ortable-level (exclusion constraints), as well as referentialintegrity (foreign keys)

– For multiple table logic (or single table logic), triggers can doalmost anything - use responsibly

I Disadvantages– Documentation can be overwhelming– Cascading triggers can throw you in a loop (remove the

WHEN conditions on trip_period_integrity trigger)– Performance overhead– Cascading triggers are difficult to debug and the actions are

not always intuitiveI Advice

– Write unit tests (pgTap) to validate that a changed / newtrigger doesn’t break your functionality

23

ReferencesI Docs:

– Triggers - https://www.postgresql.org/docs/current/static/sql-createtrigger.html

– Constraints - https://www.postgresql.org/docs/current/static/ddl-constraints.html

– Exclusion constraints -https://www.postgresql.org/docs/current/static/

sql-createtable.html#SQL-CREATETABLE-EXCLUDE

– Examples of exclusion constraints on ranges -https://www.postgresql.org/docs/current/static/

rangetypes.html#RANGETYPES-CONSTRAINTI Other presentations on constraints and triggers

– Jim Mlodgenski (more on triggers) -http://www.slideshare.net/jim_mlodgenski/

an-introduction-to-postresql-triggers

– Robert Haas (triggers cheat sheet) - http://www.slideshare.net/pgconf/introduction-to-triggers

– Magnus Hagander (exclusion constraints) -https://www.hagander.net/talks/BeyondUNIQUE.pdf

24

References (part 2)I More on exclusion constraints

– Depesz - https://www.depesz.com/2010/01/03/waiting-for-8-5-exclusion-constraints/

– Davis Jeff -http://thoughts.davisjeff.com/2010/09/25/

exclusion-constraints-are-generalized-sql-unique/

I Github examples:– For this talk -

https://github.com/adrianprelipcean/PUG_Stockholm

– For the tracking system mentioned at the beginning (database+ mobile apps + annotation UI) -https://github.com/Badger-MEILI

25

Thank you for your attention!Questions and Discussions

Adrian C. Prelipceanhttp://adrianprelipcean.github.io/acpr@kth.se@Adi Prelipcean