Unstructure:: smashing the boundaries of data ::

2014-03-07 - SxSWi WorkshopIan Varley - @thefutureian

Chapter 0: Intro & Logistics

Hi. I’m Ian Varley.

I live in Austin, TX.

I work for Salesforce.com, doing data engineering.(Note: this presentation is entirely my own work and opinions, and doesn’t imply anything about Salesforce’s products.)

→ @thefutureian, ianvarley.com

About me:

- BA in Philosophy- MS in Software Engineering- 15+ years database experience

Not really an authority on data structure, but

"You teach what you want to learn".

Why are we here?

To grok the structure of data, and then smash it.

Some logistics:

- 150 minutes, no breaks. (Feel free to get up, use the facilities, or leave if you're bored.)

- This will be dense.(Lots to cover, so we'll move fast.)

- But! Do interrupt at any time with questions.(If you’re lost, you’re not the only one.)

All materials are available:

- This presentation: http://tiny.cc/unstructure-sxsw14-slides

- Live notes:http://tiny.cc/unstructure-sxsw14-notes

- Code & samples: https://github.com/ivarley/unstructure-sxsw14

There’s some setup required, but we won’t need it right away.

Probably best if everyone starts trying to download & install stuff now, and do it in the

background as I’m talking.

Setup, part 1: git

- Download & Install Githttp://git-scm.com/book/en/Getting-Started-Installing-Git

- Clone my repo: $ cd ~$ git clone https://github.com/ivarley/unstructure-sxsw14.git

Setup, part 2: heroku

- Download & Install Heroku Toolbelthttps://toolbelt.heroku.com/

- Create Heroku Accounthttps://id.heroku.com/signup

Setup, part 3: CouchDB

- Download & Install CouchDBhttp://couchdb.apache.org/

Setup, part 4: miscellaneous

- Google Chromehttps://www.google.com/intl/en/chrome/browser/

- JSONView Pluginhttp://goo.gl/K07fFs

Quick Survey: - occupation: coders? designers? other? - technical skill: low / medium / high - know what a relational database is? - know SQL? - know what NoSQL means? - have used a NoSQL database? - have read Aristotle? :)

There’s a spectrum in an audience like this ...

The Hacker: Wants hands on, doesn’t care

about theory

The Academic: Wants heady concepts, not

comfortable with code.

The “Hackademic”: Wants enough theory to be grounded, and enough hacking to

know when something is bullshit.

What I’m aiming for:

Where we’re headed:

Chapter 1 - Warming Up Chapter 2 - Hierarchy Chapter 3 - Relation Chapter 4 - Mutation Chapter 5 - Conclusion

Any questions before we get started?

Chapter 1: Warming Up

Talking about data can be a little … dry.

So, we’re going to use an example that most people can relate to easily.

Music.Photo credit: Josh Haner/The New York Times

● It's a domain we all know about.○ But, informally (not usually for work or study)

● Lots of meaty concepts to think about:○ Recording, Performances, Compositions, Bands, Venues, etc ...

● There are lots of music sites with data APIs:○ Do512, EchoNest, Songkick, Sched.org, MusicBrainz, 7Digital, etc.

● It's on everyone's mind during SxSW.● Also, I'm a musician and I felt like it.

Why music?

Running Example:

● listen.up - Internet company for “all things music”.● Duh, this is fake, there’s no “.up” TLD

● What do we do? Everything! ● Including:

● Recorded music catalogs, streaming, purchase...● Live music performance, booking & tickets...● Licensing, royalties, compositions, lyrics …● Instruments, lessons, repairs, classifieds …● Anything else you can think of.

Running Example:

● listen.up - Internet company for “all things music”.● Duh, this is fake, there’s no “.up” TLD

● What do we do? Everything! ● Including:

● Recorded music catalogs, streaming, purchase...● Live music performance, booking & tickets...● Licensing, royalties, compositions, lyrics …● Instruments, lessons, repairs, classifieds …● Anything else you can think of.

Is this a good business model? No, but who cares!

You are my team of

professional ontologists,you’re going to figure out

what music data is out there in the world for us to store.

(I’ll pay you in stock. It’ll be worth a fortune, trust me.)

Normally, this is where we might do a big group brainstorm.

But it turns out those don’t work.

In fact, they hurt more than they help.

(https://en.wikipedia.org/wiki/Brainstorming#Challenges_to_Effective_Brainstorming)

So, we’ll do 3 steps:

1. Take 60 seconds and type as many music-related concepts as you can.

- concepts, not proper nouns (i.e. "band", "composer", "instrument"; not "Radiohead", "Beethoven", "guitar", etc.)

- make them singular ("band", not "bands")- not sure if it’s music-related? put it anyway.

2. Dump them into a shared google doc.3. I’ll lowercase, dedupe, and publish.

Go!I’ll add a few prompts in case you are getting stuck ...

So, we’ll do 3 steps:

1. Take 60 seconds and type as many music-related concepts as you can.

- concepts, not proper nouns (i.e. not "Radiohead", but "band"; not "Beethoven" but "composer", not "guitar" but "instrument", etc.)

- make them singular ("band", not "bands")- not sure if it’s music-related? put it anyway.

2. Dump them into this google doc: http://tiny.cc/unstructure-sxsw14-terms

3. I’ll lowercase, dedupe, and publish here.

Now, your job is to make some sense of this.

1. Break into groups of ~3 neighbors.a. Introduce yourselves like civilized human beings.

2. Organize this list however you want to!a. Group things togetherb. Indent thingsc. Draw lines in a drawing programd. etc.

Go!We’ll take about 10 minutes for this.

Let’s discuss a few sample things people did.(Goal is to "sample", not for everyone to share! We don’t have all day.)

● How did you sort or group the terms?● Did you end up with a flat list, or hierarchy?● Did anything not fit in?● Any higher level organization of terms?

(Note: there’s no right answer here ... yet.)

Good job. is off to a great start.

Now it’s time to get into the meat.What is structure? What is data?

This is a story in three parts:

HierarchyRelationMutation

These are 3 successive viewpoints thatwill gradually open our eyes to the

deep structure of data.

“Structure? Isn’t this workshop supposed to be about unstructured data?”

Poppycock. You want to see some real unstructured data?

What most people mean when they say unstructured data is:

“flexibly structured data”

Or possibly:

“data we don’t know the structure of yet”

(We’ll get to both of those; hold your horses.)

There are lots of boundaries to smash in the world of data.

But we have to learn to structure before we can unstructure.

Image credit: Rodrigo Diaz Aravena

Our minds are very fluid, and can connect concepts in subtle ways.

Our computers aren't. They need concrete instructions to structure & connect data.

The next three chapters will be a deep dive into how concepts are combined to let us represent the world in computers.

This is usually called “modeling”.

Modeling gets a bad rap.

Models cut away the accidental and leave the essential.

Model != diagram, drawing

Model == Skeleton, Essence, Abstraction

Most of the time, it’s good to remember:“the map is not the territory”.

But for the next couple hours, for us, the map is precisely the territory.

We are data cartographers.

- Alfred Korzybski

So let’s dive in!

photo credit: Michael Mundy

Chapter 2: Hierarchy

We all have a pretty good intuitive understanding of “data”. What’s yours?

data = plural of datumdatum = Latin for “given”

so ... data is “givens”?aka “facts”?

What’s the simplest fact? A bit:

1 / 0on / offyes / no

true / false

“Are the lights on in this room?”

When you want more complex facts, you chunk together simpler ones.

In one dimension, that's a "list":● Byte = list of bits → 01000010 = 66 = “B”● Word = list of characters → [B,e,a,t,l,e,s]● Phrase = list of words -> “The Beatles are a band”● And so forth ...

Of course, a 1-dimensional list is just one (very simplistic) way

to chunk things together.

Photo credit: http://thekingoflimbspart2.com/radiohead-setlists/radiohead-2012-setlists/radiohead-setlist-houston-texas-3032012/

So you just chunk datums together, and you get information, knowledge, wisdom … !?

Photo By Karora (Own work) [Public domain], via Wikimedia Commons

No. It’s not quite that easy.

f∆ƒ˙73f˚=£ƒ••XMbritneysp3ars-giraffe

is a complex structure, but it lacks something: “meaning”

Getting a little more haughty, we could say that the “givens” of

structured data are really pointers to concepts.

Without at least some concept, it’s not data: it’s noise.

But that raises all kinds of questions.

What are concepts?What can we say about them?

What kinds of concepts are there?What’s the difference between a

concept and the thing it points at?

Who could answer such questions?

When the epistemological going gets tough, the tough call ...

Aristotle, 384 –322 BC

Aristotle wrote a treatise called

The Praedicamenta(The Categories)

It’s not clear exactly what he was categorizing (he didn’t say), but the list

stands to this day as a pretty damn sensible way to, well, categorize.

Aristotle’s 10 "categories":

substance Stuff, essence; matter, but also universal concepts

quantity How much? How many?

quality What kind? Of what nature?

relation More, less, double, half, stronger, weaker, etc.

place Where?

time When?

position Being situated on, in, next to, sitting, touching, etc.

having Possession, state like “clothed” or “armed”

causing What did it do, make happen?

being caused What happened to it, what did it undergo?

You don’t have to agree with Aristotle’s categories (you’d be in good company).

(But you’re also unlikely to have a sudden inspiration about it that hasn’t already been the subject of 12 papers and a dissertation.)

But having some version of Aristotle’s list is hard to argue with, experientially. We

sort the world into a hierarchy of concepts; everything in its right place.

Without concepts, we don’t have data.

We just have noise.

Of course, you can’t just store a concept on a disk, or send it over a network.

So, not long after we had computing machines, folks set to work figuring out how to map and store our concepts in

the unforgiving realm of silicon.

Let’s take a trip back in time.

Photo By NASA Ames Research Center (NASA-ARC) (NIX A-28284) [Public domain], via Wikimedia Commons

The year is 1966.

NASA is working on Saturn V and Apollo rockets, and they can’t figure out how to store this ginormous bill of materials. They ask:

Could these new “computers” help?

IBM: “Hey NASA! We made an system for you to manage information!”

NASA: “Groovy! What’s it called?”

IBM: …...“Information Management System.”

Thus began the not-extremely-exciting era of

hierarchical databases.● Data is stored in records, which can have sub-records.● There's a single strict hierarchical arrangement.● To access data, you need to know the hierarchy.

For example:

Show Band Time

House Of Vans

Charlie XCX

4:15

House Of Vans

Pusha T

5:00

Show Venue Date

House Of Vans

The Mohawk

3/13/14

Chaos in Tejas

Iron And Lace

3/14/14

Band Song Order

Charlie XCX

You 1

Charlie XCX

Super Love

2

To get to the set list, you have to navigate through the show, to the

band, to the song.

And, you only get to choose one hierarchy to store things in.

That sounds arbitrarily restrictive. Why did they make it like that?

To understand,we have to talk about dimensions.

We already talked about zero dimensions:

And about one dimension:

point = bit = on/off = true/false

line = list = array

It’s pretty straightforward to see how you’d represent these,

with a series of zeros and ones.

What about 2 dimensions?

It’s a plane! (aka table, grid, matrix, spreadsheet, etc)

Tabular data is everywhere.

You're certainly familiar with the world's most ubiquitous 2-dimensional data tool ...

X dimensionY dimension

Excel!Photo credit: http://decentralist.wordpress.com/2012/10/01/libreoffice-vs-openoffice-not-always-simple/

(Or, its successor, Google Spreadsheets,

which is way better, IMHO.)((Especially in combination with Google Forms, which make it extremely useful.))

Spreadsheets are totally flexible.

This is a blessing and a curse.

They can be used well ...

Or poorly ...

(Fun read: http://www.epmchannel.com/2013/02/22/is-excel-the-most-dangerous-piece-of-software-in-the-world/)

Or awesomely ...

Image credit: http://gadgetose.com/excel-stop-motion-music-video/

But you get the point. 2-dimensional data is everywhere.

2-dimensional data doesn’t always look like a table or grid.

We just mean that it’s “conceptually planar”: two axes, each w/ a set coordinate system.

(Here, rows = “web results”, and columns = “link name”, “url”, “description”, and “image”. No link has two URLs, for example.)

However! We are now faced with a choice, because we can still only actually store things

in linear form (a single stream of bits).

So do we put rows inside columns, or columns inside rows?

To linearize the two dimensions in a table, I can either ...

<table> <row> <col>Creep</col> <col>1993</col> <col>Pablo Honey</col> </row> <row> <col>No Surprises</col> <col>1997</col> <col>OK Computer</col> </row> <row> <col>Lucky</col> <col>1997</col> <col>OK Computer</col> </row> <row> <col>Karma Police</col> ... </row></table>

Go row-wise ...

<table> <col> <row>Creep</row> <row>No Surprises</row> <row>Lucky</row> <row>Karma Police</row> <row>Fake Plastic Trees</row> </col> <col> <row>1993</row> <row>1997</row> <row>1997</row> <row>1997</row> <row>1995</row> </col> <col> <row>Pablo Honey</row> ... </col></table>

Or column wise ...

<table> <row> <col>Creep</col> <col>1993</col> <col>Pablo Honey</col> </row> <row> <col>No Surprises</col> <col>1997</col> <col>OK Computer</col> </row> <row> <col>Lucky</col> <col>1997</col> <col>OK Computer</col> </row> <row> <col>Karma Police</col> ... </row></table>

But I can’t have it both ways.<table> <col> <row>Creep</row> <row>No Surprises</row> <row>Lucky</row> <row>Karma Police</row> <row>Fake Plastic Trees</row> </col> <col> <row>1993</row> <row>1997</row> <row>1997</row> <row>1997</row> <row>1995</row> </col> <col> <row>Pablo Honey</row> ... </col></table>

?(Unless I store it twice.)

Because we read left-to-right and top-to-bottom, most of our systems store tables that

way too (row-wise).

But it’s not mandatory, of course.

Whichever way you choose, you can’t get around the fact that you have to choose an ordering of dimensions.

That makes sense for 2 dimensions.

But what about …3+ dimensions?

Same thing.

Computer science has had the idea ofmulti-dimensional arrays

since the beginning.

It's a straightforward extension to a table, conceptually. It's very hard to visualize more than 3 unless you’re on dope*.

* - This is a verrrry funny joke because multidimensional arrays use locators called dope vectors. Ha ha ha hmm.

But you’re still making it linear when you store it.

Imagine storing the venues, shows, bands, and songs in one file.

That’s 4 dimensions.

(Each venue has many shows; each show has many bands;

each band has many songs; etc.)

Venue: The Mohawk Show: Vans Day Party, 3/13/14 Band: Eagulls (12:30 p.m.) Song: Nerve Endings Song: Tough Luck Band: DJ Rashad (1:15 p.m.) Song: Holiday Song: I Can Feel It Band: Kelela (2:15 p.m.) Song: ... Band: Charli XCX (3:15 p.m.) Song: ... Band: Dum Dum Girls (4:15 p.m.) Song: ... Band: Pusha T (5:00 p.m.) Show: Venue: ... Show: ... Band: ... Song: ...etc.

Song: Nerve Endings Band: Eagulls Show: Vans Day Party, 3/13/14 Venue: The MohawkSong: Nerve Endings Band: Eagulls Show: Vans Day Party, 3/13/14 Venue: The Mohawk Show: Official Showcase, 3/14/14 Venue: The MohawkSong: Holiday Band: DJ Rashad Show: Vans Day Party, 3/13/14 Venue: The Mohawk Band: Cattle Decapitation Show: Chaos In Tejas, 3/10/14 Venue: Iron And LaceSong: ... Band ... Show: ...etc.

By venue → show → band → song By song → band → show → venue

No matter how many dimensions, storing the data still requires that you

pick a single primary orientation.

So Big Blue’s design choice makes a little more sense now, right?

BTW, If you've used an ATM recently…

you're an IMS user.

(It’s not as obsolete as it sounds.)

How do you actually store the linearized data?

You put it in a format.

Creep•••••••••••••1993Pablo•HoneyNo Surprises••••••1997OK•ComputerLucky•••••••••••••1997OK•ComputerKarma Police••••••1997OK•ComputerFake Plastic Trees1995The•Bends••

Fixed-width files were all the rage in the 1960s.

Row delimiter is a line break; column delimiter is a pre-set agreement about how many characters are in each line.

This is wasteful, brittle, and hard to read.

"Creep","1993","Pablo Honey""No Surprises","1997","OK Computer""Lucky","1997","OK Computer""Karma Police","1997","OK Computer""Fake Plastic Trees","1995","The•Bends"••

Delimited files (e.g. CSV, comma separated values):

Row delimiter is still a line break; column delimiter is variable (a comma, in this case). Optionally, also “qualifiers” (quotes, here).

This is a decent format (but, Microsoft’s version really screwed things up for everyone.)

<row> <col>Creep</col> <col>1993</col> <col>Pablo Honey</col> </row> <row> <col>No Surprises</col> <col>1997</col> <col>OK Computer</col> </row> <row> <col>Lucky</col> <col>1997</col> <col>OK Computer</col> </row>

There’s also markup (e.g. HTML)

“Tags” (<tag></tag>) give you the start and end of rows, and the start and end of columns within those rows.

SGML, HTML, XML, all follow this approach.

Song: CreepYear: 1993Album: Pablo Honey%Song: No SurprisesAlbum: OK ComputerYear: 1997%Year: 1997Song: LuckyAlbum: OK Computer...

Or even YAML (“YAML Ain’t Markup Language”)

(Like email headers.)

Column pointers (names) are inline with the values; rows have many lines, and are delimited by another character (e.g. “%”).

This is obviously more flexible, but still inherently hierarchical.

And there are a couple modern technologies that are hierarchical all

the way down.

In practice, the only form of markup people use for storing data is XML.

And the most prevalent subset of YAML people use for storing data is JSON.

(Technically JSON isn’t a subset of YAML but you shut up.)

XML was early obvious choice, because we were used to HTML, so we all "got" it.<?xml version="1.0"?><venue name="The Mohawk"> <show title="Vans Day Party, 3/13/14"> <band name="Eagulls" time="12:30 p.m."> <song order="1">Nerve Endings</song> <song title="Touch Luck" order="2" /> </band> <band name="DJ Rashad" time="1:15 p.m."> <song title="Holiday" order="1" /> <song title="I Can Feel It" order="2" /> </band> </show></venue>

But XML isn’t very human-friendly.● It’s verbose● The wrong things grab your eye● It’s somewhat complicated to parse● Distinction between attributes and tag contents

is confusing.“XML is crap. Really. There are no excuses. XML is nasty to parse for humans, and it's a disaster to parse even for computers. There's just no reason for that horrible crap to exist.” - Linus Torvalds, Yesterday (2014-03-06), on Google+

Now JSON is winning ...{ "venue": { "name": "The Mohawk", "show" : { "title": "Vans Day Party, 3/13/14", "bands": [{ "name": "Eagulls", "time": "12:30 pm", "songs": [ {"title": "Nerve Endings", "order": 1}, {"title": "Touch Luck", "order": 2} ]}, { "name": "DJ Rashad", "time": "1:15 pm", "songs": [ {"title": "Holiday", "order": 1}, {"title": "I Can Feel It", "order": 2} ]} ] } }}

JSON:● Where XML looks like a mass of text, JSON

looks spacious (little clutter)● Fast to parse, for humans and computers● Self-describing, flexible format● Extremely simple syntax (one page)

from http://www.json.org/

One more thing: so far, we’ve talked about a certain kind of hierarchical

relationship: containment.

But there’s another kind worth mentioning: generalization.

This kind of relationship is common in programming (it’s called superclassing). But it’s uncommon (at least, explicitly) in database systems.

We’ll come back to it later.

Hands on!

You’re doing some corporate espionage forlisten.up. You notice that do512.com seems to have a good way to organize data in their API. What can you steal, er, learn?

Note: the guys at do512 are friends of mine and I am in no way encouraging anyone to perform any actual corporate espionage, no matter how cool that sounds. Listen.up is a made up company, do512’s json API is open, and this is an exercise for learning; no stealing anything. :)

Hands on!

● Make sure you’ve got the JSONView extension in Chrome● Go to: http://2014.do512.com/events.json● Explore the hierarchical data that comes up● Try “-” to collapse all, click “+” signs to unfold sections● Also try:

○ http://2014.do512.com/venues.json○ http://2014.do512.com/artists.json

Hands on!

● What’s the hierarchical structure of this data?● In the same way we talked about nesting bands, venues,

shows, etc, … what are the objects being nested here?● Would you store it differently?● Are there any superclass / subclass relationships?

Questions?

Chapter 3: Relations

So, hierarchy! Pretty great, right?

Actually, no; it’s quite problematic if you use it as the method for storing data.● Lots of stuff isn't naturally hierarchical.● You can't change the organization without changing all the

code that accesses data.● Above 3 dimensions, the number of possible access paths

goes up dramatically! (Exponentially, in fact.)○ The academic literature of the 60s and 70s is full of

papers describing how to do this better or faster.

But, what are you gonna do? Computers are just boring & hierarchical, so you’d

better learn to deal with it.

Meanwhile, outside the offices of IBM, the revolution of the ‘60s was happening.

The desirability and feasibility of hierarchy, especially a single universal hierarchy, was seeming less and less

important.

Reconciliation seemed impossible.

And then ...

something wonderful happenedTHE MONOLITH IN KUBRICK'S 2001: A SPACE ODYSSEY (1968)

Specifically, the math nerds

beat the business jocks.

This is Dr. Edgar F. Codd (1923-2003)

He worked for IBM in the 1960s, and couldn’t stand the thought of our rich, interconnected world being subjugated to storage in hierarchical databases.

So he came up with a radical theory.image from wikipedia:

http://en.wikipedia.org/wiki/File:Edgar_F_Codd.jpg

Two intellectual ancestors:Set Theory Graph Theory

Set Theory:

Sets are collections of objects. You can precisely describe operations on sets:● Union● Intersection● Difference● Cartesian Product

Graph Theory:

Graphs are collections of nodes, connected by edges.

Not this:

Graph Theory:

Graphs are collections of nodes, connected by edges.

This: Think: a social network where the nodes are people and the edges are friend relationships.

Codd’s genius was combining these and proposing a declarative (rather than imperative) access model.

● The relational model is a graph of sets● Relations (tables) are sets of tuples (rows).● Some attributes (columns) are edges that let you

connect the sets in interesting ways.● You never specify “how” to get to data, just “what”

data to get, based on sets.

When he proposed this as a way to actually store data in 1969,

people thought he was from Mars.

Everyone said “It’ll never work, computers are too slow.”

His employer, IBM, said “Thanks but no thanks; we’ll just keep selling IMS.”

But the haters didn’t bank on two things.

1: Moore’s Law

That graph is exponential, not linear.

We can have plenty of CPU.

2: Programmer Time

As programmer time became more valuable than computer time, ease of representing the problem domain became a dominating factor.

Due to some tomfoolery at IBM, Codd’s “Alpha” never made it.

But another group at IBM created a quasi-relational version called SEQUEL, which looked kind of like COBOL.

Then in 1979, Larry Ellison copied the design to create Oracle and SQL (SEQUEL was trademarked). And the rest is history.

For his part, Codd waged a decades-long battle to get a more true representation of the relational model adopted.

But, it never was. SQL was king.

So what is the relational model, then?

First, one quick PSA ...

Occupational Hazard:

Abstraction Vertigo

Concrete Data

Metadata

The Structureof Metadata

Band name: RadioheadYear formed: 1985

Entity: BandAttributes: Name, Year Formed

Concepts → Entities, Attributes

Safety first.If you feel dizzy, just ask a question.

Entity Attribute

Relationship

There are 3 foundational concepts:

Entity

Attribute

Attribute

relationship

Entity

Attribute

Attribute

Show

Start time

Cover $

is at

Venue

Address

Age Req.

Vans Day Party

12pm

Free

is at

The Mohawk

123 Red River

All Ages

Notice the subtle shift here.

Neither of those entities is “inside” the other. They’re both “first class” entities,

and they’re in a relationship.

The relationships are described at the level of sets, not ad hoc. Shows can be at venues, categorically.

Entities look pretty much exactly like 2-dimensional tables, except for the

concept of a “key”.

That’s the attribute (or set of attributes) that distinguishes this row from that row.

Show

Start time

Cover $

is at

Venue

Address

Age Req.

Show Start time Cover $

1234 8pm $5

5678 11pm $9

Venue Address Age Req.

The Mohawk 123 Red Riv. 21+

Beerland 456 Red Riv. All Ages

key

key

Keys describe an entity’s identity.

In practice, most systems today use surrogate keys (i.e. IDs) to establish unambiguous identity.

eg: integers (123456), codes (X74-UUA2), GUIDs

Show

Start time

Cover $

is at

Venue

Address

Age Req.

Show Is At Start time Cover $

1234 The Mohawk 8pm $5

5678 Stubb’s 11pm $9

Venue Address Age Req.

The Mohawk 123 Red Riv. 21+

Beerland 456 Red Riv. All Ages

Then, to get connected information out,you traverse the relationships with

something called a JOIN.

Show

Start time

Cover $

is at

Venue

Address

Age Req.

Show Is At Start time Cover $

1234 The Mohawk 8pm $5

5678 Stubb’s 11pm $9

Venue Address Age Req.

The Mohawk 123 Red Riv. 21+

Beerland 456 Red Riv. All Ages

Show Is At Start time Cover $

1234 The Mohawk 8pm $5

5678 Stubb’s 11pm $9

Venue Address Age Req.

The Mohawk 123 Red Riv. 21+

Beerland 456 Red Riv. All Ages

Show Is At Address Age Req. Start time Cover $

1234 The Mohawk 123 Red Riv. 21+ 8pm $5

+

=

Tuple is a fancy word for “row”.

(It comes from abstracting … septuple, octuple, N-tuple …)

Cardinality means, how many tuples of one relation can match each tuple in

another relation?

(In English: a show is at one venue, but a show can have many bands.)

Show

Start time

Cover $

played by

Band

Name

Genre

Show Start time Cover $

1234 8pm $5

5678 11pm $9

Band ID Name Genre

RDOHD Radiohead Rock

EGLS Eagulls Indie Rock

many to many?

Show

Start time

Cover $

played by

Band

Name

Genre

Show ID Start time Cover $

1234 8pm $5

5678 11pm $9

Band ID Name Genre

RDOHD Radiohead Rock

EGLS Eagulls Indie Rock

Show ID Band ID Start Time

1234 RDOHD 9:30pm

5678 EGLS 9pm

5678 RDOHD 11pm

Normalization is the idea that if something repeats (i.e. has a cardinality of more than 1), it should be expressed as another entity, not as repeating data.

Show Date Venue Street City State

Austinist Party! - Day 1 3/13 Mohawk 123 Red River Austin TX

Austinist Party! - Day 2 3/14 Mohawk 123 Red River Austin TX

Austinist Party! - Day 3 3/15 Mohawk 123 Red River Austin TX

No:

Yes: Show Date Venue

Austinist Party! - Day 1 3/13 Mohawk

Austinist Party! - Day 2 3/14 Mohawk

Austinist Party! - Day 3 3/15 Mohawk

Venue Street City State

Mohawk 123 Red River Austin TX

Basically, normalization dictates that the same fact is never repeated

in more than one place.

Show Date Venue Street City State

Austinist Party! - Day 1 3/13 Mohawk 123 Red River Austin TX

Austinist Party! - Day 2 3/14 Mohawk 123 Red River Austin TX

Austinist Party! - Day 3 3/15 Mohawk 123 Red River Austin TX

There’s nothing wrong with this data per se, but we all know that the final 4 columns refer to the same thing.

Show Date Venue Street City State

Austinist Party! - Day 1 3/13 Mohawk 123 Red River Austin TX

Austinist Party! - Day 2 3/14 Mohawk 985 Congress Austin TX

Austinist Party! - Day 3 3/15 Mohawk 123 Red River Austin TX

There’s nothing wrong with this data per se, but we all know that the final 4 columns refer to the same thing.

What would it mean if the street address were different in one of these? Are there two Mohawks?

Normalization gives you consistency, also known as relational integrity.

Certain kinds of problems (like that one) just can’t happen, because there’s

literally only one place where you store the address of the Mohawk.

Show Date Venue

Austinist Party! - Day 1 3/13 Mohawk

Austinist Party! - Day 2 3/14 Mohawk

Austinist Party! - Day 3 3/15 Mohawk

Venue Street City State

Mohawk 123 Red River Austin TX

Normalization also means that theattributes of an entity are non-repeating.

(i.e. there’s no repeating columns, or groups of columns)

Show Start time Cover $ Band #1 Band #2 Band #3

House of Vans 8pm $5 Eagulls Cyndi Lauper Radiohead

Chaos in Tejas 11am $10 Metalface CRUD Decapitation

Show Start time Cover $

House of Vans 8pm $5

Chaos in Tejas 11am $10

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

House of Vans Radiohead 3

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

Chaos in Tejas Decapitation 3

No:

Yes:

This makes sense, because …what if you had 4 bands? Or 400?

Do people break these rules?All the time!

(We’ll get to valid reasons why they might want to do that.)

One last thing:

Schema is the structure of the database itself. It is stored as ... data!

VenueID Name Address

123 Mohawk 123 Red River

456 Stubbs 456 Red River

BandID Name # Members

4321 Eagulls 4

8765 CRUD 17

Table Column Type

Venue VenueID ID

Venue Name String

Venue Address String

Band BandID ID

Band Name String

Band # Members Integer

The final piece of the relational database puzzle:

SQL (Structured Query Language)

which is about how you get stuff out of this graph of sets.

We obviously don’t have time to really learn SQL, but here’s the gist:

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

Show Start time Cover

House of Vans 8pm $5

Chaos in Tejas 9pm $10

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

Show Start time Cover

House of Vans 8pm $5

Chaos in Tejas 9pm $10

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

Show Start time Cover

House of Vans 8pm $5

Chaos in Tejas 9pm $10

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

Show Start time Cover

House of Vans 8pm $5

Chaos in Tejas 9pm $10

$10 Metalface

SELECT S.cover, B.Band FROM Shows S INNER JOIN ShowBands B ON B.Show = S.Show WHERE S.start_time > 8pm AND B.band like ‘%face%’ ORDER BY order ASC

Show Band Order

House of Vans Eagulls 1

House of Vans Cyndi Lauper 2

Chaos in Tejas Metalface 1

Chaos in Tejas CRUD 2

Show Start time Cover

House of Vans 8pm $5

Chaos in Tejas 9pm $10

$10 Metalface 1

So that’s relational databases and SQL, in a very small nutshell.

You can imagine, relational databases get very complicated for non-trivial things.

But you now know almost all there is to know: relational databases are graphs of sets, navigated via declarative language.

Source: http://wiki.musicbrainz.org/-/images/5/52/ngs.png

Also, remember generalization? Theoretically, that’s just another type of

relationship between entities.

Band Genre

Radiohead Rock

Metalface Jazz

Entertainer Since

Radiohead 1985

Metalface 2005

Gonzo the Incredible 1968

Gob 2002

Penn & Teller 1990

Magician Style

Gonzo the Incredible Sorcery

Gob Fail

Penn & Teller Conjuring

I say theoretically because nobody actually does this in practice, since

(unlike in object oriented programming) doing this carries a performance and

complexity burden in databases.

(But, you could do it, theoretically.)

Hands on!

We’re going to spin up a relational database using Heroku, create and populate some tables, and show the data on a web site.

Hands on!About Heroku:● Super simple hosted engine for running any code online● Has built-in relational database capabilities with Postgres● Uses a version control system called git● Uses Amazon AWS to host the code● It's owned by Salesforce.com (my company) but run

separately; I'm not an expert● Also note that this example is written in Ruby and I’m

really not an expert on that. Hope it works!

Hands on!

Prerequisites:1. You should already have git installed; do now if not.2. You should have heroku toolbelt installed, and have

created a heroku login. Do now if you haven’t.3. You should have already cloned my repo, but if not do it

now.$ cd ~$ git clone https://github.com/ivarley/unstructure-sxsw14

Hands on!

Steps:1. Go to the directory you cloned the repo into:

$ cd unstructure-sxsw14

2. Create and publish the app:$ heroku login$ heroku create$ git push heroku master$ heroku open

Hands on!

Congrats! You just published a web app.

Not convinced? Edit some of the visible text in the file:app/views/welcome/index.html.erb

and republish:$ git commit -am 'made an edit'$ git push heroku master$ heroku open

Hands on!Now let’s add some data!

Heroku automatically created a Postgres database for you.$ heroku pg:psql

opens a database prompt. Create a table and insert data:CREATE TABLE band (bandid INT PRIMARY KEY, name VARCHAR, genre VARCHAR);INSERT INTO band (bandid, name, genre) VALUES (1, 'Radiohead', 'Rock');SELECT * FROM band;

Hands on!Add some more tables and data:CREATE Table venue (venueid INT PRIMARY KEY, name VARCHAR, address VARCHAR);CREATE TABLE show (venueid INT PRIMARY KEY, bandid INT, date VARCHAR);INSERT INTO venue (venueid, name, address) VALUES (1, 'The Mohawk', '123 Red River, Austin, TX');INSERT INTO show (venueid, bandid, date) VALUES (1, 1, '2014-03-07');

And create a view (basically, a saved SQL statement):CREATE VIEW shows AS SELECT b.name as band_name, v.name as venue_name, s.date FROM band b INNER JOIN show s ON b.bandid = s.bandid INNER JOIN venue v ON s.venueid = v.venueid;

Hands on!Add some more tables and data:CREATE Table venue (venueid INT PRIMARY KEY, name VARCHAR, address VARCHAR);CREATE TABLE show (venueid INT PRIMARY KEY, bandid INT, date VARCHAR);INSERT INTO venue (venueid, name, address) VALUES (1, 'The Mohawk', '123 Red River, Austin, TX');INSERT INTO show (venueid, bandid, date) VALUES (1, 1, '2014-03-07');

And create a view (basically, a saved SQL statement):CREATE VIEW shows AS SELECT b.name as band_name, v.name as venue_name, s.date FROM band b INNER JOIN show s ON b.bandid = s.bandid INNER JOIN venue v ON s.venueid = v.venueid;

Hands on!Now you can SELECT data:SELECT * FROM shows;

And you’ll get: band_name | venue_name | date -----------+------------+------------ Radiohead | The Mohawk | 2014-03-07

Note if you’re into nit-picking: I'm taking a shortcut here and saying that each record in "show" is a band/venue combination, with a date. If "show" were a proper entity (for example, if the show had a name, a promoter, etc.) then the proper “normalized” way to model it would be to create a show entity with an ID and a venue ID, show name, promoter, etc; and then have intersection tables between bands and shows (like a band_show table). But this is fine for now.

Hands on!Leave the SQL prompt (type “\q”) and reopen the app:$ heroku open

And (hopefully) voila! You see the results of:SELECT * FROM shows;

Feel free to mess around now--add more data, to see how it’ll show up. If you’re really advanced, try changing the ruby code to show different SQL statements, etc. We’ll take about 10 minutes to play around, ask questions, etc.

Questions?

Chapter 4: Mutation

So, hierarchical databases were lame. But relational databases are awesome!

SQL Rules!! Right … ?

Hrm. As awesome as they are, relational databases have a few warts.

And some of them have only become apparent recently.

● It’s really hard to work with recursive / graph relationships● Results of SELECT queries are always flat tables, which

means you have to reassemble nested structures yourself● They don’t play well with object oriented programs● They don’t support inheritance & superclassing!● Relational modeling tools are generally quite sucky.● They don’t scale well to extremely large data sets,

because they promise things you can only do on a single (non-distributed) system.

There’s a standard list of grievances:

These are all interesting points; if we had a whole semester, we could spend weeks on any one of them.

But I want to spend the rest of our time on what is (IMO) the essential problem:

mutability.(i.e. change, being mutated)

Let’s zoom way out for a minute.

By NASA [Public domain], via Wikimedia Commons

We talked about hierarchies.● Old school, classical● Single top down view of reality

We talked about relations.● Modern, networks, connections● No single privileged access path or view

But both hierarchies and relations tend towards a commitment to afixed representation of reality.

You’re modeling static concepts that exactly match the real world. Doing this with high fidelity is the crowning

achievement of the relational model.

But reality doesn’t hold still.

It changes, all the time.

Any fixed way of representing the world is doomed to become outdated.

Darwin knew what was up.The nature of reality is to be dynamic, evolving.

A species seems like a fixed thing, but that's just a label we attach.

They are fuzzy around the edges, and always changing.

Change is a fact of life at all levels of the abstraction ladder…

our understanding of structure itself

the structure of our software apps

facts about things in the real world

And it’s only getting faster.

We can either hide our heads in the sand, or we can figure out what to do about it.

So let’s talk about 3 ways to smash the boundaries of data,

in the mutation dimension:

Attribute FlexibilityNon-Destructive Mutability

Model Agility

Non-Destructive Mutability

This is an UPDATE statement in SQL:UPDATE Band SET name = 'Jaydiohead' WHERE Band ID = 5678 and name = 'Radiohead'

Result:Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Jaydiohead

You know what you can’t do?

Undo.Edits in traditional relational databases are destructive. If you want to keep the old version, you have to do it yourself.

Note for data nerds: yes, databases keep transaction logs so you can undo and redo edits as part of transactions. But this is (a) implementation dependent, (b) not typically exposed to users in the relational model, and (c) not guaranteed to persist beyond the transaction itself.

This means that everyone has to either:

● Add extra complexity to their data model, or● Accept that changes are destructive.

Both are pretty crappy options. Most people just do #2.

But, sometimes you can’t do that. Consider compliance.● Who changed my salary?● Who deleted this opportunity from the

pipeline?● Who took my stapler?

You might not care a bunch about that, but Sarbanes-Oxley sure does.

And if you don’t know what that is, count yourself lucky and go back to making pretty things.

So this …

So how do people store history now?Option #1: audit columns (partial solution)

Band ID Name

1234 Bjork

5678 Radiohead

Option #1: audit columns (partial solution)

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name Create Date Created By Modify Date Modified By Deleted?

1234 Bjork 3/6/14 2pm Ian Varley 3/6/14 3pm Jan Jones false

5678 Jaydiohead 3/5/14 1pm John Smith 3/7/14 1pm Ian Varley false

So this … becomes this.

So how do people store history now?

So this …

So how do people store history now?Option #2: History tables

Band ID Name

1234 Bjork

5678 Radiohead

So this … becomes this

So how do people store history now?Option #2: History tables

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name Version Date By

1234 Bjork 1 3/6/14 Ian V

5678 Radiohead 1 3/6/14 Ian V

5678 Jaydiohead 2 3/7/14 Ian V

1234 Fjork 2 3/7/14 Ian V

Neither of these are ideal.

They ugly up your data model, which is not just an inconvenience; it makes it harder to see the “real” stuff.

It also violates the spirit of normalization: if something means the same thing, don’t repeat it all over the place.

But what if … the database took care of this for you?

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Name

1234 Bjork

5678 Radiohead

Some newer ones do:● Google pioneered this with BigTable - time is a

privileged dimension, part of the model, stored with every datum

● NoSQL Stores like HBase follow suit● Salesforce offers “field history” out of the box,

as a meta-feature on any entity, expressed as a history table

● Document stores can store older versions

Does this use a lot of space? Yes.

But guess what? We have a lot of space.

And, it turns out that for a lot of computing applications, making it immutable makes the problem way easier to reason about.

(For more on that, see Pat Helland’s talk, Immutability Changes Everything.)

Here’s the crazy part. If we do this across the board, we end up with ... data time travel.“K-9, show me my accounts receivable as of last May, and compare it with today.”

Plain old “undo” is a subset of this. Ever wondered why Google web products all have “undo” and nobody else’s do?

(Full revision history is the fancy version. They have that too.)

My pronouncement:

Henceforth, all databases should make time a privileged dimension, and retain older versions of data in a way that supports time travel.

Attribute Flexibility

Historically, databases have been a little something of a schoolmarm.

Image: Public domain. From the 1894 Laughable Lyrics: A Fourth Book of Nonsense Poems, Songs, Botany, Music, etc. by Edward Lear.

Some things are decided and enforced directly at the database level:

- what attributes an entity can have- how many attributes there are- what type of data to store in each attribute

- (string, number, date, currency, etc.)- other constraints (max & min values, etc)

But why is this the database's job?

It’s partially a historical accident.Disk & memory used to be scarce and

highly optimized, so record formats had to be prescriptive and fixed.

It’s also partially a mindset.

Centralizing decisions about structure is very tempting; it’s easy to overestimate your ability to “get it right” the first time,

or find the “one true model” for all.

But …

What if a database's job is really to store whatever fields I give it?

What if I want to say, "Let me store any additional facts I can think of about bands, venues, shows"?

If your model imposes less, then there's less to change when the world changes.

This analogous to the difference between strongly typed languages (like C and Java)

and scripting languages (like Ruby and Python).Sometimes you want to trade safety for flexibility.

As an example, Thom Yorke’s booking agent might use address book software.

But what if they had to update the database schema for each new kind of social media service he decides to use?

You can picture just adding more attributes, like key-value pairs in JSON:

{ id: "ARP5KQF1187B9B4DD1", name: "Explosions in the Sky", genres: [{name: "post rock"}], years_active: [{start: 1999}], artist_location: { location: "Austin, TX, US", city: "Austin", region: "Texas", country: "United States" }}

You can picture just adding more attributes, like key-value pairs in JSON:

{ id: "ARP5KQF1187B9B4DD1", name: "Explosions in the Sky", genres: [{name: "post rock"}], years_active: [{start: 1999}], artist_location: { location: "Austin, TX, US", city: "Austin", region: "Texas", country: "United States" }, familiarity: 0.687572, favorite_color: "Blue"}

“Tagging” is the simplest version of this.

Each tag is a boolean (true/false), and there can be any number of them.

You can already model attribute flexibility at a meta-level in a relational database...

Band ID Name

1234 Bjork

5678 Radiohead

Band ID Attribute Value

5678 City Austin

5678 State TX

5678 Fav. Color Blue

5678 Familiarity 0.687572

After all, it’s just another degree of cardinality, right?

But writing queries against it is a massive PITA (trust me).

And, it’s the same meta problem: if you did this for every entity in your model, your model would

be impossible to comprehend.

Attribute flexibility is “table stakes” for new DBs:● Google did it with BigTable - the columns for

a row are totally flexible at run time, and the values are simple byte arrays

● Most other NoSQL stores offer this too● Some services make it the backbone of

what they offer (e.g. keen.io - 1 entity, but any set of attributes you want to send)

My second pronouncement:

Henceforth, most databases should really think about offering attribute flexibility, accepting writes and reads of “columns” that haven’t been declared in advance.

Aside: if columns aren’t predefined, how do we know their data type?● Strings, Numbers, Dates, etc?● JSON has the right idea:

○ If it’s in quotes, it’s a string○ If it’s not, it’s either:

■ a number → ■ true or false

And … while we’re at it, why not just get rid of attributes altogether, and say that entities store JSON blobs, with nesting intact?

(We’ll come back to that …)

Model Agility

"Schemas usually remain relatively stable over the lifetime of a database for most applications."

- S Navathe, 1992

"No one will need more than 637 kB of memory for a personal computer."

- B. Gates, 1979

It’s true, though. Database schemas do tend to remain stable … because it sucks so bad to change them!

● Honestly, databases have always been the least “agile” part of software development.

● It's the final frontier of "BDUF" (big design up front)

But, wait. Aren’t schemas stored as data? Can’t you just change them?

ALTER TABLE Band ADD COLUMN favorite_color STRING

In theory, that works. In practice, it doesn’t.

● You can change the schema, but you can only ever have one schema at a time

● Some changes might require downtime● For big tables, that could mean you’re offline for hours,

days, or even weeks.● And, by the way, if you follow my first pronouncement

about data and time travel, what happens when you change the schema?

What to do instead?

One option is to generalize: make your model so generic, you never have to alter a table!

This is what “architecture astronauts” do, and it doesn’t work, because you eventually end up with this model:

(You can sometimes find a sweet spot, but more likely you’re just pushing the essentialism and brittleness to another layer.)

What we really need is a system where structure changes are also non-destructive.

(We’d need that anyway if we want to time travel with the data, right?)

There’s actually an example of this!

http://couchdb.apache.org/

Apache CouchDB is a NoSQL database.

● The database is just a flat collection of JSON files● There’s no schema! Put in whatever you want!● You create views (using javascript) that “materialize”

certain access patterns across your documents.

It’s not perfect (scaling is tricky, etc.) but as an illustration of these points, it’s spot on.

CouchDB hits all 3 of our mutability desires:

● All documents are versioned. ○ Non-destructive Mutation!

● Document JSON can have any structure○ Attribute Flexibility!

● Schema-On-Read using views○ Model Agility!

It also happens to be wicked easy to get started with, so let’s do one more

hands-on exercise.

Hands on!

Let’s load some music data into CouchDB!

You should have already installed CouchDB, but if not, do so now.

Hands on!

Earlier, we used the Do512 API. This time we’ll use The Echo Nest, a huge music data repository. (Which was, incidentally, just bought by Spotify yesterday, March 6th, 2014! Too bad we didn’t get on this listen.up thing a little sooner amiright?)

They require setting up an API key for access, so as a shortcut I’ve done that part for you.

Hands on!

The file mutation/echonest-artists-austin-v1.json is data on the top 100 “most familiar” artists with a hometown of Austin, TX, from this API call:http://developer.echonest.com/api/v4/artist/search?api_key={MY_API_KEY}&format=json&results=100&start=0&bucket=familiarity&bucket=genre&bucket=artist_location&bucket=years_active&artist_location=austin

Full disclosure, I modified the result slightly, so it’d work immediately with CouchDB bulk load:● removed the outer "response" wrapper from the API● changed the name of the array from "artists" to "docs"● changed all the "id" fields to "_id" so CouchDB would use them

Hands on!Ensure CouchDB is installed and running:

http://127.0.0.1:5984/_utils/index.html

Create a new database called

“listenup”

Hands on!Upload a bunch of data:

$ cd ~/unstructure-sxsw14/mutation/$ curl -H "Content-Type:application/json" -d @echonest-arists-austin_v1.json -X POST http://127.0.0.1:5984/listenup/_bulk_docs

Reload the database web page:http://127.0.0.1:5984/_utils/database.html?listenup

Hands on!See what happens when you change a value. Let’s pick a band at random:

http://127.0.0.1:5984/_utils/document.html?listenup/ARJXDMJ11FF10D84F9

Double click, and change to: [{"name": "garage

soul"}]Then click “Save Document”.

Hands on!Now there are two versions of this document:

Non-destructive mutability, in the flesh!

Hands on!Now add a field that never existed before:

http://127.0.0.1:5984/_utils/document.html?listenup/ARJXDMJ11FF10D84F9

Click “Add Field”.

Add “Influences” as the Field name, and click

“Save Document”.

Double click the “null” next to “Influences”, and change the value to

[{"name": "Michael McDonald"}]Click “Save Document”.

Hands on!

Attribute flexibility!

Notice that so far, we haven’t once had to specify a schema.

Hands on!

Now we query the data by writing code to implement materialized views. This is actually a little complicated ...

Hands on!

(Yes, I know this is actually about Riak, not CouchDB. Why do you hate laughing?)Comic by John Muellerleile / http://thinkdifferent.ly/fault-tolerance.png

Hands on!As a simple example, create a new document:

{ "_id": "_design/application", "views": { "genre-view": { "map": "function(artist) {if(artist.name && artist.genres) {artist.genres.forEach(function(genre) {emit(genre, artist.name);});}}" } }}

Then visit: http://127.0.0.1:5984/listenup/_design/application/_view/genre-view

Hands on!Try making your own view. Some examples are here:

http://guide.couchdb.org/draft/cookbook.html

Questions?

Chapter 5: Conclusion & Future Directions

So where have we been?

Hierarchy

Relation

Mutation

Before we go off to our subsequent debauchery, if you’ll indulge me, we

actually have a tiny bit more smashing to do.

Now it’s time to go through the looking glass.

Image from Disney Wikia: http://disney.wikia.com/wiki/Alice

(Some of these thoughts are adapted from my previous presentation, I’ve Always Wanted To Data Model)

This is a technical book from the 1970s. It’s a philosophy book.

It opened my eyes to some of the real

underlying questions.

Is data “true”?

Of course not, not categorically.

Member Year Joined

Thom Yorke 1985

Jonny Greenwood, 1985

Ed O'Brien, 1985

Colin Greenwood 1985

Philip Selway 1985

Ian Varley 2014

Official Radiohead Band Members

Is data “real”?

Robin Hood’s Band Of Merry Men

Depends on what you mean.

Merry Man Merry-ness

Robin Hood High

Little John Medium

Much The Miller’s Son Medium

Friar Tuck High

Arthur a Bland Low

Maid Marian Unknown

My personal theory:

Data is “existential claims”.

The fact that this data, and not some other, is stored implies that someone (or something) is making a claim about the existential state of something. This claim may or may not correspond to the actual existential state of that something.

As cool as the boundary-smashing tools we looked at today are, it’s also wise to

remember that we don’t really know much of what’s going on. For example ...

Nobody actually knows what an “entity” really is.

“Entity” is another word for Category, in linguistics terms.

And an important property of linguistic categories is that they are slippery.

See:● Steven Pinker: The Stuff Of Thought● Douglas Hofstadter: Surfaces & Essences● George Lakoff: Women, Fire, and Dangerous Things

part: an abstract definition of a connected set of physical materials that serve some purpose

part: one instance of a manufactured item, which either does or does not meet quality standards

Images: (1) Atwood Hydraulic Surge Brake Actuator, http://www.pacifictrailers.com/Atwood-Hydraulic-Brake-Actuator-Parts-List-and-Schematic/; (2) Ford Motor Company flywheel magneto assembly line 1913, source unknown

And if you think you can “solve” the problem, I’ve got some World Trade Center insurance policies to sell you.

That said, there are a couple techniques we could adopt that would help:

● First-class Sub- / Super-Typing● First-class Scoping and Aliasing

(Not that there aren’t ways to do this in relational models, but they’re unobvious and not widely used.)

Discrete models encourage black & white thinking in a

gray world

Sometimes the deep structure is actually ambiguous.

Image credit: By Chire (Own work) [Public domain or Public domain], via Wikimedia Commons

Our current tools are s#!t.

Boxes & lines aren’t how we actually think, our spatial processing of diagrams doesn’t map well to our

temporal, spatial, and causal comprehension of data structure.

F*** THAT NOISE.

If we had the right tools, what would they look like?

They’d have native support for ...

● My mutability requirements○ Non-destructive mutation○ Attribute Flexibility○ Model Agility

● The 3 Ps: ○ Provenance, Provability, Probability

They’d have native support for ...

● Supertyping● Extensible Meta-Metadata● Semantic Zoom● Prototype Generation● Model Versioning and Diffing

Interesting direction: RAML

● Textual (YAML-based)● Strong, simple syntax● Generates useful models● Communication focussed

Could there be something like this for data models?

Who knows ...

Thanks! @thefutureianianvarley.com

(If you enjoyed this, please rate it a 5 … sxsw.com/rate)