SQL for the IoTRichard Hipp

Percona University “Smart Data”Raleigh, NC2015-02-12

versus

in addition to

Global DataLocal Data

Enterprise Data DepotApplication File

Centralized DataEdge Data

does not compete with

competes with fopen()

About SQLite

● In-process library● Direct I/O to disk● Zero-configuration● Automatic recovery● Portable C89 source code● Cross-platform, stable, well-defined file format● Public domain● https://www.sqlite.org/

SQLite By The Numbers

● 1 file of C89 code: “sqlite3.c”● 3 developers● 100% MC/DC testing● 93,000 SLOC● 500,000 bytes compiled● 1,000,000 applications use it● 163,295,153 tests● 2,000,000,000 new installs during 2014

SQLite Features

● Well-defined Cross-platform File Format● Power-safe Transactions● Triggers● Foreign Keys● Full-Text Indexes● R-Tree Indexes● Recursive Common Table Expressions● Partial Indexes

SQLite Limits

● 1 writer and N readers per database● 64-way joins● 1 gigabyte strings and/or BLOBs● 140 terabytes per database● 125 databases per connection

Flexible Datatypes

● SMALL INT will actually hold any size integer.● VARCHAR(5) will hold a 1-billion byte string● No native DATETIME format:

– ISO8601 text: '2015-02-12 14:23:51.142'

– Julian day number: 2457068.09989748

– Unix time: 1423923831.142

● Integers work as BOOLEAN, as in C/C++

sqlite.org/intern-v-extern-blob.html

Simple C/C++ Interface Summary

● sqlite3_open()● sqlite3_prepare_v2()● sqlite3_bind_xxxxx()● sqlite3_step()● sqlite3_reset()● sqlite3_finalize()● sqlite3_close()

Run-time Extensible

● Application-defined SQL functions● Virtual Tables● Application-supplied collating sequences● Pluggable filesystem interface (VFS)● Alternative memory allocators

Command-line Tool: sqlite3.exe

● sqlite3 DATABASEFILE ?SQL?● .tables● .schema● .mode csv● .import CSVFILE TABLE● .read SCRIPT● .help● https://www.sqlite.org/cli.html

sqlite3_analyzer.exe

● https://www.sqlite.org/download.html● “make sqlite3_analyzer”● sqlite3_analyzer DATABASEFILE >out.txt● Space used per table and per index● Average and max row sizes● Packing efficiencies● And so forth...

Building From Sources

● Sources from https://www.sqlite.org/src– Click on desired version

– Click on “Tarball” or “ZIP archive”

● ./configure; make TARGET● nmake /f makefile.msc

– sqlite3.c

– sqlite3.exe

– sqlite3_analyzer.exe

– test

Database Engine As Data Filter

Application DB Engine

Storage Decision Checklist

Remote Data?

Concurrent Writers?

Big Data?

Otherwise

Storage Decision Checklist FAIL!

Remote Data?

Concurrent Writers?

Big Data?

Otherwise

fopen()

No!

SQLite versus fopen()

● Power-safe atomic transactions● Cross-platform and cross-language● Higher performance● Automatic concurrency control● Incremental updating

●Uses SQL

Unstoppable Ideas Behind SQL

(1) Data Abstraction

(2) Declarative Language

H/T: Alex Scott, http://www.slideserve.com/kezia/relational-database-internalsi

“Representation is the essenceof computer programming.”

"Show me your flowcharts andconceal your tables, and I shallcontinue to be mystified. Showme your tables, and I won't usuallyneed your flowcharts; they'll beobvious."

- Fred Brooks, The Mythical Man-Month, pp102-103

"Data dominates. If you've chosenthe right data structures andorganized things well, the algorithmswill almost always be self-evident.Data structures, not algorithms, arecentral to programming."

- Rob Pike: Rule of Programming #5

"Bad programmers worry aboutthe code. Good programmersworry about data structures andtheir relationships."

- Linus Torvalds, on the Git mailing list, 2006-06-27

Unstoppable Ideas Behind SQL

(1) Data Abstraction

(2) Declarative Language

Ask the machine questions, ratherthan tell it what to do.

There was another big cultural shift at Google...The SQL-based analytics system, Dremel... made a lot of SQL converts at Google. People realized it is incrediblypowerful to just push the semantics of your query down intothe storage system and let it figure out what to do.

Alexander Lloyd at Berlin Buzzwords, 2012

Low-Volume Web Applications

● Example: https://www.sqlite.org/– 400K hits/day

– 80-85% static content

– ~200 SQL statements per page - mostly reads

– Most pages in less than 100 millisec

– 1/24th slice of a real server (Linode)

– 0.1 load average

Data Analysis

● Easily load up a one-off database from CSV or other sources.

● Powerful SQL queries● Simple integration with analysis scripts● Collaborate via email attachment, or passing

single files via USB stick● Popular in Bioinformatics

JSON

10011010010111

Pile offiles

<xml/>

Real Example: Adobe Lightroom

Real Example: MicroStation

SQLite Archiver

● http://www.sqlite.org/sqlar● Transactional● Concurrent & random access● File size similar to ZIP

CREATE TABLE sqlar( name TEXT PRIMARY KEY, -- name of the file mode INT, -- access permissions mtime INT, -- last modification time sz INT, -- original file size data BLOB -- compressed content );

SQLAR versus ZIP

-rw-r--r-- 1 drh drh 10218329 Feb 10 12:49 percona.odp-rw-r--r-- 1 drh drh 10221568 Feb 10 12:56 percona.sqlar-rw-r--r-- 1 drh drh 10175232 Feb 10 12:56 percona.zip

SQLAR is only 0.46% larger than ZIP

www.locusmap.eu

Data Transfer Format

Over-the-air Sync

Application

“Contrary to a lot of people'sviews on big data, I think themajority of the data will actuallybe analyzed and acted upon atthe edge of the network.”

John Chambers, CEO of CiscoQuoted in WSJ 2015-02-10

The database on the edgeof your network...