The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and...
Transcript of The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and...
![Page 1: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/1.jpg)
The Shifting Storage Spectrum
Tim BrayWeb GuySun Microsystems
![Page 2: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/2.jpg)
The Shifting Storage Spectrum
Tim BrayWeb GuySun Microsystems
![Page 3: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/3.jpg)
“Memory is the new disk; Disk is the new tape.”
![Page 4: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/4.jpg)
The Storage HierarchyRegisters
L* Cache
DRAM
SSD
Disk
Tape
![Page 5: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/5.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 6: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/6.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Bytes
Normalized tuples
![Page 7: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/7.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
Distributed hash table
Key/Value pairs
![Page 8: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/8.jpg)
www.danga.com/memcached/For example:
![Page 9: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/9.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 10: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/10.jpg)
Application code
SQL engine
OS and Filesystem
Caches and disks
SQL
Normalized tuples
Bytes
![Page 11: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/11.jpg)
wikipedia.orgFor example:
![Page 12: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/12.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 13: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/13.jpg)
Application code
OS and Filesystem
Caches and disks
XML, JSON, plain text, media files, ...
Bytes
![Page 14: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/14.jpg)
tbray.org/ongoing/For example:
![Page 15: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/15.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 16: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/16.jpg)
Object store
Application code
OS and Filesystem
Caches and disks
Objects
Serialized objects
Bytes
![Page 17: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/17.jpg)
ruby.gemstone.comFor example:
![Page 18: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/18.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 19: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/19.jpg)
Minimal SQL
engine
Application code
O/R mapping
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 20: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/20.jpg)
krow.livejournal.com/622006.htmlFor example:
![Page 21: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/21.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 22: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/22.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 23: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/23.jpg)
Column-oriented DBMS
Application code
O/R mapping
OS and Filesystem
Caches and disks
Objects
SQL-like queries
Bytes
Normalized tuples
![Page 24: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/24.jpg)
appengine.google.comFor example:
![Page 25: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/25.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 26: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/26.jpg)
Document-oriented DBMS
Application code
OS and Filesystem
Caches and disks
Name-value pairs and blobs
Bytes
XML, JSON, plain text, media files, ...
![Page 27: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/27.jpg)
For example: incubator.apache.org/couchdb/
![Page 28: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/28.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 29: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/29.jpg)
Web space via HTTP
Application code XML, JSON, plain text, media files, ...
• WebDAV (filesystem metaphor) • AtomPub (REST)
![Page 30: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/30.jpg)
Clients Procs PostsSecondsElapsed %Idle
Postsper sec
1 10 1000 ~10.8 85 ~90
2 20 2000 ~12.5 72 ~160
3 30 3000 ~13.5 53 ~220
4 40 4000 ~14.2 46 ~280
5 50 5000 ~15.4 56 ~320
For example: tbray.org/ongoing/When/200x/2008/11/02/How-Fast-is-Fast
![Page 31: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/31.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 32: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/32.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes?!
![Page 33: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/33.jpg)
The Storage HierarchyRegisters
L* Cache
DRAM
SSD
Disk
Tape
![Page 34: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/34.jpg)
The Storage HierarchyTypicalLatency
TypicalThroughput
Registers < 1 nsec n/a
L* Cache < 10 nsec > 100 G/sec
DRAM < 500 nsec > 1 G/sec
Dist. hash table < 50 µsec n/a
SSD/Disk ? ?
Tape < 60 sec > 100 M/sec
![Page 35: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/35.jpg)
![Page 36: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/36.jpg)
![Page 37: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/37.jpg)
What Bonnie Believes• Applications need to read & write files.• Sometimes they use smart block I/O, sometimes
they process characters and rely on buffering libraries.
• Sometimes they need to update files in place.• Sometimes they need to seek around.• The OS filesystem RAM cache always eventually
runs out and you have to do real disk I/O.
![Page 38: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/38.jpg)
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPUPro 12 64.7 82.0 67.4 10.0 29.8 5.0 64.8 76.7 67.9 6.5 190 0.7
Some Bonnie Output
![Page 39: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/39.jpg)
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPUPro 12 64.7 82.0 67.4 10.0 29.8 5.0 64.8 76.7 67.9 6.5 190 0.7T2K 100 20.5 100 150.1 100 61.4 64.8 19.8 98.9 148.9 76.7 214 10.7
Some Bonnie Output
![Page 40: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/40.jpg)
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPUPro 12 64.7 82.0 67.4 10.0 29.8 5.0 64.8 76.7 67.9 6.5 190 0.7T2K 100 20.5 100 150.1 100 61.4 64.8 19.8 98.9 148.9 76.7 214 10.77410 16 125.9 98.6 260.5 56.5 90.3 28.9 106.1 98.6 237.2 38.4 1058 93.1
Some Bonnie Output
![Page 41: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/41.jpg)
-------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU /sec %CPUPro 12 64.7 82.0 67.4 10.0 29.8 5.0 64.8 76.7 67.9 6.5 190 0.7T2K 100 20.5 100 150.1 100 61.4 64.8 19.8 98.9 148.9 76.7 214 10.77410 16 125.9 98.6 260.5 56.5 90.3 28.9 106.1 98.6 237.2 38.4 1058 93.1SSD 8 44.8 66.4 69.3 12.8 51.5 10.7 73.4 94.3 246.0 27.0 7856 43.2
Some Bonnie Output
![Page 42: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/42.jpg)
The Storage HierarchyTypicalLatency
TypicalThroughput
Registers < 1 nsec n/a
L* Cache < 10 nsec > 100 G/sec
DRAM < 500 nsec > 1 G/sec
Dist. hash table < 50 µsec n/a
SSD < 200 µsec > 100 M/sec
Disk < 10 msec > 100 M/sec
Tape < 60 sec > 100 M/sec
![Page 43: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/43.jpg)
“Memory is the new disk; Disk is the new tape.”
![Page 44: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/44.jpg)
“Memory is the new disk; Disk is
the new tape.Unless it’s SSD.”
![Page 45: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/45.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes
![Page 46: The Shifting Storage Spectrum - tbray.org · Application code O/R mapping SQL engine OS and Filesystem Caches and disks Objects SQL Normalized tuples Bytes](https://reader033.fdocuments.us/reader033/viewer/2022051915/6006f4405023824d55140f85/html5/thumbnails/46.jpg)
Application code
O/R mapping
SQL engine
OS and Filesystem
Caches and disks
Objects
SQL
Normalized tuples
Bytes