Berkeley, CA: 1984
Transcript of Berkeley, CA: 1984
![Page 1: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/1.jpg)
The Land Sharks are on the Squawk Box
(Why Riding a Bicycle Across America and Building Postgres Have a Lot in
Common)
2
![Page 2: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/2.jpg)
3
Kennebago, ME: Summer 1993
• t
![Page 3: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/3.jpg)
4
Part 1: Off to a Great Start
![Page 4: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/4.jpg)
5
Anacortes, WA: Day 1 - June 4, 1988
![Page 5: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/5.jpg)
6
Berkeley, CA: 1984
• Commercial Ingres is 4 years old — And much better code than academic Ingres
• Makes no sense to continue prototyping on our code line
• We push the code off a cliff, and start something new
• So what is Postgres to be?
![Page 6: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/6.jpg)
7
Berkeley, CA: 1984
• Abstract Data Types (ADTs)
— Ingres, System R were focused on business data processing (competing against IMS and Codasyl) — Motivated by a large collection of papers of the form:
• “we tried RDBMS on X; it didn’t work, so we added Y to the relational model”
![Page 7: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/7.jpg)
8
Example from GIS
Find me all employees who overlap a specific (14, 17, 0, 16) rectangle? Select name where xmin < 17 and xmax > 14 and ymin < 16 and ymax > 0 Messy and very difficult to optimize!!!
name age xmin xmax ymin ymax
Sam 38 100 220 46 87
![Page 8: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/8.jpg)
9
Better Solution
Select name where location Overlaps MakeBox (14, 17, 0, 16)
name age location
Sam 38 Internal representation for a box
![Page 9: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/9.jpg)
10
In General
User-defined data types (Rectangle) User-defined operators (Overlaps) User-defined functions (MakeBox) Requires adding functions to DBMS engine and calling them at appropriate times during query execution…
![Page 10: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/10.jpg)
11
The Devil is in the Details
• Using new types in indexes (B-trees, hashing) • New indexes (e.g. R-trees) • Teaching the query optimizer about new types • Specifying commutativity rules • Optimization of “not” • And others
![Page 11: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/11.jpg)
Winthrop, WA: Day 3
12
![Page 12: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/12.jpg)
Rainy Pass, WA: Day 3
13
![Page 13: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/13.jpg)
Washington Pass, WA: Day 3
14
![Page 14: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/14.jpg)
15
Berkeley, CA: 1985-86
• Chris Date wrote a pioneering paper on referential integrity
• Basically what to do with foreign-key-primary-key constraints
![Page 15: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/15.jpg)
16
Berkeley, CA: 1985-86
What happens if you delete the Candy department?
dname floor sq. ft. budget
Shoe 3 500 40,000
Candy 2 800 50,000
name dept salary age
Bill Shoe 2000 40
Art Candy 3000 35
Sam Shoe 1500 25
Tom Toy 1000 23
![Page 16: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/16.jpg)
17
Date’s Solution
• Cascade the delete • Replace dept by “null” • Refuse the delete
• Plus three corresponding actions on inserts
![Page 17: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/17.jpg)
18
My Idea: A Single Mechanism
• Referential integrity • Triggers (make sure a salary update to
Sam gets cascaded to Bill) • Integrity constraints (salaries must be
non-negative)
![Page 18: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/18.jpg)
19
I Hate IF-THEN Rule Systems
• Consider two rules: — Make sure George is in the Shoe department — Move everybody over 40 from Shoe to Candy
• Order dependent!!!! • Made worse by SQL being a set-oriented
language!
• No mere mortal can understand a 20 line Prolog program!
![Page 19: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/19.jpg)
20
My Solution
• Update ALWAYS command • Make sure Bill has the same salary as Sam
— Update ALWAYS (set salary = E.salary) where E.name = “Sam” and name = “Bill”
— Semantics: the command appears to be perpetually running
• Somewhat cleaner than an if-then rules system….
![Page 20: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/20.jpg)
Marias Pass, MT: Day 15
21
![Page 21: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/21.jpg)
Traditional DBMS Crash Recovery
Two data stores with different access engines, which must be carefully synchronized!! Log code is complicated and ugly!
22
DBMS
Data Log
![Page 22: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/22.jpg)
Postgres Picture: No Overwrite
Gets rid of crash recovery code! Provides “time travel” in addition
23
DBMS
Data + Log
![Page 23: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/23.jpg)
24
The Devil is in the Details
• Log is “write optimized” • Data is “read optimized” • No overwrite must be competitive – tricky
placement problem
![Page 24: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/24.jpg)
25
Postgres Game Plan
• ADTs • Rules using an “always” command • No overwrite storage
• There was other (less important) stuff… — Complex objects — Inheritance
![Page 25: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/25.jpg)
26
Part 2: Things Aren’t Going Well
![Page 26: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/26.jpg)
27
Drake, ND: Day 26
![Page 27: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/27.jpg)
Berkeley, CA: 1986-87 Reality Sets In
• Lisp is a disaster — Lots of code over the cliff
• ADT system works well — Multiple semantics for time
• Time travel has promise, but is a tricky to tune
• “Always” doesn’t work — Lots of code over the cliff — And we go back to a exploring a
conventional rule system
28
![Page 28: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/28.jpg)
Berkeley, CA: 1986-90 Reality Sets In
• The “Wine Connoisseur” and I spend a lot of time on:
— How complex can complex objects be — What to do about time travel, rules, …
• Next 3 years was a lot of “repair” — Think of this as a “slog through the
swamp”
29
![Page 29: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/29.jpg)
Part 3: Another High Point
30
![Page 30: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/30.jpg)
Carrington, ND: The Next Afternoon Serendipity
31
![Page 31: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/31.jpg)
Berkeley, CA: 1991-92 Commercial Postgres
• In the DBMS field, the ultimate arbiter of good ideas is the commercial marketplace
• In my opinion, technology transfer is best done
through a startup — Large companies have their own agenda — Read “The Innovators Dilemma” by Clayton
Christensen
32
![Page 32: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/32.jpg)
• Recruited “Mom” and her husband “Short One” • “EMP1” and “Quiet One” came over from the
academic team, joined shortly thereafter by “Triple Rock”
• With the “Tall Shark” as interim CEO and lead investor
• We were off and running – converting the query language from QUEL to SQL, hardening the code and improving performance
33
Berkeley, CA: 1991-92 Commercial Postgres
![Page 33: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/33.jpg)
Luddington, MI: Day 38
34
![Page 34: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/34.jpg)
Berkeley, CA: 1993-94
• After a couple of miscues, we are named Illustra
• Got the first few customers • Raised more money (see slide 1) • Hired “Voice of Experience”, “Uptone”
and “Smooth” -- a real management team
• The future is looking up….
35
![Page 35: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/35.jpg)
Part 4: Another Low
36
![Page 36: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/36.jpg)
Ellicottville, NY: Day 49
37
![Page 37: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/37.jpg)
Berkeley, CA: 1994 - 1995 Reality Sets In
• Catch-22 between customers and ADT providers — Customers wanted ADTs from the large app vendors — App vendors wanted a distribution channel
• The dreaded “down round” — “a fate worse than death”
38
![Page 38: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/38.jpg)
Part 5: Looking Good
39
![Page 39: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/39.jpg)
Troy, NY: Day 56 The Last Hill
41
![Page 40: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/40.jpg)
Berkeley, CA: 1995 Serendipity
• The internet is taking off • “Uptone” turns the company on a dime
and we become “the database for cyberspace”
• Get a lot of buzz • Maybe we have turned the corner
40
![Page 41: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/41.jpg)
Part 6: More Challenges
42
![Page 42: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/42.jpg)
Berkeley, CA: 1995 Reality Sets In (Again)
• The internet bake off • On business data processing data!!!!
— Transaction processing is not in our “wheelhouse”
43
![Page 43: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/43.jpg)
Part 7: The Ending
44
![Page 44: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/44.jpg)
Wollaston Beach, MA: Day 59
45
![Page 45: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/45.jpg)
Berkeley, CA: February 1996 Serendipity
• Illustra is purchased by one of the elephants
— Solves the Catch-22 on ADTs — Solves the transaction processing problem
• We set about putting Illustra features into their system
46
![Page 46: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/46.jpg)
Why Tell a Bicycle Story?
Algorithm for a cross-country bike trip:
Until (ocean) { get up in the morning; ride east; persevere and overcome any obstacles
that arise; }
47
![Page 47: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/47.jpg)
A Bit of Abstraction…
Until (GOAL) {
get up in the morning; ACTION; persevere and overcome any obstacles
that arise; }
48
![Page 48: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/48.jpg)
And a Macro
Make It Happen
49
![Page 49: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/49.jpg)
Why Would Anybody Want to Ride Across America?
• Long and hard • Periods of depression and elation • Lots of monotony (slogging through the
swamp)
50
![Page 50: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/50.jpg)
My Career
Make It Happen (PhD) – 5 years Make It Happen (tenure) – 5 years Make It Happen (bike ride) – 2 months There is an obvious conclusion to be drawn…
51
![Page 51: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/51.jpg)
Building Postgres Required Make It Happen
Have a good idea (or 2 or 3) Make it Happen (prototype) Start a company Hire superb implementors Hire talented execs Make It Happen (product) (Have a good idea then Make It Happen for 10 years) Why would anybody in their right mind do this?
52
![Page 52: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/52.jpg)
My Personal Thoughts On Good Ideas
• I have never had any good ideas by communing with nature on the mountaintop
• Instead, I recommend: - talking to users - talking to more users - hanging around a critical environment of smart people (like MIT or Berkeley) • For my take on current DBMS ideas: see my test-of-time talk in ICDE ‘15
53
![Page 53: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/53.jpg)
Closing Thoughts
• Where does Make It Happen come from? - I have no idea!
• What about serendipity?
- Really important in North Dakota! - Really important to Illustra! - Often a very significant factor - As is anti-serendipity!!!!
54
![Page 54: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/54.jpg)
Boston, MA: Present Day
• Our bicycle (Boston Bound) sits in our basement gathering dust
• It has not been ridden since Wollaston Beach
• I still love to take on physical challenges (climbing the NH 4000 footers)
• Illustra was successfully integrated into the Informix code base and is still available
55
![Page 55: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/55.jpg)
Boston, MA: Present Day
• Postgres ADTs have been added to most RDBMSs
• The biggest impact of Postgres came from the shepherding of the open source Berkeley system
• Pure serendidity! • Postgres code has formed the starting point for
several DBMS startups
55
![Page 56: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/56.jpg)
Boston, MA: Present Day
I am still building new DBMSs (“one size fits one”): C-Store/Vertica column store for the data
warehouse market H-Store/VoltDB main-memory row store for
the OLTP marketplace SciDB/Paradigm4 array engine for the complex
analytics marketplace Each a decade (and counting) Could have used any of these in this talk!
56
![Page 57: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/57.jpg)
The Following 39 Berkeley Students and Staff
Wrote Postgres (with data model collaboration from Larry Rowe)
Jeff Anton, Paul Aoki, James Bell, Jennifer Caetta, Philip Chang, Jolly Chen, Ron Choi, Matt Dillon, Zelaine Fong, Adam Glass, Jeffrey Goh, Steven Grady, Serge Granik, Marti Hearst, Joey Hellerstein, Michael Hirohama, Chin-heng Hong, Wei Hong, Anant Jhingren,Greg Kemnitz, Marcel Kornacker, Case Larsen, Boris Livshitz, Jeff Meredith, Ginger Ogle, Mike Olson, Nels Olsen, Lay-Peng Ong, Carol Paxson, Avi Pfeffer, Spyros Potamianos, Sunita Surawagi, David Muir Sharnoff, Mark Sullivan, Cimarron Taylor, Marc Teitelbaum, Yongdong Wang, Kristen Wright, and Andrew Yu
57
![Page 58: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/58.jpg)
None of This Would be Possible Without:
• Incredibly talented system programmers - People like “EMP1,” “Triple Rock” and “Quiet One” • Start-up executives - People like “Uptone,” “Smooth” and “Voice-of- experience” - “Cueball” • The land sharks - Especially “The Believer”
59
![Page 59: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/59.jpg)
None of This would be Possible Without
My Wife and Family
59
![Page 60: Berkeley, CA: 1984](https://reader033.fdocuments.us/reader033/viewer/2022051521/5868d8871a28abd33f8c1faa/html5/thumbnails/60.jpg)
In Memory Jim Gray
60