class 2 from bits to systems - Harvard...
Transcript of class 2 from bits to systems - Harvard...
![Page 1: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/1.jpg)
from bits to systemsprof. Stratos Idreos
HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/
class 2
![Page 2: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/2.jpg)
CS165, Fall 2016 Stratos Idreos /502
logistics, goals, etc
big data & systems (cont’d)
designing a data system algorithm: what can go wrong
quick overview: logical/physical/system design
today
![Page 3: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/3.jpg)
CS165, Fall 2016 Stratos Idreos /503
Marcin Zukowski
founder @Snowflake
previously founder @Vectorwise
guest lecture
10/17
![Page 4: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/4.jpg)
CS165, Fall 2016 Stratos Idreos /504
Johannes Gehrke
Microsoft
previously prof at Cornell
guest lecture
10/24
![Page 5: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/5.jpg)
CS165, Fall 2016 Stratos Idreos /505
HV Jagadish
Prof University of Michigan
CS Colloquium
10/6
![Page 6: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/6.jpg)
CS165, Fall 2016 Stratos Idreos /506
Magdalena Balazinska
Prof University of Washington
CS Colloquium
10/13
![Page 7: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/7.jpg)
CS165, Fall 2016 Stratos Idreos /507
research oriented, discussion based, material based on research papers
![Page 8: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/8.jpg)
CS165, Fall 2016 Stratos Idreos /50
labs, sections, and cs nights
8
Sections: video only / 1-2 videos a week first 2 sections on Project 0 and basics of C are online
OH: every day 3-4pm with Stratos at MD139
Labs (& CS Night): every day help with debugging and everything else
starting Tuesday Sep 6
away next week but most likely I will be able to hold OH remotely most days monitor Piazza and learn to use Zoom
![Page 9: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/9.jpg)
CS165, Fall 2016 Stratos Idreos /509
http://daslab.seas.harvard.edu/classes/cs165/project.html
code and testing infrastructure available as of today treat it as a good starting point
you have complete design freedom no libraries allowed
collaboration encouraged! final deliverable is personal face-to-face/hands-on evaluation
stay in touch!
![Page 10: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/10.jpg)
CS165, Fall 2016 Stratos Idreos /5010
spring 2014, somewhere at Harvard, late at night working on cs165 project
James & Wilson
![Page 11: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/11.jpg)
CS165, Fall 2016 Stratos Idreos /5010
spring 2014, somewhere at Harvard, late at night working on cs165 project
James & Wilson
“This class is invaluable.”
“One of the greatest intellectual journeys to date.”
several months later…
![Page 12: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/12.jpg)
CS165, Fall 2016 Stratos Idreos /5011
“TAKETHISCLASSIFYOUWANTTOACTUALLYLEARNSOMETHINGANDBECOMEATRUECOMPUTERSCIENTIST!“
“CSisawesomeandDatabasesrock!Icameinnotbeingverycon0identinmycsskillsormyunderstandingofhowmemoryhierarchyandlargesystemsworkandnoweverythingischanged!Ialsojustfeelveryinspiredtoexploremoreintheareaofdatabasesinthefuture!Loved
thisclass!“
“Project is brutal and rewarding.”
“The course is amazing! It's relaxed, it teaches you about modern database systems, it introduces you to
current research topics, it's freaking incredible! “
“Iabsolutely,wholeheartedlyloved165!!!!”
student testimonials (from Q)
![Page 13: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/13.jpg)
CS165, Fall 2016 Stratos Idreos /50
should I take the class now or next year (or never)? it depends (but mostly up to you and your energy levels)
delaying can be better if you not senior unless you have research ambitions
12
can I take it pass fail? no
can I audit? yes (but limited slots)
FAQ
![Page 14: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/14.jpg)
CS165, Fall 2016 Stratos Idreos /50
cs165 diff from 2015
13
cleaned up DSL cleaned up starter code
give out parser code revamped testing infrastructure
performance tests (=bigger data) benchmark implementation
project 0 midway check-in 10%
class participation 20% more TFs (lab class status)
addition of new research papers from 2015/2016
![Page 15: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/15.jpg)
CS165, Fall 2016 Stratos Idreos /5014
we purposely build up slowly in class
from your side: Project 0 next week
and basic environment/tools for project then start with P1 the week after once we discuss the basics of storage
![Page 16: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/16.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
star(id, name, distance, density, …)
[1, star1, x1, y1, …] [2, star2, x2, y2, …] [3, star3, x3, y3, …] [4, star4, x4, y4, …] …
astronomer
15
![Page 17: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/17.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
star(id, name, distance, density, …)
[1, star1, x1, y1, …] [2, star2, x2, y2, …] [3, star3, x3, y3, …] [4, star4, x4, y4, …] …
data size store - access
paper - just look at it!10s/100s
data collection is the key
astronomer
15
![Page 18: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/18.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
star(id, name, distance, density, …)
[1, star1, x1, y1, …] [2, star2, x2, y2, …] [3, star3, x3, y3, …] [4, star4, x4, y4, …] …
data size store - access
paper - just look at it!
PC files - shell/excel
10s/100s
K/M
data collection is the key
learn a bit how computers work
astronomer
15
![Page 19: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/19.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
star(id, name, distance, density, …)
[1, star1, x1, y1, …] [2, star2, x2, y2, …] [3, star3, x3, y3, …] [4, star4, x4, y4, …] …
data size store - access
paper - just look at it!
PC files - shell/excel
PC files - custom
10s/100s
K/M
B
data collection is the key
need a bit more tailored analysis
learn a bit how computers work
need serious programming skills
astronomer
15
![Page 20: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/20.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
star(id, name, distance, density, …)
[1, star1, x1, y1, …] [2, star2, x2, y2, …] [3, star3, x3, y3, …] [4, star4, x4, y4, …] …
data size store - access
paper - just look at it!
PC files - shell/excel
PC files - custom
10s/100s
K/M
B
data sys. - declarativeT
data collection is the key
need a bit more tailored analysis
learn a bit how computers work
need serious programming skills
exploration - many users/updates
data-driven analysis
astronomer
15
![Page 21: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/21.jpg)
CS165, Fall 2016 Stratos Idreos /5016
“Increasingly, scientific breakthroughs will be powered by advanced computing capabilities that help researchers manipulate and explore massive datasets.
The speed at which any given scientific discipline advances will depend on how well its researchers collaborate with one another, and with technologists, in areas of eScience such as databases, workflow management, visualization, and cloud computing technologies.”
![Page 22: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/22.jpg)
CS165, Fall 2016 Stratos Idreos /5017
big data V’s (it is not about size only)
volume velocity variety veracity
actually none of that is really new…
new: our ability to gather and store machine generated data broad understanding that we cannot just manually get value out of data
![Page 23: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/23.jpg)
CS165, Fall 2016 Stratos Idreos /5018
declarative interface ask ‘’what’’ you want
db system
the system decides “how” to best store and access data
![Page 24: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/24.jpg)
CS165, Fall 2016 Stratos Idreos /5019
a data system stores data and provides access to data
& makes knowledge generation easy
![Page 25: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/25.jpg)
CS165, Fall 2016 Stratos Idreos /5019
a data system stores data and provides access to data
& makes knowledge generation easy
data system analysisdata
knowledge
![Page 26: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/26.jpg)
CS165, Fall 2016 Stratos Idreos /5019
a data system stores data and provides access to data
& makes knowledge generation easy
data system analysisdata
knowledgeX
![Page 27: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/27.jpg)
CS165, Fall 2016 Stratos Idreos /5020
cs165/265 student
data system kernel
(here is where all the magic happens!)
![Page 28: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/28.jpg)
CS165, Fall 2016 Stratos Idreos /5021
database kernel
data data data
algo
rithm
s/op
erat
ors
applications
sql
disk
memory
cpu
![Page 29: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/29.jpg)
CS165, Fall 2016 Stratos Idreos /50
a data system is a massivecollection of algorithms and
data structures
more than one ways to do the same thing
optimization: a smart way to dynamically decide which way to choose for each query
no good way to model a system
22
![Page 30: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/30.jpg)
CS165, Fall 2016 Stratos Idreos /50
a data system is a massivecollection of algorithms and
data structures
more than one ways to do the same thing
optimization: a smart way to dynamically decide which way to choose for each query
no good way to model a system
22
Research Problem 1
![Page 31: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/31.jpg)
CS165, Fall 2016 Stratos Idreos /5023
so what is a good data system?it depends…
application requirements
hardware
budget
energy profile
performance
conflicting goals
![Page 32: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/32.jpg)
CS165, Fall 2016 Stratos Idreos /5024
“Three things are important in the database world: performance, performance, and performance”
Bruce Lindsay, IBM ACM SIGMOD Edgar F. Codd Innovations award 2012
![Page 33: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/33.jpg)
CS165, Fall 2016 Stratos Idreos /5025
a “simple” example
assume an array of N integers: find all positions where value>x
data
qualifying positions
exists in all systems: sql, nosql, newsqlselect operator
even the simplest tasks are actually far from trivial
no obvious solutions; just a taste of what to come
![Page 34: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/34.jpg)
CS165, Fall 2016 Stratos Idreos /5026
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
![Page 35: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/35.jpg)
CS165, Fall 2016 Stratos Idreos /5026
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
res[j]=i; j+=(data[i]>x)
![Page 36: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/36.jpg)
CS165, Fall 2016 Stratos Idreos /5026
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 37: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/37.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 38: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/38.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
res
![Page 39: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/39.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 40: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/40.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 41: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/41.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 42: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/42.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 43: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/43.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 44: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/44.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 45: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/45.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
![Page 46: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/46.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
copy res
![Page 47: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/47.jpg)
CS165, Fall 2016 Stratos Idreos /50
data
memory
26
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if only 1% qualifies?
copy res
but how can we know?
![Page 48: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/48.jpg)
CS165, Fall 2016 Stratos Idreos /5027
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if 90% qualifies?
result size= qualifying values*w bytes
![Page 49: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/49.jpg)
CS165, Fall 2016 Stratos Idreos /5027
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if 90% qualifies?
result size= qualifying values*w bytes
bit vector for res? 1 0 0 0 1 1 1 0 1
vs
00000001 00000101 00000110 00000111 00001001
![Page 50: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/50.jpg)
CS165, Fall 2016 Stratos Idreos /5027
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if 90% qualifies?
result size= qualifying values*w bytes
bit vector for res? 1 0 0 0 1 1 1 0 1
vs
00000001 00000101 00000110 00000111 00001001
if statements= bad, bad, bad
![Page 51: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/51.jpg)
CS165, Fall 2016 Stratos Idreos /5027
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
what if 90% qualifies?
result size= qualifying values*w bytes
bit vector for res? 1 0 0 0 1 1 1 0 1
vs
00000001 00000101 00000110 00000111 00001001
and we haven’t even started discussing about how to find the qualifying values…
if statements= bad, bad, bad
![Page 52: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/52.jpg)
CS165, Fall 2016 Stratos Idreos /5028
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
thread1, core1
thread2, core2
thread3, core3
…
logically partition
![Page 53: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/53.jpg)
CS165, Fall 2016 Stratos Idreos /5028
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
thread1, core1
thread2, core2
thread3, core3
…
logically partition
NUMA architectures? SIMD functionality?
& what about result writing?
not as simple as spinning off N threads…
![Page 54: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/54.jpg)
CS165, Fall 2016 Stratos Idreos /5029
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
q1,q2,q3
N>>1 queries in parallel
![Page 55: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/55.jpg)
CS165, Fall 2016 Stratos Idreos /5029
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
q1,q2,q3
N>>1 queries in parallel
![Page 56: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/56.jpg)
CS165, Fall 2016 Stratos Idreos /5029
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
q1,q2,q3
N>>1 queries in parallel
![Page 57: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/57.jpg)
CS165, Fall 2016 Stratos Idreos /5029
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
q1,q2,q3
N>>1 queries in parallel
q4
![Page 58: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/58.jpg)
CS165, Fall 2016 Stratos Idreos /5029
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
q1,q2,q3
N>>1 queries in parallel
q4
q4
![Page 59: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/59.jpg)
CS165, Fall 2016 Stratos Idreos /5030
assume an array of N integers: find all positions where value>x
res=new array[data.size]
j=0 for (i=0; i<data.size; i++)
if data[i]>x res[j++]=i
res=new array[10] j=0
if data[0]>x res[j++]=0 if data[1]>x res[j++]=1 if data[2]>x res[j++]=2 if data[3]>x res[j++]=3 if data[4]>x res[j++]=4 if data[5]>x res[j++]=5 if data[6]>x res[j++]=6 if data[7]>x res[j++]=7 if data[8]>x res[j++]=8 if data[9]>x res[j++]=9
vs
![Page 60: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/60.jpg)
CS165, Fall 2016 Stratos Idreos /50
cost: data touched & computation
31
time
speed cpu
mem
![Page 61: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/61.jpg)
CS165, Fall 2016 Stratos Idreos /50
random access & page-based access
32
…
need to only read x… but have to read all of page 1
page1 page2 page3
data value x
registers
on chip cache
on board cache
memory
disk
CPU
data
mov
e
![Page 62: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/62.jpg)
CS165, Fall 2016 Stratos Idreos /5033
assume an array of N integers: find all positions where value>x
option 1: scan all data
option 2: use a tree (do not consider tree generation costs)
which one is best?
groups 3-4 students: try to keep noise level down, take turns speaking, etc.
![Page 63: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/63.jpg)
CS165, Fall 2016 Stratos Idreos /5034
it is not obvious that the tree would give the best result even if we ignore the building cost
random access at the granularity of one page at a time
data movement is a HUGE cost component
it depends on the query and the data
>1 ways to have a tree (implicit tree, linked leaves or not, etc.)
and happens with updates, >>1 concurrent queries, etc.
![Page 64: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/64.jpg)
CS165, Fall 2016 Stratos Idreos /5035
actually this was your second open research problem of the day!
systems are so fast= too little time to optimize
we will do all of these in great detail: the examples so far are meant to demonstrate the kinds of problems and discussion we will be doing
![Page 65: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/65.jpg)
CS165, Fall 2016 Stratos Idreos /5036
it all starts with how we store data every bit matters
design space
![Page 66: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/66.jpg)
CS165, Fall 2016 Stratos Idreos /50
C
37
and no libraries
![Page 67: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/67.jpg)
CS165, Fall 2016 Stratos Idreos /50
MAIN-MEMORY OPTIMIZED DATA SYSTEMS
38
![Page 68: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/68.jpg)
CS165, Fall 2016 Stratos Idreos /50
MAIN-MEMORY OPTIMIZED DATA SYSTEMS
38
![Page 69: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/69.jpg)
CS165, Fall 2016 Stratos Idreos /50
MAIN-MEMORY OPTIMIZED DATA SYSTEMS
38
![Page 70: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/70.jpg)
CS165, Fall 2016 Stratos Idreos /5039
history/timeline
~1960sphilosophy/ sciences
papyrus/ paper
dbs
dbs vs
hand code
dbs vs
OSs dbs …
dbs vs
nosql
~2015
declarative/physical/logical independence
recovery/consistency
dbs
![Page 71: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/71.jpg)
CS165, Fall 2016 Stratos Idreos /50
why noSQL
40
db noSQL
as apps become more complex
as apps need to be more scalable
complex legacy tuning
expensive fast …
simple clean
just enough cheap/free
slow
newSQL
![Page 72: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/72.jpg)
CS165, Fall 2016 Stratos Idreos /50
why noSQL
40
db noSQL
as apps become more complex
as apps need to be more scalable
complex legacy tuning
expensive fast …
simple clean
just enough cheap/free
slow
newSQL
regardless of system name/data model it is the same design space
and the same design principles
![Page 73: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/73.jpg)
CS165, Fall 2016 Stratos Idreos /50
design
41
logical design
physical design
system design
![Page 74: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/74.jpg)
CS165, Fall 2016 Stratos Idreos /5042
piazza forum
all announcements & discussions (link on class website)
![Page 75: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/75.jpg)
CS165, Fall 2016 Stratos Idreos /5043
slides are not notes!slides are mainly there to trigger discussion
note keeping is your task: starting class 3 we will do collaborative note taking:
http://tinyurl.com/CS165-Fall2016-Notes
![Page 76: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/76.jpg)
CS165, Fall 2016 Stratos Idreos /50
and check syllabus/website!
44
![Page 77: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/77.jpg)
CS165, Fall 2016 Stratos Idreos /5045
NEXT CLASS ON SEP 12 4pm, Pierce 301
stratos away next week NO CLASS ON SEP 7 TFs will hold a lab
NO OH NEXT WEEK every day labs + OH with Stratos on demand via Skype or Zoom
![Page 78: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/78.jpg)
CS165, Fall 2016 Stratos Idreos /50
reading
46
textbook: chapters 1, 3 (-3.5), 5 (-5.8,-5.9) intro + relational model + SQL
browse “the Fourth Paradigm”
next couple of classesdata models low level details - system design data layouts and column-stores
![Page 79: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/79.jpg)
CS165, Fall 2016 Stratos Idreos /5047
readings for next 3 classes
Architecture of a Database System (Sections 1,2,3,4)by J. Hellerstein, M. Stonebraker and J. Hamilton
The Design and Implementation of Modern Column-store Database Systemsby D. Abadi, P. Boncz, S. Harizopoulos, S. Idreos, S. Madden
![Page 80: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/80.jpg)
CS165, Fall 2016 Stratos Idreos /50
games
48
W. White, A. Demers, C. Koch, J. Gehrke and R. Rajagopalan Scaling games to epic proportions
ACM SIGMOD International Conference on Management of Data, 2007
thousands of users - tons of data frequent updates - tons of queries
analytics in the background for game intelligence
![Page 81: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/81.jpg)
CS165, Fall 2016 Stratos Idreos /5049
traditional algorithm analysis does not capture all costs
both data movement and processing matter
“mundane” steps such as incrementing a loop counter
or materializing results can have a huge effect
there are no simple tasks
all systems are the “same”: just data and access patterns
Notes to remember
![Page 82: class 2 from bits to systems - Harvard Universitydaslab.seas.harvard.edu/classes/cs165/doc/class_slides/... · 2017-07-27 · CS165, Fall 2016 Stratos Idreos /50 labs, sections, and](https://reader034.fdocuments.us/reader034/viewer/2022050609/5fafe344b7dfcc0bd3324f25/html5/thumbnails/82.jpg)
from bits to dbs
DATA SYSTEMSprof. Stratos Idreos
class 2