Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions,...
Transcript of Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions,...
![Page 1: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/1.jpg)
Thinking Like A Data ScientistEm Grasmeder
ThoughtWorks Data Witch@emgrasmeder
![Page 2: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/2.jpg)
About Me● Name: Em● Pronoun: they/them● Job: ThoughtWorks
data witch● Background: Graduate
research in economics
● Generalist within data space
![Page 3: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/3.jpg)
<Data Science Identity Crisis>
![Page 4: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/4.jpg)
What Does a Data Scientist Do
● Predictions, categorization, clustering
![Page 5: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/5.jpg)
What Does a Data Scientist Do
● Predictions, categorization, clustering
● Write software
![Page 6: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/6.jpg)
What Does a Data Scientist Do
● Predictions, categorization, clustering
● Write software● Visualizations
![Page 7: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/7.jpg)
What Does a Data Scientist Do
● Predictions, categorization, clustering
● Write software● Visualizations ● ...magically fix the
business model??
![Page 8: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/8.jpg)
How is a Data Scientist UsefulModelsCode Visualization
??
?
?
??
?Business Value
![Page 9: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/9.jpg)
Controversial opinions about data scientists● They should be good software and API developers● They should be competent at continuous delivery,
making and managing pipelines, and writing infrastructure as code
● They should speak the language of the business and be involved in conversations about KPIsIf not...
● They might not be very useful
![Page 10: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/10.jpg)
So how do data scientists actually think?
![Page 11: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/11.jpg)
Let’s answer that question with a story about
Cholera!
![Page 12: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/12.jpg)
Cholera Facts (yay!)Deadly bacteria that can kill within hours
The water in your body just comes out from everywhere
![Page 13: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/13.jpg)
Cholera Facts (yay!)Deadly bacteria that can kill within hours
The water in your body just comes out from everywhere
![Page 14: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/14.jpg)
Cholera Facts (yay!)
Deadly bacteria that can kill within hours
The water in your body just comes out from everywherePretty much curable (90% of cases) with salty, sugary water that costs $0.10
Used to be a problem, for example, in London; is still a problem in some places
![Page 15: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/15.jpg)
The cause of cholera and how it
spreads was unknown 1854.
They thought it was “miasma”
literally, bad air
![Page 16: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/16.jpg)
Deadly, exploding cesspits
Waste from houses, slaughterhouses and factories dumped in the Thames
“The Great Stink”
![Page 17: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/17.jpg)
“The Great Stink”
![Page 18: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/18.jpg)
I can certify that the offensive smells, even in that short whiff, have been of a
most head-and-stomach-distending nature
Charles Dickens
“The Great Stink”
![Page 19: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/19.jpg)
f(a, b, c, ...) + ε = y
f(proximity to bad air, sinful, too much blood, other old fashioned belief) + ε = probability of contracting cholera
![Page 20: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/20.jpg)
The Broad Street Cholera Outbreak of 1854
John SnowThe data
![Page 21: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/21.jpg)
💡
![Page 22: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/22.jpg)
Formally write your hypothesis
● H0 is called the Null Hypothesis● In 1850s England, the Null Hypothesis is “bad airs”
![Page 23: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/23.jpg)
Formally write your hypothesis
● H0: Thing is normally distributedOR
● H0: Thing is uniformly distributed● H1: Thing is distributed differently because reason
* A model is another way of writing a hypothesis
![Page 24: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/24.jpg)
Central Limit Theorem
![Page 25: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/25.jpg)
Formally write your hypothesis
● H0: Thing is normally distributedOR
● H0: Thing is uniformly distributed● H1: Thing is distributed differently because reason
* A model is another way of writing a hypothesis
![Page 26: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/26.jpg)
Formally write your hypothesis● H0: People living in equally
odorous parts of town will have a uniform likelihood of contracting cholera
![Page 27: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/27.jpg)
Preposterous!What if we actually talked to the poor
people?
![Page 28: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/28.jpg)
Collecting more data Workers at brewery were unaffected while their families diedChildren of some families died while their families lived There was this one woman, a complete outlier, the only person in her neighborhood to die
![Page 29: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/29.jpg)
Formally write your hypothesis● H0: People living in equally
odorous parts of town will have a uniform likelihood of contracting cholera
![Page 30: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/30.jpg)
Formally write yourhypothesis● H0: People living in equally
odorous parts of town will have a uniform likelihood of contracting cholera
● HA: People who drink contaminated poo-water have a uniform likelihood of contracting cholera
![Page 31: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/31.jpg)
Formally write yourhypothesis● H0: People living in equally
odorous parts of town will have a uniform likelihood of contracting cholera
● HA: People who drink contaminated poo-water have a uniform likelihood of contracting cholera
![Page 32: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/32.jpg)
![Page 33: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/33.jpg)
The greatest value of a picture is when it forces us to notice what we never expected
to see. - Tukey “Exploratory Data Analysis” 1977
![Page 34: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/34.jpg)
Collecting more data Workers at brewery drank beer which required boiling the water The children who died went to school near the infected well, far from their homes and familyThat one woman, she just loved the flavor of the water from that poo-contaminated cholera well
![Page 35: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/35.jpg)
So what are the lessons?Data is good. More data is better
Visualize your data!Do we really need machine learning for this?
![Page 36: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/36.jpg)
Data Exploration: Refining your mental model
John SnowThe data
![Page 37: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/37.jpg)
Data Exploration
![Page 38: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/38.jpg)
Let’s talk about models
f(a, b, c, ...) + ε = y
![Page 39: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/39.jpg)
Data is good. More data is betterTry to move as much as possible from the ε into the function
f(a, b, c, ...) + ε = y
Maybe b comes from an external APIMaybe c is too complicated and needs to be split into d and eMaybe g is derived from a function/calculation based on other records or parameters a and b
![Page 40: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/40.jpg)
Data is good. More data is better. Unless it’s not
f(a, b, c, ...) + ε = y
Sometimes b and c are just confusing the algorithmMethods of dimensionality reduction or principle component analysis help extract a signal from noise, and help prevent overfitting
![Page 41: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/41.jpg)
The data
Your model ->
It’s, you… you’re the scientist in this
metaphor now
Thinking about the end user
![Page 42: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/42.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
![Page 43: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/43.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
![Page 44: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/44.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
![Page 45: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/45.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
![Page 46: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/46.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
![Page 47: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/47.jpg)
“Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.”
- George Box(“one of the great statistical minds of the 20th century”)“Empirical Model Building and Response Surfaces”, 1987
Thinking like a data scientist means making
pragmatic choices
![Page 48: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/48.jpg)
● Fun to do● Difficult to master● Not very different from
science throughout the ages
● Not the hard part
Building Models
![Page 49: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/49.jpg)
Hamburg Cholera Outbreak of 1892
● Hamburg was an independent city-state
● They bought into “laissez-faire” economics
● Business would have been temporarily hurt by sanitation modernization
![Page 50: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/50.jpg)
Hamburg Cholera Outbreak of 1892
● Other cities (e.g. nearby Altona) which invested in sanitation didn’t have an outbreak
● 1.5% of Hamburg’s population died
● People rioted
![Page 51: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/51.jpg)
● Internal politics● Overcoming bureaucracy● Positioning you model to
deliver value to users quickly
● Getting trust from internal stakeholders
● Building something meaningful, something good
The Hard Part of doing Data Science
![Page 52: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/52.jpg)
Fun fact! The word “statistics” came about in the 18th century and meant "science dealing with data about the condition of a state or community"
![Page 53: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/53.jpg)
![Page 54: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/54.jpg)
![Page 55: Thinking Like A Data Scientist - files.gotocon.com · What Does a Data Scientist Do Predictions, categorization, clustering ... So how do data scientists actually think? Let’s answer](https://reader030.fdocuments.us/reader030/viewer/2022041108/5f0be7ab7e708231d432cb7e/html5/thumbnails/55.jpg)
Thank you!
I’m Em Grasmederthe ThoughtWorks Data Witch@emgrasmeder on Twitter