Data Visualization at Twitter
103
Krist Wongsuphasawat / @kristw visualization at Twitter data
-
Upload
krist-wongsuphasawat -
Category
Data & Analytics
-
view
4.827 -
download
0
description
My talk at the Hacks & Hackers Meetup SF at Twitter HQ on Oct 8, 2014
Transcript of Data Visualization at Twitter
Krist Wongsuphasawat /@kristw
visualizationat Twitter
data
Krist Wongsuphasawat /@kristw
Krist Wongsuphasawat /@kristw
Bangkok, Thailand
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
Chulalongkorn University
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
Programming + Soccer
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
Programming + Soccer
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
Programming + Soccer
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
M.S. in Computer ScienceUniv. of Maryland
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
PhD in Computer ScienceUniv. of MarylandInformation Visualization
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
PhD in Computer ScienceUniv. of MarylandInformation Visualization
IBMMicrosoft
Krist Wongsuphasawat /@kristw
Computer EngineerBangkok, Thailand
PhD in Computer ScienceUniv. of MarylandInformation Visualization
IBMMicrosoft
Sr. Data Visualization ScientistTwitter
at Twitterdata visualization
at Twitterdata visualization
visualizationat Twitterdata
visat Twitterdata
at Twitterdata
“Tweets”
at Twitterdata
“Tweets”
#eventsWorld Cup ElectionOscars
TV Shows New Year
Breaking news
EarthquakeSuper Bowl
Protest
…
at Twitterdata
“Tweets”
#eventsWorld Cup ElectionOscars
TV Shows New Year
Breaking news
EarthquakeSuper Bowl
Protest
…
#curiositySleep patternHuman behaviorLanguage …
at Twitterdata
“Tweets”
#eventsWorld Cup ElectionOscars
TV Shows New Year
Breaking news
EarthquakeSuper Bowl
Protest
…
What could we learn from the Tweets?
#curiositySleep patternHuman behaviorLanguage …
visat Twitterdata
“Tweets”
Tell stories about an event, Pursue curiosity or inspiration
Goal:
visat Twitterdata
“Tweets”
Tell stories about an event, Pursue curiosity or inspiration
(with deadline)
Goal:
Challenge accepted
visat Twitterdata
“Tweets”
Get data
1
easy?
Having all TweetsHow people think I feel.
How people think I feel. How I really feel.
Having all Tweets
• Too much data
• Want only relevant Tweets
• hashtag: #BRA
• keywords: “goal”
• Need to aggregate & reduce size
• Long processing time (hours)
Challenges
Hadoop ClusterVertica
Pig / Scalding (slow) SQL
Data Storage
Tool
Workflow
Hadoop ClusterVertica
Pig / Scalding (slow) SQL
Data Storage
Tool
Workflow
Hadoop ClusterVertica
Pig / Scalding (slow) SQL
Data Storage
Tool
Smaller datasetYour laptop
Workflow
Hadoop ClusterVertica
Pig / Scalding (slow) SQL
Data Storage
Tool
Final dataset
Tool node.js / python / excel (fast)
Your laptop
Workflow
Smaller dataset
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
• Peek into data
• Check data & test ideas
• Decide how to visualize
• Guided by data type
• Choose tools
• Start building
Visualize
• Peek into data
• Check data & test ideas
• Decide how to visualize
• Guided by data type
• Choose tools
• Start building
Visualize
Tableau Yeoman
d3R
(+ media)photos, videos
What?
Where? When?
GEO TIME
TEXT
data
What?
Where? When?
GEO TIME
TEXT
Visualize Data
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Time Tweets/second
Time Tweets/second
Time Tweets/second + Annotation
http://www.flickr.com/photos/twitteroffice/5681263084/
What?
Where? When?
GEO TIME
TEXT
Visualize Data
GeoHeatmap
Low density
High density
GeoSan Francisco
flickr.com/photos/twitteroffice/8798020541
Low density
High density
GeoSan Francisco
Rebuild the world based on
tweet volumes
twitter.github.io/interactive/andes/
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Text Word cloud of Tweets right after the 1st goal
www.wordle.net
It was an “own” goal.
Text WordTree [Wattenberg & Viégas 2008]www.jasondavies.com/wordtree
www.jasondavies.com/wordtree
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Time + Geo
blog.twitter.com/2011/global-pulseyoutu.be/SybWjN9pKQk
Japan Earthquake 2011
Time + Geo Tweet pattern [Rios & Lin 2012]
Night
Late night
Daytime
Night
Late night
Daytime
Night
Late night
Daytime
Night
Late night
Daytime
Time + Geo Tweet pattern [Rios & Lin 2012]
Night
Late night
Daytime
Night
Late night
Daytime
Time + Geo Tweet pattern [Rios & Lin 2012]
Night
Late night
Daytime
Night
Late night
Daytime
Time + Geo Tweet pattern [Rios & Lin 2012]
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Geo + Text Real-time Tweet map
Geo + Text Real-time Tweet map
most frequent
term
Geo + Text Real-time Tweet map
Gmail was down Jan 24, 2014
Geo + Text Real-time Tweet map
Nelson Mandela passed away Dec 5, 2013
Geo + Text Real-time Tweet map
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Time + Text
UEFA Champions League
Biggest tournament for European soccer clubs
Many Tweets during the matches
UEFA Champions League
Dortmund Bayern MunichTeam 1 Team 2
Time + Text
UEFA Champions League
Dortmund Bayern MunichTeam 1 Team 2
Time + Text
UEFA Champions League
Dortmund Bayern MunichTeam 1 Team 2
Time + Text
UEFA Champions League
Dortmund Bayern Munich
Count Tweets mentioning the teams every minute
Team 1 Team 2
Time + Text
Time + Text UEFA Champions League
+ “goal” count + context
Time + Text UEFA Champions League
+ “offside”
Time + Text UEFA Champions League
+ players
Time + Text UEFA Champions League
A B C D
A C
C
Competition Tree
vs vs
vs
A B C D
A C
C
Competition Tree
vs vs
vs +
A B C D
A C
C
Competition Tree
vs vs
vs + =
What?
Where? When?
GEO TIME
TEXT
Visualize Data
Time + Text + Geo State of the Union
twitter.github.io/interactive/sotu2014
1) timeline + topic from Tweets
4) Density map of Tweets about selected topic
3) Volume of Tweets by topics
during selected part of the SOTU
2) context (speech)
twitter.github.io/interactive/sotu2014
Time + Text + Geo State of the Union
World Cup 2014Time + Text
Time + Text + Geo World Cup 2014
What?
Where? When?
GEO TIME
TEXT
Visualize Data
What?
Where? When?
GEO TIME
TEXT
Visualize Data
+Non-Twitter data
CONTEXT
Time + Text New Year 2014
Time + Text New Year 2014
Time + Text + Geo (c) New Year 2014
twitter.github.io/interactive/newyear2014/
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
Iterate!
• Self
• Peer feedback
• Non team members / Potential audience
Evaluation
visat Twitterdata
Get data
1
Visualize
2
Evaluate
3
visat Twitterdata
Get data
1
Visualize
2
Evaluate
3
big data => small data
visat Twitterdata
Get data
1
Visualize
2
Evaluate
3
big data => small data
What? Where? When?
visat Twitterdata
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?• users • followers graph • logs • etc. !
• derived data: language, sentiment
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?Who? …• users
• followers graph • logs • etc. !
• derived data: language, sentiment
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?Who? …• users
• followers graph • logs • etc. !
• derived data: language, sentiment
(with deadline)
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?Who? …• users
• followers graph • logs • etc. !
• derived data: language, sentiment @kristw / https://interactive.twitter.com
(with deadline)
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?Who? …• users
• followers graph • logs • etc. !
• derived data: language, sentiment @kristw / https://interactive.twitter.com
(with deadline)
+ visualizations by @philogb, @miguelrios & @trebor
Questions?
visat Twitterdata
“Tweets”
Get data
1
Visualize
2
Evaluate
3
big data => small data self, peer, external
What? Where? When?Who? …• users
• followers graph • logs • etc.
@kristw / https://interactive.twitter.com
(with deadline)
+ visualizations by @philogb, @miguelrios & @trebor
Thank you
![An Admin's Guide to Data Visualization - USENIX · An Admin's Guide to Data Visualization Caskey L. Dickson caskey@{microsoft,gmail,twitter,github,...}.com. @caskey [citation needed]](https://static.fdocuments.us/doc/165x107/5e553704b030cd298c6b4e86/an-admins-guide-to-data-visualization-usenix-an-admins-guide-to-data-visualization.jpg)
