How LinkedIn Democratizes Big Data Visualization
-
Upload
chi-yi-kuan -
Category
Data & Analytics
-
view
240 -
download
0
description
Transcript of How LinkedIn Democratizes Big Data Visualization
![Page 1: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/1.jpg)
ORGANIZATION NAME
Photo: Courtesy of O'Reilly Conference on Flickr
How LinkedIn Democratizes
Big Data Visualization
![Page 2: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/2.jpg)
Democratizes
Big Data Visualization
How
Jonathan Wu
Praveen Neppalli Naga
Chi-Yi Kuan
![Page 3: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/3.jpg)
313,000,000 Members
End of Q2 2014
![Page 4: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/4.jpg)
25,000,000,000 Page Views
Q2 2014
![Page 5: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/5.jpg)
3,000,000+ Endorsements
![Page 6: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/6.jpg)
3,500,000+ Companies
![Page 7: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/7.jpg)
What can we do with Linkedin data
?
![Page 8: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/8.jpg)
Sales
Talent flow between companies
![Page 9: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/9.jpg)
Product & engineering
![Page 10: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/10.jpg)
Is it simple?
Member attributes Page View events data
![Page 11: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/11.jpg)
Photo Credit: https://www.flickr.com/photos/johnjoh/1060267344
Data is the new vineyard
![Page 12: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/12.jpg)
Photo Credit: https://www.flickr.com/photos/johnjoh/1060267344
Data is the new vineyard
![Page 13: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/13.jpg)
Data infra: collect & prepare data
Collect & Prepare Data Mysql, Oracle, Kafka + Hadoop
Serve Data Pinot
Taste Data Easy-to-use visualization
![Page 14: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/14.jpg)
Data Computation
ETL
HDFS
Y
A
R
N
Map-Reduce Spark Tez
Pig Hive Cubert
Kafka Data Stores
Hadoop
![Page 15: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/15.jpg)
Data infra: Serve data
Collect & Prepare Data Kafka + Hadoop
Serve Data Pinot
Taste Data Easy-to-use visualization
![Page 16: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/16.jpg)
Products for members/customers with real-time
interactive analytics
• Who’s Viewed Your Profile
• Ads Reporting
• Jobs Analytics
Categories of interactive analytics products
Interactive business analytics for internal use
• How feature X is performing
Real-time business monitoring
• Page view changes across mobile devices in different
regions
![Page 17: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/17.jpg)
Requirements for real-time interactive analytics
Slice and dice billions of records,
hundreds of dimensions
End to end freshness of minutes
not hours
Sub-second query response times
e.g. Which are top regions that contribute to my profile views? Which
industries in those regions?
![Page 18: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/18.jpg)
Pinot
Distributed Analytics Infrastructure that
serves Interactive Analytics products at
Linkedin.
![Page 19: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/19.jpg)
Data
Indexes
Distributed
System
Ingestion
What is Pinot?
Compressed Columnar indexes
(supports Mmap and In-memory)
Apache Helix for cluster
management
Apache Kafka (for near real-time)
and Hadoop
![Page 20: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/20.jpg)
Data Indexes
Single Value
Index
Multi Value
Index
Inverted Index
• Fixed bit length encoding
• Sorted Index
• Secondary Sorted Index
• Multi-value Fixed bit length encoding
• BitMap Multi-value Index
• P4Delta
• Modified P4Delta
• BitMap
![Page 21: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/21.jpg)
Cluster Management
• Create Resources
• Update Resource meta data
• Expand/Contract partitions dynamically
• Query Router
![Page 22: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/22.jpg)
Data Ingestion
Kafka for Realtime
Hadoop for Historical
![Page 23: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/23.jpg)
High Level Architecture
PINOT
Hadoop Kafka
Historical Realtime
CLUSTER MANAGER
Controller
Helix
Zookeeper
Broker 1 Broker 2
Server 1 Server 2 Server 3
![Page 24: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/24.jpg)
Core Features
Low latency and high QPS OLAP Queries
with real-time ingestion
Support complex dimensions
Operational simplicity
Data bootstrapping & reconciliation
![Page 25: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/25.jpg)
Usage @ Linkedin
About 18 member facing products
on Linkedin.com
Internal Reporting
Open-source.…coming soon
![Page 26: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/26.jpg)
Reporting UI: serve & taste data
Collect & Prepare Data Kafka + Hadoop
Serve Data Pinot
Taste Data Easy-to-use visualization
![Page 27: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/27.jpg)
I want to access big data without
running SQL
Business need
![Page 28: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/28.jpg)
Start a new dashboard with one click
![Page 29: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/29.jpg)
Select what metrics/dimensions you want
![Page 30: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/30.jpg)
Charts are rendered in just a few seconds
![Page 31: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/31.jpg)
Zoom into a single chart
![Page 32: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/32.jpg)
Filter on various dimensions
![Page 33: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/33.jpg)
Access everywhere
![Page 34: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/34.jpg)
Portal that connects dashboards, internal reports,
and internal Wiki Pages
Enterprise analytics portal
![Page 35: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/35.jpg)
Scale of the data
Pinot for interactive analysis
Self service visualization for insights
Summary
![Page 36: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/36.jpg)
![Page 37: How LinkedIn Democratizes Big Data Visualization](https://reader034.fdocuments.us/reader034/viewer/2022052600/55853a31d8b42a4f018b469d/html5/thumbnails/37.jpg)
We are hiring
Jonathan Wu
www.linkedin.com/in/pneppalli
www.linkedin.com/in/jiyewu
www.linkedin.com/in/chiyikuan
Praveen Neppalli Naga [email protected]
Chi-yi Kuan [email protected]
650-605-2184
650-962-3299
650-426-6301