[B1]real time large data at twitter
-
Upload
naver-d2 -
Category
Technology
-
view
4.404 -
download
0
description
Transcript of [B1]real time large data at twitter
![Page 1: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/1.jpg)
real-time large data
@ra!deview - 17 september 2012
![Page 2: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/2.jpg)
![Page 3: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/3.jpg)
![Page 4: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/4.jpg)
![Page 5: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/5.jpg)
there are over400 million tweetsa day
![Page 6: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/6.jpg)
a second4600 tweets
0.2 msa tweet≈
![Page 7: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/7.jpg)
Pull Push
Targeted twitter.comhome_timeline API
User / Site StreamsMobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
![Page 8: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/8.jpg)
Write API
Fanout
Tim
elin
e C
ache
Redis
Ingester
Sear
ch C
ache
Redis
Push
Com
pute
HTTP Push
Bat
ch C
ompu
te
Hadoop
![Page 9: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/9.jpg)
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
![Page 10: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/10.jpg)
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
![Page 11: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/11.jpg)
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
![Page 12: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/12.jpg)
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
![Page 13: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/13.jpg)
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
Monorail
T-Bird
T-Flock + Haplo
Flock(s)
Darkwing
![Page 14: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/14.jpg)
Macaw+Swift
Story Service
Macaw+Disco
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
T-Bird
T-Flock + Haplo
Flock(s)
Darkwing
Tweetypie
Gizmoduck
TLS
Social Graph Service
Monorail
Woodstar
TFE
![Page 15: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/15.jpg)
![Page 16: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/16.jpg)
timeline delivery statistics⇢30b deliveries / day (~20m / min)
⇢3.5 seconds @ p50 to deliver to 1m
⇢~350k deliveries / sec
![Page 17: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/17.jpg)
![Page 18: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/18.jpg)
![Page 19: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/19.jpg)
![Page 20: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/20.jpg)
![Page 21: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/21.jpg)
![Page 22: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/22.jpg)
![Page 23: [B1]real time large data at twitter](https://reader034.fdocuments.us/reader034/viewer/2022042813/54897d26b47959d30c8b5988/html5/thumbnails/23.jpg)
#JoinTheFlock