Learning to Build Distributed Systems the Hard Way
-
Upload
iconara -
Category
Technology
-
view
737 -
download
0
description
Transcript of Learning to Build Distributed Systems the Hard Way
![Page 1: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/1.jpg)
LEARNING TO BUILDDISTRIBUTED SYSTEMS
THE HARD WAY
@iconara
![Page 2: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/2.jpg)
speakerdeck.com/u/iconara(real time!)
![Page 3: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/3.jpg)
Theo / @iconara
![Page 4: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/4.jpg)
Chief Architect at
![Page 5: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/5.jpg)
let’s make online advertising a great experience
![Page 6: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/6.jpg)
MAKING THIS
![Page 7: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/7.jpg)
INTO THIS
![Page 8: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/8.jpg)
HOW HARD CAN IT BE?
![Page 9: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/9.jpg)
TRACKING AD IMPRESSIONS
track page views and all their adstrack visibility and send updates on changes
track events, track activity, sync cookies,and track visits
![Page 10: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/10.jpg)
track page views and all their adstrack visibility and send updates on changes
track events, track activity, sync cookies,and track visits
LOADED
VISIBLE
HIDDEN
VISIBLE
LOADED
![Page 11: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/11.jpg)
ASSEMBLING SESSIONS
assemble ad impressions, page views and visits,to be able to calculate things like total visible durationmix in demographics, revenue, and third-party data
![Page 12: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/12.jpg)
assemble ad impressions, page views and visits,to be able to calculate things like total visible durationmix in demographics, revenue, and third-party data
WASLOADED
BECAMEACTIVE
BECAMEVISIBLE
WAS HIDDEN
BECAMEVISIBLEAGAIN
A CLICK! { "user_id": "M9L6R5TD0YXK", "session_id": "MAI3QAGNAIYT", "timestamp": 1347896675038, "placement_name": "example", "category": "frontpage", "embed_url": "http://example.com/", "visible_duration": 1340 "browser": "Chrome", "device_type": "computer", "click": true, "ad_dimensions":"980x300"}
3rd PARTY DATA & OTHER GOODIES
![Page 13: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/13.jpg)
ANALYTICSprecompute metrics, count uniques, build visitor histories for attribution
![Page 14: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/14.jpg)
precompute metrics, count uniques, build visitor histories for attribution
![Page 15: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/15.jpg)
HOW HARD CAN IT BE?
![Page 16: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/16.jpg)
25K REQUESTSPER SECOND
~1 billion requests per day, 1 TB raw data
![Page 17: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/17.jpg)
ONE VISIT CAN CHANGE UP TO
100K COUNTERShundreds of millions of individual counters per day,
plus counting uniques and visitor histories
![Page 18: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/18.jpg)
IN REAL TIMEor near real time, if you want to be pedantic
![Page 19: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/19.jpg)
START WITH TWOOF EVERYTHINGgoing from one to two is the hardest
![Page 20: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/20.jpg)
GIVE A LOT OF THOUGHT TO YOUR
KEYS AND IDSit will save you lots of pain
![Page 21: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/21.jpg)
MANLO0 JME57Zmonotonically increasing,
sorts nicely
a timestampsomething random
![Page 22: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/22.jpg)
JME57Z MANLO0uniformly distributed,
works nicely with sharding
something randoma timestamp
![Page 23: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/23.jpg)
PUT BUFFERSBETWEEN LAYERS
queues can even out peaks, let you scale layers independently,
and let you restart services without loosing data
![Page 24: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/24.jpg)
SEPARATE PROCESSING
FROM STORAGEthat way you can scale each independently
![Page 25: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/25.jpg)
PLAN HOW TO GET RID OF YOUR DATAdeleting stuff is harder than you might think
××××
××
×
![Page 26: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/26.jpg)
NoDBkeep things streaming
×
![Page 27: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/27.jpg)
STREAM PARTITIONING
![Page 28: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/28.jpg)
RANDOMLYwhen you have no interdependenciesbetween things it’s easy to scale out
(or round robin, it’s basically the same)
![Page 29: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/29.jpg)
CONSISTENTLYwhen there are interdependencies you needto route using some property of the objects,but make sure you get a uniform distribution
![Page 30: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/30.jpg)
NUMEROLOGY
![Page 31: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/31.jpg)
12
![Page 32: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/32.jpg)
2 | 123 | 124 | 126 | 12
![Page 33: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/33.jpg)
8 | 245 | 60
![Page 34: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/34.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 35: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/35.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 36: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/36.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 37: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/37.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 38: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/38.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 39: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/39.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 40: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/40.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 41: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/41.jpg)
12, 60, 120, 360superior highly composite numbers
![Page 42: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/42.jpg)
for maximal flexibility partition with multiples of 12
![Page 43: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/43.jpg)
for maximal flexibility partition with multiples of 12
![Page 44: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/44.jpg)
A SHORT DIVERSION ABOUT COUNTING TO 60
the reason why there’s 60 seconds to a minute,and 360 degrees to a circle
![Page 45: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/45.jpg)
3 SEGMENTSON EACH FINGER
= 12
![Page 46: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/46.jpg)
3 SEGMENTSON EACH FINGER
= 12
FIVE FINGERSON OTHER HAND
= 60
![Page 47: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/47.jpg)
log2(366) ≈ 31
![Page 48: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/48.jpg)
$-$(ASCII code 36)-----
![Page 49: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/49.jpg)
log2(366) ≈ 31
![Page 50: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/50.jpg)
log2(366) ≈ 31six characters 0-9, A-Z can represent 31 bits,which is kind of almost very close to four bytes
![Page 51: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/51.jpg)
MANLO0
![Page 52: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/52.jpg)
MANLO0a timestamp
Time.now.to_i.to_s(36).upcase
![Page 53: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/53.jpg)
DO YOU REALLYNEED A BACKUP?
if you got 3x replication over multipleavailability zones, is that backup really worth it?
![Page 54: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/54.jpg)
PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT
when thousands of things happen every second, new, weird and unforeseen things happen all the time,
no test can anticipate everything(but testing is good anyway, just don’t think you got everything covered)
![Page 55: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/55.jpg)
KTHXBAI@iconara
github.com/iconaraarchitecturalatrocities.com
burtcorp.com
![Page 56: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/56.jpg)
COME TO SWEDEN IN MARCH AND
TALK ABOUT BIG DATAscandevconf.se/2013/call-for-proposals
![Page 57: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/57.jpg)
IDEMPOTENCE
![Page 58: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/58.jpg)
f(f(x)) = f(x)doing something again doesn’t change the outcome
![Page 59: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/59.jpg)
IDEMPOTENCEif you don’t have to worry about things accidentally happening twice, everything becomes much simpler
![Page 60: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/60.jpg)
COUNTING UNIQUESwhen adding to a set it doesn’t matter how many
times you do it, the end result is the same
![Page 61: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/61.jpg)
INC X VS SET Xincrements are not idempotent, and very scary,if you can avoid non-idempotent operations, try
![Page 62: Learning to Build Distributed Systems the Hard Way](https://reader033.fdocuments.us/reader033/viewer/2022051608/54514212af79591b038b4bf1/html5/thumbnails/62.jpg)
KTHXBAI@iconara
github.com/iconaraarchitecturalatrocities.com
burtcorp.com