The Web Scale
-
Upload
guille-bisho- -
Category
Documents
-
view
951 -
download
3
description
Transcript of The Web Scale
![Page 1: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/1.jpg)
The Web ScaleTuenti architecture to withstand1500+ million pageviews / day
Guillermo Pérez - [email protected] Security & Backend Architecture Tech Lead
![Page 2: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/2.jpg)
What is a scalable system?
![Page 3: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/3.jpg)
![Page 4: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/4.jpg)
![Page 5: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/5.jpg)
What is scalability
![Page 6: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/6.jpg)
Some Tuenti stats
![Page 7: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/7.jpg)
Tuenti Stats
13M usersREALLY ACTIVE
50%+ active weekly>1h browsing per DAY!
![Page 8: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/8.jpg)
Tuenti Stats
- Each month, over:40,000 M pageviews50,000 M requests100 M new photos2,000+ Tb served photos
- On peaks:1,600 million pageviews/day35,000 requests/second6,000 million served photos/day
![Page 9: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/9.jpg)
Tuenti Stats
- 1200+ servers~500 FEs~300 DBs~100 MCs~100 image serversOthers: Chat, HBase, Queues, Processors...
![Page 10: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/10.jpg)
How to scale?
![Page 11: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/11.jpg)
No silver bullet
![Page 12: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/12.jpg)
MonitorKnow your toolsEvolve, iterate
Learn
![Page 13: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/13.jpg)
Monitoring
- Your crystal ball!Glimpse of the futureAnswer questions
- Detect bottlenecks- Detect what needs to be optimized
The 90/10 RuleNo premature optimization
- Detect bad usages- Detect browser patterns- Detect changes, issues
![Page 14: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/14.jpg)
Monitoring
![Page 15: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/15.jpg)
Monitoring
![Page 16: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/16.jpg)
Monitoring
![Page 17: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/17.jpg)
MonitorKnow your toolsEvolve, iterate
Learn
![Page 18: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/18.jpg)
Know your tools
- Stop reading blogs- Read internals documentation- Test software- Test hardware- Experiment
![Page 19: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/19.jpg)
Know your tools
- Mysql (innoDB) IS fastphotos table (photo_id, user_id, ...)
PK photo_id, KEY user_idPK user_id, photo_id, KEY photo_idUsage: select * from photos where user=X
sortingcovering indexEven No SQL :)Hardware limits, replication
![Page 20: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/20.jpg)
Know your tools
![Page 21: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/21.jpg)
Know your tools
- MemcacheTons of persistent TCP conns eats your ram
UDP performance issuesSingle thread for UDPMultiport patch
proxiesStresses the network to the max
Driver issues, configurationVariable performance with net devices
![Page 22: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/22.jpg)
Know your tools
- No SQLNot magic!Good for heavy write loadsGood for data processingStill needs tweaking partitioning, schemas
![Page 23: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/23.jpg)
MonitorKnow your toolsEvolve, iterate
Learn
![Page 24: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/24.jpg)
Evolve, iterate
- All architectures scale till certain point- Then you must rethink everything
Then, and only then!Remember premature optimization?Scale != efficientFuture is hard to predict
![Page 25: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/25.jpg)
MonitorKnow your toolsEvolve, iterate
Learn
![Page 26: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/26.jpg)
Learn
Learn from:Experience
FailureOthers
![Page 27: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/27.jpg)
Architecture
![Page 28: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/28.jpg)
Architecture
- Basic rules:Static: Add layers (easy caching)Dynamic: Move responsibility to edgesGeneral: Decentralize, redundancy
![Page 29: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/29.jpg)
Architecture
- Design for failure:Support disablingNice degradation, fallbacksControlled launches
- Test with dark launches- Think on storage operations- Be able to migrate live- Focus on your core, use CDNs
![Page 30: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/30.jpg)
Architecture
- Move work to the browser:Request routingTemplatesCachePefetch
- Move remaining to your FEs:Data relationsConsistencyPrivacy, access checkLive migrationsKnowledge of the storage infraestructure
![Page 31: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/31.jpg)
Architecture
- All teams involvedFrontend
Good JS, templating, caching, prefetchingBackend
Data design, parallelization, optimizationsSystems
Iron benchmarks, tunning, networking
![Page 32: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/32.jpg)
Dynamic site example
![Page 33: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/33.jpg)
Scaling a website
- Setup: 1 server- Bottleneck: cpu - Solution: Add fronteds- Changes: Share sessions
![Page 34: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/34.jpg)
Scaling a website
- Setup: N fronteds, 1 DB- Bottleneck: DB Reads - Solution: Add DB slaves- Changes: Split reads to slaves or DB proxy
![Page 35: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/35.jpg)
Scaling a website
- Setup: N fronteds, 1 DB Master + N Slaves- Bottleneck: Limited # of slaves, so DB Reads - Solution: Chain replication / Add cache layer- Changes: Big ones!
Some caches in certain places is easyBut for dynamic app, Memcache as storageMakes your DB nor relational
![Page 36: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/36.jpg)
Scaling a website
- Setup: N FEs, 1 DB Master + N Slaves, Caches- Bottleneck: DB Writes - Solution: Split tables into DB clusters- Changes: Add some DB abstraction
![Page 37: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/37.jpg)
Scaling a website
- Setup: N FEs, N DB clusters, Caches- Bottleneck: DB Writes on certain table - Solution: Partition tables- Changes: DB abstraction and big changes
DB no longer relational, more key basedPartition key limits queriesDenormalization, duplicity
![Page 38: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/38.jpg)
Scaling a website
- Setup: N FEs, N partitioned DBs, Caches- Bottleneck: Disk space, DB cost - Solution: Archive tables- Changes: DB abstraction + migration scripts
![Page 39: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/39.jpg)
Scaling a website
- Setup: N FEs, N partition+archive DBs, Cache- Bottleneck: Internal network traffic - Solution: 2 level caches, split services, cache affinity- Changes: Cache abstraction, browsers
![Page 40: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/40.jpg)
Scaling a website
- Setup: N FEs, N partition+archive DBs, multilayered Cache, services- Bottleneck: Datacenter - Solution:
Split servicesPartition users data
- Changes: Big ones!Greater replication lags, inconsistencies
![Page 41: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/41.jpg)
The Tuenti Backend Framework
![Page 42: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/42.jpg)
Backend Framework
- Our mission:Provide easy to use, productive, easy to debug, testable, fast, extensible, customizable, deterministic, reusable, instrumentalized (stats) framework and tools to ease developers daily work and manage the infraestructure.
![Page 43: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/43.jpg)
Backend Framework
- From Request routing to Storage- Simple layers, clean responsibilities- Clean, organized codebase- Using:
convention over configurationconfiguration over coding
- Queuing system for async execution- Gathering stats from all levels
![Page 44: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/44.jpg)
Backend Framework
- Request routing:Multiple entry pointsFast request parsers route to AgentsData centric agentsPrinters
![Page 45: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/45.jpg)
Backend Framework
- Domain Api:Expose top-level business actionsClean, semantic ApiNo state, no magic, all data in paramsCheck privacy (the right place!)
![Page 46: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/46.jpg)
Backend Framework
- Domain Backend:Implement public/internal business actionsClean, semantic ApiNo state, no magic, all data in paramsCoordinate transactionsNo privacy
![Page 47: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/47.jpg)
Backend Framework
- Domain Storages (ORM like)Configure storage access for a table
Fields, validation, partitioning, primary key, caching techniques, custom queries.
Provide access to storage via standard apis:CRUD actionsCached ListsCached Queries+ Custom
Data container
![Page 48: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/48.jpg)
Backend Framework
- Storage StrategiesCRUDCached ListsCached QueriesCUD Observers for custom actions
![Page 49: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/49.jpg)
Backend Framework
- Storage ServiceProvides access to the different storage services:
mysql, memcache, hbase...Coordinates transactionsAbstract the infrastructure complexities:
partitioning, read/write, weights, hostsHandles transactions
![Page 50: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/50.jpg)
Backend Framework
- Storage Services (concrete ones)Abstract the infrastructure complexities:
partitioning, read/write, weights, hostsApi close to real one:
Memcache: set, get, cas...Mysql: insert, select, update...
![Page 51: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/51.jpg)
Backend Framework
- Storage Drivers (concrete ones)Read configManage PHP driversEnhance API
![Page 52: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/52.jpg)
Love challenges?
![Page 53: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/53.jpg)
We are hiring!http://jobs.tuenti.com
And... Stay tuned for our
Tuenti Challenge 2!http://contest.tuenti.net
![Page 54: The Web Scale](https://reader038.fdocuments.us/reader038/viewer/2022103109/546cac18af795985168b5fb4/html5/thumbnails/54.jpg)
Thanks!
?
Guillermo Pérez - [email protected] & Backend Architecture Tech Lead
Images Creative Commons from flickr:heydanielle, eschipul, deanfotos66, nrbelex, mikolski, fdecomite, guldfisken