SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016...
Transcript of SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016...
![Page 2: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/2.jpg)
![Page 3: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/3.jpg)
INSTAGRAM HISTORY
2010
201114M users
2012/4/3Android release
2012/4/9Facebook acquisition
2014/1
![Page 4: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/4.jpg)
INSTAGRAM EVERYDAY
300 Million Users
4.2 Billion likes
95 Million photo/video uploads
100 Million followers
![Page 5: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/5.jpg)
![Page 6: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/6.jpg)
SCALING MEANS
Scale out
Scale up
Scale dev team
![Page 7: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/7.jpg)
SCALE OUT
![Page 8: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/8.jpg)
SCALE OUT
“To scale horizontally means to add more nodes to a system, such as adding a new computer to a distributed software application. An example might involve scaling out from one Web server system to
three.”
- Wikipedia
![Page 9: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/9.jpg)
MICROSERVICE
![Page 10: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/10.jpg)
SCALING OUT
—>
—>
—> vertical partition horizontal sharding
![Page 11: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/11.jpg)
SCALING OUT
![Page 12: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/12.jpg)
![Page 13: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/13.jpg)
![Page 14: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/14.jpg)
INSTAGRAM STACK
Tuesday, June 25th, 2013
memcache
RabbitMQ
PostgreSQL
Cassandra
Celery
OtherServicesDjango
![Page 15: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/15.jpg)
STORAGE VS. COMPUTING
• Storage: needs to be consistent across data centers• Computing: driven by user traffic, as needed basis
![Page 16: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/16.jpg)
SCALE OUT: STORAGE
Tuesday, June 25th, 2013
-Masterless-Async, low latency-Multiple data center ready-Tunable latency vs consistency trade-off
user feeds, stories, activities, and other logs
![Page 17: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/17.jpg)
SCALE OUT: STORAGE
Tuesday, June 25th, 2013
user, media, friendship etc
• One master, replicas are in each region• Reads are done locally• Writes are cross region to the master.
![Page 18: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/18.jpg)
COMPUTING
Tuesday, June 25th, 2013
![Page 19: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/19.jpg)
Tuesday, June 25th, 2013
Django
RabbitMQ PostgreSQL
CassandraCelery
Django
RabbitMQPostgreSQL
CassandraCelery
memcacheDC1 DC2memcache
![Page 20: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/20.jpg)
MEMCACHE
Tuesday, June 25th, 2013
• Millions of reads/writes per second• Sensitive to network condition• Cross region operation is prohibitive
![Page 21: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/21.jpg)
feed
get
Django
User R
DC1
Django
PostgreSQL memcache
User Ccomment
setinsert
![Page 22: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/22.jpg)
Django
memcache PostgreSQL
User Ccomment
insertset
DC1
Django
memcachePostgreSQL
User R
feed
get
DC2
replication
![Page 23: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/23.jpg)
Django
memcache PostgreSQL
User Ccomment
insertset
DC1
Django
memcachePostgreSQL
User R
feed
set
DC2
replication
Cache invalidate
Cache invalidate
get
![Page 24: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/24.jpg)
COUNTERS
select count(*) from user_likes_media
where media_id=12345;
100s ms
![Page 25: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/25.jpg)
Tuesday, June 25th, 2013
COUNTERS
![Page 26: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/26.jpg)
COUNTER
Tuesday, June 25th, 2013
select count from media_likes where media_id=12345;
10s us
![Page 27: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/27.jpg)
Cache invalidatedAll djangos try to access DB
![Page 28: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/28.jpg)
MEMCACHE LEASE
d1 d2 memcache dbtime
lease-get
filllease-get
wait or use stale
read from DB
lease-set
lease-get
hit
![Page 29: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/29.jpg)
INSTAGRAM STACK - MULTI REGION
Tuesday, June 25th, 2013
Django
RabbitMQ
PostgreSQL
Cassandra
Celery
memcache
Django
RabbitMQ
PostgreSQL
Cassandra
Celery
memcache
DC1 DC2
![Page 30: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/30.jpg)
SCALING OUT
Tuesday, June 25th, 2013
• Capacity• Reliability• Regional failure ready
Requests/second
![Page 31: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/31.jpg)
LOAD TEST
Servers
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24CPU instructions
Loaded
Regular
Load Balancer
DjangoServers
![Page 32: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/32.jpg)
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24
User growth Server growth
![Page 33: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/33.jpg)
“Don’t count the servers, make the servers count”
![Page 34: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/34.jpg)
SCALE UP
![Page 35: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/35.jpg)
SCALE UP
Use as few CPU instructions as possible
Use as few servers as possible
![Page 36: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/36.jpg)
SCALE UP
Use as few CPU instructions as possible
Use as few servers as possible
Scale up
![Page 37: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/37.jpg)
CPU
Monitor
Optimize
Analyze
![Page 38: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/38.jpg)
COLLECT
struct perf_event_attr pe;
pe.type = PERF_TYPE_HARDWARE;
pe.config = PERF_COUNT_HW_INSTRUCTIONS;
fd = perf_event_open(&pe, 0, -1, -1, 0);
ioctl(fd, PERF_EVENT_IOC_ENABLE, 0); <code you want to measure> ioctl(fd, PERF_EVENT_IOC_DISABLE, 0); read(fd, &count, sizeof(long long));
![Page 39: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/39.jpg)
DYNOSTATS
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24
Follow
Feed
Explore
![Page 40: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/40.jpg)
REGRESSION
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24
![Page 41: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/41.jpg)
GRADUAL REGRESSION
0
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24
![Page 42: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/42.jpg)
With new feature
Without new feature
![Page 43: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/43.jpg)
![Page 44: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/44.jpg)
CPU
Monitor
Optimize
Analyze
![Page 45: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/45.jpg)
PYTHON CPROFILE
import cProfile, pstats, StringIO pr = cProfile.Profile()
pr.enable() # ... do something ... pr.disable() s = StringIO.StringIO() sortby = 'cumulative' ps = pstats.Stats(pr, stream=s).sort_stats(sortby) ps.print_stats() print s.getvalue()
![Page 46: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/46.jpg)
![Page 47: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/47.jpg)
CPU - ANALYZEcontinuous profiling
generate_profile explore --start <start-time> --duration <minutes>
![Page 48: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/48.jpg)
CPU - ANALYZEcontinuous profiling
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24
Caller
Callee
![Page 49: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/49.jpg)
CPU - ANALYZEdecorator
def get_photos(): ……
def feed(): get_photos()
@log_stats def get_follows(): …… def follow(): get_follows()
@log_stats
![Page 50: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/50.jpg)
get_follows get_photos
feed follow
log_stats
![Page 51: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/51.jpg)
get_followsget_photos
feed follow
Keeping Demand in Check
![Page 52: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/52.jpg)
![Page 53: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/53.jpg)
CPU
Monitor
Optimize
Analyze
![Page 54: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/54.jpg)
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s300x300/12345678_1234567890_987654321_a.jpg
![Page 55: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/55.jpg)
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s150x150/12345678_1234567890_987654321_a.jpg
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s400x600/12345678_1234567890_987654321_a.jpg
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s200x200/12345678_1234567890_987654321_a.jpg
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s300x300/12345678_1234567890_987654321_a.jpg
![Page 56: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/56.jpg)
CPU - OPTIMIZE
![Page 57: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/57.jpg)
igcdn-photos-d-a.akamaihd.net/hphotos-ak-xpl1/t51.2885-19/s300x300/12345678_1234567890_987654321_a.jpg
150x150
400x600
200x200
![Page 58: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/58.jpg)
CPU - OPTIMIZE
C is really faster
• Candidate functions:• Used extensively• Stable
• Cython or C/C++
![Page 59: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/59.jpg)
CPU - CHALLENGE
cProfile is not free
False positive alerts
Better automation
![Page 60: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/60.jpg)
Use as few CPU instructions as possible
Use as few servers as possible
Scale up
![Page 61: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/61.jpg)
SCALE UP: MEMORY
(memory budget /process) X (# of processes) < system memory
Less memory budget/process
===> Dies sooner
===> More processes
![Page 62: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/62.jpg)
LOAD TEST
Servers
20
40
60
80
100
0 2 4 6 8 10 12 14 16 18 20 22 24CPU instructions
Loaded
Regular
Load Balancer
DjangoServers
![Page 63: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/63.jpg)
SCALE UP: MEMORY
Code
Large configuration
![Page 64: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/64.jpg)
SCALE UP: MEMORY
• Run in optimized mode (-O)• Use shared memory• NUMA• Remove dead code
![Page 65: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/65.jpg)
SCALE UP: LATENCY
Synchronous Processing model
===> All user experience impacted
===> Worker starvation
Single service degradation
===> Fewer CPU instr executedLonger latency
![Page 66: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/66.jpg)
Stories
FeedDjango
Feed
Stories
SuggestedUsers
ASYNC IO
![Page 67: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/67.jpg)
Use as few CPU instructions as possible
Use as few servers as possible
Scale up
![Page 68: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/68.jpg)
SCALE DEV TEAM
![Page 69: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/69.jpg)
SCALING TEAM
30% engineers joined in last 6 months
Bootcampers - 1 week
Hack-A-Month - 4 weeks
Intern - 12 weeks
![Page 70: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/70.jpg)
Comment Filtering
Self-harm Prevention
Windows App
Story Viewer Ranking
Video View Notification
Save Draft
First Story Notification
![Page 71: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/71.jpg)
Which server?
NewTable or New Column?
What Index?Should
I cache it?
Will I lock up DB?
Will I bring down Instagram?
![Page 72: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/72.jpg)
WHAT WE WANT
• Automatically handle cache• Define relations, not worry about implementations• Self service by product engineers• Infra focuses on scale this service
![Page 73: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/73.jpg)
TAO
USER1
USER2
USER3mediaposted
posted bylikes
liked by
likes
liked by
![Page 74: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/74.jpg)
SCALE DEV - END OF POSTGRES
![Page 75: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/75.jpg)
SHIPPING LOVE
60-80 daily diffs
>120 engineers committed code last month
![Page 76: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/76.jpg)
RELEASE
• Master, no branch• All features developed on master
gated by configuration• Continuous integration • No branch integration overhead
• No surprises• Iterate fast, collaborate easily• Fast bisect and revert
![Page 77: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/77.jpg)
Once a week?
40-50 rollouts per day
Once a day?Once a diff!!
![Page 78: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/78.jpg)
CHECKS AND BALANCES
Code reviewunittest
Code acceptedcommitted Canary To the Wild
Dark launchLoad test
![Page 79: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/79.jpg)
![Page 80: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/80.jpg)
TAKEAWAYS
Scaling is a continuous effort
Scaling is multi-dimensional
Scaling is everybody’s responsibility
![Page 81: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/81.jpg)
QUESTIONS?
![Page 82: SCALING INSTAGRAM INFRA - QCon San Francisco … INSTAGRAM INFRA Lisa Guo— Nov 7th, 2016 lguo@instagram.com INSTAGRAM HISTORY 2010 2011 14M users 2012/4/3 Android release 2012/4/9](https://reader031.fdocuments.us/reader031/viewer/2022022511/5ade0a4c7f8b9aeb668dc1a4/html5/thumbnails/82.jpg)