(NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
-
Upload
amazon-web-services -
Category
Technology
-
view
829 -
download
2
Transcript of (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC
![Page 1: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/1.jpg)
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
@Sumbry
Director of Cloud Services
Twilio.com
October, 2015
NET409
Movin' On Up to the VPCHow Twilio Migrated its Infrastructure from
EC2-Classic to EC2-VPC
![Page 2: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/2.jpg)
Purpose of this talk
- Learn about Twilio
- Review legacy infrastructure
- Why EC2-VPC?
- How we built the Twilio Cloud
- How we migrated
- Internal tools developed
- Lessons learned
![Page 3: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/3.jpg)
What Is a Twilio?
- A global communications company
- A real-time communications API
- Used by over 500,000 developers
- Requires low-latency resilient infrastructure
- Has lots of infrastructure on EC2-Classic
![Page 4: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/4.jpg)
Who are Twilio customers?
![Page 5: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/5.jpg)
Legacy Twilio
![Page 6: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/6.jpg)
What did Twilio look like yesterday?
- Twilio has used AWS since 2008
- Three products
- All infrastructure located in us-east-1
- Hundreds of instances
- 10/8 shared private network
- Non-consecutive EIPs
![Page 7: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/7.jpg)
Before global
![Page 8: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/8.jpg)
What is going global?
- Launched outside US
- Global provisioning
- Route traffic between regions
- Low-latency communications
- Global service discovery
![Page 9: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/9.jpg)
The network after global
![Page 10: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/10.jpg)
Problems with going global
- Overlapping 10/8 networks
- Proxies not ideal, point-to-point
- Routing around failovers
- Need low latency connectivity
![Page 11: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/11.jpg)
Why EC2-VPC?
![Page 12: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/12.jpg)
What is EC2-VPC?
EC2-VPC is the next major revision of the EC2 platform:
- Software Defined Network
- Elastic Network Interfaces
- HVM and SR-IOV
![Page 13: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/13.jpg)
What is a software defined network?
- Define your own network
- VPC and subnet routing tables
- Network Access Control Lists
- Provision networks like virtual machines
- Protects data-in-transit
![Page 14: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/14.jpg)
What are elastic network interfaces?
- Public and Private EIPs
- Multiple Private EIPs per interface
- Multiple ENIs per instance
- Security groups follow an ENI
- ENI has a MAC address
![Page 15: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/15.jpg)
What are HVM instances?
- Hardware Virtualized Machine instances
- PCI Express speeds to network adapter
- Low-latency access to network adapter
- Up to 10 GB network speeds
![Page 16: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/16.jpg)
Why move to EC2-VPC?
- SDN solves overlapping 10/8 networks
- Route tables eliminates proxies
- Routing around failovers is an API call
- HVM solves low latency connectivity problem
![Page 17: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/17.jpg)
The Twilio Cloud
![Page 18: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/18.jpg)
What is the Twilio Cloud?
- Iteration 2.0 of our infrastructure
- Addresses many EC2-Classic limitations
- Connectivity between data centers
- Automatic failover and redundancy
- Provider agnostic
![Page 19: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/19.jpg)
What does the Twilio Cloud look like?
![Page 20: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/20.jpg)
What about routing?
![Page 21: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/21.jpg)
We built it, did they come?
We solved all previous issues but no one used it:
- Twilio Cloud was isolated from EC2-Classic
- Existing services had no migration path
![Page 22: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/22.jpg)
Data center migration
![Page 23: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/23.jpg)
Why is a migration like moving data centers?
- Separate infrastructure from EC2-Classic
- Need to migrate all your compute
- Zero downtime
![Page 24: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/24.jpg)
The networks
![Page 25: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/25.jpg)
What problems do we need to solve?
- Move an instance from Classic to VPC
- Network connectivity
- Instance discoverability
- No service interruptions
![Page 26: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/26.jpg)
Classic deploy
![Page 27: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/27.jpg)
VPC deploy
![Page 28: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/28.jpg)
Kill Classic
![Page 29: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/29.jpg)
Steps to migrate a service
![Page 30: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/30.jpg)
Wait - you just invented a bunch of stuff …
- Bridge EC2-Classic and VPC?
- Global Service Discovery?
- Multiple Service Deployments?
- WTF!
![Page 31: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/31.jpg)
Migration tools
![Page 32: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/32.jpg)
What are the tools for migrating to EC2-VPC?
We modified existing internal tools:
- IP Tunnel Manager / ClassicLink
- Global Service Discovery
- HAProxy Distributed Load-Balancing
- Config-Renderer
![Page 33: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/33.jpg)
What is IP Tunnel Manager ClassicLink?
ClassicLink allows you to link
your EC2-Classic instance to
a VPC in your same account
in the same region.
It provides network
connectivity between EC2-
Classic and EC2-VPC
instances.
![Page 34: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/34.jpg)
What is Global Service Discovery?
GSD stores IP addresses for any service in the cluster and
serves them on-demand.
![Page 35: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/35.jpg)
What is distributed load balancing?
Every instance in the cluster runs its own instance of
HAProxy. It load balances requests to any downstream
services.
![Page 36: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/36.jpg)
What is Config-Renderer?
Config-Renderer renders configuration files filled with data
from Global Service Discovery, like HAProxy Configs!
![Page 37: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/37.jpg)
What about deploying services?
Our internal
provisioning tool
called BoxConfig lets
us deploy services
with the click of a
button.
![Page 38: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/38.jpg)
How does it all work?
![Page 39: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/39.jpg)
Unix philosophy
We use lots of small tools and combine them:
- Twilio Cloud to route
- ClassicLink to bridge
- HAProxy for distributed load-balancing
- Global Service Discovery for IP info
- Config-Renderer to write HAProxy configs
- BoxConfig to deploy
![Page 40: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/40.jpg)
In conclusion
![Page 41: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/41.jpg)
Where are you today?
- The Twilio Cloud is live today
- Routes traffic through nine virtual data centers
- Over 100 IPSEC Mesh links
- Automatic region failover thanks to EIGRP
- 35% of Twilio infrastructure is in EC2-VPC
- We can complete the migration in 2015
![Page 42: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/42.jpg)
What are some lessons learned?
- Properly subnet your VPC. You have one shot.
- No need to do a giant migration all at once.
- Tools need to work both ways in case you screw up.
- Less complexity always wins.
![Page 43: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/43.jpg)
Thank you!
![Page 44: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/44.jpg)
Remember to complete
your evaluations!
![Page 45: (NET409) How Twilio Migrated Its Services from EC2-Classic to EC2-VPC](https://reader034.fdocuments.us/reader034/viewer/2022042908/58f1afba1a28ab3b7a8b4579/html5/thumbnails/45.jpg)
Related Sessions