Amazon EC2 to Amazon VPC: A case study (CPN301) | AWS re:Invent 2013
RedMart Migrating from EC2 to VPC with Chef
-
Upload
ritesh-angural -
Category
Technology
-
view
404 -
download
4
description
Transcript of RedMart Migrating from EC2 to VPC with Chef
![Page 1: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/1.jpg)
EC2 to VPC with Chefand some other CI stuff...
+
![Page 2: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/2.jpg)
About
● Leading online grocery store in Singapore.
● Proprietary Ecommerce platform
● `redmart10` for 10% off. :)
![Page 3: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/3.jpg)
EC2 to VPC
![Page 4: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/4.jpg)
EC2 to VPC - Goals
● Automate our infrastructure.● Security● Maintain (or improve) workflow● Migrate with zero downtime
![Page 5: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/5.jpg)
Security ☐
![Page 6: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/6.jpg)
VPC
● Public Subnets● Private Subnets● Gateway● NATs
![Page 7: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/7.jpg)
VPC InfraPublic Subnet10.0.0.0/24
Private10.0.1.0/24
Frontend,ELBs
Backend,Api, Servicesetc.
GatewayInternet. Mostly cats.
HOP
Mongo(Ecom)
Mongo(Fulfillment)
RedMart VPC (10.0.0.0/16)
Scale/ERP
Scale DB
Private (10.0.2.0/24)
![Page 8: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/8.jpg)
Why VPC - without
● Securityo Security groups
Node
![Page 9: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/9.jpg)
Why VPC - With
● Securityo Security groupso Network ACLso Route Tableso VPN
● Easier to manageo Intra VPCo External
NodeNode Node Node
Gatewayor
NAT
![Page 10: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/10.jpg)
Security ☑Maintain (or improve) Workflow ☐
![Page 11: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/11.jpg)
Workflow 1 - Deployment
1. Code2. Push to git3. Relax
![Page 12: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/12.jpg)
Developer
1. Transfer Artifacts to S3
Push
Build
Fail
Pass
Emails, Slack
SSH
Chef Server
`sudo chef-client`
Download
2. Get nodes, SSH
![Page 13: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/13.jpg)
Workflow 2 - Bootstrapping
1. Clone boilerplate project repo2. Clone boilerplate cookbook & role
3. knife create server -r “role[shiny_new_app]”4. Code!
![Page 14: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/14.jpg)
1. Transfer Artifacts to S3
Chef ServerDevOps
`sudo chef-client`
Bootstrap
Download
![Page 15: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/15.jpg)
Developer
1. Transfer Artifacts to S3
Push
Build
Fail
Pass
Send Emails
SSH
Chef ServerDevOps
`sudo chef-client`
Bootstrap
Download
2. Get nodes, SSH
![Page 16: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/16.jpg)
Travis process
1. Tests and Builds your project2. If success:
o Uploads project to S3o Finds nodes relevant to your project (chef search)o Triggers `chef-client` on relevant nodes
![Page 17: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/17.jpg)
Chef Process
1. Bootstrapping a node `knife bootstrap`
a. creates a server with specified role & environment
2. Converging a node `sudo chef-client`
a. role based recipe eg. recipe[golden_admin]
b. redmart deployment recipe eg. recipe[base_redmart::deploy]
i. download artifacts based from s3ii. extractiii. run start script eg. eg. `sh /ci/start.sh`
iv. symlink release
![Page 18: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/18.jpg)
Everyone deploys the same way.
Security ☑Maintain (or improve) Workflow ☑
Migrate with zero downtime ☐
![Page 19: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/19.jpg)
Stateful services
● Mongo (superfun)● Redis● Rabbitmq (not really stateful)
![Page 20: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/20.jpg)
Mongo with six hosts
Apps
private
Apps
VPC Internet
public
![Page 21: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/21.jpg)
Redis Daisy chaining
● Crucial to ensure user sessions not losto slaveof IPADDRESSo slave-read-only no
private
VPC Internet
public
![Page 22: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/22.jpg)
Rabbitmq
● Keep alive till queues die out.
Apps
private
Apps
VPC Internet
public
![Page 23: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/23.jpg)
Non stateful apps
● sudo chef-client● Hooray for chef!
![Page 24: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/24.jpg)
DDay
● Duplicate version of entire redmart.com in VPC.o chef.redmart.como chefapi.redmart.com
● Change DNS of everything in public subnet● TTL!
![Page 25: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/25.jpg)
Security ☑Maintain (or improve) Workflow ☑
Migrate with zero downtime ☑
![Page 26: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/26.jpg)
… Cache busted!
● Full chef-client takes - 1min● 3 nodes deployed sequentially - 3 mins● Deploying one node at a time not okay.● Cache busting on Backbone.js apps
Solution: parallel triggering of chef & specific deployment recipe - 5 secs
![Page 27: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/27.jpg)
What’s worked for us
● Infra as code is awesome● One command to deploy is awesome● Chef search is awesome● Chef for entire lifecycle (bootstrap +
deployment) is awesome
![Page 28: RedMart Migrating from EC2 to VPC with Chef](https://reader033.fdocuments.us/reader033/viewer/2022061202/547bebf15906b59a798b46ab/html5/thumbnails/28.jpg)
Thank you
We’re hiring! - redmart.recruiterbox.comWe now have bread! - redmart.comSlides - slideshare.net/riteshangural