Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS...

48
IP Expo Manchester April 2019 Terraform for Serverless. Best Practices. Lessons Learned.

Transcript of Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS...

Page 1: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Terraform for Serverless.

Best Practices. Lessons Learned.

Page 2: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

DevOps Landscape. Daily Challenges.

Page 3: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

If That Was Not Enough…

Introducing Serverless Landscape

Page 4: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Serverless Landscape. Daily Challenges.

Page 5: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Raise your hand if...

this feels like

TOO MUCHto keep up with

Page 6: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Eugene

ISTRATI

@eistrati

About Presenter

• CTO, Tech Partner @ Mitoc Group

• Ex-AWS, ex-Hearst, ex-GrubHub

• Certified AWS Solutions Architect

• 20 Years in IT; 10 Years in Cloud

Computing; 5 Years in Enterprise IT

• Focusing on: Automation, DevOps,

Serverless

Page 7: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Agenda

Exponential increase

in services and tools

for cloud & serverless

Provide reusable and

cloud native solutions to

large organizationsThe Devil is in Details

Page 8: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

The Problem

Exponential Increase

in Services and Tools

for Cloud & Serverless

Page 9: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

The Opportunity

$49B customer

$33B customer

$11B customer

+ other customers

Page 10: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Enterprise Customers Love Terraform

$49B customer

$33B customer

$11B customer

+ other customers

Page 11: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Page 12: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Prerequisites: Terraform For Serverless

1. Understand IT-as-a-Service Spectrum

1. Understand DevOps Spectrum

2. Understand Scope & Boundaries

Page 13: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

1. Understand IT-as-a-Service Spectrum

On-Prem

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Colocation

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Hosting

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

IaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

PaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

SaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Managed by Customer Managed by Provider

Page 14: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

What Is Serverless?

Page 15: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Serverless in IT-as-a-Service Spectrum

On-Prem

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Colocation

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Hosting

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

IaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

PaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

SaaS

Data

Application

Databases

Operation System

Virtualization

Physical Servers

Network & Storage

Data Center

Managed by Customer Managed by Provider

Serverless Architecture

not in scope

Page 16: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

2. Understand DevOps Spectrum

Page 17: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

3. Understand Scope & Boundaries

A

B C

Page 18: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Terraform For Serverless

A

B C

B == Terraform

A + B + C == Terraform

For Serverless

Page 19: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Terraform For Serverless

Best Practices.

Lessons Learned.

Page 20: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #1 (of 8)

Adopt microservices architecture; aim for 1-to-1 relationship

between serverless resources and terraform configurations

Page 21: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #1 (of 8)

Adopt microservices architecture; aim for 1-to-1 relationship

between serverless resources and terraform configurations

Page 22: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #1 (of 8)

Adopt microservices architecture; aim for 1-to-1 relationship

between serverless resources and terraform configurations

Page 23: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #2 (of 8)

Pass variables between resources using terraform remote state

Page 24: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #2 (of 8)

Pass variables between resources using terraform remote state

Page 25: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #2 (of 8)

Pass variables between resources using terraform remote state

Page 26: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #3 (of 8)

Avoid code build using local provisioner or external data; instead

use hooks provided by terraform orchestration tools

Page 27: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #3 (of 8)

Avoid code build using local provisioner or external data; instead

use hooks provided by terraform orchestration tools

Page 28: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #3 (of 8)

Avoid code build using local provisioner or external data; instead

use hooks provided by terraform orchestration tools

Page 29: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #4 (of 8)

Execute in parallel your automated terraform workflows; don’t

ignore terraform configurations dependencies

Page 30: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #4 (of 8)

Execute in parallel your automated terraform workflows; don’t

ignore terraform configurations dependencies

$ terraform apply -parallelism=100var.account_id

Allowed AWS account ID, to prevent you

from mistakenly using an incorrect one

(and potentially end up destroying a live

environment)

$ terrahub apply --auto-approve[api_gateway_rest_api] terraform apply -auto-approve

[cognito_identity_pool_es] terraform apply -auto-approve

[cloudtrail] terraform apply -auto-approve

[codebuild] terraform apply -auto-approve

[codepipeline] terraform apply -auto-approve

[cognito_user_pool_client] terraform apply -auto-approve

[cognito_user_pool] terraform apply -auto-approve

[db_subnet_group] terraform apply -auto-approve

Page 31: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #4 (of 8)

Execute in parallel your automated terraform workflows; don’t

ignore terraform configurations dependencies

$ terraform apply -parallelism=100var.account_id

Allowed AWS account ID, to prevent you

from mistakenly using an incorrect one

(and potentially end up destroying a live

environment)

$ terrahub apply --auto-approve[api_gateway_rest_api] terraform apply -auto-approve

[cognito_identity_pool_es] terraform apply -auto-approve

[cloudtrail] terraform apply -auto-approve

[codebuild] terraform apply -auto-approve

[codepipeline] terraform apply -auto-approve

[cognito_user_pool_client] terraform apply -auto-approve

[cognito_user_pool] terraform apply -auto-approve

[db_subnet_group] terraform apply -auto-approve

Page 32: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #5 (of 8)

Optimize automated terraform workflows with git diff

Page 33: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #5 (of 8)

Optimize automated terraform workflows with git diff

$ terrahub run --dry-runProject: Security_Terraform

├─ IamIdp

├─ IamRoleForADadmins

├─ IamRoleForADcompliance

├─ IamRoleForADdevelopers

├─ IamRoleForADdevops

├─ IamRoleForADguests

├─ IamRoleForApiGateway

├─ IamRoleForGlue

├─ IamRoleForLambda

└─ IamCrossAccountRoleForTerrahub

$ terrahub run --dry-run --git-diff master...devProject: Security_Terraform

├─ IamRoleForApiGateway

└─ IamRoleForLambda

Page 34: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #5 (of 8)

Optimize automated terraform workflows with git diff

$ terrahub run --dry-runProject: Security_Terraform

├─ IamIdp

├─ IamRoleForADadmins

├─ IamRoleForADcompliance

├─ IamRoleForADdevelopers

├─ IamRoleForADdevops

├─ IamRoleForADguests

├─ IamRoleForApiGateway

├─ IamRoleForGlue

├─ IamRoleForLambda

└─ IamCrossAccountRoleForTerrahub

$ terrahub run --dry-run --git-diff master...devProject: Security_Terraform

├─ IamRoleForApiGateway

└─ IamRoleForLambda

Page 35: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #6 (of 8)

Logically separate environments using terraform workspace

Page 36: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #6 (of 8)

Logically separate environments using terraform workspace

dev == default

Page 37: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #6 (of 8)

Logically separate environments using terraform workspace

dev == default

Page 38: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #7 (of 8)

Overwrite environment specific values using variables precedence

Page 39: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #7 (of 8)

Overwrite environment specific values using variables precedence

https://www.terraform.io/docs/configuration/variabl

es.html#variable-precedence

https://www.terraform.io/docs/configuration/variables.html#variable-precedence
Page 40: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #7 (of 8)

Overwrite environment specific values using variables precedence

https://www.terraform.io/docs/configuration/variabl

es.html#variable-precedence

https://www.terraform.io/docs/configuration/variables.html#variable-precedence
Page 41: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #8 (of 8)

Get comfortable with lots of terraform code; or use terrahub cli

Page 42: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #8 (of 8)

Get comfortable with lots of terraform code; or use terrahub cli

$ find . -name ‘*.tf*’ | xargs wc -l | grep total

33998 total

$ find . -name ‘.terrahub*.yml’ | xargs wc -l | grep total

22118 total

Page 43: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practice #8 (of 8)

Get comfortable with lots of terraform code; or use terrahub cli

$ find . -name ‘*.tf*’ | xargs wc -l | grep total

33998 total

$ find . -name ‘.terrahub*.yml’ | xargs wc -l | grep total

22118 total

Page 44: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Best Practices Summary

1. Adopt microservices architecture; aim for 1-to-1 relationship

1. Pass variables between resources using terraform remote state

1. Avoid code build using local provisioner or external data

1. Execute in parallel your automated terraform workflows; don’t ignore

terraform configurations dependencies

1. Optimize automated terraform workflows with git diff

1. Logically separate environments using terraform workspace

1. Overwrite environment specific values using variables precedence

1. Get comfortable with lots of terraform code; or use terrahub cli

Page 45: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Lessons Learned Summary

1. Adopt microservices architecture; aim for 1-to-1 relationship

1. Pass variables between resources using terraform remote state

1. Avoid code build using local provisioner or external data

1. Execute in parallel your automated terraform workflows; don’t ignore

terraform configurations dependencies

1. Optimize automated terraform workflows with git diff

1. Logically separate environments using terraform workspace

1. Overwrite environment specific values using variables precedence

1. Get comfortable with lots of terraform code; or use terrahub cli

https://github.com/

TerraHubCorp/terrahub

https://github.com/TerraHubCorp/terrahub
https://github.com/TerraHubCorp/terrahub
Page 46: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Terraform For Serverless

DEMO

https://github.com/TerraHubCorp/

demo-terraform-automation-aws

https://github.com/TerraHubCorp/demo-terraform-automation-aws
https://github.com/TerraHubCorp/demo-terraform-automation-aws
Page 47: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Page 48: Terraform for Serverless.€¦ · $ terraform apply -parallelism=100 var.account_id Allowed AWS account ID, to prevent you from mistakenly using an incorrect one (and potentially

IP Expo Manchester

April 2019

Terraform for Serverless.

Best Practices. Lessons Learned.

Eugene Istrati @eistrati

[email protected]

Thank You!


Effective terraform

Effective terraform

Terraform at Adobe

Terraform at Adobe

Building infrastructure with Terraform (Google)

Building infrastructure with Terraform (Google)

Elite Brochure | Wadhwa Terraform Realty Thane

Elite Brochure | Wadhwa Terraform Realty Thane

An NCC Group Publication SERVERLESS ARCHITECTURE: THE ... · An NCC Group Publication | Serverless Architecture 3 Scalability in Serverless Computing Serverless computing offers the

An NCC Group Publication SERVERLESS ARCHITECTURE: THE ... · An NCC Group Publication | Serverless Architecture 3 Scalability in Serverless Computing Serverless computing offers the

Securing Serverless Architectures - AWS Serverless Web Day

Securing Serverless Architectures - AWS Serverless Web Day

Infrastructure as Code with Terraform · State Terraform stores state about your managed infrastructure and configuration. Used by Terraform to map real world resources to your configuration

Infrastructure as Code with Terraform · State Terraform stores state about your managed infrastructure and configuration. Used by Terraform to map real world resources to your configuration

GroupHealth Insurance Presentation-Terraform

GroupHealth Insurance Presentation-Terraform

Serverless & PHPthe future serverless? is PHP's future serverless? try! learn! share! bref.sh null serverless-php.news @matthieunapoli. Title: Bref - February 2020 Created Date:

Serverless & PHPthe future serverless? is PHP's future serverless? try! learn! share! bref.sh null serverless-php.news @matthieunapoli. Title: Bref - February 2020 Created Date:

Application Repository AWS Serverless · AWS Serverless Application Repository Developer Guide What Is the AWS Serverless Application Repository? The AWS Serverless Application Repository

Application Repository AWS Serverless · AWS Serverless Application Repository Developer Guide What Is the AWS Serverless Application Repository? The AWS Serverless Application Repository

Terraform at Scale

Terraform at Scale

Serverless Architecture Patterns - Manoj Ganapathi - Serverless Summit

Serverless Architecture Patterns - Manoj Ganapathi - Serverless Summit

Refactoring terraform

Refactoring terraform

Terraform Basics - Adfinis SyGroupTerraform Basics Be smart. Think open source. Write, Plan, and Create Infrastructure as Code Agenda Introduction to Terraform Setup HCL My first Terraform

Terraform Basics - Adfinis SyGroupTerraform Basics Be smart. Think open source. Write, Plan, and Create Infrastructure as Code Agenda Introduction to Terraform Setup HCL My first Terraform

Terraform: Configuration Management for Cloud Services

Terraform: Configuration Management for Cloud Services

Terraform Modules and Continuous Deployment

Terraform Modules and Continuous Deployment

Terraform Earth - QCon · Builds Terraform state files by fetching remote resources, think `$ terraform refresh` Manual and distributed changes easily reconciled when AWS is the source

Terraform Earth - QCon · Builds Terraform state files by fetching remote resources, think `$ terraform refresh` Manual and distributed changes easily reconciled when AWS is the source

HashiCorp Certified - Terraform Associate Certification

HashiCorp Certified - Terraform Associate Certification

Terraform - Infrastructure as Code

Terraform - Infrastructure as Code

Nascent Notions On Serverless Computing€¦ · 2 Serverless Computing 3 Server Based Model vs Serverless Model 4 Why to use Serverless? 5 Leading Serverless providers 6 The Serverless

Nascent Notions On Serverless Computing€¦ · 2 Serverless Computing 3 Server Based Model vs Serverless Model 4 Why to use Serverless? 5 Leading Serverless providers 6 The Serverless