Microservices: moving parts around
72
Microservices: moving parts around Chris Winters @cwinters Turnitin
-
Upload
chris-winters -
Category
Software
-
view
184 -
download
1
Transcript of Microservices: moving parts around
Microservices: moving partsaround
Chris Winters
@cwinters Turnitin
What are we doing here?Talk about moving parts behind microservices
Why are we using microservices, and what are our plans?
How do they help and hinder?
Initial questionsDocker
Microservices
AWS/Azure/Google/...
My biases and contextSmall teams, small-ish companies
My biases and contextDeeply skeptical of magic
Relational database pro, ORM con
My biases and contextKernighan on debugging applies to more than code
Ease cognitive load
Side-effects killA change in feature X also changes feature Y
Second order effect: fear, uncertainty, and doubt
Response: slather on that process!
Revision AssistantPittsburgh company! (previously:
Lightside Labs)
Use machine learning to helpstudents write better
Focus on drafting process
Where we startedSingle app taking all requests; one per EC2
Single worker taking all async jobs; two-ish per EC2
Deployment via Ansible
Where we are nowApp and worker => Docker containers
...still serve a lot of traffic
Created associated app served by separate microservicesand tables
Re-architecting the workers to scale
Where we want to beMore pieces of the single app out into services
Merge duplicated data
Autoscale individual services
Why not?Operating a single app is way easier
Easier to propagate patterns within single codebase
Everything is in one place... code and admin tools (Django)
One Approved Way to do things
Why not?Reaching into another table is easy
No network call boundaries (latency)
Why?Keep a thing in your head
Independence (concurrent projects)
...both of which help with speed
Disentangle via forced separation
Why?Make side-effects explicit
Small vectors for experimenting
Scale at more granular level
Why? (Conway's Law)...doesn't really apply to us, yet
But... I can do that nowAre microservices the only way to do these?
...but few teams exhibit discipline to do them in monolith
Constrain areas ofuncertainty
Help where you're prone to get things wrong
...and enable freedom to experiment in others
What are the moving parts?Deployment and configuration
Routing and load balancing
Monitoring
Logging
Recovery
12 Factor EchoSmart people have already this designthought about
You really need to internalize this document and goals.
Just a few:
3. Config: Store in the environment
4. Backing services: Treat as attached resources
11. Logs: Treat as event streams
ECS basics: ClusterEC2 instances providing compute pool of compute
Can be in single AZ or spread across
ECS basics: Task definitionImmutable metadata to run container
- Resource constraints
- Environment
- Docker image tag
- Linked containers
- ...generally, arguments to docker run...
ECS basics: Schedulers runtasks
Given resource constraints, where to put the task?
"Run task": go now, n times, and leave when you're done
"Service": go now, n times, and stick around...
...updates, rolling deploys, ELB, and autoscaling
How we use itSingle region, per-AZ clusters
production-app-1a, production-app-1c, ...
More overhead: duplicate deploys
Bonus: switch off an entire AZ
ECS: What's missing?Deployment and configuration: OK to Amazing
Routing and load balancing: So-so
Monitoring: Meh
Logging: Meh
Recovery: OK
Developing a featureFeature branch: work work
work => push
Slack: make me a colorenvironment
Slack: run end-to-end-tests
Slack: make me someclasses
Test test test...
Developing a featurecode review
PR to staging, merge => deploys
PR to production, merge => deploys
Slightly different for front-end and services
EnvironmentsLocal + "Color"
All containers on one host
Includes datastores
Front-end deployed to S3
Extra containers for router, docs, mail
"Color"?Deployable with slack
Specify different branches for different services
Constrain at Docker, assume other differences minimizedfor these purposes
DeploymentLocal development can be in container, or not
CI/CD uses Codeship: builds and tests in containers
We currently do not use that same container to deploy
Does it matter?
Types of servicesBasic flow from API call through the system
External servicesPostgres, Redis, S3, Simple Queue Service, Simple
Notification Service...
All configured through environment and URL
Logs collected on per-host container and forwarded toLoggly
AlertingTwo inputs: Loggly and NewRelic
Loggly: watch log entries for patterns
NewRelic: watch performance and exceptions for patterns
Both go to Slack (maybe elsewhere in the future?)
Service discovery: eventsEvents: coming up and going down
Docker knows about those!
Registrator picks them up and publishes name + address +port...
Service discovery:propagation
...to Consul. Distributed k/v store and service registry
Consul Template listens for Consul changes...
...and rewrites nginx config with contents, then gives it aHUP
Routing: central vsdistributedTwo flavors of nginx:
1. Central: in front of the application routing from theinternet
2. Distributed: on every host, acting as a client-side loadbalancer
Which is better?"It depends"
Central is easier to monitor and manage, but SPOF
Distributed spreads out risk and load, but harder tomanage and monitor
AWS recommendsCluster of EC2 instances across AZs
ECS will spread containers across AZs
ELB per service (lots of services, lots of ELBs)
Ensure datastores are Multi-AZ
Sidebar: Missing factorDocumentation
First idea: one document to rule them all
What do you really want?
One place to readDifferent than one place to write
Your instinct is to control at the wrong level
It's what you're comfortable with
You are mutable
Solve THAT problemControl the format and API
Tool to collect and merge
Everything is a container
Control allows leverage
Constraints over controlWhat does that even mean?
Constraints: define the boundaries of a thing, hands offwithin
Control: define both what a thing must do and how it mustdo its job
What you knowControl illusory if not at right level
You'll probably get it wrong with something new
Developers: control coding standard, module layout, etc.
What problem are you trying to solve?
Example: ORMsWhy do you use them?
Small service: few entities, few relationships
Are you really going to switch databases?
Shouldn't you know SQL?
Constrain data accessGet away from integrating at data level - they live forever
Wrapping API around queries means you can controlarbitrary access
Fewer connections, less to keep in your head
Few data lessonsEverything is relational (except caches)
...with some JSONB tables
We are using a single database... for now
Few data lessonsServices cannot peek into other services tables...
...except for reporting (still thinking about this)
Data affinity for read-only.
Emergent simplicity"You get these emergent simplicities when the thing starts
to take shape..." (Dan North)
As you pull small systems out of larger systems, keep youreyes open for commonalities or things to eliminate.
Left to doClient per backend service
Testing service-to-service interactions
Monitoring improvements
Move to AWS blessed architecture?
Resources and referencesJames Lewis and Martin Fowler:
which is a excerpted form of (Thoughtworks)Fight Club:
; Sam Newman: (Thoughtworks) and the Lewis/Fowler
article above also has a section titled "Organized aroundBusiness Capabilities" that describes it well
Werner Vogels print interview(2006); also from the same time.
Microservices in aNutshell this article
"Stop trying to control everything and just letgo..."Conway's Law (wikipedia) DemystifyingConway's Law
"You build it, you run it"this podcast interview
Resources and references, plus a on them from
@caseywestCasey also gave his talk atDevopsdays Minneapolis a week and a half ago.Adrian Cockroft has given loads of talks, this is a kind ofbest-of: ,which includes a brief discussion of building per-serviceclients (slide 122).
12 Factor Apps Cloud Native spin
Minimum Viable Platform
Migrating to Cloud Native with Microservices
Resources and references
Dan North: ;you should just go ahead and watch
, including (from 2009!) that talksabout the relationship between architecture and teamsMy pinboard links: | | |
(you see the pattern...)
Steve Yegge's platform rantMicroservices: Software that fits in your head
his otherpresentations this one
microservices architecture dockeraws
