MongoDB Europe 2016 - Powering Microservices with Docker, Kubernetes, and Kafka
-
Upload
mongodb -
Category
Data & Analytics
-
view
226 -
download
3
Transcript of MongoDB Europe 2016 - Powering Microservices with Docker, Kubernetes, and Kafka
Powering Microservices with Docker, Kubernetes & Kafka Andrew Morgan MongoDB Product Marketing @andrewmorgan
#MDBW16
Agenda
What, Why, How?
Microservices 01 Kubernetes, Mesos,…
Orchestration 03 Docker Kafka
Containers 02
Why, How?
MongoDB 04 Who, Why?
Use Cases 06 When to Use 05
#MDBW16
Why Use Microservices? (tl;dr WebScale)
Team = Components No committees
Empower Build MVP quickly
Speed Simple components
Isolated impact
Maintain Rapid iterations React to market
Change Product Team
Scale
#MDBW16
1990s & Earlier
Monolithic
Cross-Team
Tight Coupling
Small Change
Huge Impact
System Retests
#MDBW16
Microservices Example
Twitter Ingest Google+
Ingest
Snapchat Ingest
Feed merge
Facebook Ingest
#MDBW16
Microservices Example
Twitter Ingest
Snapchat Ingest
Feed merge
Facebook Ingest
Whatsapp Ingest
#MDBW16
Microservices Example
Twitter Ingest
Snapchat Ingest
Feed merge
Facebook Ingest
Whatsapp Ingest
Snapchat Ingest Snapchat
Ingest
#MDBW16
Alternate Microservices Example
• Much larger Microservices 1. User Account 2. Product Catalog 3. Inventory 4. Orders
• Best practice is each microservice has its own database
“ Conway’s Law 1967
Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.”
#MDBW16
Containers – Powering Microservices
Real world shipping containers • Road, rail & sea • Contents untouched • Ubiquitous & standardized • Simple • Contents protected • Constraints
#MDBW16
Containers – Powering Microservices
Software containers • 1 image -> Many containers
• Laptop, DC, cloud • Dev, QA, production, support
• Simple, efficient • Isolation • Constraints
#MDBW16
VMs vs. Containers
VM VM VM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container Container Container
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service Service Service
#MDBW16
VMs vs. Containers
VM VM VM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container Container Container
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service Service Service
#MDBW16
VMs vs. Containers
VM VM VM
Bare Metal
Host Operating System
Hypervisor
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Guest OS
Libraries
Apps
Service
Container Container Container
Bare Metal
Host Operating System
Docker Engine
Libraries
Libraries
Apps
Libraries
Apps
Service Service Service
#MDBW16
Docker
• Simple to use • 100K+ images on Docker Hub • Build images from images • Platforms
• Linux, OS X, Windows • Laptop, VM, Cloud,… • Cloud services
#MDBW16
Security
Opportunities • Isolation • Constrain resources • Narrow roles • Keep images/containers current • root: so restrict access
#MDBW16
Microservice Architectures Built on Containers
Many small, focused containers -> sophisticated services • Well defined APIs • Independent languages & libraries • Modular: easy maintenance + reuse • Fault tolerant • Scalable
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
987
1 2 3 ...
Topic A
Consumer
Producer Consumer
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
987
1 2 3 ... Partition 0
Topic A
Consumer
Producer Consumer
435
1 2 3 ... Partition 1
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
LEADER
Topic A / Partition 0
Broker 1
FOLLOWER
Topic A / Partition 1
FOLLOWER
Topic A / Partition 0
Broker 2
LEADER
Topic A / Partition 1
#MDBW16
Connecting the Microservices – Apache Kafka
Producer
Producer
Producer
987
1 2 3 ...
Partition 0
435
1 2 3 ...
Partition 1
732
1 2 3 ...
Partition N
Topic A
Topic B
765
1 2 3 ...
Partition 0
New ß Old
Consumer
Consumer
#MDBW16
Orchestration
Automated deployment, connecting, and maintenance of multiple containers • Provision hosts • Containers
• Instantiate • Reschedule • Link • Scale Out/In
• Expose services
#MDBW16
Kubernetes
Created by Google, feature-rich and widely adopted • Deployment and ‘replication’ • On-line scale out/in • Rolling upgrades • High Availability • Persistence • Ports • Load balancing • Google Compute Engine
#MDBW16
Apache Mesos
10,000s of physical servers; used by Twitter, Airbnb & Apple • Code (“frameworks) vs. declarative • Less feature rich than Kubernetes • Kubernetes as a Mesos framework • Foundation for distributed systems
• Apache Aurora, Chronos, Marathon
#MDBW16
Choosing an Orchestration Framework
• What you have: • Skills? • DevOps frameworks? • Number of hosts? • Bare metal, VMs, or cloud?
• Lifecycle • Features
• Automated High Availability? • Grouping and load balancing? • As a service??
#MDBW16
Why MongoDB is a Good Fit For Microservices
Scalability Monitoring &
Automation
Redundancy Flexible Data Model
Simplicity
#MDBW16
Orchestrating MongoDB Using Kubernetes
Distributed, stateful application
• Persistent volumes
• External IP addresses for internal comms
• Init MongoDB replica set
• Monitor
• Backup
Volumename: mongo-persistent-storage1
pdName: mongodb-disk1
mongodb-disk1
Containername: mongo-node1image: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:
name: mongo-persistent-storage1mountPath: /data/db
DockerHubRegistry
Podlabels: [name: mongo-node1; instance: rod]
ReplicationControllername: mongo-rc1labels: [name: mongo-rc]replicas: 1 selector: [name: mongo-node1]
Service:LoadBalancername: mongo-svc-a labels: [name: mongo-svc-a]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node, instance: rod]Ex
ternalIP
Address
104.1.1.1
App
104.1.1.1:27017
ReplicationControlername: mongo-rc-europelabels: [name: mongo-europe]replicas: 1 selector: [name: mongo-node]
Volumename:
mongo-persistent-storagepdName: mongodb-disk-europe
mongodb-disk-europe
Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:
name: mongo-persistent-storagemountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-europelabels: [name: mongo-svc-europe]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]
ReplicationControlername: mongo-rc-asialabels: [name: mongo-asia]replicas: 1 selector: [name: mongo-node]
Volumename: mongo-persistent-storage
pdName: mongodb-disk-asia
mongodb-disk-asia
Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:
name: mongo-persistent-storagemountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-asialabels: [name: mongo-svc-asia]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]
ReplicationControlername: mongo-rc-uslabels: [name: mongo-us]replicas: 1 selector: [name: mongo-node]
Volumename: mongo-persistent-storage
pdName: mongodb-disk-us
mongodb-disk-us
Containername: mongo-nodeimage: mongocommand: mongod –replSet my_replica_setcontainerPort: 27017volumeMounts:name: mongo-persistent-storagemountPath: /data/db
Podlabels: [name: mongo-node]
Service:LoadBalancername: mongo-svc-uslabels: [name: mongo-svc-us]ports: [port: 27017, targetPort: 27017]]selector: [name: mongo-node]
#MDBW16
References
• Enabling Microservices – Containers & Orchestration Explained
https://www.mongodb.com/collateral/microservices-containers-and-
orchestration-explained
• Microservices: The Evolution of Building Modern Applications
https://www.mongodb.com/collateral/microservices-the-evolution-of-
building-modern-applications