Microservices: Organizing Large Teams for Rapid Delivery
-
Upload
pivotal -
Category
Technology
-
view
416 -
download
0
Transcript of Microservices: Organizing Large Teams for Rapid Delivery
![Page 1: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/1.jpg)
Microservices: Organizing a Large Team for Rapid DeliveryBy Jason Goth, Micah Blalock, Patricia Anderson
@jgothtx, @micah_blalock, @patricia_sooner
![Page 2: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/2.jpg)
About Us
Jason [email protected]
@jgothtx
Micah [email protected]
@micah_blalock
Patricia [email protected]
@patricia_sooner
![Page 3: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/3.jpg)
Microservices are great hard
3
![Page 4: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/4.jpg)
Managing teams developing microservices is not so great much harder
4
![Page 5: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/5.jpg)
Objectives
• Share lessons learned on a few real world microservices projects:• Several customer projects• One internal project
• Topics:• Organizational structure for rapid delivery• Design/Development practices to support these organizations• Communication across multiple teams
5
![Page 6: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/6.jpg)
Case Study … Custom Analytics Platform
6
We are here in the story
2014 2015 2016
AugOct
![Page 7: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/7.jpg)
The Problem
• Suite of applications:• Analytics for hospitals • Multiple products/users• Products share core services
(e.g. how much does X cost? how many of these did I use?)
• Multiple product owners• Billions of rows of legacy data• Huge roadmap of features and
functionality (that they’d already promised…)
7
Lots of Legacy Data
ProductOwner
ProductOwner
Customers
Customers
App
App
App
SharedStuff Customers
System
![Page 8: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/8.jpg)
They Promised This
8
We are still here in the
story
2014 2015 2016
AugOct
Product 1 Product 2 Product 3 Product 4 And So On
![Page 9: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/9.jpg)
We Decided To Do Something Like This
• Reuse of services ? • Add applications easily ?• Scale out forever ? • Cool factor? • Multiple teams can deliver products
and features super fast ? Umm, we’ll come back to that...
9
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
![Page 10: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/10.jpg)
Let’s Use Spring!
10
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . . SpringBoot
SpringFramework
SpringSecurity
SpringData
SpringBatch
SpringAMQP
Spring Cloud
Service Registry
Configuration Server
![Page 11: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/11.jpg)
It Worked Great!
• Built the first application in the suite and the first few services in a few months
• Spring-* provided a lot of the plumbing• We had one team (8-10 people) for the
application and the services• Big success
11
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . . DevTeam
![Page 12: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/12.jpg)
This Is Easy … Let’s Do Another One
12
Phase 2 StartsStart
Big Success
2014 2015 2016
AugOct AprMar
![Page 13: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/13.jpg)
It Worked Ok …
• Second team (8-10 people) brought on board to build the next application in the suite
• Did it all in 3 months this time• Another big(-ish) success
• Teams still did their own thing• More friction, but manageable
13
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
![Page 14: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/14.jpg)
Let’s Do 2 More Apps … in Only 3 Months
14
“Phase 3” Starts
Start
Big Success
2014 2015 2016
AugOct Mar Jun July
Big(-ish) Success
![Page 15: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/15.jpg)
We Just Need To Add Some People, Right?
• Continue trying to scale out as before:• Add new teams for new features
and services• Letting teams do their own thing
15
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
![Page 16: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/16.jpg)
Two Months In …
16
StartBig
Success
2014 2015 2016
AugOct Mar Jun
Big(-ish) Success
We are here
Sep
![Page 17: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/17.jpg)
It All Came to a Screeching Halt
17
![Page 18: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/18.jpg)
Why?
18
![Page 19: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/19.jpg)
It Really Looked Like This
• Dependency hell:• Each feature changed many
services• Changes cascaded everywhere• Teams were stepping on each
other• Versioning helped; but versioning is its
own special hell
19
Clients
Gateway
App 1
Svc 1
App 2 App 3 App 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc 2 Svc 3 Svc 4
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
![Page 20: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/20.jpg)
Team Health Suffering
• Unequal workloads • Training issues:
• Unfamiliar systems • Not always the right skillset
• We had sooooo many meetings:• Many to coordinate with each
other• Up to 50% of people’s time
20
![Page 21: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/21.jpg)
Some Things Outside Our Control
• “Legacy” processes added friction:• “Architecture Review Board”• “Change Control Board”
• Shared environments • QA, Load Testing, etc.• Customer doesn’t use any cloud
services L
21
![Page 22: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/22.jpg)
Isn’t this supposed to be faster?
22
![Page 23: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/23.jpg)
It can be faster, if you align the solution and the organization
23
Main Pointof This Talk
![Page 24: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/24.jpg)
Alignment
• Conway’s Law is the starting point• When each team “owns” their own services
• Easy to change• Coordination costs low
• Sharing exposes stress points, coordination costs increase:
• Adding features to existing services• Cascading changes to service contracts
• Coordination costs + partitioning of work = efficiency of change.
24
SvcSvc Svc
Svc
Svc
Svc
Stress point
![Page 25: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/25.jpg)
Partitioning … Time vs Workers
25
PartitionedTasks
Non-partitioned Tasks
Partitioned TasksHigh Coordination
Time
Workers Workers Workers
Time
Time
Source: The Mythical Man-Month, Frederick P. Brooks, Jr., 1995
![Page 26: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/26.jpg)
Better Partitioning
• Refactor to create surface area• Similar to the open/closed
principle• Package code by features not layers
• Add packages for new features• Grow code horizontally not
vertically
26
// This is your clue to refactor
if( purchaseType.equals(“Lab”)) {savings = getLabSavings()
} else if (purchaseType.equals(“Rx”)) {savings = getRxSavings()
} else ...
![Page 27: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/27.jpg)
Handling Cascading Changes
• Remove semantic coupling• Shared concepts,data types or domain
entities• Behavioral dependencies
• The more effects and side-effects a service has, the tighter the coupling with its consumers
See: http://www.michaelnygard.com/blog/2015/04/the-perils-of-semantic-coupling/
27
Oracle Redis Elastic SearchOracle
Gateway
Svc 2
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
Svc 1
App 1
![Page 28: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/28.jpg)
Example ... Semantic Coupling
28
Invoices
Customer
FinanceApp
Spend
FinanceApp
![Page 29: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/29.jpg)
Eliminate Shared Dependencies
• Warning architecture purists … you will not like this…• Shared dependencies increase coordination costs:
• Some redundancies are ok• GARY (Go Ahead, Repeat Yourself)
29
![Page 30: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/30.jpg)
Example … GARY
30
// Does every service need to include a shared component (JAR) to validate plan types?
import java.utils.Arrays;
private static String [] PLAN_TYPES= {“HMO”, “PPO”};
public static boolean isValidPlanType(String planType) {return Arrays.stream(PLAN_TYPES).anyMatch(planType::equals)
}
What if different services accept different types of plans? Are all services using Java 8?
![Page 31: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/31.jpg)
Example … GARY
31
// Service 1if(planType.equals(“HMO”) ||
planType.equals(“PPO”) {return true;
}
// Service 2if(planType.equals(“HMO”) {
return true;}
• Why not this?
Great discussion here: http://blog.cognitect.com/blog/2016/6/16/the-new-normal-team-scale-autonomy
![Page 32: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/32.jpg)
Your design will evolve constantly … refactor ruthlessly
32
![Page 33: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/33.jpg)
Ok, now let’s address the team problems
33
![Page 34: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/34.jpg)
Reorganizing Teams
• Refactored design allows us to organize around customer needs (applications/features):
• Single Product Owner• Few overlapping changes• Much lower coupling
• Split into smaller teams (~4 people)
• Focused on the “full stack”… everyone you need
34
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
. . .
![Page 35: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/35.jpg)
Addressing Our Training Problems
• Put some standards in place:• Just enough of them• Familiarity when switching between components
• No onerous review/governance process:o That’s an organizational dependencyo Trust people to follow standards
• Train team on all components and standards:• Sample code/generators/snippets• “Real” documentation, not shelf-ware
35
![Page 36: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/36.jpg)
Addressing Our Workload Problems
• Warning Scrum Masters ... you will not like this…
• We shift our resources all the time• Our standards and consistency
make this easy to do• Do you want to predict velocity
or do you want to havevelocity?
• Over time it averages out
36
Clients
Gateway
App 1
Svc 2Svc 1
App 2 App 3 App 4
Svc 3 Svc 4
Service Registry
Configuration Server
Oracle Redis Elastic SearchOracle
Svc N-2Svc N-3 Svc N-1 Svc N
![Page 37: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/37.jpg)
Addressing our Meeting Problems
• Elect one person per team as “Team Lead”:• Could be anyone• Coordinate any cross-team questions • Facilitate inter-team questions
• External Team Communication:• Designate someone as the sacrificial
offering to the Legacy Process Overlords
• You may need more than one person• Let everyone else work!
37
Legacy ProcessOverlord
![Page 38: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/38.jpg)
So Where Are We In The Story?
38
Start
Big Success
2014 2015 2016
AugOct Mar Jun
Big Success
Sep Feb
Things Stopped
We are hereLots of refactoring, documentation,
reorg, etc.
![Page 39: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/39.jpg)
We Started Gaining Momentum
• Teams working (mostly) independently• Meeting frequency greatly reduced • Not as much cross-team coordination• Legacy Process Overlords appeased
39
![Page 40: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/40.jpg)
And Now We Are Here
40
Start
Big Success
2014 2015 2016
AugOct Mar Jun
Big(-ish) Success
Sep Feb
Things Stopped
Back on Track
Tons Accomplished
Huge Success
Jun
Lots of refactoring, documentation,
reorg, etc.
4 new applications in the suite in just over 4 months80+ enhancements to existing applications
![Page 41: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/41.jpg)
Have a Need for Speed?
41
So, have a need for speed?
![Page 42: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/42.jpg)
Summary
• Align solution and the organization• Improve partitioning • Reduce coupling and dependencies• GARY
• Create standalone teams with singular focus• Have “just enough” standards/process• Be flexible; its ok to move people around• Have people responsible for communications• Others? We’d love to hear from you
42
![Page 43: Microservices: Organizing Large Teams for Rapid Delivery](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ed417b1a28ab99298b4661/html5/thumbnails/43.jpg)
Learn More. Stay Connected.
@springcentralspring.io/blog
@pivotalpivotal.io/blog
@pivotalcfhttp://engineering.pivotal.io