Microservices and the Inverse Conway Manoeuvre - James Lewis - Codemotion Rome 2017

1

Microservices and Conway’s Law James Lewis @thoughtworks

ROME 24-25 MARCH 2017

CONSULTANTS!HELL YEAH!

5

BDD

8

“…organizations which design systems … are constrained to produce designs which are copies of the communication structure of those organizations”Melvyn Conway, 1968

The mirroring phenomenon is consistent with two rival causal mechanisms. First, designs may evolve to reflect their development environments. In tightly-coupled organizations, dedicated teams employed by a single firm and located at a single site develop the design. Problems are solved by face-to-face interaction, and performance “tweaked” by taking advantage of the access that module developers have to information and solutions developed in other modules. Even if not an explicit managerial choice, the design naturally becomes more tightly-coupled.

By contrast, in loosely-coupled organizations, a large, distributed team of volunteers develops the design. Face-to-face communications are rare given most developers never meet. Hence fewer connections between modules are established. The architecture that evolves is more modular as a result of the limitations on communication between developers.

10

http://www.hbs.edu/faculty/Publication%20Files/08-039_1861e507-1dc1-4602-85b8-90d71559d85b.pdf

"Exploring the Duality between Product and Organizational Architectures : A Test of the “Mirroring” Hypothesis"

loosely-coupled organizations ⇒ the architecture is more modular

11

tightly-coupled organizations ⇒ the design becomes more tightly-coupled.

organised around business capabilities

componentisation via services

products not projects

smart endpoints and dumb pipes

decentralised governance

decentralised data management

infrastructure automation

evolutionary design

designed for failure

How big are they?

18

microservices should be:

cheap to replace

and should allow you to go as “fast as possible”?

quick to scale

resilient

A TYPICAL COMPANYI got approximately 5 problems out of a maximum of about 20- I reckon

19

develop-ment

20

testing

architects

ops

PMO

HR

sales

marketingfinance

develop-ment

22

testing

architects

ops

PMO

HR

sales

marketing

finance

23

Project(process(

1(

Business

IT Solutions

dev

QA

Support

Idea

High level Estimation

and decision to proceed

Deployment

Testing

Business Requirement Document

Business Case

Project Brief

Functional spec

Initiation Doc

reqs. matrix

Detailed Estimate

Project Approve

d

TDD

PM docs

Test Script

s

Change Request

Requirements

clarification and

estimation

1010100010100101011010 Dev

Kick off Coding (Dev)

Additional <thing> raised

>10%

CAPEX

<10%

CAPEX

Support handover

User Acceptance

Testing

Normal Flow

Data Flow

Alternate Process

Flow

KEY

Dropped requirements become

Enhancements

develop-ment

25

testing

architects

ops

PMO

HR

sales

marketing

finance

26

each of these chords represents at least one queue

“There is nothing so useless as doing efficiently that which should not be done at all” Peter Drucker

29

SYSTEMS WOT I HAVE SEENa synthesis of many different experiences…

30

31

A capability is a combination of people, processes, systems that provides value to customers (internal or external)

The what of the business, not the how

32

delivering stuff selling stuff

Often capabilities are built by large teams

33

delivering stuff selling stuff

Using project-thinking, not product thinking*

* http://lmgtfy.com/?q=GDS+service+thinking

http://lmgtfy.com/?q=GDS+service+thinking

34

And the nicely decoupled capabilities end up looking like this

Feature starts dev

iteration 2

iteration 1

iteration 3

Feature deployed

regression testingperformance testing

deployment tests

cycle time

Anyone seen 6 week periods of “hardening”?

Unit TestsBuild Service TestsService A

BuildService B

End-to-end Tests

BuildService C

BuildService D

Service TestsUnit Tests

Unit Tests

Unit Tests

Service Tests

Service Tests

Anyone seen “fan-in” to end to end tests?

Anyone had to deploy everything all at once?

write spec

write code

test release

Without deploying into production, inventory is built up - inventory costs money and the more we have the more risky our deployments

?

38

beware the distributed monolith!

39

this is a recipe for a nervous breakdown

this is not “as fast as possible”

WHAT MIGHT GOOD LOOK LIKE?

41

Pauline Cafferkey

44

Team at the Royal Free

infectious

45

lab services

facilities

domestic

physio

micro biology

estates

pharmacy

finance

Royal Free Hospital

Capabilities

<other><other>

infectious

46

lab services

facilities

domestic

physio

micro biologyestates

pharmacy

finance

47“The approach favors agility over raw power <snip>”http://en.wikipedia.org/wiki/OODA_loop

48

1 Dr Rebecca Lester, Infectious diseases, Specialist registrar

2 Dr Damien Mack, Microbiology, Consultant

3 Dr Ruaridh Buchanan, Infectious diseases, Specialist registrar

4 Dr Simon Warren, Infectious diseases, Consultant

5 Oliver Carpenter, Infectious diseases, Nurse clinical practice educator

6 Breda Athanasopoulos, Infectious diseases, Senior matron lead high level isolation unit

7 Dr Tabitha Mahungu, Infectious diseases, Specialist registrar

8 Susan Coath, Infectious diseases, Ward sister

9 Zenaida Pandio, Domestic services, Housekeeper

10 Claire Harawa, Domestic services, Domestic team leader

11 Renato Pandio, Domestic services, Deep cleaner

12 Sheila Connolly, Patient environment, Head of patient environment

13 Peter Williams, Facilities, Security manager

14 Marchito Tumibay, Domestic services, Support worker

15 Michael Campo, Facilities, Porter

16 Lauren Geddes, Physiotherapy, Clinical lead cardiorespiratory care

17 Victoria Maddox, Laboratory service, Senior health care scientist

18 Nikki Williams, Infectious diseases, Ward sister

19 Dr Sanjay Bhagani, Infectious diseases, Consultant

20 Sarah Stanley, Patient at risk team, Nurse consultant

21 Dr Susan Hopkins, Infectious diseases and microbiology, Consultant

22 Torrain Exall, Physiotherapy, Clinical lead critical care

23 Dr Stephen Mepham, Infectious diseases and microbiology, Consultant

24 Dr Michael Jacobs, Infectious diseases, Consultant

25 Simon Rattenbury, Laboratory service, Head of laboratory service

26 Bryan O’Farrell, Pharmacy, Senior Pharmacist

27 Adrien Cooper, Estates, Director of estates

28 Robert Elliot-Cooke, Pharmacy, Senior pharmacist

29 Allan Harris, Estates, Engineering supervisor

30 Sumal Kuranaratne, Facilities, Support services manager

31 Dave Reavy, Facilities, Head of security and support services

32 Garba Ibrahim, Domestic services, Domestic team leader

33 Stephen King, Royal Free Charity, Head of finance

49


































50


































“If you imagine all the functions a hospital has to have, not only the doctors and nurses and physiotherapists and pharmacists but estates management and food and facilities and laundry — they all have to be involved, and they all have to be specially trained to be safe to work in that environment. “ Dr Michael Jacobs - Consultant, infectious disease

51

infectious

52

lab services

facilities

domestic

physio

micro biology

estates

pharmacy

finance

Royal Free Hospital

Capabilities

<other><other>

infectious diseases

53

Ebola treatment

54

single purposecross-functional teams

focussed entirely on ‘customer’ outcome

55

In our study at a large Australian Telco, we found that stories leaving a team cycled 12 times slower than those that stayed in the team.

56

This is true for Trauma teams in hospitals

And Formula 1 teams

And “reinforced battle groups”

A SOFTWARE EXISTENCE PROOF…

57

develop-ment

58

testing

architects

ops

PMO

HR

sales

marketing

finance

x-func teams organised around lines of business

59

ops

HR

sales

marketing

finance

insurance company

separate lines of business

home motor life

separate lines of business

and cross-cutting capabilities

home motor life

my account

cross-functional teams delivering lines of business


66

ops

HR

sales

marketing

finance


67

ops

HR

sales

marketing

finance


68

ops

HR

sales

marketing

finance


69

ops

HR

sales

marketing

finance

72

DEVOPS!!!Well, build it, run it…


73

ops

HR

sales

marketing

finance

74

ops

HR

sales

marketing

finance

build it / run it

build it / run it

75

HR

sales

marketing

finance

each of these capabilities can be tested and deployed independently

home motor life

my account

build it / run it

77

HR

sales

marketing

finance

79But can we go further?

getting there…

“If you imagine all the functions a hospital has to have, not only the doctors and nurses and physiotherapists and pharmacists but estates management and food and facilities and laundry — they all have to be involved, and they all have to be specially trained to be safe to work in that environment. “ Dr Michael Jacobs - Consultant, infectious disease

80

build it / run it

81

HR

sales

marketing

finance

HR

sales

marketing

finance

build it / run it

83

http://nomad8.com/squadschaptersguilds/https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/

84

market segments

85

Value streams

86

teams

each teamowns one or more services

~160-200~10-20

multiples thereof

89Thomas J. Allen, 1977

0 10 20 30 40 50 60 70 80 90 1000

0.05

0.10

0.15

0.20

0.25

0.30

m

Probability of weekly interaction

x

x x

xxxxxxxxxxx

x xx

x xx x

The effect of distance on communication

communication frequency varies with the inverse square of distance

co-locate as much as possible

take advantage of serendipitous conversations

as you chunk up from team to value stream to market segment to organisation, use Conway’s Law, Allen’s research and Dunbar’s number to preserve low-coupling and high cohesion

and remember - whenever work leaves your team, you incur cost

92

The “chunking up from microservices to teams to value streams to market

segments to organisations" practice onion*

*I might need a better name for this

My hypothesis is that you can use organisational boundaries to reason about which testing types to apply and which integration patterns to use

94

Low change rate

inter-market segment

High stability

Semantic Versioning

Contract Testing

Tolerant Reader“Conversational change”

95

higher change rate

inter-value stream

lower stability

Semantic Versioning

Contract Testing


96

higher change rate

inter-team

lower stability

Semantic Versioning

Contract Testing


97

highest change rate

intra-team

lowest stability

Semantic Versioning

Contract Testing


https://www.flickr.com/photos/snippets101/2686592689/https://creativecommons.org/licenses/by-nd/2.0/

Thoughts to finish

successfully adopting the microservice-style of architecture requires more than just componentisation via services

organised around business capabilities

componentisation via services

products not projects

smart endpoints and dumb pipes

decentralised governance

decentralised data management

infrastructure automation

evolutionary design

designed for failure

The Inverse Conway ManoeuvreDesign the organisation you want, the architecture will follow (kicking and screaming).Evan Bottcher, ThoughtWorks

CONWAY’S LAW SENDS US SIGNALS

•constantly ordering work across capabilities

•splitting stories between teams

•moving people around

•deploying in lock-step

• fan-in for end-to-end testing

THERE IS A HIGH COST ASSOCIATED WITH WORK LEAVING YOUR TEAM

• raising tickets with the “platform team”

•handing off work to other departments

•constantly starting and stopping projects

the last word

[email protected] @boicy

thanks!https://creativecommons.org/licenses/by-nc-sa/3.0/

mailto:[email protected]?subject=

Microservices and the Inverse Conway Manoeuvre - James Lewis - Codemotion Rome 2017

Technology

Transcript of Microservices and the Inverse Conway Manoeuvre - James Lewis - Codemotion Rome 2017