Disruptive Innovation: how do you use these theories to manage your IT?

Disruptive Innovation: Past, Present, Future (how to use these theories to manage your IT) February, 2016

Mark Madsen - @markmadsen - http://ThirdNature.net

© Third Nature

Innovation: The Cargo Cult of Management Consulting

© Third Nature

The go-to innovation company is Google

You can’t get fired for doing what everyone else did (but you can get fired for not getting the results they did)

© Third Nature

If you do what google did you could:

Make a data center out of shipping containers.

▪ That didn’t work out.

Build your own servers.

▪ Made out of “razor blades and hate”

▪ Start fires in the data center

Build your own environmental cooling data center

▪ Generating fog and rain inside the data center

As Dan Luu noted, at least copy current engineering practices rather than things done in 1999.

By the way, are you using MapReduce?

© Third Nature

Saying there’s a process you can follow to be innovative is like saying there’s a recipe that will make you a chef.

© Third Nature

You keep using that word.

I do not think it means

what you think it means.

Innovation?

© Third Nature

Innovation is not “add new features”

© Third Nature

“Better experiences, not more features.” Roland Rust

“When technology

delivers basic needs,

user experience

dominates”

Don Norman

© Third Nature

Value is not in the product, it’s in the practice

Innovation is not a characteristic of things

© Third Nature

Innovation is change. Change is often not appreciated.

© Third Nature

Paradox: Innovation becomes best practice

Innovation isn’t reproducible. Only the conditions that permit it are

© Third Nature

HOW DOES THE MARKET WORK AND WHAT IS HAPPENING TO OUR TECHNOLOGIES?

© Third Nature

Commoditization of Computing Technology is the Driver

“There is no reason anyone

would want a computer in

their home.”

Ken Olson, CEO of DEC, 1977

“…by 2008 we will be producing

one billion transistors for every

man, woman and child on earth”

Semiconductor Industry Association, 2007

© Third Nature

How significant is the computing improvement?

1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

1010

10 9

10 8

107

106

105

104

103

102

101

10

10-1

01-2

10-3

10-4

10-5

10-6

Ca

lc

ula

tio

ns p

er se

co

nd

p

er $

10

00

Data: Ray Kurzweil, 2001

10,000 X improvement

DW architecture and

methods start here in

the mid 80s Term “BI” coined

Mechanical Relay Vacuum tube Transistor Integrated circuit

© Third Nature

Don’t worry about performance, just buy more hardware

10,000 X performance

improvement, soon 100K

© Third Nature

There are always limits

“If the automobile had followed

the same development as the

computer, a Rolls-Royce would

today cost $100, get a million

miles per gallon, and explode

once a year killing everyone

inside.”

Robert Cringely

Time

Anything

Reality

© Third Nature

RIP Moore’s Law. Data is growing faster than compute. This forces an architectural shift.

© Third Nature

We’ve reached another generational technology shift

1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

1010

10 9

10 8

107

106

105

104

103

102

101

10

10-1

01-2

10-3

10-4

10-5

10-6

Ca

lc

ula

tio

ns p

er se

co

nd

p

er $

10

00

Mechanical Relay Vacuum tube Transistor Integrated circuit

Data: Ray Kurzweil, 2001

Multicore and networked

parallelism is the next wave

© Third Nature

What’s different?

Parallelism

We’re not getting more CPU power, but more CPUs.

There are too many CPUs relative to other resources, creating an imbalance in hardware platforms.

Most software is designed for a single worker, not high degrees of parallelism and won’t scale well.

© Third Nature

Reality: you must assume distributed architecture

Why by default? Because the upgrade between single node and distributed is a major change to designs. It carries new component linkages and complexities. A new ecosystem.

The future holds cloud provisioning, software-defined environments and a lot less single-server provisioning.

Slide 21 Copyright Third Nature, Inc.

(a) Scaling up with a larger server (b) Scaling out with many small servers (aka “++ungood)” (aka “the future”)

The future is already here, it just isn’t evenly distributed yet.

© Third Nature

An important cloud computing benefit

Scalability is free (if you have the right software)

If your task requires 10 units of work, you can decide when you want the results:

10 servers, 1 unit of time

Cost is the same. Not true of the conventional IT model

Time

1 server, 10 units of time

X X

© Third Nature

We are in a transitional phase in IT architecture

Then State of Practice Now, forward

Architecture Timeshare Client/server Cloud

Data Core TXs All TXs, some events, docs

All data

Rate of change Slow Rapid Continuous

Uses Few Many Everything

Latency Daily+++ < daily to minutes

Immediate

Data platform Uniprocessor SMP, cluster Shared nothing

© Third Nature

Majority use of computing over time

1930s-1950s: Calculate

1960s-1980s: Automate

1990s-2010s: Informate

2010s+: Analyze and Actuate

Computing technology has become a tool of observation

Ris

ing

org

aniz

atio

na

l co

mp

lexity

© Third Nature

Evolution of data

50s-60s: data as product

70s-80s: data as byproduct

90s-00s: data as asset

2010s +: data as substrate

The real data revolution is in business structure and processes and how they use information.

© Third Nature

Not hype: another round of infrastructure change

Mainframe c/s cloud

Batch online event driven

Infrastructure takes a long time.

Value is driven by new capabilities used to do new things, less by doing old things better or cheaper

© Third Nature

Disruption Time

The Internet forced a new architectural evolution.

IT has had a hard time keeping up, and new entrants in many markets are taking advantage of the new architecture to change how IT work is done.

Any time you have a backlog of resisted innovations, the pressure will eventually force wholesale change.

© Third Nature

UNDERSTANDING INNOVATION AND COMMODITIZATION PROCESSES*

This is how things change.

An amalgam of Everett Rogers, Yochai Benkler, Geoffrey Moore, Clayton Christensen, Stephen Gould, Eric von Hippel and others.

aka I like S curves

*the very short version

© Third Nature

COMMODITIZATION

© Third Nature

Four phases of technology commoditization

Early adopters

show results

Market

growth

Innovation

Development

Maturation

Saturation

Time

Early mainstream

starts to pay attention

Mainstream buy-in

This axis could be considered market penetration, adoption, product maturity

Invention /

discovery

© Third Nature

Time

Characteristics of software as it evolves

Innovation

Unique

Custom built

High value

High cost

Differentiator

Not well understood

High rate of change

Few vendors

© Third Nature

Time

Oddity known

Focus on integration not

building, customizable

products

High value

Lowered costs or

cost/speed/fit tradeoffs

Point of competition

Better understood

Slowed rate of change

Growing then shrinking

vendor count


Maturation

© Third Nature

Time

Ubiquitous

Configurable product

High to low value*

Low cost

Barrier to entry

Purpose and limitations well

understood

Negligible rate of change

Few, large vendors


Saturation

© Third Nature

Time

Compete on

differentiating

value

Vendor strategies (in general) vary by phase

Maturation

Compete on

product &

features

Compete on

process

Saturation Innovation

Market

growth

© Third Nature

Should you be a first mover or fast follower?

Time

Little product

substitution is

possible here.

Few competitive

bids or RFPs.

Maturation

Uncertain

tradeoffs here.

Competitive

bids for unlike

products. Early

it’s less “what

feature” and

more “how to

accomplish my

task”, later it’s

the opposite.

Predictable

cost and

feature

comparison

until practices

change. That

change can

take a long

time to occur.


Market

growth

© Third Nature

Time

Few product

choices

The vendor landscape changes over time

Many, expanding

product choices

Many, contracting

product choices

Relatively few

product choices

Market

growth A particularly

dangerous time

to pick vendors

© Third Nature

We see this pattern in evolutionary processes

Sa

tura

tio

n, co

mp

etitio

n, e

nv.

co

nstr

ain

ts Evolution in most complex

systems goes through periods

of rapid change followed by

periods of general stability,

referred to as “punctuated

equilibrium”

Invertebrates

Vertebrates

Bacteria

Insects

Innovation – Adaptive radiation – Selection – Convergence

3.5b Bacteria (Cell)

2.5b Sponge (Body)

0.7b Clams (Nerves)

0.5b Trilobites (Brains)

0.1b Mammals

Timing

© Third Nature

The same model can be applied to technology

Sa

tura

tio

n, co

mp

etitio

n, e

nv.

co

nstr

ain

ts

Copyright Third Nature, Inc.

Innovation – Adaptive radiation – Selection – Convergence

© Third Nature

With a long view a pattern emerges

Evolution in most complex systems goes through periods of rapid change followed by periods of general stability, referred to as “punctuated equilibrium”

New technologies take the place of old, establishing new ecosystems which are in turn disrupted by newer technologies and ecosystems.

Chaotic Stable Chaotic Stable Chaotic

Time

© Third Nature

Activities, products and practices evolve over time

Source: Simon Wardley

© Third Nature

Technology doesn’t just fulfill a need. It generates new needs and new problems. Business practices and technology co-evolve.

© Third Nature

As practices evolve based on new capabilities…

A new level of complexity develops over top of the older, now better understood processes, leading to new needs.

© Third Nature

Evolution and the Salaman-Story Paradox


© Third Nature

Evolution and the Salaman-Story Paradox

”Survival requires efficient exploration of current competencies and ‘coherence, coordination and stability’; whereas innovation requires discovery and development of new competencies and this requires the loosening and replacement of these erstwhile virtues”


© Third Nature

As a technology moves from emerging to commodity the nature of acquiring, using and managing it should change

Generate

options

Innovation

Novel practice

Maximize value

Maturation

Standardize /

minimize choice

Acquisition

Best practice

Minimize costs


e.g. BI which went from many tools to a few vendors, now being

disrupted by new technologies and capabilities

Constrain

choices

Adaptation

Good practice

Optimize

© Third Nature

In terms of technology, we are in a tough position because the ecosystem is in a disjoint state

Maturation Saturation Innovation

Big data and

analytics is here

BI / DW is here

© Third Nature

ADOPTION: ENOUGH ABOUT THE ADOPTEES, WHAT ABOUT ADOPTERS?

The Enterprise IT Adoption Cycle

Wardley IT adoption reality

Adoption cycle graphic © 2012 Simon Wardley and CC BY-SA 3.0

****

https://twitter.com/swardley

https://twitter.com/swardley

http://creativecommons.org/licenses/by-sa/3.0/



© Third Nature

The Incredible Rate of Technology Change

Big data?

OMG!

© Third Nature

The Incredible Rate of Technology Change

We told you about

it in 2004…

© Third Nature

Time

Adoption

Rate

Some Innovation Adoption Theory

End of Life New innovation Time

Adoption

Rate

End of Life New innovation

© Third Nature

Adopter Categories

Innovators Late

Majority

Early

Majority

Early

Adopters

Laggards

© Third Nature

Ability to adopt is governed by people & organizations

Innovators Late

Majority

Early

Majority

Early

Adopters

Late adopters

People here tend to view a technology as a means to capitalize on future opportunities*

e.g. big new projects, process change

People here tend to view technology as a means to resolve present problems.

e.g. more focused projects, process improvement


*adopter status is based on the person/org and a given technology, it’s not a blanket statement

© Third Nature

Slowing it down: innovation is gated by ability to adopt

No technology stands

entirely alone – these

dependencies slow

adoption, stretching

the maturation phase.

This ecosystem effect

is what creates

technology regimes

that can last decades.


Younger companies

have a relative

advantage when it

comes to absorbing

new infrastructure.

© Third Nature

Time

Cumulative

Adoption

Market Adoption

Hard work

Tipping point

© Third Nature

Product

Maturity

Some Ideas Aren’t That Good

End of Life Time New innovation

Some ideas aren’t that

good, like object

databases in the 1990s

© Third Nature

These Curves Can Explain a Lot

Time

Product

Maturity

Analyst revenue

predictions

Executive interest “Gartner Gap”

© Third Nature

The “experts” often have a foreshortened view

“Open source is not worth paying attention to.”

A Gartner analyst talking about the database and analytics market, January, 2006.

Where the analysts are on the

adoption curve

© Third Nature

Crossing the Chasm (1991)

© Third Nature

Geoffrey Moore’s Ideas

Built on Rogers’ ideas, extended them to tech marketing and product management. The original focus was on the development of technology (gray).

Just say no

Stick with the proven

Stick with the herd

Stay ahead of

the herd

Just try it

© Third Nature

Core BI / DW technology is mainstream-stable

The data management market has many segments, some new, some mature, some being rejuvenated.

Platforms (this

should scare

everyone)

Databases* Reporting

&

ETL and DI

Analytics

© Third Nature

Product evolution in software markets

PC

1 2 3

4 5 6 Image: Geoffrey Moore, Dealing With Darwin”

© Third Nature

INNOVATION

© Third Nature

Innovation and Commoditization

This section isn’t really a summary

© Third Nature

Image: Harvard Business Review, “Skate to Where the Money Will Be”

Theory of Disruptive Innovation

i.e. you don’t pay attention and do

what you always did and the other guy

eats your market from below

© Third Nature

Disproving Christensen

a) 9% of the cases fit the model

b) Disruptive innovation <> success; banks disruptively innovated debt products and we know how that turned out

c) The model fails to predict failure too:

In 2007, Christensen told Business Week that “the prediction of the theory would be that Apple won’t succeed with the iPhone,” adding, “History speaks pretty loudly on that.” In its first five years, the iPhone generated a hundred and fifty billion dollars of revenue. In the preface to the 2011 edition of “The Innovator’s Dilemma,” Christensen reports that, since the book’s publication, in 1997, “the theory of disruption continues to yield predictions that are quite accurate.”

d) Oh

© Third Nature

Types of innovation

Incremental or “sustaining”

▪ Incremental is based on existing concepts, framing; smaller changes within the same general framework

Disruptive

▪ Based on new concepts, science, principles; requires new knowledge, skills; over time has significant consequences to market

Architectural – the third path

▪ Changes how the parts are related. It devalues advantage of experience, knowledge, usefulness of prior knowledge, but doesn’t affect the existing knowledge. (Christensen missed this one)

© Third Nature

Adoption and decline – everything gets old

For most businesses, nearly 80% of IT budget is dedicated to basic infrastructure

…and more than 60% of IT labor cost goes to keep things running, i.e. basic operations and support.

Strategic

Commodity

© Third Nature

It Wasn’t Always This Way

As technologies mature and spread to competitors, they cease to be differentiators. Unfortunately, this is what packaged software vendors do to your “best practice.”

Commodity Commodity

The old advantages becomes the new focus of cost reduction.

For example, your data warehouse.

Strategic Strategic

© Third Nature

Adoption and decline

Rarely does anyone talk about the core problem: preexisting conditions

You have something new. How does it affect the old?

▪ Replaces it?

▪ Adds something new?

▪ Overlaps it, forcing you to make hard decisions about what parts to keep, change, throw away?

The heart of this problem is the process of architecture: integrating changes to systems over time. The integration is not purely technical, it’s practices of use, operation, deployment.

© Third Nature

How IT strategies evolved with commoditization

Time

Equipment

Expensive: outsource to reduce equipment cost

Labor

Affordable: insource for control, innovation

Dirt cheap: outsource to reduce labor cost

76

© Third Nature

The cost flip in the business intelligence world

Cost factors traded positions 1990 - 2010

Equipment

Software

77

Cost

Labor

For small to mid-sized organizations it’s very affordable

© Third Nature

TCO and BI

What can you control?

▪ Labor effort is almost identical across BI products.

▪ Hardware use by BI tools is similar across products.

▪ You can negotiate the software costs.

3 Year BI TCO Cost Categories

Source: Third Nature Open Source cost study

© Third Nature

This is an old problem

BI tools are better, but the model being applied in most organizations is not different from the past.

November 2010 Mark Madsen

If BI is a commodity, why does it cost so much?

Processes Applications Data Integration Storage EDM / BRM Delivery Consumers

Purchasing

Distribution

Manufacturing

Sales &

Service

ERP Data warehouse

ODS

Stream db / cache

Content store

Identify

Analyze

Debt<10% of Income Debt=0%

Good

Credit

Risks

Bad

Credit

Risks

Good

Credit

Risks

Yes

YesYes

NO

NONO

Income>$40K

Predict

Batch ETL

EII

SCM

SFA

CRM ESB

EDR

Monitor

Explore

Data mart

Low-lat ETL

BP

M /

Work

flow

BRE

CEP

Prescribe

Data services

Transaction services

Manual feedback

Automated feedback

© Third Nature

Lessons to take from this

1. Business intelligence is still expensive for many organizations, with the largest proportion of cost being labor.

2. Business intelligences is not a technology problem, or the failure rate and costs wouldn’t be so high.

3. BI tools being a commodity does not make BI a commodity.

4. Architecture has an outsized impact on your ability to adopt and adapt.

5. What you remove is as important as what you add.

82

© Third Nature

Questions to ask

Why innovate?

▪ Usual answer: profit

▪ Proper answer: solve a problem

Innovation for what?

▪ A product or service you are selling to customers

▪ Internal products and services, how you run your business or department

© Third Nature

Reinforcing relationships resist change, despite radical technology and practice shifts

Note how only one third is tech

ArchitecturalRegime

Methodology Technology

Organization

Organization defines where the work is done and the roles.

Technology defines what work can be done in a given area. Methodology

defines how work is done and what that work is.

Slide 85

© Third Nature

Designing for data: monolithic vendor technology-based classifications of the ecosystem won’t help

These types of eye charts provide a categorization of what’s available, not what you need. They ignore the contexts of use that are most important.

86

© Third Nature

It’s tough making it decisions in a turbulent market


If you’re here you probably

don’t want to be making

long term technology or

vendor commitments.

© Third Nature

Today: repeating the experience of the 80s & 90s

This is the turbulent

phase of the market

as it goes through

rapid development,

then product and

service changes.


The Internet combined with commodity computing is forcing a new

architectural evolution, already well underway.


© Third Nature

Methods change too, one size doesn’t fit all


Agile &

exploratory

methods

6 Sigma &

efficiency

methods

© Third Nature

How procurement decisions are made

Deliberation ▪ Actions are consciously

chosen. Don’t attribute to malice what you can attribute to stupidity, and don’t attribute to stupidity what you can attribute to laziness.

Rationality ▪ People make logical

decisions. Sure they do.

Order ▪ System are understandable

and the results of actions predictable.

© Third Nature

“Choose Boring Technology”

You only get so many chances to make big changes at a company. Don’t waste them.

You can spend X time focusing on the goal and worry less about the known tech, or you can spend X time learning the new tech and less time focusing on the goal.

The important thing is not the choice of tech, it’s knowing when the time is right to make a new tech choice.

© Third Nature

In other words… Software is like puppies. Getting a puppy is easy, raising one is hard.

“The short term benefits of using a new [type of] database exceed the long term cost of operating it.”

Dan Mckinley

© Third Nature

Where does innovation come from?

“It has long been assumed that product innovations are typically developed by product manufacturers. …it now appears that this basic assumption is often wrong.”

Eric von Hippel

© Third Nature

Being innovative and culture

Myth of process – there is no “process for innovation”, only principles and exceptions

e.g. “It’s best to work in small teams, keep them crowded and foster serendipitious connections.” – Eric Schmidt

It depends on creative problem solving, and solving problems people care about.

Removing deviance removes change, so you have to be careful about best practices.

© Third Nature

No silver bullet

It’s culture-dependent, and creative and messy and idiosyncratic and slow and hard, no process, just survival bias and heuristics and principles.

© Third Nature

“The future, according to some scientists, will be exactly like the past, only far more expensive.” ~ John Sladek

© Third Nature

Further Reading

Further Reading:

Manager’s Theories About Innovation, Salaman & Storey, 2002 Democratizing Innovation, Eric von Hippel, http://web.mit.edu/evhippel/www/books/DI/DemocInn.pdf Sources of Innovation, Eric von Hippel, http://web.mit.edu/evhippel/www/sources.htm The Wealth of Networks, Yocahi Benkler An introduction to value chain mapping, http://blog.gardeviance.org/2015/02/an-introduction-to-wardley-value-chain.html

The diffusion of infrastructure dependent technologies: A simple model http://www.dime-eu.org/files/active/0/vanderVoorenAlkemade.pdf

Architectural innovation: the reconfiguration of existing product technologies and the failure of established firms http://dimetic.dime-eu.org/dimetic_files/HendersonClarkASQ1990.pdf

What the Gospel of Innovation Gets Wrong http://www.newyorker.com/magazine/2014/06/23/the-disruption-machine

How Useful Is the Theory of Disruptive Innovation? http://sloanreview.mit.edu/article/how-useful-is-the-theory-of-disruptive-innovation/

Slide 101

http://web.mit.edu/evhippel/www/books/DI/DemocInn.pdf

http://web.mit.edu/evhippel/www/books/DI/DemocInn.pdf

http://web.mit.edu/evhippel/www/sources.htm

http://web.mit.edu/evhippel/www/sources.htm

http://blog.gardeviance.org/2015/02/an-introduction-to-wardley-value-chain.html












http://www.dime-eu.org/files/active/0/vanderVoorenAlkemade.pdf





http://dimetic.dime-eu.org/dimetic_files/HendersonClarkASQ1990.pdf




http://www.newyorker.com/magazine/2014/06/23/the-disruption-machine





http://sloanreview.mit.edu/article/how-useful-is-the-theory-of-disruptive-innovation/

















© Third Nature

Image Attributions

Thanks to the people who supplied the images used in this presentation: indonesian angry mask phone - Erik De Castro Reuters.jpg

egg_face1.jpg - http://www.flickr.com/photos/sally_monster/3228248457

chicken_head2.jpg - http://www.flickr.com/photos/coycholla/4901760905

snail1.jpg - http://flickr.com/photos/7147684@N03/1037533775/ wheat_field.jpg - http://www.flickr.com/photos/ecstaticist/1120119742/

http://www.flickr.com/photos/ecstaticist/1120119742/

© Third Nature

About Third Nature

Third Nature is a consulting and advisory firm focused on new and

emerging technology and practices in information architecture, analytics,

business intelligence and data management. If your question is related to

data, analytics, information strategy and technology infrastructure then

you‘re at the right place.

Our goal is to help organizations solve problems using data. We offer

education, consulting and research services to support business and IT

organizations as well as technology vendors.

We specialize in information strategy and architecture, so we look at

emerging technologies and markets, evaluating how technologies are

applied to solve problems.

© Third Nature

About the Presenter

Mark Madsen is president of Third Nature, a technology research and consulting firm focused on business intelligence, analytics and performance management. Mark is an award-winning author, architect and former CTO whose work has been featured in numerous industry publications. During his career Mark received awards from the American Productivity & Quality Center, TDWI, Computerworld and the Smithsonian Institute. He is an international speaker, contributing editor at Intelligent Enterprise, and manages the open source channel at the Business Intelligence Network. For more information or to contact Mark, visit http://ThirdNature.net.

http://thirdnature.net/

Disruptive Innovation: how do you use these theories to manage your IT?

Technology

Transcript of Disruptive Innovation: how do you use these theories to manage your IT?