Value Driven Development by Dave Thomas

3/11/17

1

© 2014 Bedarra Research Labs. All rights reserved.

Value Driven Development

Maximum Value, Maximum Speed –Targeted Innovation Trumps Systemic

Transformation

Dave Thomaswww.davethomas.net

Chief Scientist, Kx Systems. Chairman YOW! Conferences,

Carleton University, School of Computer Science, Canada

Outline

• Scaling Agile – Can we? Should We?

• Process Driven Development

• Value Driven Development

• Techniques to reduce time, increase value

• Innovations Opportunities

• Legacy and New Product Case Studies

3/11/17

2

Scaling Agile – Can it be Done?

Yes! If and only if:

1. Has strong consistent and respected leadership.

2. The business adopts a product culture versus a project culture.

3. Implements Continuous Integration and Delivery.

4. Implements a comprehensive measurementsystem to enable continuous improvement.

5. Establishes a Technical Ladder and Communities of Practice


Leadership Matters!The Foundation for Trust and Respect

A leader has integrity; stands by values, Leads by example, Executes with transparency, Has the wisdom of experience , Articulates the vision, Says when they are wrong, Judges fairly, Promotes trust, Delegates authority, Coaches versus directs, Promotes constructive diversity, Makes timely consistent decisions, Sense of humour, Positively motivates and Educates, Always tries to find the best wrong answer, Has time to listen…

©2011 Bedarra Research Labs. All rights reserved.

“Agile Leadership is Fragile” Nigel Dalton, GM Lonely Planet, now CIO Realestate.com

3/11/17

3

Scaling - Your Challenges => Potential Solutions 3:45 PM Track 3

Lean and Agile in the Large - An informal Q&A for attendees to bring forward their challenges and learn how these challenges have been dealt with in our experience with Lean and Agile in the Large Transitions.

Experiences from Eclipse, Instrumentation, Online Travel, Mobile Telecom, Medical, Business Intelligence, Insurance and Banking, Enterprise Software, Defense and Embedded Systems


Scaling Agile - Should it even be the goal?

Agile is not about increased - revenue; customer interactions; productivity; cost savings; new practices which is where most companies have acute demands.

Systemic change requires substantial sustained investment in changing the way people do things. It seldom has a major impact on value or cost which is the place companies most need it.

Scaling Process often dilutes training to following prescribed steps rather than encouraging thinking based on measured continuous improvement.


3/11/17

4

Process Driven Development Lean and Agile in the Large works but

• 10-15% improvement in productivity, 25 %+ quality, little change in productivity

• Seldom has the Impact or Agility expected by executives!

• Encourages micro improvements, rather than high impact changes! Sprinting to Technical Debt..

• Biased to tools and technologies of the past!

• Systemic change expensive to implement and sustain!

You can’t lead using yesterdays process from someone else! Process lags best practice by 3 – 5 years!

A generic solution is very seldom as effective as a strong specific solution

We have great confidence in your abilities to meet our competitive challenges. Hence we have given you Objects, Agile, Tools, Hardware, New Offices, Mentors because we believe you need these to do your best. Here is what the business needs from you:

1. We need to be much faster in selecting and executing high value projects.

2. We need to turn up the dials on productivity while maintaining quality and features.

CEO – All Hands Challenge

As long as we keep building software the same way as everyone else were have no competitive advantage.

…. THIS IS YOUR MISSION…* THIS MESSAGE WILL SELF DESTRUCT IN30 SECONDS….

* Mission Impossible

3/11/17

5

Better, Faster, Cheaper – A New Road?We must adapt and improve our way of building software to

meet the challenge.

1.Focus on Value – Target resources and innovations to where they will make a difference.

2.Advance our practices – use alternative techniques to better communicate, design, estimate, build, test and deploy.

3.Refactor our organization to streamline and enable more concurrency and reduce cycle time without reducing quality. We need leverage what works and not be constrained by current best practices. If it is slow it has to go!

4.Explore and Experiment – we need to envision alternatives and evaluate them quickly before betting too much on any approach. We need to fail fast to maximize ROI and time.

© © Dave Thomas Bedarra Corp 2016

Value Driven Development

1. Identify a high value bottleneck which if accelerated will make a large impact on the business.

2. Quickly envision a solution and experiment• Think Out of the Box (Innovate)• Use the appropriate HW• Use appropriate SW and practices

3. Validate at scale – this is the major reason clever solutions fail to go into production.

4. Complete the work in 3 – 4 months


3/11/17

6

Management Buy In – ROI and Risk MitigationBusiness

• Strong Senior Business Sponsor

• Clear and tangible measureable goal

• ROI model shows significant business value ( >15%)

• Implementation Timeline of 3 – 4 months

• Minimal Disruptive Impact on Business Operations

Technical• Single small team tech plus business with track record• Localized code changes• Proof of Concept validation in weeks • Proof of Scale validation in weeks• Easy to monitor SLA Life cycle costs...• Straight forward deployment• Minimum dependency• Independent Acceptance Testing

Some Techniques For Fast Delivery with Cases

1. Envisioning

2. Leverage Innovation

3. Rigorous Testing

4. Simplicity

5. Fast Hardware

6. Ship Less Code – Klocs Kill

7. Table and Data Driven

8. Reduce APIs – Improve Integration

9. Loose Coupling

10. Simple Data Architecture

11. Collection Oriented Programming (just enough FP)

© © Dave Thomas Bedarra Corp 2016

3/11/17

7

Envisioning => Tangible Requirements

• easily expressed by Business

• easily understood by IT

• acceptance easily understood by Business

• acceptance easily understood by IT

• acceptance criteria => acceptance test (AT)


StoryCompetitive Delta

AnalysisCustomer Field

Studies & Interviews

TechnologyEvaluations

Market & ProductAnalysis Brainstorming

& Visioning

QFDHouse of Quality

PrototypingAcceptanceCriteria

StoryAcceptance

Criteria

Envisioning Practices

Envisioning – Requirements Through DesignThe Waterfall Pitfall

Plan ‘everything’ before you do ‘anything’ until…

It’s 2 years late, 173% over budget, kind of

buggy, & costs $32,345.99…

But “it does everything you’d ever want to do!”

The ‘Potential’ Agile Pitfall“Planning, shamming, I need to add 1+1, so let’s just

build it!” On time, on budget, and we’re giving it away as a beta. “It’s bug-free, it works, and it adds 1+1!” Darn… now how do we add all the features we never had time to think about?

Envisioning gives Agile some breathing room…Allows us to understand enough of the vision of ‘tomorrow’…

3/11/17

8

Low-Fidelity Envisioning Prototype ExampleLow-fidelity prototype

§ Initially rough and then later refined drawings§ Interactive branching allowed walkthrough§ User model, task model, task flows§ 3 structure and navigation alternatives§ 2 visual form alternatives

Concept iterations§ 6 iterations (expanding from 8 to 48 screens)§ 3 sprints§ 3 internal / 2 external customer sessions

Detail iterations§ 3 iterations (148 screens)§ 8 sprints§ 3 internal / 1 external customer sessions

Investment§ Less than 2% of overall effort

Leverage Innovations

Improved Business Practices• Simplification, Partnering, Regulatory …

Improved Hardware • Performance, Capacity, Latency, Functionality – atomic

clock, mobility, voice, cloud …

Improved Software• Usability, Productivity, Flexibility – Database; Fault

Tolerance – isolation, Languages, Robust Libraries

Improved Practices• Property Based Testing, Micro Services, Continuous

Deployment, Functional Programming, Algorithms, Self Service …


3/11/17

9

Testing is still 30 – 50% of Development

Test Soon – TDD, Inspections, Acceptance Tests• It all starts with acceptance criteria (including ilities – there are

no nonfunctional requirements!)

• SBE/Cucumber Good | Bad News?

Side by Side Comparison of Two Full or Partial Implementations

Dog Fooding for UI

Property Based Testing


Testing is still 30 – 50% of Development

Property Based Testing • QuickCheck is currently the best exemplar

• Generates random tests from a spec• Collapses False Positives to minimum fail test sequence• RIAK, Level DB experience

Dog Fooding Still Best for UI Testing• Every time a significant amount of code has changes, Freeze the

code for 2 – 3 days• Have developers test and log bugs on the code of others.• Fix the bugs for, usually 2 – 3 days.


3/11/17

10

Simplicity! - The Road Not Taken?

© 2012 Bedarra Research Labs

ComplexitySimplicity

Fast Software

FAST Software = FAST Hardware + Simplicity

Simplicity – Software Sympathy*

• Simplicity in Code

• Simplicity in Architecture

• Functional Thinking

* Mechanical Sympathy – Martin Thompson


3/11/17

11

Let the Hardware Do The Work!$15,000 buys a computer 1 TB RAM with 100 TB disk and 64 cores!

• Automated Build and Test

• All interesting data is in memory! Data Base is an oxymoron . Soon 10TB or persistent memory in a PC!

• Flexibility of Data versus Code • Inexpensive Data Conversion/Translation• Data Compression and Encryption is “free” on multi-

core

• Speed and Memory enable Simpler Algorithms and Architecture

• Enable End User Computing e.g. Excel and R versus C#, Java


Ship Less Code!


Klocs Kill!• The more code you ship the more code you need test and fix!

Abstractions Bloat!• Frameworks inject dependencies into your code!• Object Bloat Costs

• Development Time• Execution Time and Space

Dependencies Strangle!!• Increase Integration Testing• Increase Build and Deploy Time• Frameworks pull in much than required

Object Refactoring harder than Changing Data and Functions!

Use Less Objects and Less Code !

3/11/17

12

Table Driven Programming A picture is 1000 words, a table 400 and a diagram 50

Advantages

• Easily understood by Business, BA, Dev and QA

• Easy to create, refactor and extend using Excel

• Modularity through structured tables

• Consistency /Completeness Checking

• Easy to version and Diff

• Efficient Automated Data Driven implementation

• Data Driven means changes can be “hot deployed” to a running application

ApplicationsInsurance, Banking, Taxation, Healthcare, ATC, Real-time...



Rules Decision TableCalculation SpreadsheetData Validation Domain and Range TableMapping Lookup TableFlow Data, Work Flow, MessageEvents, Matches State TableProcess, Reports Input-Output TableAcceptance Criteria Domain Models Entity-Relationship Attribute Data Dictionary

Table Driven Programming Examples

3/11/17

13

Reduce Integration Time and $$$

Avoid Slow and Complex APIs• Leverage stable physical data formats – pages; logs

• Make R/O SQL replicas for fast reporting

• Put ATOM/RSS feeds on our legacy/partner systems – journal files, events …

• REST and JSONify your services

• Use ODBC as a simple interface to complex server systems

• Use a simple Mashup tool against to deliver a integrated application view


Enabling Loose Coupling

• All APIs are value based and where possible stateless

• Isolation of services in separate processes/machines

• Simple Pipes and Filters when possible (data flow)

• Occasionally Disconnected – replication and sync; event source..

• Simple efficient implementations using co-routines..

• Orchestration/Composition using Scripting• Service Bus, Messaging.. ZeroMq, Rabbit• Node.js e.g. Azure Mobile Services• Erlang, Actors, Scala Akka


3/11/17

14

Batching, RDB + HDB, Event Source, Lambda Architecture, Gartner HTAP, Forrester

Translytical DB


• Using batching to improve through put

• Real-time DB + Historical DB (kdb+)

• Real-time (Twitter Storm) + Batch DB• Immutable Data

For Space and Speed AVOID Binary Formats

1. Textual data formats such as XML, JSON, CSV are very expensive to parse

2. Textual data formats are bulky (10X) to store

3. Make sure your format has a version id for every format (aka schema change) else working with schema less data is problematic.

4. NOSQL – Easy to Write, Expensive to Store and Process!!!


3/11/17

15

Collection Oriented Programming

• Programs written with a functional idioms of map, filter, reduce… are :• More concise and easier to read.• Reduce side effects (leverage immutability)• More amendable to optimization• Data + Functions

• Examples• Underscore.js; CoffeeSript, Elm.., ES6• LINQ, Rx, Reactive Streams, Python ..• Clojure, Scala, Haskell, Erlang, Ocaml, q, Java 8+, C# 4.0 ,

F#…


Innovation Opportunities/Insertion Points


Data and Flow Interfaces

• Database Interface• File Interface• Sterilization Interface• Messaging Interface• Functional Transformers - ETL; Map

Reduce; GPU …• Query Interface• Disk/San Interface• Shared Memory Interface• Log Interface – Event Sourcing; Historical

DB (Batch) + Real-time DB (Stream)• Sync Replicate Interface - CDN leverage;

Mobile Occasionally Disconnected; BASE Fault Tolerant

• Reactive MVC Interface

• Small computational bottle necks

• Highly structured rules/calculations

• Points of high variability/constant change

• Points with large numbers of defects

Selective Code Focus

3/11/17

16

Insurance Legacy + SOA + EAP

ChallengeLegacy L1, L2, L3,L4; Vendor EAP V1, V2,V3,V4,V5;Integration Services I5, I6; Commercial Insurance V1, L1Personal Insurance L2; Vehicle Insurance V2

Rating Engine L2, L3, V3; Billing System V4Policy System of Record L4 + I5 + V5Enterprise Software Bus I6

Solution1. Outstanding BAs define all products in tables2. Agile experts generate applications from BA tables3. SI experts build simple interfaces to ESB + Interface

Acceptance Tests 4. All vendors required to deliver acceptance test

HR System Bottleneck

Challenge• calculations very complex.

• analysts capture in Excel.

• 100 Devs in COBOL too slow

• Agile OO gave only 15% in productivity

Solution• Retain 2 Agile OO experts

• Excel rule checker in java

• Spreadsheet in Java on mainframe

• Legacy team deployed on other systems

3/11/17

17

Product at Risk – Massive Data Migration

Challenge Legacy Database Migration required due to DB Vendor Risk

Product Vendor dependent on DB vendor Customers want improved query and reportingCustomers have massive data locked in DB vendor at risk

Solution1. Product Team implements ODBC interface to legacy data2. Product Team migrates product to ODBC and provides

improved query and reporting . . .BUT Customers and Product locked to legacy physical data!and Product needs to change schema!

3. Expert SI retained by Product to perform high performance bulk conversion to modern database. (Agile DB Refactoring)

4. Independently developed data comparison program used for acceptance testing

© 2010 - 2011 Bedarra Research Labs. All rights reserved.

Modernize Legacy Factory Automation

Challenge

$$$ investment in proprietary automation system

Must to integrate new systems and new GUI-based control rooms

Solution

1. Modify interface to control units using TCP/IP

2. Implement HTTP and ATOM

3. Client devices are Web 2.0 appliances or PCs

3/11/17

18

Cyber Analytics – Analyst for Kx

1.Orderofmagnitudeimprovementinproductivity§ Interactive,Incremental,Exploratorywithrapiddeploytoproduction

§ Scripting/DomainOrientedProgrammingbySpecialistAnalyst

§ Analyst/EndUserVisualQuery,TransformationandVisualizationatscale

§ Providefullprovenanceforallartefactsandanalysis§ GranularCrossJurisdictionalCollaboration2.FastData

§ Efficientcomputationoverlargevolumesofdata(atleast10s– 100sTB)

§ MinimalImpedancebetweenpersistent(historical)andlivedata(real-time)

§ EnableAnalyticsfromSensorstoCloud§ Interoperablewithexistingdataformatsandtools

Analyst and Expert Programming Models

• 100:1ratioanalyststoexperts• ExpertssurfacenewfunctionalitytoanalystsasDSLlibraryextensions

Expert Programming Model

•Wide Spectrum Functional Vector Language• Fully Interoperable with current technologies – C to OO to Web

Analysts Application Programming Model

• Wide Spectrum DSLs ( SQL, Spreadsheets...)

• Narrow Domain Specific DSL (IP packets, finance, geo, cultural ...)• Leverage existing standards and user models

• Interoperable with R, MatLab, MS Office...

Core VEE Platform – High Performance Vector Functional Runtime

• Data Store Features

• Interoperability Features• Core Libraries

New Functionality

3/11/17

19


Deliver Value

Identify the High Business ValueFind the simplest innovative solution to delivery quickly

Shukriya!`

Value Driven Development by Dave Thomas

Engineering

Transcript of Value Driven Development by Dave Thomas