HOW LEAN HELPED US PUT QUALITY BACK AT

THE HEART OF OUR AGILE PROCESS?

Renaud Wilsius

bisam.com

2

I have been a Quant, then a developer for 10 year. Strong believer in Agile and now R&D director at BISAM

BISAM IS LEADER SOFTWARE EDITOR INTO PERFORMANCE, ATTRIBUTION & COMPOSITES ANALYTICS

WHO ARE WE?

3

BISAM has 10+ years of disciplined Agile practices- eXtreme Programming in 2005- Scrum in 2010- Lean in 2014

AGILE MANIFESTO REMINDER

4

We are uncovering better ways of developping software by

doing it and helping others do it.

Through this work we have come to value :

Individuals and Interactions over processes and tools

Working Software over comprehensive documentation

Customer Collaboration over contract negociation

Responding to change over following a plan

SCRUM REMINDER

5

BUT, WHAT ABOUT QUALITY?

6

WELL ….

7

Client :

• I will not encourage any client to take the 3.9release.

• They don’t believe us anymore.Installations• A lot of packaging problems on upgrades and

patches. We have not tested the rightcombinations.

Support• Our tickets went into the big black hole.

5.7/10

• Customers say we have a very goodproduct. Very comprehensive.

• Customers like the upgrade in 20 minutes.• They really like the new functionality.

With a dangerous trend on Issues

APPROACH

8

TWO OPTIONS

9

Empower Outsource

LEAN

10

Lean approach• Specialized consulting firm • Put our client in the centre• Focus on the product• Bring departments together

Define visual performance Indicators

Deliver one good version first

Obeya

WHAT HIDES BEHIND “QUALITY”?11

ACTIONS

12

OUR TYPICAL LEAN JOURNEY

13

We had a crisis.

We made the problem very specific.

We demanded immediate results (3 Months).

OUR TYPICAL LEAN JOURNEY14

Stream Mapping

OUR TYPICAL LEAN JOURNEY15

Stand Up Meetingsand Visual

Management

OUR TYPICAL LEAN JOURNEY

16

Quality Wall

OUR TYPICAL LEAN JOURNEY

17

Red Bins

OUR TYPICAL LEAN JOURNEY

18

PDCA

OUR TYPICAL LEAN JOURNEY19

PDCAAgain

OUR TYPICAL LEAN JOURNEY20

Delivery Platform

ZERO INSTALL AND UPGRADE ISSUES

21

Test as a client• Take client’s database• Take client’s O/S• Follow Migration Guide steps

to the letter as a client

What• 4 different clients upgraded• 3 initial versions: 3.8.0-n,

3.8.5-f, 3.6.0-aa• 2 Operating systems (2 Linux,

2 Windows)

Definition of a successful upgrade• Key Processes run smoothly• Acceptable upgrade time• No issue browsing the application

Found• No issue• Upgrade from3.8.0-l to 3.9 on

client production data (2 billion EQA results) took 15 min

APPROACH

OUTCOME

ZERO REGRESSION ISSUES 22

Test as a client• User Interface Tests• Non Regression Tests• Challenge and review our

own test packChallenge and review our own test pack• 15,316 unit tests & 1,874

functional tests• Adapted functional tests to

run not only on HSQLDB but also Oracle & SqlServer

• => Found one major issue

It is all about Data• On client data• On client site pre-test

User Interface Tests• Opening and testing 1,050+ screens on 4

different client databases=>Found and fixed 79 issues/ improvements with one critical KPI issues

Non Regression Tests• Run 31 client specific test procedures on

client upgraded environment• Take client major blocking and critical

issues that might have been impacted by 3.9.0 and retest them

• =>Found and fixed 2 issues

APPROACH

OUTCOME

ZERO PERFORMANCE ISSUES

23

Performance Lab• Isolate Lab from any other activities• Invest 15,000 euros on SAN bay to

harbor Client Databases

Performance Campaign• Run performance campaign

throughout the version lifecycle• Check performance non-regression

again at the end of the version.

Web Services load testing• Run Total Return Simulation

web service via SoapUI• Iterations: 1,000 // 8

concurrent calls (threads)

Excel Formula load testing• Open 6 large excel reports.

One of them containing12,000+ formulas

APPROACH

OUTCOME Action Result

Performance Campaign Found and fixed 1 issue (BON-4003)

Web Service load testing On average 15% improvement

Excel load testing No Issue found

3 MONTHS LATER

24

Efficient Quality Wall

3 MONTHS LATER

25

DailyObeya

3 MONTHS LATER26

Team Performance

3 MONTHS LATER27

Success!

LESSON LEARNED

28

Keep doing

• Systematic packaging testing on Client configuration (Major & Minor Release)

• Extend our unit and functional tests coverage

• Run Performance campaign during development cycle

• Enhance Product, Migration & Performance Documentation

Start doing• Have Client satisfaction

measures/feedback before, during and after upgrade

• Develop and formalize Beta Partnership Program (BPP)

• Acquiring Client database and test plans• Monitor Support tickets stock, Number

of Regression, Number of successful/failed upgrade

• Run Client database migration during development cycle

• Test feature from GUI on Client upgraded version during development cycle

• Foster cross-department communication and collaboration

3 YEARS LATER…

29

BETA PARTNERSHIP

30

Version after version, the number of Beta Partner per version keeps growing.

3 weeks to releasing any generalavailability version, we started enrollingactive Beta Partners who:

- Are heavily testing new features

- Ensuring non regression in a real client environment

- Have direct access to R&D SME 0 0

1 1 1

2

3

0

1

2

3

4

3.7Nov 13

3.8Apr 14

3.9Dec 14

3.10May 15

3.11Nov 15

3.12May 16

16.2Nov 16

# Beta Partner

RELENTLESSLY INCREASE OUR AUTOMATED TESTS

31

Over the last three years , we have exceeded 18,600 Unit tests in B-One v 16.2 (+21%), while increasingthe number of End to End Tests by 74%.

0

500

1 000

1 500

2 000

2 500

3 000

3.7Nov 13

3.8Apr 14

3.9Dec 14

3.10May 15

3.11Nov 15

3.12May 16

16.2Nov 16

# of Functional Tests per version

1374

2389

WHILE REDUCING OUR NUMBER OF OPEN ISSUES

32

Every year the R&D has KPI objectives to reduce the number of open issues (Client / BISAM identified)

Last two years, the numberof open issues in B-Onedropped by 77%.

0

20

40

60

80

100

120

140

160

3.7Nov 13

3.8Apr 14

3.9Dec 14

3.10May 15

3.11Nov 15

3.12May 16

16.2Nov 16

# Open Bugs

# Open BugsReported by Clients

# Open BugsReported by BISAM

RESULTING IN FASTER PROMOTION OF RELEASES

33

Client are going live with early B-Oneversion (-a, -c) .

We kept track of the first minor release which went to production on a given release, and measure how many days passed between the official release and this production ready minor release. It takes today 3 times less than 3 years ago.

106

74

55

1

34 32

0

20

40

60

80

100

120

3.7Nov 13

3.8Apr 14

3.9Dec 14

3.10May 15

3.11Nov 15

3.12May 16

# Open days to receive go live version

CONCLUSION

34

CONCLUSION

35

Trust your people.

Empower your people.

Teach lean thinking and skills to everyone: “Fall seven times. Stand up eight.”

Support Lean practices at each level of the hierarchy.

Develop sustainable and sticky pattern for long term customer benefit.

bisam.com

THANK YOU

Follow us on