Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS....

44
Test Gap Analyse bei der Munich RE 27.10.2016 Uwe Proft

Transcript of Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS....

Page 1: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

27.10.2016Uwe Proft

Page 2: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

3. Evolution

4. Anliegen

2TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 3: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TQETechnical Quality Engineer

Static analysis of source code

Peer Review Lightweight code review process

TGATest Gap Analysis

Dynamic code analysis during test

TSATest Smell Analysis

Natural-language analysis of test descriptions

TASATest Automation Smell Analysis

Static analysis of script code

Page 4: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGATester

VCS

Test Environment

Profiler

TASATFSRanorex

Tester

TQEVCS

Build Developer

ReviewTFS

Developer

TSAQC/TFS

Tester

Page 5: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Munich Re – IT organization and systems

IT HubAmericas

IT HubEMEA

IT HubAPAC

Global IT Organization

Applications: 987 (Global / Local) Decreasing Projects: 369

Americas18 Locations

~6,500 Application Users

EMEA13 Locations

~7,000 Application Users

APAC14 Locations

~500 Application Users

Remarks: Figures are based on 2012 TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 5

Page 6: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

IT Management

Give Strong Support

Team Coach

BusinessRepresentatives

ServiceManager

Project Management

…ServiceManager

Requirement Engineering

…ServiceManager

Technical Architecture

…ServiceManager

Offshore

Development

…ServiceManager

Offshore

Test

Collaborative Project Team:Taking responsibility = our product, our release, our success,

my commitment, contribution and expertise

IT Services(our Line Organization and vendors)

Agile software development paired with outsourcing

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 6

Page 7: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

• Unterstützung klassischer Testphasen (Regression- / Release-Test)

• Analyse von Regressionstest Suiten

• Hotfixes besser absichern

• Prozess (Kommunikations) Defizite (in Pilotphasen) aufdecken

• Slice Test Gap des Tests in Iterationen

3. Evolution

4. Anliegen

7TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 8: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Unterstützung klassischer Testphasen (Regression- / Release-Test)

8TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 9: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Unterstützung klassischer Testphasen (Regression- / Release-Test)

9TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 10: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Unterstützung klassischer Testphasen (Regression- / Release-Test)

10TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 11: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Unterstützung klassischer Testphasen (Regression- / Release-Test)

11TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 12: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Unterstützung klassischer Testphasen (Regression- / Release-Test)

12TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 13: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

• Unterstützung klassischer Testphasen (Regression- / Release-Test)

• Analyse von Regressionstest Suiten

• Hotfixes besser absichern

• Prozess (Kommunikations) Defizite (in Pilotphasen) aufdecken

• Slice Test Gap des Tests in Iterationen

3. Evolution

4. Anliegen

13TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 14: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Analyse von Regressionstest Suiten

14TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 15: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Analyse von Regressionstest Suiten

15TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 16: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Analyse von Regressionstest Suiten

16TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 17: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

• Unterstützung klassischer Testphasen (Regression- / Release-Test)

• Analyse von Regressionstest Suiten

• Hotfixes besser absichern

• Prozess (Kommunikations) Defizite (in Pilotphasen) aufdecken

• Slice Test Gap des Tests in Iterationen

3. Evolution

4. Anliegen

17TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 18: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Hotfixes besser absichern

18TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 19: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Hotfixes besser absichern

19TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 20: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Hotfixes besser absichern

20TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 21: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

• Unterstützung klassischer Testphasen (Regression- / Release-Test)

• Analyse von Regressionstest Suiten

• Hotfixes besser absichern

• Prozess (Kommunikations) Defizite (in Pilotphasen) aufdecken

• Slice Test Gap des Tests in Iterationen

3. Evolution

4. Anliegen

21TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 22: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

3. Evolution

4. Anliegen

22TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 23: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA ist funktional gereift

ABAP

SAP Data Warehouse Systeme

Summarische All Dashboards

Unit Test Ergebnisse

Trends

Genauigkeit (z.B. ABAP Export)

ConQAT Teamscale

Individuelle Dashboards (Filtern: Zeit, Scope)

23TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 24: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Funktionsumfang ist nicht alles

Assistenztool

Freiwillige Nutzung verpflichtend

24TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 25: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Transparenz schaffen

25TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 26: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Transparenz schaffen

26TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 27: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Transparenz schaffen

27TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 28: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Alle Stakeholder einbinden

As a Test Analyst• I want to know if my testing missed some code which my developer colleague created for the slice we are working on.

As a Test Lead• I want to know if testing progress in the current iteration keeps up with developed changes and where our gaps are.

• I want to know the progress of my release test and make conscious decisions for remaining gaps.

• I want to make sure that we don‘t release any untested code in our hotfixes.

• I want to make sure my regression test set’s (functional) coverage is in sync with my code base.

• I want to make sure that different test levels complement each other.

As a Test Service Manager• I quickly get an overview over the test scope and progress of a project.

As a Developer or a Technical Architect I can help my test colleagues to understand the code and its criticality so that they can create missing test cases.

As a Requirements Engineer who also tests, I can quickly see the impact of my tests on the code level.

As a Project Manager who needs to understand testing in the project, I can quickly get an overview and use the tool in discussions with my testers.

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 28

Page 29: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Report

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 29

Page 30: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Report

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 30

Page 31: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Process – Tester & Developer

System

• Developer changes system code.Tester executes test cases for system.

• Dashboard shows untested changes.

• Tester regularly checks dashboard and closes test gaps with help of developer.

Dashboard

31

Tester & Developer

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 32: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Process – Quality Engineer

System

• Developer changes system code.Tester executes test cases for system.

• Dashboard shows untested changes.

• Tester regularly checks dashboard and closes test gaps with help of developer.

• Quality Engineer assesses remaining Test Gaps in collaboration with team.

• Quality Engineer proposes specific tasks to improve test process.

• Findings and tasks are captured in Test Quality Report. Reports are prepared regularly, synchronized with test phases.

Dashboard

32

Tester & Developer

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Quality EngineerTask List

Page 33: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Process – Project Manager

System

• Developer changes system code.Tester executes test cases for system.

• Dashboard shows untested changes.

• Tester regularly checks dashboard and closes test gaps with help of developer.

• Quality Engineer assesses remaining Test Gaps in collaboration with team.

• Quality Engineer proposes specific tasks to improve test process.

• Findings and tasks are captured in Test Quality Report. Reports are prepared regularly, synchronized with test phases.

• Project Manager receives quality reports and schedules tasks.

• Project Manager follows up on findings, tasks and Test Gap trend.

Dashboard

33

Tester & Developer

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Quality EngineerTask List

Test QualityReport

Project Manager

Page 34: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

TGA Quality Goals

QG 4: No test gaps

QG 3: No test gaps in changes that affect requirements with high or medium priority

QG 2: No test gaps in changes that affect requirements with high priority

QG 1: Don’t care

43

2

1

TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016 34

Page 35: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Monthly (light weight) Assessments

35TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 36: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Funktionsumfang ist nicht alles

Assistenztool

Freiwillige Nutzung verpflichtend

Transparenz schaffen

Alle Stakeholder einbinden

Report & Prozess

Quality Goals

Monthly assessments

Interner Kümmerer

36TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 37: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Test Gap Analyse bei der Munich RE

1. Kontext

2. Nutzungsszenarien

3. Evolution

4. Anliegen

37TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 38: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Mit Microsoft kann man über fehlende Features reden

38TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 39: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

... und hat Transparenz über den Backlog.

39TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 40: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Anliegen an CQSE

40TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 41: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Anliegen an CQSE

41TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Transparenz über den Produkt-Backlog für TGA (alle Quality Tools) (für die Kunden)

Page 42: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

User Forum?

42TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 43: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

User Forum?

Ich bin gern bereit, unsere Erfahrungen zu teilen UNDich würde gern aus Ihren Erfahrungen lernen:[email protected]

43TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016

Page 44: Test Gap Analyse bei der Munich RE - CQSE · Ranorex. Tester: TQE. VCS. Build. Developer. TFS. Review: Developer. QC/TFS. TSA. Tester. Munich Re – IT organization and systems. IT

Zusammenfassung

TGA, dynamische Codeanalyse, Teil einer Toolbox

Transparenz effektives & effizientes Testen

Mächtiges & komplexes Werkzeug

Veränderung & Akzeptanz

Gemeinsam Test Gap Analyse (die Quality Tools) voran bringen:Offener Backlog & User Forum

44TGA @ Munich RE, CQSE Wks., Uwe Proft, 27.10.2016