1

Scaled Agile Testing in a DevOps Environment By: Reindrich Geerman

October 02nd 2017

Reindrich Geerman

2

Digital QA Architect

3

KPN & Digital

Team Evolution

Test & Tech. Evolution

We Own

4

5

6

7

8

Customer Master KPN-ID BI & Analytics

9

KPN & Digital

Team Evolution

Test & Tech. Evolution

We Own

The Waterfall (a.k.a The Past)

10

Requirement

Design

Development

Deployment

Testing

Time in months

Finding bugs

11

X

Manual Tests Through UI

Integration Test

Unit Tests

Agile Scrum

12

Develop

Testing

Evaluation/Prioritisation

Requirements

Design & Analysis

Potentially Shippable Product

Time in days

Agile Scrum

13

Sprint 1 Sprint 2 Sprint 3 Sprint 4

Continue

The Agile Scrum Waterfall

14

Requirement

Design

Development

Requirement

Design

Development

Testing

Deployment

Sprint 1 Sprint 2

Continue

Team Evolution

15

Fase 0

16

Outsourcing Partners Big Contracts

4 Deployments per Year Big Test Teams (30+)

Fase 1

17

Started with Scrum Teams Started with Test Automation

1 Deployment every other Sprint (12 per Year) 4 Developers

2 Testers

Fase 2

18

“Big” Scrum Team Continuous Testing

Started with Automation

1 Deployment per Sprint (2 weeks) 6 Developers

3 Testers

Fase 3

19

Small Scrum Team Complete CI/CD

Started with DevOps

More than 1 Deployment per Day (10 or more per sprint) 5 Dev/Test

Fase 4

20

Small Scrum Team Platform Team

Started with Eco Team

Deployment on each Code Commit 5 Dev/Test

Support Team

21

KPN & Digital

Team Evolution

Test & Tech. Evolution

We Own

KPN Digital Stack

22

Backends

Databases

Python/Django

OR

Fail Often23

Fail Early

Fail Fast

Dev/Test Strategy

Initial Test Framework

24

DEV

TEST

ACC

PROD

SOURCE

CONTROL

JENKINS

BUILD

SYSTEM

RUN TEST

OR

25

26

E2E Integration Testing

27

“The purpose of carrying out end-to-end tests is to identify system dependencies and to ensure that the right information is passed between various system

components and systems.”

Back-end’s Databases

28

Slow Tests

29

Cross Team Collaboration

30

Agile Testing

31

Understanding Value, Priority & Risk

Quality is a Team Responsibility

Zero-Touch Testing

Continuous Quality Monitoring

Prevent Bugs

32

X

Manual Tests Through UI

Integration Test

Unit Tests

X

Automated UI Tests

Automated Integration Test

Unit Tests

Exploratory Testing

Find bugs Prevent bugs

33

We Are All Testers

34

DevTest DevTest DevTest

You Build it, You Automate it, We Break it

Dynamic Settings

35

API Client

(Browser)

SETTINGS MAPPING

Backend backend1.tst1.kpn.com

Internet & Bellen …

Mobile …

SSO_enabled …

ETCD

Backend 1 HTTP Request backend.tst1.kpn.com

Backend 2

backend.tst2.kpn.com

HTTP Header Backend = Backend 2 (backend.tst2.kpn.com)

Detox

36

Tester Role?

37

Shift Left

Test Coach/Consultant • Code Review (unit test, code) • Create User Story (BDD) /review • Acceptatie criteria • (Test) Automation • Writes Test cases (TDD) • …

…

Shift Right

Technical/Operational Dev. • (Test) Automation • CI/CD Engineer • Release management • OPS • Aftercare • Testing the NFR’s • …

BDD User story

38

As a user I want to be able to login to KPN Selfcare environment, so that I can manage my Personal Detail information

GIVEN The user is on the login pageAND Enters the correct KPN-ID Credentials

WHEN The user clicks on the Login ButtonAND The user clicks on the Change Details Button

THEN The user is able to change his personal detailed information

Robot Framework

39

Robot Framework Architecture

40

Robot Framework

Test Scripts

Flow Object

Page Object

Libraries

Automated Visual Testing

41

Detailed Test Report

42

Detailed Test Report

43

Zero Touch Deployment

44

DEV

TEST

ACC

PROD

SOURCE CONTROL

< / >

•  Unit, •  Functional, •  Multibrowser, •  Performance, •  Security Testing

45

KPN & Digital

Team Evolution

Test & Tech. Evolution

We Own

Continuous Customer Journey Monitoring

46

PROD

RUN TEST

Rerun Failed Test

1x

Wait x min

Customer Experience Monitoring

47

Social Studio Usabilla

Technical Monitoring

48

Technical Monitoring

49

DevOps Tools

50

51

Reindrich Geermanm: +31657497774 e: reindrich.geerman@kpn.com in: reindrichgeerman t: @reinst42

52

Question? August 31st 2017