08.10.08 1

Agile TestingAgile Testing

Tiina KiuruReaktor Innovations

08.10.08 2

Agenda● What is agile?

– Agile values, principles and practices● How to organize testing?● How to implement testing?

08.10.08 3

What is agile?

08.10.08 4

Why Agile?● Top 5 reasons:

– Software based on real and most valuable requirements - business needs and value drive development

– Shorter time to market– You always know where your projects are - more

transparency and controllability– Focus on quality– Lower project risk

➔ Better return on investment

08.10.08 5

Agile Manifesto

AgileValuesAgile

Values

AgilePrincipl

es

AgilePrincipl

es

Copyright Reaktor Innovations 2007 6

These are important

Processes and tools

Comprehensive documentationContract negotiation

Following a plan

These are even more importantIndividuals and interactions

Working software

Customer collaboration

Responding to change

Agile ValuesAgile ManifestoAgile

Values

AgilePrinciple

s

08.10.08 7

Agile Principles● Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software.

● Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

● Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

● Business people and developers must work together daily throughout the project.

● Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

● The most efficient and effective method of conveying information to and within a development team is face-to-face conversation

Agile ManifestoAgileValues

AgilePrinciple

s

08.10.08 8

Agile Principles● Working software is the primary measure of progress.

● Agile processes promote sustainable development.The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

● Continuous attention to technical excellence and good design enhances agility.

● Simplicity--the art of maximizing the amount of work not done--is essential.

● The best architectures, requirements, and designs emerge from self-organizing teams.

● At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly

Agile ManifestoAgileValues

AgilePrinciple

s

08.10.08 9

Values

Principles

Practices

Defines a set ofways to meet

the values

Defines in detail howthis is implemented

in practice

Defines the setof most importantValues of what istruly important

Copyright Reaktor Innovations 2007 10

Product Backlog Sprint

Backlog

Potentially shippable product increment

Sprint1 - 4 weeks

24 hours

Daily Scrum meeting

S printP lanningM eeting

Demo, S printR eview &R etros pec tiveM eeting

Scrum

08.10.08 11

Scrum in 5 Bullets

● Iterative, incremental, cyclic process● Time-boxed development● Evidence-based planning● Inspect and adapt - feedback loops● Cross-functional, self-managing teams

08.10.08 12

How to Organize Testing?

08.10.08 13

Basics of Agile Testing● Individuals and interactions over processes and tools

➔ Tester integrated part of the team● Working software over comprehensive documentation

➔ Testing right after feature has been implemented● Customer collaboration over contract negotiation

➔ Customer participates in testing, continuous feedback● Responding to change over following a plan

➔ Exploratory testing and test automation

08.10.08 14

Test Process● Principles:

– Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale

➔ Purpose is to help in building quality into system➔ No separate testing phase, continuous testing ➔ Test early and often: when a feature is implemented, it is

tested

08.10.08 15

Test Process● Principles:

– Working software is the primary measure of progress.– Business people and developers must work together daily

throughout the project➔ Feature is not ready before it has been tested➔ Direct feedback as soon as possible➔ Bugs are fixed as they are found

“If you don’t have meaningful feedback then you’re not agile. You’re just in a new form of chaos.”

-Brian Marick

08.10.08 16

Between Agile and Iterative● Agile transformation is demanding → currently testing is

often something between iterative and agile, eg:● Testing done in the end of sprint or after it

– This is not agile!– Customer most probably does not get working

software– Bugs remain → tasks are not actually finished by the

end of sprint – Creates endless quality debt → stabilization sprints

08.10.08 17

Between Agile and Iterative● Few test releases in each sprint

– A lot better than testing in the end of sprint– Slow feedback loop → tasks get ready late, pressure

at the end of sprint– Creates quality debt, if not enough time to fix bugs

08.10.08 18

The Role of Tester ● Principles:

– Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

➔ Tester is like any other team member➔ Quality belongs to everyone, tester is just a facilitator➔ Quality contributor instead of gate keeper

”Testaajien yleisin virhe on kaapata laatukeskustelu kokonaan itselleen”

-Erkki Pöyhönen

08.10.08 19

The Role of Tester● Principles:

– Business people and developers must work together daily throughout the project.

➔ Testers work with business representatives to understand their needs

➔ Testers work with developers to understand the system and to provide feedback

➔ All essential stakeholders contribute to testing

08.10.08 20

Do You Need Specialized Testers?

● No

– In a crossfunctional team everyone does everything and the responsibility is shared, tester is just another team member

– Developers have the technical knowledge and customer has domain knowledge → they can do testing

● Yes

– Testing of complex systems is demanding and requires testing expertise

– Testers need a different mindset → it is hard to switch between development and testing

➔ Responsibility and knowledge is shared, but also specialized experts are needed

08.10.08 21

Test Reporting and Documentation

● Principles: – The most efficient and effective method of conveying

information to and within a development team is face-to-face conversation

➔ Direct ongoing feedback to developers➔ Communication with business representatives in order to

understand their needs

08.10.08 22

Test Reporting and Documentation

● Principles:– Simplicity--the art of maximizing the amount of work

not done--is essential. ➔ Forget the test documentation templates and IEEE 829!➔ Automate reporting as much as possible➔ Diaries, tables, mind maps instead of word documents

08.10.08 23

Do You Need a Bug Management Tool?● No

– Feedback is given directly– A failing automated test is a bug report– Bugs are fixed as they are found → no need to manage their state– An unmanageable amount of bugs is just an indication of bad practices

● Yes

– All bugs are not found only with automated tests– Often testing starts too late → lots of bugs found in one sprint– All bugs are not fixed → they have to be controlled in some way

➔ In an ideal agile project bug management tool is not needed, but in reality it often has to be taken into use.

08.10.08 24

Challenges and Common Mistakes

● Tasks marked as done before testing is completed (Definition of Done)

● Testing is not done from the beginning of the project

● Only features are tested, not the system as a whole

● Regression testing is neglected● Testers are not part of the team

08.10.08 25

Test Process and Scrum

Product Backlog Sprint

Backlog

Potentially shippable product increment

Sprint1 - 4 weeks

24 hours

Daily Scrum meeting

S printP lanningM eeting

Demo, S print R eview &R etros pec tiveM eeting

Testing activities defined in backlogs

Tester participates and contributes asa team members

Testing tasks distributed among teammembers, not only to tester

Only testedfeatures in demo

08.10.08 26

Scrum and Testing in 5 Bullets

● Iterative, incremental, cyclic process➔ Testing in the same pace with development

● Time-boxed development➔ Time-box includes QA, QA included in the

Definition of Done

08.10.08 27

Scrum and Testing in 5 Bullets● Evidence-based planning

➔ Concerns testing as well; tools and practices are evaluated by the team

● Inspect and adapt - feedback loops➔ Testing practices changed as needed

● Cross-functional, self-managing teams➔ Testers are team members, quality belongs to

everyone

08.10.08 28

How to do it in Practice?

08.10.08 29

Agile Testing Areas

By Brian Marick

08.10.08 30

Unit Testing● Cheapest way to build quality● Done by developers● Continuous integration● Unit tests ignore user's point of view → unit

tests do not substitute system level tests!● Frameworks: JUnit, TestNG, RSpec,

JDave

08.10.08 31

Test Driven Development● Tests written first● Tests serve as documentation

– Behavior Driven Development● Red-Green-Refactor

– Red: Failing test– Green: Write code so that test passes– Refactor: Code with better quality

08.10.08 32

Acceptance Testing● Should be automated and run frequently● Every team member should have access to

test automation tools and scripts● Automated regression tests

”Traditional test automation tools don’t work for an Agile context because they solve traditional problems, and those are different from the challenges facing

Agile teams.”.- Elisabeth Hendrickson

08.10.08 33

Acceptance Test Driven Development (ATDD)

● Tests for a feature are defined before development starts

● Tests are specifications or examples of the feature

● Feature can be verified right after it has been implemented

● Usually keyword driven

08.10.08 34

Keyword Driven Automation● Tests defined in natural language, eg. in

html tables● Customers and business people can

specify tests

● Testers or developers take care of the implementation

● Frameworks: Fit, Robot, Concordion

08.10.08 35

Example: Keyword Driven Automation

Test case:Test case:

Test implementation:

08.10.08 36

Acceptance Testing - Challenges● Maintenance costs of automation● Automated system tests are slow● It's not possible to automate everything

cost-efficiently

➔ Unit tests and acceptance tests support each other

➔ Continuous running

08.10.08 37

Usability and Exploratory Testing● Tests that cannot be automated● Question acceptance tests

– Are acceptance tests valid?● Finding new essential features

08.10.08 38

Exploratory Testing “Exploratory software testing is a style of software

testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

08.10.08 39

Exploratory Testing● Purpose to find bugs, new features and to

learn● No detailed predefined tests● Based on testers' expertise● Can be done in pairs with customer,

developers, users etc.● Not equal to ad hoc!

08.10.08 40

Example: Exploratory Testing

Session handling & authentication

Credentials

OK Wrong password Wrong user

name

Expiration

Configuration

Re-opening

Accountlocking

Configuration

Basic case

08.10.08 41

Usability and Exploratory Testing - Challenges

● Keeping testing exploratory, not ad hoc● Finding appropriate level of documentation● Getting usability feedback as early as

possible → protos, simulations, early usability tests

08.10.08 42

Property Testing● ~ Non-functional testing, excluding

usability● Reliability, Portability, Supportability,

Security● Early testing and feedback● These issues have to be

considered during development

08.10.08 43

Property Testing - Challenges● Usually requires very specialized

knowledge● Has to be done also on complete system● Specialized tools, that are not available for

all team members➔ Open source tools➔ Early profiling➔ Awareness

08.10.08 44

Summary● Collaborate● Adapt● Explore● Automate