Bug Hunting in practice - Nordic Testing...
Transcript of Bug Hunting in practice - Nordic Testing...
06/06/20161 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Bug Hunting in
practiceINTERNATIONAL TESTING CONFERENCE
Nordic Testing Days
JUNE 1 - 3, 2016 IN TALLINN, ESTONIA
• Szilard Szell, (Marko Kuisma)
06/06/20162 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Introduction
Marko Kuisma
•2005-2015 working in system testing and
performance testing area in Nokia and NSN
•Owner for ~1000 fault reports during testing
specialist career for six years in Nokia
•Several project manager positions in testing area
•Line manager of different test teams for four years
•Bachelor of Engineering in Telecommunications,
University of Applied Science
•Advocate for manual testing with human errors
Szilárd Széll
•2000 – various Testing Related engineering and
management positions in Nokia
•2015 – Test Coach in Central Org.
•2014 – President of Hungarian Testing Board
Studies
•MSc in IT Management, Central European
University
•MSc in Programming and Mathematics, University
of Szeged
•Certified Scrum Master, ISTQB CT-AL Test
Manager, CT-FL Agile Tester, IREB CPRE, Lean
Six Sigma Green Belt
3 © Nokia 2016
Nokia internal use
Highlights
• Core on Cloud: Open MSS, vTAS, TAS-E
• SDM Solutions
• GSM-R
• Cloud Infrastructure
• Serve at Once Intelligence, Device Mgmt, Cloud Application Mgmt
• All Core Customer Support
• T&I: SDN, SON, CEM, Big Data and Network Mgmt
Ecosystem
•Symbiotic cooperation with major ICT Universities in the region
•Strategic partnership with Hungarian Government
•European ICT hub: Ericsson, Epam Systems, Morgan Stanley, NNG, Lufthansa, SAP, Evosoft, Bosch, Siemens, IBM, Prezi, LogMeIn, BalaBit, T-Systems
•Partnering with and supporting Hungarian Testing Board
Technology Expertise
• Cloud computing and architecture - Core Virtualization
• (SDN) Software Defined Networking
• 2G/ 3G/4G Technology Expertise
• Network Function Virtualization
• Self Organizing Networks
• Big Data Management
Budapest – Core on Cloud Center
4 © Nokia 2016
Nokia internal use
Nokia as a house of Testing
• Well defined Testing Processes on Company level
• Well defined, standardized Requirements as input
• ISO9001 and TL9000 compliant processes
• Lean Six Sigma
• Complex, Highly available (99.999%), Robust Products on IT Cloud and
Telco HW
Cornerstones of System Verification
06/06/20165 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Test Case Design TechniquesAs used during Product Development
Defect-based techniques
• Taxonomies
Specification-based or Black-box
techniques
• EP
• BVA
• Decision table testing
• State transition testing
• Orthogonal arrays / all pairs tables
• Use case testing
Structure-based or White-box techniques
• Statement testing
• Decision testing
• Condition / Multiple condition / Condition determination testing
• Path testing
• LCSAJ
Experience-based techniques
• Error guessing
• Checklist-based testing
• Exploratory testing
• Attacks
06/06/20166 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
• Exploratory Testing can be described as a goal oriented wandering
• There is a mission/goal described in a charter, but there is no planned route
you need to take
• Create a charter describing what and how and which way you want to test
• Describe duration of your test
Exploratory Testing(One) Definition
”Exploratory Testing is an interactive process of
concurrent product exploration, test design and test
execution. To the extent that the next test we do is
influenced by the result of the last we did, we are doing
exploratory testing.”
James Bach, Satisfice
06/06/20167 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Exploring BarcelonaET is a goal oriented wandering
06/06/20168 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
06/06/20169 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Explaining Exploratory testStory Cubes – the game
“Once upon a time there was a flash, into who a fish has
moved in (to live). The fish has asked the flash - Why is there a
pyramid behind you? Why are you striking it? He (the flash)
said - because an airplane is just going to fly by 200.000
millimeters below (us). Then, the passengers in the plane has
asked the pilot - Why are we turning all around (left and
right?)? The pilot said - Because I am reading a book. Then
they (the passengers) asked the waitresses - Why don’t we
get something to eat? Because (the waitresses said) a
tortoise jumped into the soup. And then the tortoise said:
Look, the sheep is parachuting out there!”
Szonja, 7 years
06/06/201610 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
1. Gather Developers, Testers, Customers, Business people to the same table
2. Show them a Story Cube pack
3. Start collecting 9x6 areas of the system that is important
4. Make Cubes for your own
5. Play the game, roll the dice and tell a story by reading
the pictograms
6. You just created an exploratory test case
Explaining Exploratory testStory Cubes – in testing
06/06/201611 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Cube 1 Cube 2 Cube 3 Cube 4 Cube 5 Cube 6 Cube 7 Cube 8 Cube 9
Theme Unit State
Change
Load
Balancer
“CRUD” Output Transacti
on
Analysis Attribute
Anal.
Inter-
face
Side1 STU WO RE H.248
MSS
Create /
Add
Log LocUp Barring Routing A/Iu
Side2 CHU SP RE Read /
Interrogate
Alarm HandOve
r
User
plane
CHA Mc
Side3 VLRU CSWO SIP Update /
Modify
ErrorLog MO Digit EOS MAP
Side4 CM FSWO M3UA Delete /
Remove
ClearCod
e
MT EOS Service Nc,
BICC/SI
P
Side5 SU DISK H.248
MGW
Measure
ment
Forward Function AIF ISUP
Side6 IPDU POWER CHA
Ticket
SS Pre./Ext. IN CAMEL
Explaining Exploratory testStory Cubes – cubes “created” by experts
06/06/201612 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
• Rolled cubes: CMM -> Faulty SWO -> ClearCode -> A/Iu interaface
• Test Case scope: CMM faulty switchover during traffic
• Pre-requisite: MSS is running with traffic. WO/SP-CMM are running
smoothly.
• Execution: Issue faulty CMM switchover for WO-CMM (simulate hw
failure)
• ExpRes: CMM is 2N redundant unit, CMM is the heart of the system (main
DB copies are stored in its memory). If WO-CMM goes faulty switch over
should happen without any traffic outage. SP-CMM should take over the
traffic and the tasks smoothly
Explaining Exploratory testStory Cubes – a tale by an expert (Balazs Tenyi)
06/06/201613 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Explaining Exploratory testfrom scripted testing
15
Test setup for
SUT
TEST CASE
Test setup for
SUT
Abnormal setup
to SUT
misconfigure
feature X
Use of abnormal
conditions in
variables
Test executionSimultaneous
test event
Distracting the
SUT
Modify
configuration
Test execution
OK result NOK result
Test execution
OK result
minor
faultmajor
fault
critical
fault
traditional pathexploratory path
14 © Nokia 2016
Nokia internal use
Bug Hunting
06/06/201615 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
• When a new release of software is ready for test, a Bug Hunt will very
clearly read the temperature ~ quality of the software.
• As an entry-criteria for new phases. View it as a ”smoke test” executed by
people, instead of automated test, if you don´t have any of these.
• As team-motivation, when test execution becomes day to day work, and
the auto-pilot is taking over, a Bug Hunt can be what you need to add extra
adrenalin to your test.
Bug HuntingWhen can we apply Bug Hunting - Based on “Go on Bug Hunt” –
from Klaus Olsen, FiSTB Testing Assembly, 2013
06/06/201616 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
• 4 dissimilar test systems
• 4x test pairs
• 4 x 45 minutes testing rounds
• Exploratory Testing
• Scope of Hunting
• Chat tool
• Referee, Coach
• Prize
Bug HuntingIngredients
SUT1
SUT2
SUT3
SUT4
06/06/201617 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Bug HuntingSteps
1) Define
Pre-conditions for bug
hunting: scope,
environments, duration,
limitations, known issues
2) Go for Hunting
Use all the possible
methods and scenarios.
Do fault attacks, error
guessing, pair working,
parallel testing..
4) Judge
Referee makes notes of all
issues and informs if issue
is duplicate.
5) Analyze
Categorize issues.
Investigate more for
unclear issues
3) Inform
Briefly report all findings to
referee. Don’t pause
hunting for fault
investigation
6) Report
All clear faults reported
06/06/201618 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Performance Test / Robustness Test teams (ESPOO):
• ORANGE Team: (SUT86); 2 person – I, I, I, I, I
• BLACK Team: (SUT33); 2 person – I, I, I
• MAGENTA Team: (SUT64); 4 person – I, I, I, I, I, I, I, I, I, I, I
Performance Test / Robustness Test teams (BUDAPEST):
• RED Team: (SUT76): 3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I
• BLUE Team: (SUT87): 4 person – I, I, I, I, I, I, I
E2E product teams (BUDAPEST):
• GREEN Team: (SUT5/6):3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I
• Yellow Team: (SUT7-8): 4 person – I, I, I, I, I, I, I
• PURPLE Team: (SUT71-72) :4 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I
• Azure Team: (SUT53-54): 4 person – I, I, I, I, I, I, I
Bug HuntingGo for Hunting - Groups and Environments
~ 95 issues found
3 Major, 30 Minor Faults
06/06/201619 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Bug hunting Metrics
Different issues detected
with 3 hours of bug huntingFault reports created
by 6 Product Lines so
far with 8 bug hunting
events
Fault reports created after 3
hours testing by whole team
90+
Issues detected by most
effective pair during 3
hours of bug hunting
20+Fault reports done daily by
system testing team with
traditional way
1-333 100+
06/06/201620 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
Planning
• Plan all Sessions up-front as part of project
• Estimate time for the session, preparation and analysis
• Communicate to stakeholders
Schedule
• As part of Sprint Review
• Before Delivering to next stage (or customer)
• As quick entry and exit quality check
Administration
• Each Session is 1 Test Case in Test Management tool, with fix time (estimate)
• Report faults
• Create new test case for opened faults as needed
Bug HuntingPlanning, scheduling and administration
06/06/201621 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public
• Negative scoring as a punishment for not following rules
• “Hunting session” takes a month
• Using only Pre-Automated test cases
• Connect Real Incentive to results
• Collecting Bugs beforehand and keep them in your pocket until the Bug
Hunting Event
Bug HuntingMisuse – to be avoided
22 © Nokia 2016
Nokia internal use
Lessons Learned
• helps tester to think out of the box
• improves team spirit and expertise
• need to be planned up-front in Project Plan, not an ad-hoc event
• the event need preparation and time to analyze issues found
• complements requirement based testing
…is FUNA Bug Hunting event …
06/06/201623 © Nokia 2014 1.0 Kuisma_Szell - DocID
Public