Do you test the way your customer acts?
-
Upload
andre-verschelling -
Category
Software
-
view
63 -
download
0
description
Transcript of Do you test the way your customer acts?
Madrid, 26th to 28th May 2014
Do you test the way your customer acts?
Real-life test; Real-life situations
A practical example of the Statistical Usage Test
Madrid, 26th to 28th May 2014
Customer satisfaction
– Number one need:Involvement• Customers want to feel like
you care about their needs,and that you’re emotionallyinvested in helping themsolve their problems(Source: http://www.businesspundit.com/top-10-things-that-all-customers-want/)
2Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Therefore, we test
– Because testing …
… is a process that provides insight into, and advice on, quality and the related risks
– And quality …
… is the totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs
– Thus, …
… the customer gets what he really needs, right? 3Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
What is happening
4Real-life test; Real-life situations
Focus shift
Madrid, 26th to 28th May 2014
Does that satisfy the customer?
– “On system level we do exploratory testing”
– Does that really address the customer needs …
… and does it solves his/her problems?
5Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
What about test design techniques?
6Real-life test; Real-life situations
Test Design Technique
Useful for Test Design Technique
Useful for
Decision Table Verification Evolutionary Algorithms
Verification
Classification Tree Method
Verification State Transition Test Verification
Elementary Comparison Test
Verification Real-life test: Statistical usage Test
Verification & Validation
Error Guessing Verification Rare Event Test Verification
Exploratory Testing Verification & Validation
Semantic Test Verification
Data Cycle Test Verification Syntactic Test Verification
Process Cycle test Verification Use Case Test Verification & Validation
Madrid, 26th to 28th May 2014
Validation Techniques
– Exploratory Testing• Only explores what the testers has interest in• Tester in center stage, instead of user• Actually verifying, not validating
– Use Case Testing• Fixed and pre-defined workflows to represent user
behavior• Based upon defined actors/users
– Real-life test: Statistical Usage Test• Statistically correct user behavior randomly applied• Based upon operational usage of all users
7Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Statistical Usage Test – A Real Life Test
– Operational Usage is captured in Operational Profiles• Which operation is most likely to be performed under which
circumstances?– ‘Behaviors’ (events) – ‘States’ (history classes)– Probability of HistoryClass-Event combination
8Real-life test; Real-life situations
Event 1 Event 2 … Event m
History Class 1 P1,1 P1,2 .. P1,m
History Class 2 P2,1 P2,2 .. P2,m
… .. .. .. ..
History Class n Pn,1 Pn,2 .. Pn,m
Madrid, 26th to 28th May 2014
Real-life test; Real-life situations 9
Statistical Usage Test
– Example of an operational profile (VCR)Events ►
History classes ▼evStop evRewind
(X tics)evPlay(X tics)
evFastForward(X tics)
evRecord(X tics)
Standby 0 0.2 0.5 0.2 0.1
Rewind 0.3 0 0.6 0.1 0
Play(counter < 15000)
0.5 0.3 0 0.2 0
Fast forward 0.5 0.1 0.4 0 0
Record 1 0 0 0 0
Play(counter ≥ 15000)
0.8 0.15 0 0.05 0
Madrid, 26th to 28th May 2014
Real-life test; Real-life situations 10
Gather the Operational Profile (1)
– Formal analysis / specification• Differentiate customers• Differentiate users• Differentiate system modes• Differentiate functions• Specify operational profile
for each [customer, user, system mode, function]
– Learn by example• Watch and record• Interviews
Madrid, 26th to 28th May 2014
Real-life test; Real-life situations 11
Gather the Operational Profile (2)
– Better: Use production data• Log files• Transaction data• Usage data
– Most systems are evolutions instead of revolutions• Evaluate predecessors• Evaluate comparable systems
Madrid, 26th to 28th May 2014
Real-life test; Real-life situations 12
Statistical Usage Test
– Test case starts with initial state (i.e. History Class 1)
– Randomly choose next event – Example Test Case for the given Operational
Profile (= sequence of transitions):
– Test case must continue until execution profile corresponds with Operational Profile
Standby(current count = 15000) | evRewind(10000 Tics) | Rewind(5000) | evPlay(6000) | Play(11000) | evStop | Standby(11000) | evPlay(5000) | Play(16000) | evRewind (8000) | Rewind(8000) | evPlay(3000) | Play(11000) | etc., etc., etc.
Madrid, 26th to 28th May 2014
– Everything your users do is done prove it
– No fixed sequence, butrandom selection at runtime statistically correct
– Thus, caring about the users’needs
– But what if the real user finds an anomaly? 13Real-life test; Real-life situations
Run the test
Madrid, 26th to 28th May 2014
OK, there’s more to it
– Given a system (mobile X-ray system) with:• Intermittent failures• Unclear problem descriptions like
– “Sometimes the connection between stand and Mobile Viewing Station is lost”
– “Sometimes images are lost”
• Problems could not be reproduced by functional tests
– Root cause of system failure not yet found– Log file contains not enough information– Data analysis assumes hw errors
14Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Solving a customers problem
– Increasing number of repairs
– Wear-out effects and error pareto shift
15Real-life test; Real-life situations
1% 2%1%6%
61%
24%
2%4%
1st timeBattery
CF card defect
ETX
Mechanical damage
No repair performed
Other
PSU
SCSI disk
1% 2%
17%
52%
23%
2%
2%
2nd timeBattery
CF card defect
ETX
Mechanical damage
No repair performed
Other
PSU
SCSI disk
4%
26%
30%
26%
15%
3+ timeBattery
CF card defect
ETX
Mechanical damage
No repair performed
Other
PSU
SCSI disk
Madrid, 26th to 28th May 2014
Gather the Operational Profile
– Log file does contain usage information
– Define most important parameters:• Switch on / switch off
system• Start / stop low dose
fluoroscopy• Start / stop high
definition fluoroscopy• Store images
Log file excerpt10-02-10 14:05:34 S INIT > > > > INIT_Main:
Starting application < < < < 10-02-10 14:05:34 S INIT Build by nlybstqvp4-
sys24 on NLYBSTQVP4DT037, 10-02-10 14:05:34 M IPDR DFI Unit Serial Number:
[00-00-10-6B-22-03-01]10-02-10 14:05:34 M IPDR DFI Unit Versions: IN [0x1] PROC [0x1] CTRL
[0x5] DISP [0x1]10-02-10 14:05:34 S IPDR Starting DFI selftest
10-02-10 14:38:55 M SYS USR switch on RIGHT10-02-10 14:38:55 M SYS Xray mode continuous
fluoro10-02-10 14:38:55 M USR START FLUORO RUN RIGHT 10-02-10 14:38:55 M SYS USR switch off at 187683 10-02-10 14:38:56 M USR STOP FLUORO RUN 10-02-10 14:38:56 M FILE Stored [1] new image(s)
on disk10-02-10 14:38:56 M FILE Deleted [0] image(s)
from disk
16Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Gather the Operational Profile
– Data from log file
17Real-life test; Real-life situations
Case#
Start datedd-mm-yy
Start Timehh:mm:ss
End Datedd-mm-
yyEnd Timehh:mm:ss
Run Timehh:mm:ss
Left Fluo
#
Right Fluo
#
Images Stored
#
Total Fluo time (sec)
Avg Fluo time (sec)
1 10-02-10 14:05:34 10-02-10 14:08:16 0:02:42 0 0 0 0 0
2 10-02-10 14:08:16 10-02-10 14:57:08 0:48:52 2 24 24 15 0.576923077
3 12-02-10 7:12:11 12-02-10 10:27:44 3:15:33 9 74 75 63 0.759036145
38 9-03-10 7:15:20 09-03-10 13:57:47 6:42:27 60 1 18 58 0.950819672
39 10-03-10 7:37:01 10-03-10 7:37:37 0:00:36 0 0 0
Totals 61:27:51 565 480 521 1065
Average 1:34:34 14 12 13 28
Madrid, 26th to 28th May 2014
Gather the Operational Profile
– Interpreting the data determine history classes – Basic profile
• A ‘case’ is from system start to system stop• Sometimes no fluoroscopy is done at all• Most of the times a case consist of some Fluoroscopy
(LDF, HDF or both)18Real-life test; Real-life situations
History class Start system
Stop system
Make fluoroscopy run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 0
Madrid, 26th to 28th May 2014
Gather the Operational Profile
– Add more details for fluoroscopy history classes• During a case the following figures apply for low
dose fluoroscopy (LDF)– Average number of LDF runs is known: 11.63– Assume normal distribution applies: sigma 1 = 66% - 133% of
average– “normal” maximum: 45 LDF runs– Real maximum: 270 LDF runs
• Same analysis for high definition fluoroscopy (HDF)
19Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Gather the Operational Profile
– Final profile
20Real-life test; Real-life situations
History class Start system
Stop system
Make fluoroscopy run(s)
System switched off 1 0 0
System started 0 0.08 0.92
Fluoro Runs completed 0 1 0
1 – 7 * LDF 8 – 16 * LDF 17 – 45 * LDF 46 – 270 * LDF
Standby 45% 29% 24% 2%
+1 – 7 * HDF 8 – 16 * HDF 17 – 78 * HDF
Standby 40% 40% 20%
Madrid, 26th to 28th May 2014
Executing the test: AutomateSwitch on the systemDecide on the number of fluoroscopy runs @nr_fluoL = 0 @nr_fluoR = 0 # start throwing the dice for the left
fluoro runs @leftSel = rand() case @leftSel when 0 .. 0.41 @nr_fluoL = 0 when 0.42 .. 0.67 @nr_fluoL = 1 + rand(7) when 0.68 .. 0.84 @nr_fluoL = 8 + rand(9) when 0.85 .. 0.99 @nr_fluoL = 17 + rand(28) else @nr_fluoL = 46 + rand(225) end
# then for the right fluoro runs (= images being stored)
@rightSel = rand() case @rightSel when 0 .. 0.20 @nr_fluoR = 0 when 0.21 .. 0.59 @nr_fluoR = 1 + rand(7) when 0.60 .. 0.78 @nr_fluoR = 8 + rand(9) else @nr_fluoR = 17 + rand(62) end
Perform the selected number of runsSwitch off the system
And start all over again (endless loop)
21Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Statistical Usage Test results
– System starts showing intermittent failures– When continuing, test results in a Fatal Error
(hard disk broken)
– 13 days of test versus 1 month of real-life usage• Images stored: 80719 versus 521• Total Fluoroscopy time: 135995 sec versus 1065 sec
22Real-life test; Real-life situations
Run Timehh:mm:ss # Left Fluo
# Right Fluo
# Images Stored
Total Fluo time (sec)
Customer system 61:27:51 565 480 521 1065
Test system 198:19:31 71873 89548 80719 135995
Madrid, 26th to 28th May 2014
Statistical Usage Test results
– Problem reproduced– Test results have proven the data analysis
– Possible improvements on the Operational Profile and/or the test• Combine more user data (e.g. more log files, more
customers)
• Define actual contribution on available data (no assumption of normal distribution)
– Automation is mandatory, e.g. using FitNesse 23Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
Conclusion
– Real-life testing is doing what your customer wants to do Paying attention to thecustomer needs
– Real-life testing helps findingissues Help solving thecustomers’ problems
24Real-life test; Real-life situations
Madrid, 26th to 28th May 2014
IS YOUR CUSTOMER SATISFIED?
Unless you have 100% customer satisfaction…you must improve.Horst Schulze