TESTING 4 - s3.amazonaws.coms3.amazonaws.com/JuJaMa.UserContent/6499e9ec-e042... · (Win, Mac,...
Transcript of TESTING 4 - s3.amazonaws.coms3.amazonaws.com/JuJaMa.UserContent/6499e9ec-e042... · (Win, Mac,...
1
TESTING 4.0
THE NEXT GENERATION OF QUALITY
Powered by Crowd & Cloud Technologies
2
Agenda
Introduction
Functional testing
(Example 1)
Usability test
(Example 2)
3
Testbirds
Introduction
UK Country Manager
Vision: “The World’s Leading Company for Innovative
Testing Services”
Founded in 2011
Origins in the enterprise software testing market
Customer satisfaction 9.2 out of 10
Quality assurance and UX / usability testing utilising
crowd and cloud technologies
4
Device Diversity
Do You Have All of the Devices to Test On?
5
Device Diversity
How Do You Validate Localisation?
6
Target Group Relevance
Do You Know Your Target Groups‘ Needs?
7
Current Challenges in the Market
Insufficient QA, Poor Market Analysis and Time Constraints
Organisational blindness
Is everything really fine? Unbiased
users can test differently.
Device diversity
Do we really have all of these
devices to test with?
Target group relevance
Target groups have different
expectations and abilities.
Lack of resources
Time pressure, limited budget and
priority of other projects.
8
CROWD
150,000+ people in 193 countries
380,000+ devices
110,000+ found bugs
Any target audience (60+ criteria / demographics)
Real world testing without artificial environments
Client specific feedback even prior to release
Flexible and efficient ad-hoc availability
CLOUD
Any combination of virtual machine
(Win, Mac, Linux) and emulator (iOS, Android)
Modern technologies exclusively created by
Testbirds
High scalability and individual configuration
Flexible, efficient and dynamic
No maintenance or development costs
Servers only hosted in Germany for maximum
data security
Testing 4.0
Powered by Crowd & Cloud Technologies
9
Nest Demo
10
Scenarios for Crowdtesting in the Automotive Industry
Multiple Possibilities for Boosting Quality
Brand portal
Car configurator
Content
Connected car app
(Example 2)
Connected car
web portal (Example 1)
Used car portal
Other apps
…
11
User-centric Testing in the Automotive Industry
Assuring Functionality and Improving Usability in a Connected Cars Project
Objectives
Functional test on various
devices/browser combinations before
Go-Live
Localization check across different global
markets
Generation of user input for requirement
definition phase
Benefits
Highly efficient coverage of device/browser
matrix
Low effort for project integration / internal
resources
Comprehensive view of customer needs
Higher customer satisfaction
Setup
Three semi-structured functional tests
runs before release (Test Pre-Live,
Retest Pre-Live, Test Live)
Integrated localization tests on up to 70
country – language combinations
Functionality checked on up to 80 device
/ browser combinations
UAT outside the development/rollout
process with 80 vehicle owners (model
year > 2009) across all major models and
important Go-Live markets (DE, FR, UK,
NL, US, IT)
Results
List of functional issues after each test that
were fixed before the next test run
Bug-Retesting in test run 2 and 3
List of spelling and translation issues from
up 70 country – language combinations
Feedback from actual vehicle owners
regarding the usability and content of the
web portal
Summary with all the feedback and
information from the testers in a compact
and clear final report
Customer
OEM
Headquarter
Location
Germany
Industry
Automotive
Product
Connected
car web
portal
12
User-centric Testing in the Automotive Industry
Assuring Functionality and Improving Usability in a Connected Cars Project
Functionality
Example
1
Usability
Product: Connected car service portal
Three semi-structured tests runs before
each release (Test Pre-Live, Retest Pre-
Live, Test Live)
Functionality checked on up to 80 device /
browser combinations
Integrated localisation tests on up to 70
country – language combinations
Product: Connected car app
Usability study outside the
development/rollout process with 81
vehicle owners, across all major models
and important Go-Live markets (DE, FR,
UK, NL, US, IT)
Exploratory (use cases)
Structured (use cases)
Localisation
End-to-End
Beta / Live
Example
1
13
Process Overview
A
Preparation
B
Testing phase
C
Reporting
D Follow-ups
1 – 3 days
Phase
Time-
frame 1 – 3 days 1 day ongoing
14
Agenda
Introduction
Functional testing
(Example 1)
Usability test
(Example 2)
15
A - PreparationFunctionality
Topic Contact person
Test environment On which environment will be tested?
How is it accessible?
XY
Test date When will be tested?
What results will be needed by when?
XY
Test content What will be tested?
Which test cases / use cases?
Is there a requirements document?
In scope / out of scope
XY
Country/language combinations Which native languages and which countries? XY
Devices and browsers Desktop / Smartphones / Tablets
+ Browser (versions)
XY
Testers Which tester demographics? XY
VINs Which VINs to use? Pre-live, Live etc. XY
Release plan Are there ongoing deployments? XY
…
16
B Test Case Test
Testbirds Platform
Test Cases implemented directly on the platform
Results documented per device
Several steps per Test Case – if one step fails the whole test case is highlighted and a bug is recorded
Device the Test Cases were
performed on
Test Case 1 failed on Device 1
Functionality
17
B Test Case Test
Testbirds Platform
Test case steps:
• Pass = green
• Fail = red > bug is submitted
• N/A = yellow > not possible to perform due to previous bug
Functionality
Same steps all worked on
different device
Step 2 failed – bug was submitted
Step 3 could not be performed because of
step 2
18
B Test Case Test
How Bugs Are Displayed
Overview of all bugs submitted by testers
Possibility to export to xls. or bug tracking tools
Functionality
Export function
19
B Test Case Test
Bug Example
Functionality
Bug
description
Visual
documentation
General information
(device, category, severity)
Title
20
C Results
Testbirds Platform
Detailed list with all devices tested
• 49 devices (device type / OS /
browser) – 28 different devices
Device #
Windows 7 (Chrome) 4
Windows 7 (Firefox 28) 3
Windows 7 (Firefox 29) 1
Windows 7 (Firefox 32) 1
Windows 7 (Firefox 34) 1
Windows 7 (Firefox 35) 3
Windows 7 (IE 11) 3
Windows 7 (IE 9) 1
Windows 8 (Firefox 28) 1
Windows 8 (IE 11) 1
Windows 8.1 (Chrome) 5
Windows 8.1 (Firefox 35) 2
Windows 8.1 (IE 11) 1
Windows 8.1 Touchscreen (IE 11) 2
Mac OS 10.10 (Firefox 35) 1
Mac OS 10.10 (Retina) (Chrome) 1
Mac OS 10.10 (Retina) (Safari) 2
Mac OS 10.10 (Safari) 1
Mac OS 10.10.1 (Chrome) 1
Mac OS 10.10.1 (Safari) 1
Mac OS 10.6 (Safari) 1
Mac OS 10.7 (Chrome) 1
Mac OS 10.9 (Chrome) 1
Device #
Tablet
Apple iPad 2, iOS 7.0.3 1
Apple iPad 4, iOS 8.1.0 1
Apple iPad Air (iPad 5), iOS 8.1.2 1
Smartphone
Apple iPhone 5S, iOS 8.1.0 1
Samsung Galaxy S i9000, Android 2.3.3 1
Total 49
Functionality
21
C Results
Technical Issues – Examples of Defects
Functionality
Defect ID Defect Title
Spelling / Grammar mistake
ID-29354 (FR/FR) Service overview - Bad translation of "Board book" tab
Translation issue
ID-29331 (FR/FR) My vehicles - Status message not translated
ID-29344 (ES/ES) Service overview - Service offers in English instead of Spanish
ID-29352 (NL/NL) Service overview - Service offers in English not Dutch
ID-29381 (FR/FR) Service overview - Service offers in English instead of French
ID-30433 (GB/EN) Service overview - Dealer Search in German (was already closed but apparently is not fixed ID 29179)
ID-30557 (Belgium/Dutch) Service overview - tabs in English instead of Dutch
Other
ID-29167 Service overview - Google link in Service Offers available
ID-29170 (US/EN) Dealer Search - Pop up not translated to English
ID-29304 Dealer Search - Encoding Problem in Print Version
ID-29334 (FR/FR) Dealer Search - GUI issue in result details of dealer search
ID-29422 (Canada/EN) Service overview - Dealer Search in English
ID-29660 Dealer search - Position and layout of error message not aligned
ID-29683 (NL/Dutch) Dealer Search - GUI issue in result details of dealer search
ID-30434 Service Overview - Service offers: Price in Euro's whilst text in English
22
Agenda
Introduction
Functional testing
(Example 1)
Usability test
(Example 2)
23
A Usability Test
Initial Situation
This usability study focused on the current production version of the client’s app / service
Content: 12 questions to establish the users’ opinion of the app, what he / she likes, what he / she misses
Goal: Gauge user acceptance and optimise the app before local market launch
Participants: 81 car-owning testers in 14 locations
Usability
Country # of participants Country # of participants
Austria 7 Netherlands 3
Belgium 4 Portugal 4
Canada 5 Romania 2
France 3 Singapore 2
Germany 28 Spain 5
Hungary 3 Sweden 4
Mauritius 1 United Kingdom 7
24
B Usability TestUsability
Use case
documentation
Visual
documentation
Export
Example
25
C Usability Test
Results
Goal: Identifying potential for improvement of the app / needs of the customer
Setup:
• 81 customers from 14 countries
• 24 hours
• 12 questions
Results:
• Overall Satisfaction
• Registration Process
• Visual Appearance
• Navigation
• Loading Times
• Recommendation
• Content
Usability
3,5
5,1
4,5
4,3
3,3
3,6
4,4
Is there anything you miss on the current app?
• “Information about my last service check. When should I change
the oil, brakes and other parts.”
• “Information about my own car. Next Service, last Bills, ... There
are only marketing Info about selling new products or add-ons
what I could buy.”
• “Guidance/Ideas/Services that help me to either update my
vehicle, unlock new features etc.”
• “The portal gives a great overview about my vehicle, but no real
function to configure/improve my car.”
• “I miss recall information.”
• “Only the basic contact details are show: No picture of the dealer's
building or some additional services or special offers (e.g. free
winter check up from 1.10. - 15.12.) are shown here. Also a rating
for the dealer could be shown like XXX does.”
26
B Usability Test
Results - Report
Usability
The results are
delivered in a
detailed final
report
27
Discussions
Any remarks / questions /
comments?
Looking for a PoC!