Post on 18-Jan-2017
Continuous Quality in the Digital Age
Bob RandallVirgin Media
Gidi PridorPerfecto
A FEW HOUSEKEEPING ITEMS
1. Submit questions using the Q&A panel.
2. Slides and recording sent after the webinar.
3. Please complete survey after the presentation.
Continuous Quality in the Digital Age
Bob RandallVirgin Media
Gidi PridorPerfecto
4
554% would switch banks based simply their satisfaction with the bank’s mobile appCNBC Apr 2015
WHO ARE VIRGIN MEDIA?
Double Play : 19%
• 4.2m fixed telco customers• 3.1m Mobile customers
• 4.5m broadband customers • 4.9M TV & VOD customers
Internet & Data TV & VideoTelco & Mobile
1
Double Play : 19% Double Play : 19%
OUR TESTING SERVICE AT A GLANCE
UK
Indi
a
75 FTE 197 FTE
…We’re 72% offshore, in India…Our testing service is now
4 years old
We’re responsible for the Test and Quality across all IT projects delivered
Mobile Broadband
Digital TV Mobile AppsBusiness Engineering
Corporate
Access
Transformation
TMMi Level 5accreditation awarded in 2015
Innovation programmes delivered ensuring best in class tooling and methodology
We run 100k test cases per year and find on average, 13k P1-P4 defects
One of only 9 organisations worldwide to achieve this top tier accreditation
30
2
42
69
102
150
2012 2013 2014 2015
9393 projects already delivered as at Q3 2015
IT P
roje
cts
per a
nnum
We deliver 250% more projects now compared 2012
+70%Aspiration for mobile test automation
90% reduction in P1-P2s in production
increase in the volume of defects identified prior to ‘go-live’, with only 1 P1 and 1 P2 since 2013 in production
55%
We have Transformed our Digital Quality 3
Managing Documents in VM Cloud
Our customers are Going Mobile
Watching TV
Controlling Set Top Boxes
Browsing Our Websites
Managing Their Accounts
Finding Virgin Wi-Fi Hot Spots
What Next?
4
Customer Behaviours are Driving Change
• Mobile browsing doubled in a 12 month period from January 2013
• Mobile conversions track 10% greater than browse
• Conversions doubled within 18 months
• Growth of mobile is constant
• Smart devices are now the norm, mobile first services are the default
• Desktop based orders will be the minority by December this year
THE RISING IMPORTANCE OF MOBILE FOR VIRGIN MEDIA
5
Old Process, New Challenges
100% Manual Testing (hand-held devices)
Devices were procured or borrowed individually by each team
Devices were insecure, often lost/missing and unavailable when needed
No corporate standard for mobile quality & test coverage
All testing had to be conducted on-site, no option to offshore
Quality Efficiency
Overhead Control
• Limited coverage• Poor oversight
• Slow cycle• QA bottleneck• Scalability• Availability
• Logistics• Cost• Can’t Offshore• Can’t keep up
• Security• Governance• Standardization• Audit• Reliability
6
TRANSFORMATION
We Had To Do More For Less, A Lot Less…
Transform the way we approached mobile testing
• Scale & Speed
• Reduce Costs
• Improve Quality
CQContinuous Quality
7
Key Requirements for Transformation
Robust Automation Sustainable Device Agnostic Unattended
Central Cloud Platform
Remote testing - offshore
Avoid device mgmt. Governance
Flexibility to Scale Add users Add use cases Update devices
Enterprise-grade
StandardSecurity SLA Reliability
Future Readiness Easily integrated Any environment Any Framework ,
CI etc.
Investing in technology was the only realistic way.
SPEED
SCALE
EFFICIENCY
COVERAGE
8
Cloud Based Continuous Quality Lab
Proprietary & Confidential
Dev
/Tes
t E
nviro
nmen
ts (+
CI/C
D)
9
Use cases
Functional application
testing
Performance testing
Web responsive testing
Customer set-top box/home hub/mobile
device testing
Hosted testing services for Liberty
Global affiliates
Service Desk mobile fault
analysis support
India
Netherlands
UK
10
How has it changed what we do?
We have embraced Continuous Quality as a practice
Mobile and Responsive Web
Testing
• Centralized enterprise test lab
• Virtualised devices as a cloud service
• Maximized test coverage
• Accelerated using Automation
Mobile Application Security Testing
• centralised Security Testing capability
Networks Virtualisation and Testing
• Network virtualisation
11
2012 2013 2014 - 2015
Transforming to Stay Ahead
Standardising Processes and Methodologies Enabling Efficiency Transformation & Innovation
Standardisation & Industrialisation Delivery Quality Cost Effectiveness
HP LOAD RUNNER STANDARDISATION
HP QUALITY CENTRE STANDARDISATION
STANDARDISED TESTING ACROSS VIRGIN MEDIA
TRANSITIONED 239 SILOS INTO STRATEGIC TCOE
CENTRALISED AND STABILISED TEST ENVIRONMENTS SV
REMOTE HFC MODEM ACCESS
DATA MASKING
MONITORING AND ANALYSIS
ENABLING OFFSHORE MOBILITY TESTING
TEST ENVIRONMENT VIRTUALISATION
ENABLING OFFSHORE TV TESTING
EARLY ENGAGEMENT – STATIC TESTING
STRATEGIC TEST AUTOMATION
TASKTOP SYNC: DEV AND TEST COLLABORATION
SAUCELABS DIGITAL CLOUD EXECUTION PLATFORM
TEST INNOVATION AND TRANSFORMATION PROGRAMME
TEST ENVIRONMENTS DASHBOARD
HP SAAS HOSTED APPLICATION LIFECYCLE MANAGEMENTHP SAAS HOSTED PERFORMANCE TESTING SUITE
FORMALISED RISK BASED TESTING
NEW MEDIA PERFORMANCE TEST ENVIRONMENT BUILT
TMMI LEVEL 5 ACCREDITATION
QUALITY MANAGEMENT
NON FUNCTIONAL PROCESS IMPROVEMENTS
TASKTOP DATA: REAL TIME DATA REPORTING
NEOLOAD: CLOUD PERFORMANCE TESTING
Embracing technology and tooling has been key to our success
SECURITY TESTING CAPABILITY
AGILE READY TEST CAPABILITIES
INTEGRATED MOBILE TEST AUTOMATION
12
Results
Perfecto’s tight integration with Selenium has helped us develop our strategic mobile automation framework, which will be our primary mechanism for automating all mobile application testing
Perfecto has enabled us to extend our Responsive Test Automation platform to use real devices instead of emulators to get increased confidence that we’re testing the real customer experience
ResponsiveTesting
Framework
The on-premise capabilities of Perfecto have enabled us to be able to test Connected Home type services from India,
On premise
capabilities
By utilising Perfecto, we’ve been able to take a significant portion of our mobile applications testing to India
Perfecto has enabled us to conduct 1st level application support through our offshore application support team
13
Our current challenges
CHANGING DELIVERY MODELS – THE SHIFT TO BI-MODAL IT
• Transformation to a Digital Business is demanding a change in IT delivery approaches
• The increasing need for agility is driving a need for Bi-Modal IT, through embracing Agile delivery methods for faster paced programmes
• The demands on our Testing service are changing• Increased levels of Automation, broader technical skill
sets, DevOps and changing governance controls and processes are needed
MODE 1 – WATERFALL
MODE 2 – AGILE / ITERATIVE
14
How Perfecto Mobile contributes to solving the challenges ahead 1
5
Integration of Perfecto Mobile within our new DevOps reference architecture for
Continuous Delivery operations
Deployment of Perfecto Local Link to enable more flexible testing in scrum
team environments
Further deployment of Perfecto Local Link to enable “Location Based” testing
Integrated automation through Selenium scripting
Requirement Management ALM Defect Management
SoftwareConfiguration Management Build and Packaging
Automated Testing Data Analysis/CodeQuality
Continuous Integration Artifact Repository Continuous DeliveryOrchestrates
Virgin Media Continuous Delivery Framework
Quality Maturity Model
1
Initializing Definition Transitioning Integration Continuous Quality
2 3 4 5
ReleaseFrequency
TestCoverage
Automation %
Your TestProcess
Your Environment
2-3/year Quarterly Every 45-60 days Monthly When done
<30% coverage -50% coverage 50-90% coverage 90%+ coverage 100% coverage
+/- automation Regression testing at end of sprint
<33% automation Some automation with legacy
QA tools-50% automation Up to 75% automation >75% automation
Agile developmentProcess +Waterfall
QA
Separate QA and Dev ALM tools
Test cycles take less than a week
Centralized testing platform
Performance testingProduction monitoring
No performance testing
No monitoring in production
Applying desktop testing approach to mobile
Isolated test environments (no lab)
Using dev tools for automated testing
(Java, C#)
Automation framework
Performance testing several times per year
Build process runs automated tests (CI)
Optimized automation framework
Managed lab
16
Future 17
• Perfecto Mobile is a key component in our strategy to implement CI and CD, we see this capability evolving and maturing in 2016, alongside further automation development
• Device types are proliferating – as new mobile and wearable technologies become relevant we’ll use Perfecto Mobile to host and test against these new devices
• Security is going to be more and more important as we adopt mobile devices into every aspect of our lives. We have a responsibility to ensure our customers and their data is kept as secure as possible so we intend to use Perfecto Mobile to support our device security strategy
• Virgin Media is a TMMI Level 5 accredited Test Centre of Excellence and we want to exploit the good practice, technologies, processes and approaches we’ve developed to support the wider Liberty Global family of companies
Summary 18
Change is the only constantTransformation isn’t easyAutomate to keep upBuild Enterprise grade solutions
Any Questions?