Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.”...
Transcript of Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.”...
![Page 1: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/1.jpg)
0© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Gareth Lynn
SOASTA
Webinar
![Page 2: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/2.jpg)
1© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Rapid Delivery
Agile development
Continuous
delivery
DevOps principles
Mobile and Web
Online everything
Mobile commerce
Legacy modernization
Complexit
y
Composite apps
3rd party
services
API proliferation
Unprecedented
Scale
100’s to millions
Local to global
100%
availability
![Page 3: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/3.jpg)
2© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer/Tester/Ops Collaboration
![Page 4: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/4.jpg)
3© 2014 SOASTA CONFIDENTIAL - All rights reserved.
![Page 5: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/5.jpg)
4© 2014 SOASTA CONFIDENTIAL - All rights reserved.
50,000 Apps 1,000,000 Apps 1,500,000 Apps
![Page 6: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/6.jpg)
5© 2014 SOASTA CONFIDENTIAL - All rights reserved.
![Page 7: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/7.jpg)
6© 2014 SOASTA CONFIDENTIAL - All rights reserved.
“We didn’t have enough
testing for a very
complicated project”
US Health and Human Services
Secretary, Kathleen Sebelius
![Page 8: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/8.jpg)
7© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Mobile Test
Automation
Performanc
e Testing
Real Test
Devices
Solid
Integration
![Page 9: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/9.jpg)
8© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Does it work as
designed?
Does it perform
under extreme
circumstances?
Approve materials Real Drivers
What happens
in actual
conditions?
Assembly Line
![Page 10: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/10.jpg)
9© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Functional
validation
Performance,
reliability,
scalability tests
Real Users/Real Devices
Build
What happens
in actual
conditions?
Unit-tested code
![Page 11: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/11.jpg)
10© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Rapid & accurate test development
o Reliable, maintainable test automation
o Stability across mobile OS platforms
o Device performance measurement
o Real-Time feedback via Continuous Integration
![Page 12: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/12.jpg)
11© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Fully managed device cloud installed
behind the firewall
o Real-time device inventory, status and
availability
o Online access for all your teams
o 24x7, secure, remote access to devices
o Manual & Automated functional test
o Remotely access to selected mobile
device and any mobile app
• Use devices for manual testing and monitoring
• Remote screen-share devices for collaboration
on design or issues
![Page 13: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/13.jpg)
12© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Dedicated Public Device Cloud
o Access to the latest market leading
devices (iOS/Android)
o Automatic installation, maintenance
and upgrade
o Hosted devices in a secure data
center
o Stock devices, not jail
broken/rooted
o Seamless integration with
TouchTest and CI environments
o API available to install apps,
reboot/wake-up devices, launch
testsPerfect for 24x7 Automated Mobile Regression Testing
![Page 14: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/14.jpg)
13© 2014 SOASTA CONFIDENTIAL - All rights reserved.
s
Memory leaks
Test Scale
Sta
ge / T
eam
Dev
& Test
Release
& Deploy
Network
& OpsCDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab Staging Production (100% +++)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Cloud Testing at All StagesPerformance validation from development to production
Continuous tuning for full-scale…and well beyond
![Page 15: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/15.jpg)
14© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
SOASTA
Global
Test Cloud
Access
for all
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
“Conductor”
CDNCredit
Card
Web
analytic
s
Other
3rd pty
svcs
Chicago
(Rackspace)
![Page 16: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/16.jpg)
15© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Manage to a
performance
baseline
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual
![Page 17: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/17.jpg)
16© 2014 SOASTA CONFIDENTIAL - All rights reserved.
![Page 18: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/18.jpg)
17© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Ut
Unit Testing
Pe
Performance
Testing
Fu
Functional
Testing
Development Testing Operations
TIME
Test
Dev Ops
Traditional Linear Approach
Continuous Testing Approach
![Page 19: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/19.jpg)
18© 2014 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Customers
“4 months of manual tests now run in 1 hour, overnight.”
“25% reduction in time to market for production
rollouts.”
Steven Winter, Director of Quality
FIS Mobile – Mobile banking platform to over 1000 banks
![Page 20: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/20.jpg)
19© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o 250 hours of manual tests running in just over 1
hour every night
o Regression Tests not run until feature complete
now run nightly – 4 months to overnight
o Fixing bugs earlier – Detailed test results for
developers
o 25% Reduction time to market
o Reduced field failures
![Page 21: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/21.jpg)
20© 2014 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Customers
“TouchTest has been critical in improving both the speed to deliver
and the quality of our mobile apps.”
“We decreased our test cycles from three weeks to a couple of days.”
Thierry Chassaing, SVP of Product & Engineering.
ItsOn - the leader in Mobile Smart Services.
![Page 22: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/22.jpg)
21© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
![Page 23: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/23.jpg)
22© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
![Page 24: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/24.jpg)
23© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
![Page 25: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/25.jpg)
24© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
![Page 26: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/26.jpg)
25© 2014 SOASTA CONFIDENTIAL - All rights reserved.
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
![Page 27: Gareth Lynn SOASTA...“We decreased our test cycles from three weeks to a couple of days.” Thierry Chassaing, SVP of Product & Engineering. ItsOn - the leader in Mobile Smart Services](https://reader034.fdocuments.us/reader034/viewer/2022051912/6002a477e71d3b7cf33eb172/html5/thumbnails/27.jpg)
www.SOASTA.com