1© CSS Corp | Confidential | www.csscorp.com© CSS Corp | Confidential | www.csscorp.com
Integrating effective testing & automation for successful adoption of DevOps
20th May, 2014SCQAA-Orange County, CA
2© CSS Corp | Confidential | www.csscorp.com
Authors
Anil Parambath
Vice President - Solution Engineering
PRabhu Meruga
Director - Solution Engineering
3© CSS Corp | Confidential | www.csscorp.com
Agenda
• What is DevOps-History, Evolution and Prominence
• Why DevOps now? Today’s Application development needs
• Deployment statistics of todays modern world
• Build Failures impact
• Development life cycle survey results
• Challenges – Collaboration, Release and Integration testing
• Drivers for DevOps
• Elements of DevOps
• Testing framework for DevOps
• Integrating continuous testing into DevOps – understand the differences
• how to integrate testing into DevOps? What are the tools available
• Difficulties, Barriers & Benefits
• Q&A
4© CSS Corp | Confidential | www.csscorp.com
DevOps – history, evolution and prominence
Development
(software engineering)
Technology Operations (IT teams)
Quality Assurance
(testing)DevOps
Started in Belgium as “DevOps”
days in 2009 by Patrick
• Cross departmental integration concept
• Idea • Culture• Practice
5© CSS Corp | Confidential | www.csscorp.com
Build, listen. Mash-up scenarios. Mobile first
Develop Fast. Succeed Fast
Ease of use. Avoid feature overload. Don’t replace Simplicity
Speed
Innovation
Quality, Performance, UsabilityCoverage
Why DevOps? Today’s dynamic application development needs
6© CSS Corp | Confidential | www.csscorp.com
Deployment challenges of modern world
• Social networking sites such as face book on average have minimum 2 builds per day and maximum 7 builds per day
• Open source code sharing firms like github.com have 500 builds/day and 100 deploys/day
7© CSS Corp | Confidential | www.csscorp.com
Development methodology survey results from Agile conference
Gartner: By 2015, DevOps will emerge as niche strategy employed by large cloud providers into a mainstream strategy for 20% of the global 2000 Organizations.
8© CSS Corp | Confidential | www.csscorp.com
Collaboration challenges between Dev, QA and Ops
Development completed
Deployment completed
Incorrect version deployed
Checked it again – it’s the right version
Oops – we picked the wrong code
Now we got the right one
Scen
ario
1Sc
enar
io 2 Request for new
environment setupPlease wait we are busy
Scen
ario
3
Request for new environment setup
I have daily buildCannot support daily build
Consistent build deployment needed
…and many other scenarios….
9© CSS Corp | Confidential | www.csscorp.com
Release flavors
Hourly Daily Weekly Fortnightly Monthly Quarterly Half Yearly Yearly
Business Critical systems-Social Media, Space, Defense, e-Commerce, financial
• Some of the changes in business critical systems are pushed on an hourly basis
• Release vary based on the criticality up to quarterly releases
Large scale ERP updates, Operating system patches, Anti virus updates, Embedded systems, traditional internal applications
• Some of the changes are very slow depending on the nature of business and components involved. Eg Hardware/embedded
10© CSS Corp | Confidential | www.csscorp.com
Integration testing becomes more complex
Dev1
Dev2
Dev3
Continuous Integration server
Submit Code branch
ERP
ch
ange
sPa
tch
u
pd
ates
Mo
bili
ty
enh
ance
me
nts
Deployment
Production Farm
QA Lab
Performance Lab
UAT Lab
Operations• Lack of controls in Parallel Dev methodology
• Heterogeneous channels – PC, laptop, Mobile
• Systems Integration – Mobile apps, ERP, Web, Client
• API testing ( too many bugs escape downstream)
• Lack of performance engineering & automation
• Multiple environments to setup and maintain• Huge number of production applications
monitoring• Lack of visibility into customer experience
11© CSS Corp | Confidential | www.csscorp.com
Drivers for DevOps implementation
My enterprise has waterfall modelWe are in agile mode
• Need for on the fly multiple environments creation and maintenance
• Code synchronization challenges across environments
• Business demanding frequent releases to catch up with Users demand
• Increased usage of mobile devices and applications
Dri
vers
12© CSS Corp | Confidential | www.csscorp.com
Elements of DevOps
Continuous Integration
Continuous Deployments
Test Driven Development
Continuous Automation testing
13© CSS Corp | Confidential | www.csscorp.com
DevOps can resolve the challenges and ensure quality of applications.
• Continuous integration tools- Jenkins, Selenium grid
• Automated build deployment process• Automation build
qualification(testing)
Developers
IT Operations
QA/Testing team
User Community
• Automated monitoring of the deploys
• Jump into action in case of deadlocks
• Automated daily builds into production• Faster release cycles for new features• No dependency on IT operations human
resources• Alert management systems
14© CSS Corp | Confidential | www.csscorp.com
With and Without DevOps – understand the differences
• Automated monitoring of the deploys leveraging DevOps tools
• Has visibility into application monitoring for better root cause analysis
• Reduces dependency on operations
• Very focused on development and coding
• Line of responsibility does not encourage cross skill set training
• Raises tickets to deploy code, monitor application logs, set up environment
• Very focused on waiting for the code deploy
• Line of responsibility does not encourage creating code snippets for any kind of automation
• Raises tickets to deploy code, set up environment
• Continuous automation testing for regression
• Has visibility into security testing and application monitoring
• On the fly environments creation for QA needs
• Very focused on operations alone
• No perspective on developing deployment scripts
• Low focus on validation and purpose of monitoring
• Takes time to create environments
• Motivated to create or design code for various support activities
• Environment creation and monitoring made easy
• Roll back, Patch updates – no dependency on Development teams
Complements additional skill set
Sharing of efforts and cross pollination DevOps
Without DevOps – Silo operations
Dev QA Operations
15© CSS Corp | Confidential | www.csscorp.com
Testing framework for DevOps implementation
Development(agile/iterative)
QA Testing(PC, Laptop & Mobile)
UAT Production
Build Failure
Build Pass
Defect Management Tool & process
QA Pass UAT Pass
Test Cases repository (Automated scripts) –leverage Selenium, Test link integration
Continuous integration coupled with continuous automation
16© CSS Corp | Confidential | www.csscorp.com
What will it look like after DevOps implementation?
Sprint 1 Sprint 2 Sprint N
Development
Functional AutomationNon Functional Automation
Automated Unit Testing
Automated Unit Testing
Automated Unit Testing
Functional AutomationNon Functional Automation
Functional AutomationNon Functional Automation
Continuous Automated regression testing
Continuous Automated integration testing
Release
…..
…..
………
17© CSS Corp | Confidential | www.csscorp.com
Tools for DevOps implementation
Deployment automation : Fai, KickStart, preseed, Cobbler
Configuration management : Puppet, Chef, cfengine, bcfg2
Build Automation :Jenkins, ant, Cruise control, Hudson
Test Automation : QTP, Autoit, Testlink, Test Complete, CA LISA Test
18© CSS Corp | Confidential | www.csscorp.com
How do I start Dev Ops implementation
Perks and Benefits – Starbucks cards etc
Join Developer, QA and Operations team members
Start small with a mini project, assess the success factors
19© CSS Corp | Confidential | www.csscorp.com
Difficulties & Barriers for implementing DevOps
Common Cultural barriers• 49 percent of respondents who
had no plans to implement DevOps identified a lack of manager buy-in as a blocker—this was the most frequently identified barrier for this group, followed by "lack of team buy-in" (38%).
• 48 percent of all respondents indicated that one of the biggest difficulties in implementing DevOps was that the value wasn't understood outside of their group.
Source: Puppetlabs
20© CSS Corp | Confidential | www.csscorp.com
Benefits of DevOps – What kind of metrics will help assess DevOps?
• Achieve 25 times more frequent code deployments
• Reduce build failures/outages by 60 times
Key Metrics• Number of Deployments• Number of Successful Builds• Change Deployment Lead time• Validation time per change• Quality of deployed change
21© CSS Corp | Confidential | www.csscorp.com
What is skill set needed for DevOps teams?
Pre-requisites: Cross pollination/training of Developers, IT and QA team
People Management Skills
Version Control Systems/Automation testing tools
22© CSS Corp | Confidential | www.csscorp.com
Where was/is DevOps implemented? Success stories!!!
US elections: Scott VanDenPlas, the head of the Obama's
DevOps group, put it"4Gb/s, 10k requests per second, 2,000 nodes, 3
datacenters, 180TB and 8.5 billion requests. Design, deploy, dismantle in 583 days to elect the President”
Source: http://gizmodo.com/5960790/inside-obamas-team-techExtract from Keynote, Velocity Conference, Santa Clara, Jun 16, 2011 by Jenkins, Jon;
Amazon AWS: 10,000 maximum deployments/hour; Meantime between deployments: 11.6 seconds;Outages due to build failure: 0.0001%;
23© CSS Corp | Confidential | www.csscorp.com
Big Picture – in simple terms
Each man, in isolation, examines a different area of the elephant — the leg, the tusk, the tail, and so on. When they compare observations, they discover that their conclusions are completely different. Only when they collaborate and bring their ideas together do they begin to understand the full picture.Source: BMC.com
24© CSS Corp | Confidential | www.csscorp.com
FAQs
1)Is there a benchmark of team size to start implementing DevOps?
2)Is Agile same as DevOps?3)Does DevOps mean Developers learning
testing and operations skill set and vice versa?
4)What will happen to the core skill set of the team members?
5)How is application quality and time to market improving through DevOps?
25© CSS Corp | Confidential | www.csscorp.com
Q&A
26© CSS Corp | Confidential | www.csscorp.com
© CSS CorpThe information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
Thank You
27© CSS Corp | Confidential | www.csscorp.com
Upcoming Webinar from CSS Corphttps://www.csscorp.com/events/are-you-ready-mobile-revolution