Lessons Learned from Large Scale Adoption of DevOps for IBM z Systems Software

Lessons Learned from Large-Scale Adoption of DevOps for IBM z Systems Software

Per KrollDirector Offerings Mgmt, Development & Support DevOps for Enterprise Systems

1

My journey…

Creator and Chief Architect

Extended Core Team Transforming +20,000 developers Practice library for agility@scale Deployment assets… Initiator and Project lead

Product Manager and Development Manager

Rational Unified Process

Development and Support ExecutiveDevOps for Enterprise Systems

Discussion Topics

• Our context

• A journey – all things agile

• Running an agile business: Offerings Management and IBM Design Thinking

• Adopting DevOps

• Driving large-scale organizational change

• Parting thoughts

2

Why transform

• Innovation – exceed customer expectations and beat competition

• Delight customers through an iconic user experience

• Quality

• Speed

• Cost

• Job satisfaction

3

Complexities we face as a product organizationCommon in the industry• Extremely distributed• Varying need for business agility and investment levels• Mixture of tools, including homegrown• Dispersed cultures, many acquisitions

Specific for our product development organization• Multi platform (multiple OSs, DBs, App Server, browsers, ….) – 100s of permutations to test• Translations to dozens of languages• Legal and packaging constraints

Specific for mainframe development• Many commercial and open source tools do not support mainframes• Attitude: ‘Mainframe is different’, average age is higher 4

Discussion Topics

• Our context



• Adopting DevOps



5

Continuous Improvement: An Agile Non-negotiable

6

• There is no such thing as “100% Agile”

• Stop living with pain - leverage known practices to alleviate your pain• Automated build and deployment• Shift left testing: test early, test often• Loosely coupled architecture• ….

“If you can’t tell me you’re better this month than you were last month, don’t tell me you’re ‘Agile!’”

All things agile

UserExperience

Development

Business

All things agile

UserExperience

Development

Business

Design Thinking Offerings Management

DevOps

Innovation

Discussion Topics

• Our context



• Adopting DevOps



9

Problem: Business Process

• Our business process, IPD, was too heavy– A lot of overhead– Encouraged over specification– Discouraged early experimentation– Forced too early commitment to a specific solution– Expensive business change process

• Solution: Introduce Offerings Management Discipline– A lighter-weight business process – Centered around the principles for the LeanStartup

Offerings Management Discipline: Inspired by the Lean StartUpLight-weight artifacts, with focus on learning and pivoting

11

Learn and Pivot

Light-weight artifacts

Focus on markets and business outcomes

Design Thinking

Design Thinking

Understand

Explore

Prototype

Evaluate

HillsInvest for market outcomes

Sponsor UsersUnderstand the user experience

PlaybacksCollaborate. Align. Engage!

IBM is increasingly recognized for design leadership

Discussion Topics

• Our context



• Adopting DevOps



15

Adopting DevOps: 2015 Focus Areas

• Build and deployment automation: Fast and extremely high ROI– Weeks or months, not quarters or years

– Key products: Rational Team Concert Enterprise Edition, Jenkins, and Urban Code Deploy

• Test automation: Crucial, but labor intensive– Incremental adoption, targeted investments: default priority order

1. Build Verification Test – Automate most frequently run tests

2. New features – Stop digging a deeper hole…

3. Components with many defects – Immediate quality benefits

4. Components with low test coverage – Incrementally bite the bullet

– If tests never fail and component not updated => Do you need to automate?

– Key products: RDz zUnit, Rational Test Workbench, Application Delivery Intelligence

• Not a focus: Already mature areas – Collaborative Development: Rational Team Concert (RTC)

– Modern Development: Modern IDEs / Problem Determination tools (RDz / PD Tools) 16

• Cross discipline feature teams• Establishing rhythm, continuous

integration, definition of done

• Shift left testing

• Complex testing run continuously

• Automation part of development

• Anything done manually analyzed for elimination or automation

• Retrospectives, eliminate friction points

17

Plan Develop Develop Develop Test/Fix/Test4 weeks 4 weeks 4 weeks

Warmup Sprint 1 Sprint 2 End Game…

Develop Develop Test/FixTest

Plan

retrospectiveSprint 1 Sprint 2 End Game

4 weeks4 weeks

Quarterly Releases

Ship ready with each Sprint

Create a culture committed to automating everything…

Continuously Test Using Automation and VirtualizationExample: Collaborative Lifecycle Management Team

Example: Increasing Test Automation in IBM Rational Developer for z

Problems• Long release cycles for both new releases and service• Development process more waterfall than agile• Too much effort spent on manual testing

Progress made in 2015H2• Educate team (JUnit and Rational Functional Tester)• Ensure JUnits are added for all new capabilities• Optimize BVT suite to run in 15 min or less, triggered by a build. Publish results.• Analyze components to understand where to invest: Quality issues, test coverage, churn• Apply swat team for rapid increase in test automation• Adopt strategic automation tooling: UrbanCode Deploy• Incorporate data-driven frameworks to automate complex areas

18

Example: Increasing Test Automation in IBM Rational Developer for z

Results so far (work in progress)• Automated test coverage increased from 20% to 50%• Daily BVT uncovered defects earlier in the cycle, resulting in quality improvement• Reduced regression testing for each deliverable from 3 weeks to 2 weeks• Shortened time to test while growing number of test cases and improving quality• Test team able to handle more deliverables and cover more integrations

19

Executes 10000s of Junits & security scans

Develop Test (continuous) Interactive Testing

Usability Test

Security Pen testing

Unit Test

Production

Jazz.net

System Test

Performance Test Integration Test

Function Test

Build

Production-Like Environment Production-Like Environment

Global dev/test/ops team ~100 daily commits/deploys using our tools to build our tools

191,000 jazz.net community members, 50K sandbox instances, >105K page views

DeploymentsDeploy Golden Topology

IBM BluemixDevOps Services

20

Developers launch test from their

workspace Rapid install and smoke TestsFunctional and systems tests on Golden Topologies

BVT

“Simply put, things always had to be in a production-ready state: if you wrote it, you darn well had to be there to get it running!”

- Mike Miller, Co-founder and Chief Scientist at Cloudant,an IBM Company

Continuous Integration and Test

Build a Delivery Pipeline Leveraging Tools-As-a-Service Example: IBM Collaborative Lifecycle Management Team

Continuous Delivery Pipeline DashboardExample: IBM Collaborative Lifecycle Management Team

21

• Automated quality gates• Single view of quality• Ability to quickly drill down on problems• View the live dashboard at: jazz.net

IBM CICS Tools (Ant / RTC Build)

IBM CICS Explorer (Ant / RTC Build)

Example: IBM z Systems DevOps TransformationKey Challenges (2015, Q2):• Products get built, tested & delivered

separately• Inconsistent build technologies &

infrastructure• Lack of synergy in driving product

compatibility• Lots of manual interventions (e.g.

testing, deployment)

Impact to customers:• Difficult for customers to figure out what

version of which products work together• Late discovery of defects, especially

related to cross-product integrations• Some products may not work well

together 22

DevIESCEXDB2IES Dev SCM Build Test

Feedback

IBM Rational Developer for z Systems (Eclipse PDE Build)

IBM Problem Determination Tools (Maven / Jenkins Build)

IBM z/OS Explorer (Ant / RTC Build)

IM Download site(RDz)

IM Download site(PD Tools)

Eclipse p2 Update Site (CICS Explorer)

Eclipse p2 Update Site (z/OS Explorer)

IM Download site(CICS Tools)

“Internal” DevOps (Phase 1) delivers IBM ADF for z Systems

23

Eclipse 4.4 Luna

z/OS Explorer V3.0 Aqua

Rational Developer for z Systems

Application Delivery Foundation for z SystemsApp Performance

AnalyzerDebug Tool

File Manager

Fault Analyzer

CICS Explorer

z/OS Connect EE

RTC

CICS ToolsSingle IM

Composite Site

Single Eclipse p2 Update

Site

IBM Explorer for z/OS Aqua Release Highlights

A single “Release Train” delivery pipeline (more than a dozen products on z/OS Explorer) Common build technology & infrastructure Common software stack & dependencies Improved cumulative build time

Improved integrated testing across products Improved overall product compatibility & quality A single download site for IM and Eclipse p2 installs Improved user experience with a single “MainframeDev” site on IBM DeveloperWorks for all products

IBM Application Delivery Foundation for z Systems V1.2

24

• A comprehensive set of z/OS tools with a modern, integrated & consistent user experience

• Best of breed capabilities in a single bundle enables the delivery of high quality z/OS applications quickly & efficiently

– Accelerates development, test & maintenance of z/OS applications

– Increases debugging efficiencies & reduces application development lifecycle

– Helps isolate system constraints quickly & maximize application response time

– Expedites the identification & analysis of system and application failures

– Simplifies manipulation of data in data stores such as DB2, IMS, CICS, MQ, data sets & HFS files

• A single & attractive pricing model for the complete bundle

• Accelerates the DevOps transformation of z/OS application delivery

RDz

DT

FA

APA

FM

IBM z Systems DevOps Transformation – Q2 2016 (Phase 2)

Phase 1 – Dec 2015 Phase 2 – Q2 2016

Theme – “Extend DevOps through automation”• Extend existing Continuous Integration pipeline to support Test & Deploy• On-board & streamline product delivery to be on the common pipeline • Increase focus & investment on test case automation (e.g. BVT, smoke test, IVT)• Integrate with IBM UrbanCode Deploy to orchestrate test automation & deployment

Expected Outcome & ROI• Reduce TCO (total cost of ownership) by replacing manual work with automation• Improve speed of product delivery across our portfolio• Further improve product quality & compatibility through continuous/automated testing

Single Eclipse p2 Update Site

Single IM Composite Site

ADF: Business Value of our DevOps Transformation

26

Our adoption of DevOps improves speed, reduces waste & enhances quality

Improvement Area Q1, 2015 Q2, 2016 Improvement Benefits

Release frequency 12‐18 months 1 month 90‐95%

‐ Accelerates innovation‐ Improves Time‐To‐Market (TTM)‐ Continually drive higher product quality‐ Enables continuous delivery

Cumulative build time ‐‐ ‐‐ 30%‐ Earlier detection of issues‐ Shortens development lifecycle

WASTE REDUCTION Build & delivery process

5 different processes

1 consolidated process 80%

‐ Eliminates duplications of work‐ Deflects customer reported issues

Automation: BVT & Integration Test 0% 100% N/A‐ Earlier detection of issues‐ Enables continuous testing & delivery

Automation: Deployment 0% 100% N/A‐ Reduced labour‐ Enables continuous testing & delivery

Dependency Management Ad‐hocCommon repository N/A

‐ Earlier detection of issues‐ Enhances product compatibility

SPEED

QUALITY

Discussion Topics

• Our context



• Adopting DevOps



27

Driving Large-Scale Organizational Change

Key factors • Provide clarity in objectives• Drive the right culture• Gain executive buy-in• Establish bottoms-up communities• Encourage self-assessments• Drive tools standardization while allowing appropriate levels of freedom• Automate everything: Build, Test, Deploy• Govern through transparency: Aid, remove roadblocks, and establish accountability

28

Becoming agile is a journey. Enjoy your journey.

Drive the right culture: Values and PrinciplesHelp install values through social contracts

29

Principles

Make self-organization work: Clarity of Outcome

30

Establish bottom-up communities

31

+10,000 members

Several events every week

Easy access to experts

Learn from, teach and interact with people in your area of interest

Line of Business

Rational Doors Next Generation (RDNG)Rational Team Concert (RTC)Request for Enhancement (RFE)

Rational Quality Manager (RQM)Rational Test Workbench (RTW)Rational Test Virtualization Server (RTVS)jUnit Automation Tool (JAT), JazzSM-Dash

Jenkins

RTC EE, RTC, ANT, Jenkins

Rational Team Concert (RTC)Rational Developer for the Enterprise

(RDz, RDi, RDp, RD A&L)Git

Pipeline automationRational Team Concert (RTC)Jenkins

UrbanCode Deploy (UCD)

zService, SPoRT

Golden Topology: Driving tools standardizationwhile allowing appropriate levels of freedom

15

Self Assessments and Roadmaps to Address Gaps

Project Management

Project Tracking Design & Design Thinking

Builds Technical Debt Automated Testing

WholeTeams

Provisioning Automated Deployments

Monitoring of Environments

Collaboration Customer Focus Monitoring Customer Usage

1 1 1.5 1.25 1.5 3 0.5 5 6 2 0 1.75 0.5PLANNING BASIC SOFTWARE ENGINEERING PRACTICES DEPLOYMENTS TO TEST AND

PRODUCTION (as applicable)DEV & OPS WORKING TOGETHER

CUSTOMERS

3.5 11.25 8 0 2.25

Area: XXXContact: XXXModel assessment results – as of 04/09/2015

No actions PlannedFuture actions TBD

Areas being addressed

Area Summary of actions Target date Target Metric

Automated Testing • Building test center of competency • Adopting test driven development• Created detailed action deck already being executed on

• 06/30/15• 11/01/15

1

Automated Deployments • Investigate UCD usage• Utilize for new offerings• Adopt for existing products and include provisioning

• 07/01/15• 12/1/15• 06/30/16

3

Dependency • Requires ability to staff Mexico team

Risks • Continued escalations and test fixes taking resource from these efforts

Issues • None

Effective governance through transparency: Aid, remove roadblocks, and establish accountability

Central dashboards providing visibility into• Improvement actions• Metrics• InhibitorsView by organization• Product• Area• ExecutiveUsed to • Identify common problems so we can address them strategically• Help teams to structure their improvement efforts• Provide accountability

34

35

The wrong metrics can incent negative behavior

Leverage Outcome and In-Process Metrics

Outcome Metrics• Determines whether you achieve your

end goals or not• Responds slowly to change• Examples from my organization:

– Delivery time of defect fix release– Delivery time of major release– Customer-reported quality issues– Customer satisfaction (survey)– Employee satisfaction (survey)

In-Process Metrics• Determines whether you are effectively

adopting the right practices• Responds rapidly to change• Examples from my organization:

– Code quality at sprint end– Point of customer feedback for release

(e.g. 20% into a project)– % of test automation– % of build verification test automation– % of regression test automation– Duration of full regression testing (manual

as well as automated)

36

Discussion Topics

• Our context



• Adopting DevOps



37

Driving Large-Scale Organizational ChangeWe are here to help!

• DevOps requires an end-to-end integrated tool set– We are the only vendor providing an end-to-end, cross platform DevOps solution

• Driving real change requires a cultural transformation– We have many experts and assets to help you in your transformation

• Start with our Deployment Project Office– Work with your IBM account team, and contact Jon Sayles ([email protected]) if

uncertain who to talk to– Example: We can offer ongoing advisory sessions between our and your executives and

senior leaders

38

Becoming agile is a journey. Enjoy your journey.

How to Get Started IBM DevOps Workshop

Wasted effortLegend:

Think Product-based

Agile

Automated

Collaborative

Optimizing

Morepredictable

Moretransparent

Morecontinuous

Process-based

Process-heavy

Manual

Silo-ed

Code/Test

Run

Manage

Inefficient LeanerLeaner and

Smarter

Productive effort

Workshop Objectives

– Define business drivers for DevOps, and see what is currently possible for your organization

– Identify existing or planned DevOps initiatives, and map them to the business drivers

– Determine the top inhibitors within the current software lifecycle

– Create an adoption roadmap for these DevOps practices, including milestones, time frames and known adoption challenges.

Overview

– No charge!– Led by IBM DevOps Solution Architects– Audience : For senior IT and LOB executives and

managers in application development and IT Operations

Sign up for a free DevOps Innovation & Optimzation

workshop

39

Notices and Disclaimers

40

Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.

41

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your

smartphone, laptop or conference kiosk.

Lessons Learned from Large Scale Adoption of DevOps for IBM z Systems Software

Software

Transcript of Lessons Learned from Large Scale Adoption of DevOps for IBM z Systems Software