Ensure Application Quality with Vendor Management Vigilance

Stephanie Moore

Vice President and Principal Analyst

Forrester Research, Inc

Ensure Application Quality with Vendor Management Vigilance

CAST Confidential 2

Speakers

Stephanie Moore

Vice President and Principal Analyst

Forrester Research, Inc

Lev Lesokhin

Vice President, Worldwide Marketing

CAST

© 2011 Forrester Research, Inc. Reproduction Prohibited 3 © 2009 Forrester Research, Inc. Reproduction Prohibited

Transparency as an Antidote to Global Sourcing Risk

© 2011 Forrester Research, Inc. Reproduction Prohibited 4

Technology is more critical to the business than ever before; it’s a market maker, not just an enabler.

Bottom Line: IT has never been so important to the business, and

yet due to years of cost cutting and “optimization”, so ill-

equipped to support business needs

64% believe:

“Technology is

CENTRAL to

how we

differentiate

ourselves”

69% believe:

“Technology is a

CORE

COMPONENT

of our products

or services”

87% believe:

“Technology is

CRITICAL to

serve

and support our

customers”


Software is everywhere and growing more complex

Product Lines of code

Order entry system 1.7 million

F-22 Raptor 1.7 million

Space Shuttle 2 million

Microsoft Word 2 million (27,000 in first release)

F-35 Joint Strike Fighter 5.7 million

Airline reservation system 6.1 million

Boeing 787 Dreamliner 6.5 million

S Class Mercedes-Benz radio with

navigation system

20 million

http://www.google.com/imgres?imgurl=http://www.icars.sg/wp-content/uploads/2009/04/2010-mercedes-s-class-leaked-9.jpg&imgrefurl=http://www.icars.sg/2009/3728/2010-mercedes-benz-s-class-leaked-brochure/&usg=__D2BZjDw7dLOxZrPEu_iaGuJAPM8=&h=333&w=500&sz=62&hl=en&start=7&itbs=1&tbnid=L6ZqXNVLwjRThM:&tbnh=87&tbnw=130&prev=/images?q=mercedes+s+class+2010&hl=en&gbv=2&tbs=isch:1

http://wallpaper-s.org/24_~_F-35_JSF_(Joint_Strike_Fighter),_UK_Royal_Navy.htm


65% The business

says says: “I buy

technology and

technology

services without

involving IT.”


“I download applications

to do my job because my

company doesn’t provide

them.”

34%


The gap between business requirements and IT’s ability to deliver is widening

How are IT and the business

filling this the gap?

In many cases, through third

party IT service providers


Understanding the trends in global outsourcing


IT Industry Perspective: Key Trends Today

Outsourcing is viewed as an essential tool for companies to reduce costs,

improve service and help the business innovate

The large scale, top tier IT services and outsourcing providers begin to look

the same

– Whether you work with Accenture, IBM, Infosys or TCS, much of your

applications work will be performed offshore

While offshore outsourcing continues to grow, satisfaction levels decrease

Managed outcome services are recognized as essential to optimizing

outsourcing relationships –from a cost and quality perspective. But, lack of

transparency into code quality and structure is a challenge for many

customers

Domestic outsourcing emerges due to a productivity-price trade off.

Sometimes it is faster and cheaper to outsource locally due to the need for

contextual, not just technology, understanding.


Global outsourcing: India is the destination of choice

Market is mature and large, cost is low and quality is high

However, quality is on the decline (C-Player and “fresher” syndrome)

Escalating costs are narrowing gap relative to US costs

Wage inflation, COLAs

Cost of gaining requisite contextual knowledge

Rework often required once code is repatriated to US

High employee turnover due to constrained labor market

Higher onsite component required for communications and context

Increased difficulty and expense in landing offshore resources onsite

using H1B and other visas

Communications challenges

Time zone challenges

Difficulty in performing early lifecycle work where business, process and

vertical knowledge is required


Managing your vendors for success


17%

40% 41%

48%

0%

10%

20%

30%

40%

50%

60%

Quality of work done is poor

2007 2008 2009 2010

Customers’ outsourcing frustration is rising

Which are your firm's biggest challenges with your existing outsourcing relationships?

POLL QUESTION 1

What types of business risks do you most closely associate with product quality?

Increased cost 28%

Slower time to market

22% Loss of revenue

12%

Loss of customers 18%

Damage to company brand

20%


So, vigilance required in managing global outsourcing vendors

Quality standards and certifications: are they real and are they real for

your activity?

Resource qualifications and certifications

– How proficient are the teams you engage?

– Do they have necessary certifications?

– Do they really know how to code software?

Continuous evaluation of vendors’ HR and recruiting and retention

capability

Continuous evaluation of vendors’ training capability

Innovation capability: show me your vertical and domain expertise


The importance of vendor training investments


Vigilance required in managing global outsourcing vendors (cont.)

Productivity vs. price trade offs

– How to measure

Evaluating deliverables from managed service engagements

– Is the code maintainable gong forward?

– Is it efficiently built?

Vendor viability assessments a must

– This is much more difficult for small and privately held vendors

– Clients have to be clever in researching their vendors and their vendors’

performance and track record.

– Employment websites

– Social media

POLL QUESTION 2

Do your outsourcing/customer SLAs include metrics on the technical/structural quality of code being delivered?

Yes 19%

No 37%

Sometimes 25%

I don't know 19%


Managed Outcomes


Manged services model: benefits and risks

Client buys

people

Client buys an

outcome directly

And

manages

them to

deliver an

outcome

Staff Augmentation Model

Managed Services Model


Why is a managed services model better?

Better savings including a productivity boost beyond labor cost savings

– Potentially a one-time benefit, however according to case history examples,

savings can range from 10% to 25%

Movement of risk from client to vendor

Greater potential for process improvement

– Allows your IT services vendor to operate at its own level of process maturity

Improved foundation for knowledge retention

– Knowledge does not walk out the door with the “person”

Improved knowledge transfer in terms of time and money expenditure

– Vendor is responsible for its own staffing

But, how do you ensure your black box is filled with good things?


Keys to Success for Managed Services

A mechanism for auditing code quality and its adherence to standards.

– This transparency is critical to both the client and the vendor

Change in the way suppliers are managed

– Managing vendors instead of programmers

– Having an effective governance model which includes vendor participation

– Ability to review vendor performance with links to penalties or rewards grounded

in the Master Services Agreements (MSA) and Service Level Agreements (SLAs)

Change in the way work is organized for outsourcing.

– A portfolio-driven view of potential work to be outsourced and the ability to batch

work where possible

Ability to envision service level requirements and outcomes and document

them in statements of work


Transitioning to managed outcome relationships requires an SLA “starter set”

Start with SLAs that are simple to track:

– Number of incidents, enhancements, patches

– Availability

– Quality of support/responsiveness and resolution

– First time right

Allow for a “true-up” with the supplier in a mutually agreeable time frame.

– Allow time to build up historical data patterns.

Create a path forward to more demanding SLAs for the future – it seems

like magic.

– Quality improvements

– Productivity improvements

– “Innovation”

Because you can’t manage what you don’t measure

May 2012

Actionable Application Intelligence for IT Executives

CAST Confidential

Market leader in Software Analysis & Measurement

25

Ambitious

Mission

Rock Solid

Foundation

Market Leader

Introduce fact-based transparency into application development and

sourcing to transform it into a management discipline

Broad market presence in Europe, North America and India

Strongly endorsed by software industry gurus and long term investors

Over $100 million of investment in R&D, driven by top talent in

computer science and software engineering

Pioneer and recognized market leader since 1999

CAST Research Labs, the world’s largest R&D facility dedicated to the

science of software analysis & measurement (SAM)

“CAST metrics have become the de facto standard for measuring the quality

and productivity of application services.” – Helen Huntley, Research VP, Gartner

CAST Confidential

Product Quality Metrics

Most enterprises measure everything but

the product delivered to the business

CAST measures the product itself

Robustness Performance Security Changeability Transferability Size

Planning Estimation Scheduling Time Tracking Cost Tracking

Product Process

Time & Duration

Effort & Budget Function &

Scope

Quality

&Size

Requirements Earned Value User Acceptance Usability

CAST AIP

CAST Confidential

Driving software measurement in the ADM industry

27

Key Influencers Recognize CAST

250 Global Leaders Rely on CAST

Institutions Engage CAST SIs Resell CAST SIs Use/Resell CAST

Top technology

First in business IT

Biggest benchmark DB

CAST Confidential

CAST dashboards, reports & benchmarks

28

CAST Highlight

Portfolio Analysis Size

Complexity

Risk

Technical debt estimation

Zero Deployment No centralized source

code collection

Portal results

Full analysis report

CAST Application Intelligence Platform

Risk Drivers

Robustness

Performance

Security

Cost Drivers

Transferability

Changeability

Alerts, trending, root cause analysis

Discovery Portal

Automated

App Blueprint

Discover, modernize

and change

applications

Function Point Manager

• Automated

FP counts

• Technical

Sizing

• Effort

Estimation

Function Point Changes Due to a Sequence of Change Requests

0

5

10

15

20

25

30

35

40

0 50 100 150 200

Cumulative Effort (Staff Hours)

# F

un

cti

on

Po

ints

1 5 2 3 4

Benchmarking Services

Compare to industry

business process

and technology

CAST Confidential

Arc

hit

ec

ture

Co

mp

lia

nce

Enterprise-grade analysis requires a 3-tier approach

29

Intra-technology architecture

Intra-layer dependencies

Module complexity & cohesion

Design & structure

Inter-program invocation

Security Vulnerabilities

Module Level

Integration quality

Architectural compliance

Risk propagation

simulation

Application security

Resiliency checks

Transaction integrity

Function point & EFP

measurement

Effort estimation

Data access control

SDK versioning

Calibration across

technologies

System Level

Data Flow Transaction Risk

Code style & layout

Expression complexity

Code documentation

Class or program design

Basic coding standards

Program Level

Propagation Risk

Java

JSP

EJB

PL/SQL

ASP.NET

Oracle

SQL

Server

DB2

T/SQL

Hibernate

Spring

Struts .NET

C# VB

COBOL

C++

COBOL

Sybase IMS

Messaging

Java Web

Services

APIs

1

2

3

CAST Confidential

Project #n Project #3

Project #2

Example: Managing multiple ADM vendors at global telecom company

230,000 employees, almost $100 billion revenue, 40 million customer accounts

Billing & OSS Solutions

– 120 billion call records and 1 billion invoices per year

– Also, SAP, Siebel, all front end apps that power e-commerce sites

Running one or two times per quarterly release

– SLAs in contracts based on CAST

– Aggregation of CAST metrics into C-level management dashboards

Software analysis

Vendor facility

Team #1

Team #2

Team #3

Team #4

Neutral & independent vendor unit is running the CAST AI Center for the customer

Project #1 • 150+ applications • 4 apps silos, with 50+ CAST consumers

plus management in each

CAST Dashboard

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1.1 2.0 2.5

Vendor A

Vendor B

Vendor C

Vendor A

CAST Confidential 31

Structural quality impact at a major services brand

Measured impact in a complex enhancement-heavy environment

304 222 196

385 401

231 198 242 279

167 112

258 274

149 140 245

188

61 56 78 97 62 57 81 75 87 40

260

181 167

225 265

220

151

195

232

154

150

295

385

228 163

223

186

148 84

121 136

96 38 11 10 8 11

0

100

200

300

400

500

600

700

R1

R1

.1

R1

.2 R2

R2

.1 R3

R3

.1 R4

R5

R6

R7

R7

.1 R8

R9

R9

.1

R9

.2

R1

0

R1

0.1

R1

0.2

R1

0.3

R1

1

R1

1.1

R1

1.2

R1

1.3

R1

2

R1

3

R1

4E

Code No RC Non Code Projected Count

Str

uctu

ral

qu

ality

S

yste

m t

est

defe

cts

Trend line

Before CAST implementation

0

500

1000

1500

2000

2500

3000

3500

R1

R1

.1

R1

.2 R2

R2

.1 R3

R3

.1 R4

R5

R6

R7

R7

.1 R8

R9

R9

.1

R9

.2

R1

0

R1

0.1

R1

0.2

R1

0.3

R1

1

R1

1.1

R1

1.2

R1

1.3

R1

2

R1

3

R1

4E

Order Management Inventory Management Billing Customer Service

New critical violations

CAST Analysis starting point

CLIENT STUDY OVER 24 MONTHS

CAST Confidential

CLIENT ADM VENDOR

32

Structural quality metrics can help both clients and vendors

Takes the guesswork out of

quality

Improved code acceptance criteria

Objective performance

measurements

Take control of accountability to

the business

Improve collaboration between

multisourcing parties

Justify the quality of work

Benchmark applications against

industry and clients applications

Better release management

Increase productivity and improve

bottom line

Remove risk from subcontracting

decisions

Improve customer satisfaction

Improve collaboration between

multisourcing parties

Transparency between customer and vendor encourages a healthier

and more risk-free relationship

CAST Confidential

Contact Information

Lev Lesokhin Stephanie Moore

[email protected] [email protected]

www.castsoftware.com www.forrester.com

blog.castsoftware.com http://blogs.forrester.com/

@OnQuality @forrester

slideshare.net/castsoftware

Ensure Application Quality with Vendor Management Vigilance

Business

Transcript of Ensure Application Quality with Vendor Management Vigilance