1

Paper 36 Presentation

Quality & Risk Management Challengeswhen acquiring enterprise systems

Ying Ki Kwong, PhD, PMPPhilip Lew, PhD, PMP

October 8, 2018

2

Presenting Today

Ying Ki KwongPNSQC speaker (2008, 2016, and 2018)

� Statewide QA Program Manager

Office of the State CIO, State of Oregon� Relevant specialties and passions

o Quality & risk management

o Enterprise IT project managemento Complex systems and complexity

o Volunteering for local nonprofits and travel

3

Presenting Today

Philip LewPNSQC Board Member and Program Chair

� CEO, XBOSoft

� Relevant specialties and passionso Software quality process, evaluation,

measurement and improvement

o Software quality in use / UX design

o Mobile User Experience and usabilityo Cycling and travel

4

Alternate title for this paper…

Inconvenient Truth:SDLC (including Agile) is one facet of quality

when acquiring enterprise systems

5

Characteristics of projects expected to benefit fro m Agile

LowExternal dependency *

AvailableProcesses & tools to support Agile *

HighStaff knowledge / skills in Agile

SmallProject team size *

Co-locatedTeam Location *

High

Short

High

Low

Project Characteristic

Requirement stability *

Product novelty or innovation

Time to delivery

Demand for work product visibility or user involvement

* Challenge : Agile or not, major enterprise projects often invol ves unexpectedly volatile requirements, large cross-fun ctional teams in different cities, external dependencies, and low maturity of processes & tools to support iterative development.

66

Characteristics of software contractors / system in tegrators

Prime contractor perspective on project quality

Profitability As high as possible

Meet business needs • doesn’t matter if not in contract

• matters but may cost extra

Data conversion convert cleansed data, but happy to redo at an extra cost

System integration testing (SIT)

• best if SIT finishes quickly so customer can start UAT• legacy systems not in scope

Test Data no problem with using converted data – “saves time / more realistic”

Defect severity level during UAT

better if defects have lower severity level – “acceptance soonest good”

* Challenge : Agile or not, the prime contractor and the acquirin g organization have different motivations and so different perspec tives when come to quality & risks.

7

Context of a major enterprise IT projectfrom the perspective of the acquiring organization

8

� Business Case� Business model or operational paradigm changes (if any)� Business drivers and high level business requiremen ts� Solution approaches (relative cost vs benefit vs risks)� Solution approach chosen & key success factors

� Available solution approaches� Often built around a base system (COTS or transfer) with

customization� “Gaps” as a measure of project complexity� Custom system as solution approach of last resort

� Costs� Life cycle cost important

Major enterprise IT projects is an investment

99

Solution Approach - Top 5 IT Projects in Oregon Stat e Government

$ 20 M

$ 28 M

$ 90 M

$130 M

$335 M

Budget*

COTS with customization

COTS with customization

COTS with customization

Transfer with customization

Transfer with customization

Solution ApproachAgency / Project

Dept. of Human Services

Dept. of Justice

Dept. of Transportation

Health Authority

Dept. of Administrative Services

� Most enterprise projects adopt a solution approach based on customizing a base system (COTS or transfer).

� Minimally viable product (MVP) has expected cost th at is a substantial fraction of overall budget, partly due to base system cost.

� Gap analysis as crucial measures of project complex ity• “as is” vs “to be” operational paradigms• “as is” vs “to be” business process • base system vs “to be” system features-functions

* Budget does not include operations & maintenance costs.

10

� Capital costs� Prime contractor / system integrator� Supporting contractors (e.g. PM, analysts, and test ers)� Internal Staff� Hardware, software licenses, networking & telecom

� Ongoing costs� Operational support & maintenance - contracted� Operational support & maintenance - internal� Financing or debt services

� “Technical debt” that reduces net benefit� Non-optimal business functions or processes � Non-optimal technical architecture � Software defects

Life Cycle Cost

11

� General scope� Enterprise specific features-functions not in base system� Interfaces or integration with existing or legacy s ystems� Conversion of data in systems being replaced

� Nature of technical work� Configuration, Scripting, Custom code� Systems integration

� Important considerations� Cost & Time of customization vs. expected benefits� Complexity and deviation from base system models � Support & Maintenance

� Bifurcation from base system releases� Ongoing software upgrades and patching� Technical environments (e.g. dev, test, prod, etc.)

Customization

12

Requirementsincluding user stories, use cases, features…

13

Agile or not, are requirements knowable / discovera ble?

Cost

Scope/QualityRequirements

BusinessCase

OrganizationalDynamics (OD)Considerations

Normal ProjectManagement

Resources

Schedule

SystemsEngineering

Considerations

For “normal” projects, yes…• Stable enterprise architecture, dominant design, or maturity model• Stable business case, with Requirements less influe nced by OD• “project triangle” – mainly system engineering conside rations

For projects with organization paradigm shifts, may be…• Uncertain enterprise architecture, dominant design, or maturity model• Uncertain business case, with Requirements influenc ed by OD• “project triangle” – not just system engineering consi derations

14

Agile or not, are requirements reliable / stable?

Cost

Scope/QualityRequirements

BusinessCase

OrganizationalDynamics (OD)Considerations

Normal ProjectManagement

Resources

Schedule

SystemsEngineering

Considerations

• May be. Note risks associated with subject matter e xperts…• Schedule – vital to business operations and may not be available

when needed• Cost – may not know tradeoffs between cost / risk of customization

vs. possible benefits• Subject matter expertise – experts of “as is” business process / rules,

but not necessarily expert of “to be” business proces s / rules• Authority – may not be empowered, e.g. change staff roles & duties• Objectivity – may be resistant to change or level of change

15

Organizational Readiness

16

Prime contractor roles in enterprise IT projects• Focus on core competence

• Insufficient resources

• Lack of skills…

BusinessTransition

SystemsIntegration

BusinessAnalysis

Support &

MaintainSystems

Design&

DevelopSoftware

DataConversion

Planning&

ProjectMgt.

Training

SystemAnalysis

� Is the organization ready to bring on a prime contr actor?

17

Internal Project

Manager

ProjectManagement

Team

Business Analysis

Team

SystemAnalysis

Team

ProjectManagement

ContractorBA Contractor

SAContractor

TestingTeam

TestingContractor

ProjectSteering

CommitteeIndependent QA

ContractorPrime Contractor

Executive

Prime ContractorProject Manager

Prime Contractor& Subcontractors

Teams

Internal Project Management Team

Agile or not, is the organization ready?

� Project charter, updated business case, plans, and RFP in good shape?� Lessons learned in similar enterprises?� Familiarity / training in contract administration, SDLC, risk management, organizational change management, and related skills ?

� Governance and internal PM team.

18

Contracting model

19

Contractual Considerations

Procurement Model • Time & Materials• Deliverables based – fixed price or max not-to-excee d price• Hybrid

Statement of Work (SOW)• Acceptance criteria – often source of disputes• Schedule slack to enable refinements of user storie s, use

cases, features-functions, requirements• Contract deliverables consistent with SDLC

2020

Iterationor

Iterations

input output

Congruency between contract model and SDLC: Waterfa ll vs “Iter-fall”

…

SI DetailRequirements

& Design

SI Testable Code;State Test Scripts

& Test Results

Iterationor

Iterations

input output

SI DetailRequirements

& Design

SI Testable Code;State Test Scripts

& Test Results

�Iterative / Agile development on a waterfall contra ct is not a good idea.� Need to align iterations with contract deliverables .

Detailed Requirements, Design, Dev, Implementationinput

Requirements

output

UATDetailed Requirements

Design“Waterfall”

“Iter-Fall”

21

Business Aware Testing

22

Scope of Testing Testing by Types

• Functional Testing: user stories, user scenarios, use cases, …, features-functions, requirements

• Non-Functional Testing: performance, load, stress, …• Government Required Testing: connectivity, certific ation, …

Testing by Focus Areas• Accessibility Testing• Security Testing• Data Conversion Testing• System integration Testing (including interfaces to Legacy)• End-to-End Workflows (including interfaces to Legac y)

Testing by SDLC Time Frames• Pre-UAT • UAT• Pilot

• Enterprise-wide Rollout

23

Entry and Exit Criteria based on test results IMPOR TANT

SIT Exit Criteria• SI completed all SIT test cases• Severity Level 1 Defect = 0 and Level 2 Defect = 0

UAT Entry Criteria• SIT Exit Criteria met• State’s test cases ready

UAT Exit Criteria• Severity Level 1 Defect = 0 • Severity Level 2 Defect = 0 or as agreed with SI • Shape Metric requirement met - see next slide

Pilot Entry Criteria• UAT Exit Criteria met and Fed approval to start Pil ot received• Data conversion completed & checked• Server recovery exercise complete at hosting facili ty• Business and people ready, especially training• Rollback strategy complete

24

Defects Shape Metric

21 3 4 5

UAT of All Test Cases in Time

Defects Count

Max

50%Max

� Need empirical evidence that defects have stablized .� Disputes around defect severity level common.

25

Managing complexity & Conclusion

26

� Put in place project governance that integrates all work teams and relevant business functions.� Develop project roadmaps that integrate relevant business functions and IT.� Coordinate cross-team / cross-function dependencies methodically and carefully.� Assure end-to-end functionality of work products across all work teams and relevant business functio ns.

26

Enterprise projects must manage complexity by…

� These are elements of good systems engineering� They are important for all SDLCs, including Agile A t Scale.

27

� There are many facets of quality when acquiring enterprise systems.� Contractors and the acquiring organizations have different motivations – the meaning of quality & ris k are fundamentally different.� SDLC (including Agile and other iterative SDLC models) is an important aspect of quality but not t he only one.� A tightly coupled team doing the right work in the right way is an idealistic worldview and must be balanced by realistic expectations and pragmatism.

27

Conclusion

28

Join Us This

Fall #PNSQC18www.pnsqc.org

Thank you!

Please note related PNSQC workshop on Wednesday.

@PNSQC