Contextually-Driven System Architecture Reviews

BT2 Concurrent Session 11/14/2013 10:15 AM

"Contextually-Driven System Architecture Reviews "

Presented by:

F. Michael Dedolph Levi Deal Consulting

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Michael Dedolph Levi Deal Consulting

Michael Dedolph is self-employed as a consultant and innkeeper. For the past seven years, Michael led quality process improvement efforts at CSC and Lucent. Previously, he was a systems architecture review leader at Bell Labs (Lucent) and facilitated numerous risk identification and project retrospective sessions. Prior to 1997 Michael worked in the Risk and Process programs at the SEI where he was the technical lead for teams that developed the SCE and CBA-IPI appraisal methods. His IT career began with ten years as an Air Force computer officer, working in satellite systems, management information systems, and as an instructor of software engineering at the Air Force Institute of Technology.

1 Contextually Driven Architecture Reviews Dedolph – 10/18/2010

Better Software East Contextually Driven Architecture Reviews – Dedolph 11/14/2013 1

Contextually-Driven

Architecture Reviews

For Better Software Conference East

F. Michael Dedolph [email protected]

14 November 2013


Author’s Note

Most of this presentation material was used previously in these venues.

–  SEI’s SATURN 2009 conference –  Washington DC Area SPIN meeting in Feb 2009 –  PNSQC 2010

Architecture reviews were a good idea then, and they still are



What is “Architecture”? Ø Before we can review it,

we should know what “it” is


“A System Architecture . . .” Provides a solution to a problem for a client – Early on, it is a conceptual or potential solution

Architecture has also been called “design with constraints” – Alternatively, you could say architecture includes

a design that works within the constraints, including cost and schedule!

Ø Any given architecture is NOT the ONLY solution, but, some solutions are better than others



Where Architecture Exists •  Architecture exists at the intersection

of management, technology, and design

Management Technology

Design


Architecture Problem Statements •  The problem space has multiple

aspects: “FFETO” – Function – what it does, how well it does it. – Form – what it “looks like”; includes major

environmental interfaces – Economy – cost, including development,

maintenance, material, system retirement, etc. – Time – relationship to past, present, and future – Operational/ Developmental – Things that

pertain to development constraints and system operating environment, rather than the system



Problem Statements Ø are the basis for the review! Provide a succinct summary of critical

success criteria – Short: 2 pages is good –  Includes major constraints – Client-centric – Cover Function, Form, Economy, Time, and

(optionally) Operational – Be expressed in sufficient detail to make

judgments about the proposed solution, but are not unnecessarily proscriptive of prescriptive


Problem Statements Ø Are NOT – A requirements document, but may summarize

the most critical high level requirements

Problem statements also have unstated "always" criteria, e.g.:

– Possible to construct – Can make money/ will provide value – Solution won't result in harm – Legal/ regulatory constraints

Ø The team must understand these as well.



Prob. Statements Provide Context •  Problem statements can be applied to

almost any product or service. •  Advantage: representational

independence •  Many architecture review techniques depend on the way

the design is represented. •  But, design can be represented in many different ways

—e.g., network diagrams, UML, physical prototypes, circuit diagrams, flow charts, use cases . . .

Ø No representation captures all aspects of the system, and focusing too much on representation can actually obscure problem areas


Classical Architecture -



Taj Problem Statement Included: Function: tomb, mosque, hotel, memorial Form: symmetry, untarnished finish,

unprecedented weight, shearing force of the river, visual perspective maintained

Economy: irrelevant, it will cost what it costs Time: it will take as long as it takes, but must

last for all time Operational: expertise is needed from other

kingdoms, supply chain issues


Architecture Review Method, in a Nutshell •  Define the problem the client wants

solved •  Compare it to the architecture

(proposed solution) •  Identify the gaps (or risks) Then: Ø Let the project resolve the gaps



Architecture Review Questions Questions to answer before a review: •  Who decides? (Know the client) •  What problem are we trying to solve? •  Are key stakeholder interests represented?

Questions to answer during a review: •  How good is the proposed solution? •  What are the issues/ risks/ gaps in the solution

Questions to answer after a review: •  Which things will we (the project team) address? •  How will we (the project team) address them?


Ground Rules for Reviews Ground rules are covered in the pre-review, and again in the review meeting •  No attribution—review products, processes, and

ideas, not people •  Team is there to identify, not solve problems •  Client requests (and pays) for the review, but,

the project owns the findings and responsibility for correcting them Ø One exception, “Management Alert” (covered later)

•  Reviewers can ask questions at any time



Review Steps •  Preparation

•  Respond to initial contact/request; develop problem statement; select team and develop agenda; arrange logistics – travel, space, connectivity; pre-review call

•  Review Meeting •  Project presentations with Q&A; review team Caucus;

conduct readout; client meeting (optional)

•  Follow-up •  Write and distribute the Review Report; present

findings to lessons learned/governing groups (Board Report); review the Project’s action plans; Close the review by meeting with project and/or sponsor


Preparation – Some Thoughts Toughest part: Problem Statement •  I have seen projects cancel a review because

they “didn’t have time to develop a problem statement” How do you think the project fared?

•  In many cases, the initial review schedule was delayed while the project and client worked out the problem statement

•  The Problem Statement is reviewed in detail during the pre-review meeting, used to finalize agenda and assign pre-reading

Team selection •  Based on the Problem Statement •  Key roles – Leader, “Angel”, SMEs



Review Meeting - Overview •  Presentations – by the project team,

any format, reusing existing materials •  Questions by the review team, at any

time, moderated by the Leader or Angel

•  Strengths, Issues, Observations recorded on index cards (Low Tech)

•  Team Caucus determines findings •  Readout and (Optional) Sponsor/

Client meeting


Paper-based reviews can be

hard to transcribe! FMD

{ Write large, One thought per card, with enough detail to be useful – but not too wordy.

Save for Sequence Number

Save for Severity

Note here if card is a Strength or Observation; default is an Issue

Don’t Forget Your Initials!

Snow Card Example - 1



Review Meeting - Caucus •  Led by review Leader and Angel. •  Categorization is done to provide a

consistent “story” for the read-out –  A “typical” review produces 80–350 observations,

arranged into 10-25 topics

•  Issues are Prioritized:“Management Alert”, Critical, Major, Minor

•  Summary of each topic area written by team subject matter experts

•  Strengths should be preserved if changes are made to the architecture


Prioritization by the Review Team Management Alert: •  Goes to client/upper management •  “In the unanimous opinion of the Review Team,

the project will fail unless these issues are immediately addressed: <List of issues>”

Critical, Major, Minor, Strengths, & Suggestions go to the project •  Readout ensures no surprises in the report •  The review team will NOT withdraw a finding or

lower the priority (project owns the action plan) •  Priority may be increased at project request



Follow-up •  Readout and Sponsor/ Client meeting •  Management Alert and Critical Issue

Letters within a week Ø Most management alerts are related to cost,

schedule, and resources, NOT purely technical issues. Hence, upper management is responsible for the action plan for management alerts

•  Critical Issue letters goes to the Project, Project management is responsible for the action plan

•  Two Reports, one to the project, one to the Board.

•  Review team Leader and Angel review the action plans.


What Happens When? • . . . a project ignores a Management Alert?

• . . . a project ignores a Critical issue? • . . . a project fixes everything? • . . . the review team identifies an issue that turns out to not be a problem (“false positives”)?



What's Different (or the Same) About this Kind of Review?


What's Different? •  Focus – problem/solution congruence, Client ownership of

the decision making, ownership by project team •  Context/ Domain independent – Notation and representation independent, not

model or standard based (although this can be included in the problem statement, if needed)

•  Follow up – Clear statement of potential failure points, Project

sees and owns findings – can ignore or post on their web site



What's the Same? Outside eyes—external to team Products, processes, ideas are reviewed, not people Not a problem solving session Ø  Balances a defined review

process with extreme flexibility.


Sponsor Survey - Benefits Saved money/time •  Estimated at > $1M per large project, similar estimates

were derived from Monte Carlo simulations based on analyzing the impact from correcting review findings

•  Direct results from specific reviews, e.g., re-architecting after the review trimmed 9 months off the schedule, saved > $7 million.

Intangible benefits include: •  Preparation (especially preparing the problem

statement) forces people to think through the problem •  Cross-pollination of techniques across the organization •  Learning as a review team member •  Synching up project teams; “on the same page”



Drawbacks to the Method •  Risks of a Client-centered method

•  The Client can grow out of touch with the market, customer needs, and end user issues

•  In contracting, many of the success criteria are negotiated into the contract, and may represent political compromises that can’t be changed easily

•  Method dependencies •  Strongly oriented to face-to-face meetings, requires a

deep pool of reviewers with expertise

•  Incomplete findings/level of detail •  Any review’s findings are inherently incomplete. A high

level review is not suited for all problem spaces; 2 to 3 days will not flush out all of the “devils in the details”


Institutionalizing the Method •  Requires building a “learning culture”

Executive Champion

Review Board

Review Organization

Project Project Project Project Project Project

Oversight

Reviewers, Reviews

Push Review Leaders

“Angels”



Suggested Steps for Making Reviews Part of the Culture •  Start with a high-level champion/ executive sponsor •  Obtain central funding for an extended period – say 5 years •  Train a core team of leaders and angels to do reviews •  Charter a board; recruit advocates. Board members are

expected to serve as review “Angels” •  Perform the reviews; capture data and lessons learned to

document the method’s value •  Move to a sponsor-based funding model after the initial

time period, when the method has proven itself •  Rigorously retain the focus on client-centered problem

statements, project ownership of findings, limited but pertinent management alerts, and non-attribution


Questions???



Presenter Info F. Michael Dedolph

412-477-7163 [email protected]

BIO: F. Michael Dedolph is currently an innkeeper and owner of a bustling B&B, and part time consultant to the IT industry for reviews, risk management, quality methods, and process improvement. From 1997 to 2004, Michael was a systems architecture review leader at Bell Labs (Lucent); he also managed and taught Lucent’s Systems Architecture Introduction class. In addition to leading architecture Review Teams, he facilitated numerous risk identification, problem solving, and project retrospective sessions. Prior to 1997, Michael worked in the Risk and Process programs at the SEI. While at the SEI, he was the technical lead for the teams that developed the SCE and CBA-IPI appraisal methods, and was the team leader for several Risk Reviews. He started his IT career by spending 10 years as an Air Force computer officer, and also spent 7 years at CSC leading process improvement teams.


References: 1. CMU/SEI-2000-TR-004, "ATAM: Method for Architecture Evaluation";

Kazman, Klein, Clements, 2000 2. Problem Seeking: An Architectural Programming Primer, 4th Ed;

Pena and Parshall, 2001, ISBN 0-913962-87-2 3. Handbook of Walkthroughs, Inspections, and Technical Reviews;

Freedman and Weinberg, 1990, ISBN 0-932633-19-6 4. IEEE Software, March/April 2005, "Architecture Reviews: Practice and Experience";

Maranzano, Rozsypal, Zimmerman, Warnken, Wirth, and Weiss 5. STQE Jul/Aug 2002 (Vol. 4, Issue 4) Feature: Measurement & Analysis

"A Blueprint for Success: Implementing an architectural review system“; Daniel Starr, Gus Zimmerman

6. CMU/SEI-2006-TR-012, "Risk Themes Discovered Through Architecture Evaluations"; Bass, Nord, Wood, Zubrow; 2006

7. CMU/SEI-2010-TN-018, “Relating Business Goals to Architecturally Significant Requirements for Software Systems”; Clements, Bass; 2010

8. CMU/SEI Webinar, “SoS Architecture Evaluation and Quality Attribute Specification (Webinar)”; Gagliardi; 2010

Contextually-Driven System Architecture Reviews

Technology

Transcript of Contextually-Driven System Architecture Reviews