Week 2

Quality Assurance

Quality Assurance in Context

Software Qualit Engineering

What is SQA• SQA is a well defined, repeatable process

that is integrated with project management and the software development lifecycles to review internal control mechanisms and assure adherence to software standards and procedures. The objective of the process is to assure conformance to requirements, reduce risk, assess internal controls and improve quality while conforming to the stated schedule and budget constraints. (Owens and Khazanchi)

More definitions (Owens and Khazanchi)

What is QA?• QA as Dealing with Defect

– Defect Prevention• Error blocking• Error source removal

– Defect Detection and Removal• Inspection• Testing etc.

– Defect Containment• Failure prevention • Failure containment.

QA: quality assurance

• Focus on correctness aspect of Q

• QA as dealing with defects– post-release: impact on consumers– pre-release: what producer can do

• what: testing & many others

• when: earlier ones desirable (lower cost)

• how =>classification as follows

Error/Fault/Failure & QA

• Removal of faults (pre: detection)– inspection: faults discovered/removed– testing: failures trace back to faults

• Failure prevention and containment:– local failure !=> global failure

• via dynamic measures to tolerate faults

– failure impact =>safety assurance

QA Classification

Defect Prevention Overview• Error blocking

– error: missing/incorrect actions– direct intervention to block errors

• fault injections prevented

– rely on technology/tools/etc.

• Error source removal– root cause analysis

• identify error sources

– removal through education/training/etc.• Focus on Product and domain specific knowledge, SW dev

knowledge and expertise, developmental methodologies, development process knowledge

• Systematic defect prevention via process improvement.

Formal Methods• Motivation

– fault present:• revealed through testing/inspection/etc.

– fault absent: formally verify.

• (formal methods =>fault absent)• Basic ideas

– behavior formally specified:• pre/post conditions, or• as mathematical functions.

– verify “correctness":• intermediate states/steps,• axioms and compositional rules.

• Approaches: axiomatic/functional/etc.

Defect Reduction:Inspection Overview

• Artifacts (code/design/test-cases/etc.) from (req./design/coding/testing/etc) phases.

• Informal reviews:– self conducted reviews.– independent reviews.– orthogonality of views desirable.

• Formal inspections:– Fagan inspection and variations.– process and structure.– individual vs. group inspections.– what/how to check: techniques .

Testing Overview• Product/Process characteristics:

– scale/order:

• unit, component, system, : : :– who: self, independent, 3rd party

• What to check:– external specifications (black-box)– internal implementation (white/clear-box)

• Criteria: when to stop?– coverage of specs/structures.– Reliability: usage-based testing

Defect Containment: Fault Tolerance Overview

• Idea originated from hardware systems.

• Focus is on reliability, availability and dependability.

• Motivation– fault present but removal infeasible/impractical– (fault tolerance ) contain defects.

• FT techniques: break fault-failure link– Recovery blocks: rollback and redo– NVP: N-version programming

• fault blocked/out-voted

Safety Assurance Overview

• Extending FT idea for safety:• fault tolerance to failure “tolerance"

• Safety related concepts:– safety: accident free– accident: failure with severe consequences– hazard: precondition to accident

• Safety assurance:– hazard analysis– hazard elimination/reduction/control– damage control

QA in Context• QA and the overall development context

– defect handling/resolution

– activities in process

– alternative perspectives

• verification/validation (V&V) view

Defect handling/resolution

• status and tracking– Logging– Monitoring

• causal (root-cause) analysis

• resolution: defect removal/etc.– Many stakeholders: testers, developer, managers,

client (except unit testing)– Part of project management activity

• improvement: break causal chain

Defect Measurement and Analysis• Defect measurement:

– parallel to defect handling– where injected/found?– type/severity/impact?– more detailed classification possible?– consistent interpretation– timely defect reporting

• Defect analyses– as follow up to defect handling.– data and historical baselines– goal: assessment/prediction/improvement– causal/risk/reliability/etc. analyses

Defect Handling in Different QA Activities.

• Among 3 classes of QA activities:– defect detection and removal activities are more

closely associated with defect handling.– Various defect prevention activities do not directly

deal with defect or discovered faults.– On defect containment the focus is not on fault

finding or removal.

• So defect handling is not closely associated with prevention and containment activities

QA in Software Processes• Development process components:

– requirement, specification, design, coding, testing, release.

• Process variations:– waterfall development process– iterative development process– spiral development process– lightweight/agile development processes such

as XP (extreme programming)– maintenance process too– mixed/synthesized/customized processes

• QA important in all processes

QA in Waterfall Process

QA in Waterfall Process: QA throughout process

• defect prevention in early phases

• focused defect removal in testing phase

• defect containment in late phases

• phase transitions: inspection/review/etc.

QA in Software Processes• Process variations (: waterfall) and QA:

– iterative: QA in iterations/increments– spiral: QA and risk management– How we can do it in agile processes?? – mixed/synthesized: case specific– more evenly distributed QA activities

• QA in maintenance processes:– focus on defect handling;– some defect containment activities for critical or highly-

dependable systems;– data for future QA activities

• QA scattered throughout all processes

V&V• Core QA activities grouped into V&V.• Validation: w.r.t. requirement (what?)

– appropriate-for-use “ doing right thing"?– scenario and usage inspection/testing;– system/acceptance testing;– beta testing and operational support.

• Verification: w.r.t. specification/design (how?)– Correct “doing things right"– design as specification for components;– structural and functional testing;– inspections and formal verification.

V&V in Software Process

V&V vs DC View

• Two views of QA:– V&V view– DC (defect-centered) view – Interconnected: mapping possible?

• Mapping between V&V and DC view:– V&V after commitment

(defect injected already)• defect removal & containment focus

– Verification: more internal focus– Validation: more external focus– In V-model: closer to user (near top) or

DC-V&V Mapping

Time for a break

Software Quality Engineering

• SQE: Software Quality Engineering

• Key SQE Activities

• SQE in Software Process

QA to SQE• QA activities need additional support:

– Planning and goal setting– Management:

• when to stop?• adjustment and improvement, etc.• all based on assessments/predictions

• Assessment of quality/reliability/etc.:– Data collection needed– Analysis and modeling– Providing feedback for management

• QA + above => software quality engineering (SQE)

SQE Process

SQE and QIP• QIP (quality improvement paradigm):

– Step 1: understand baseline– Step 2: change then assess impact– Step 3: package for improvement

• SQE as expanding QA to include QIP ideas.

Pre-QA Planning• Pre-QA planning:

– Quality goal– Overall QA strategy:

• QA activities to perform?• measurement/feedback planning

• Setting quality goals:– Identify quality views/attributes– Select direct quality measurements– Assess quality expectations vs. cost

Setting Quality Goals

• Identify quality views/attributes– customer/user expectations,– market condition,– product type, etc.

• Select direct quality measurements– direct: reliability– defect-based measurement– other measurements

• Assess quality expectations vs. cost– cost-of-quality/defect studies

Forming QA Strategy

• QA activity planning– evaluate individual QA alternatives

• strength/weakness/cost/applicability/etc.

– match against goals– integration/cost considerations

• Measurement/feedback planning:– define measurements (defect & others)– planning to collect data– preliminary choices of models/analyses– feedback & followup mechanisms, etc.

Analysis and Feedback• Measurement:

– defect measurement as part of defect

• handling process– other data and historical baselines

• Analyses: quality/other models– input: above data– output/goal: feedback and follow up– focus on defect/risk/reliability analyses

• Feedback and followup:– frequent feedback: assessments/predictions– possible improvement areas– project management and improvement

SQE in Software Processes• SQE activities development activities:

– quality planning product planning– QA activities development activities– analysis/feedback project management

• Fitting SQE in software processes:– different start/end time– different sets of activities, sub-activities,

• and focuses– in waterfall process: more staged

• (planning, execution, analysis/feedback)– in other processes:

• more iterative or other variations

SQE Effort Profile• QE activity/effort distribution/dynamics:

– different focus in different phases– different levels (qualitatively)– different build-up/wind-down patterns– impact of product release deadline (deadline-

driven activities)

• planning: front heavy• QA: activity mix (early vs. late; peak

variability? deadline?)• analysis/feedback: tail heavy (often

deadline-driven or decision-driven)

SQE Effort in Waterfall Process

Enough for this week

• Reading Tian – Chapter 3, 4 and 5.

• And lets discuss your past papers..