Slide 1

Quality Management and Testing Quality throughout the Lifecycle Testing as a Quality Practice Types of Testing Agile Testing Practices

Slide 2

2 Review: Reliability Terminology Failure Incorrect or unexpected output Symptom of a fault Fault Invalid execution state Symptom of an error May or may not produce a failure Error Defect or anomaly in source code Commonly referred to as a bug May or may not produce a fault Defects may be injected at any time in the lifecycle Recall Watts Humphrey (father of PSP): (paraphrase) A defect is anything that necessitates a change in the code

Slide 3

3 Review: The Cost of Defects Cost Development Phases Diagram below is the assumed curve (Capers Jones, 2000) Software problems are 100 to 1000 times more costly to find and repair after deployment Diagram below is the assumed curve (Capers Jones, 2000) Software problems are 100 to 1000 times more costly to find and repair after deployment Cost to Repair Software Costs to Re-test, Re-deploy, Re-* Cost of Lost Opportunities Cost of Lost Customers Cost to Repair Software Costs to Re-test, Re-deploy, Re-* Cost of Lost Opportunities Cost of Lost Customers Courtesy IBM Academic Initiative

Slide 4

4 Software Qualities Correctness Reliability Robustness Performance User friendliness Verifiability Maintainability Repairability Safety Evolvability Reusability Portability Understandability Interoperability Productivity Size Timeliness Visibility Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001 Quality is concerned about more than finding and fixing traditional bugs testing is in essence a preventive process whereas you build quality in; you create it deals with measurable and verifiable ilities of software

Slide 5

5 Verification and Validation Verification Ensure software meets specifications Internal consistency Are we building the product right? Validation Ensure software meets customers intent External consistency Are we building the right product? Formal Technical Reviews Test Planning & Review Measurement Analysis & Reporting Process Definition & Standards V & V is a whole life-cycle process

Slide 6

6 Views of QA Design, in formal notation Executable machine code Execution on verified hardware Code, in verifiable language Complete formal specification of problem to be solved Correctness-preserving transformation Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001 Design, in mixed notation Pentium machine code Execution on commercial hardware Code, in C++, Java, Ada, Mixture of formal and informal specifications Manual transformation Compilation by commercial compiler Commercial firmware An Idealistic View of QA A Realistic View of QA

Slide 7

7 Quality Practices in the Lifecycle Quality Practices Pervade Lifecycle Activities: Requirements 8 quality measures (remember these? correct, unambiguous, complete, consistent, prioritized, verifiable, modifiable, traceable) Design Does your design satisfy all the requirements? What constitutes a quality design anyway? Coding What practices can developers do to improve quality? Test Seen as the principle quality practice - but does it create quality? Deploy The ability to get the software into the hands of users Maintain What happens to quality as the system evolves over time?

Slide 8

8 Complicating the Process Four inherent complexities to building quality software: 1.Requirements 2.Inherent problem complexity 3.Accountability 4.Inertia Note that only #2 is a function of the domain The others are principally social/communicative

Slide 9

9 First Complication - Requirements No matter how sophisticated the QA process, the problem of creating the initial specification remains Goes all the way back to requirements! Real needs Actual Specification Correct Specification Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001

Slide 10

10 Second Complication - Complexity Complex data communications Electronic fund transfer Distributed processing Web search engine Stringent performance objectives Air traffic control system Complex processing Medical diagnosis system Sometimes, the software system is extremely complicated making it tremendously difficult to perform QA Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001

Slide 11

11 Third Complication - Accountability It is difficult to divide the particular responsibilities involved when performing quality assurance Project Management Development Group Quality Assurance Group Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001

Slide 12

12 Fourth Complication - Inertia Quality assurance lays out the rules You will check in your code every day You will comment your code You will Quality assurance also uncovers the faults Taps developers on their shoulders Creates image of competition Quality assurance is viewed as cumbersome Just let me code You cant tell me what to do Quality assurance has a negative connotation Notes courtesy Andre Van Der Hoek UC Irvine Copyright 2001

Slide 13

13 Quality Practices in Testing FURPS Model - sets expectations for testing Actually a general quality model, this figure shows how it fits for testing Reliability Test that the application behaves consistently and predictably. Performance Test online response under average and peak loading Functionality Test the accurate workings of each usage scenario Usability Test application from the perspective of convenience to end-user. Supportability Test the ability to maintain and support application under production use

Slide 14

Types of Testing Unit Testing (not covered) Acceptance Testing Installation Testing Regression Testing Functional (System) Testing Automated Testing Integration Testing Stress Testing

Slide 15

Acceptance Tests Purpose: Determine if the product is ready for final release Different levels of acceptance based on the user population Maps to the types of acceptance tests (next slide) May be part of a contractual basis If software development under contract specifications, then passing acceptance tests may be considered a final milestone on your deliverable. Not dedicated solely to finding implementation defects Also used to identify requirements, deployment, maintenance defects, etc.

Slide 16

Acceptance tests Benchmark Test: real-world conditions Actual or simulated real user base Commonly used for systems w/ special requirements Evaluate multiple offerings against the benchmark The benchmark represents ground truth Pilot test: install on experimental basis Uses actual end users Focuses on mainstream requirements Specialized training and education of the user base not required Alpha test: in-house pilot test Or sometimes with most trusted partners Beta test: customer pilot Parallel testing: new system operates in parallel with old Useful for side-by-side comparison Release Candidate: early access in the community Community approval that the product is ready for final release

Slide 17

Installation / Regression Testing Installation Testing Does the system work in the field as it does in the lab? How easy or hard is it to deploy and maintain? Are migration procedures accurate? Does the documentation match? This is a big deal nowadays! Regression Test Once system has passed a system test, future system tests must guarantee new functionality doesnt break working functionality When you run the Nth test, you have to run the N-1 tests before it Fits particularly well with iterative and incremental processes

Slide 18

Functional/Automated Tests Functional: Test of the system as a whole Functionally-oriented (black-box) Can include non-functional aspects Typically though, these are done as part of stress testing Not concerned with location of defects or interface problems These are tasks for integration testing Defect expensive to fix when found at this step Automated Testing and Test Tools Simulations Record-and-play tools Workflow support for test scripts Tracking and reporting results Nightly build and test tools Example: http://public.kitware.com/dashboard.php

Slide 19

Integration testing Tests complete systems or subsystems composed of integrated components Integration testing should be black-box testing with tests derived from the specification Main difficulty is localising errors Incremental integration testing reduces this problem Interface Testing Takes place when modules or sub-systems are integrated to create larger systems Objectives are to detect faults due to interface errors or invalid assumptions about interfaces Particularly important for object-oriented development as objects are defined by their interfaces

Slide 20

Interfaces types Procedural interfaces Sub-system encapsulates set of procedures called by other sub-systems Message passing interfaces Sub-systems request services from other sub-systems Parameter interfaces Data passed from one procedure to another Shared memory interfaces Block of memory is shared between procedures Types of Interface Errors Interface misuse: A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order Interface misunderstanding: A calling component embeds assumptions about the behaviour of the called component which are incorrect Timing errors: The called and the calling component operate at different speeds and out-of-date information is accessed

Slide 21

Stress testing Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light Stressing the system test failure behaviour. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded

Slide 22

The Testing Process Test data: inputs devised to test the system Test cases: Test Data plus the predicted outputs from these inputs if the system operates according to its specification Test suite: A collection of Test Cases organized around some principle or objective.

Slide 23

Test documentation Test plan describes system and plan for exercising all functions and characteristics Test specification and evaluation details each test and defines criteria for evaluating each feature Test description test data and procedures for each test Test script The actual test procedures (process description) Test analysis report results of each test

Slide 24

Example Test Script

Slide 25

Example Test Script (cont.)

Slide 26

Agile Practices Change Roles Traditional Testing role The Last Defender of quality Enforce a strict change management policy w/ rigid entry/exit criteria Heavy documentation & upfront planning ( but a dynamic project plan?! ) Agile Testing role A team member primarily focused on quality I was Loved and Appreciated overheard on a testers 1 st agile project Integrate quality practices into development process Test provides value to help reduce risk Agile Principles: Quality is everyones responsibility Developers write tests too (test-driven) Testers work in same bullpen as developers No separate, closed-door testing group Track quality with rest of development process Coverage, metrics, etc. on each build

Slide 27

Agile Testing: Continuous Testing Agile likes transparency, adaptability to change Dashboards used to show constant quality progress Include results of unit, system, load, etc. tests As well as quality measures from your metrics Compatible with other transparency measures like burndowns

Slide 28

28 Agile Testing Top 10 Best Practices 1.Write tests first: Code is not delivered to build unless it passes tests 2.Write acceptance tests: Every story defines its acceptance criteria 3.Embrace change: Agile development means frequent change 4.Reduce Test Documentation: R ecord the test and track results, thats it! Use a Wiki to capture essence, not details (must must be repeatable) 5.There is no handoff activity: Testing integrated in the dev process 6.There is separate no integration testing activity: Integration is continuous, supported with (automated) regression testing 7.Automate everything you can! Make test scripts the documentation 8.Test early (or Test First!) Add tests to development build process before feature is completed Test features before they are complete Test features before they are accessible from its external interface (GUI) 9.Testing is not a phase, its a way of life. Agile teams are test infected. How will we test it? is as important as How will we build it? 10.Testing and development are not separate teams

Slide 29

Quality and Testing Conclusions Dr. Garys Things to Remember: 1.Tested software != Quality software 2.Quality is injected, not something defined in absentia 3.But sure, testing is important, and important to do well Have clear testing objectives Dont treat it as a last step checklist 4.Respect your testers you are on the same team Dont hate the playa, hate the game--Ice T 5.Include your testers early in the process 6.Start testing early in the process 7.Verification is important, but Validation is everything 29 Program testing can be used to show the presence of bugs, but never to show their absence! --E.W. Dijkstra, 1972