2 earthsoft-quality control in software organidation

Making earth little softer

Earthsoft Foundation of GuidanceEdge-Aggressive-Reliable-Trust-Honesty-Soft-Obedient-Fun-Transparent


Earthsoft Foundation of Guidance (EFG) has uploaded following presentations at http://myefg.in/downloads.aspxBe mentor using your education, knowledge & experience to contribute for a social cause & do conduct free training seeking help of existing platforms. Kindly share with your friends•Motivation for higher study, Planning for study, Education guidance, Career guidance, Career available after SSC & HSC•Personality development – 3 files •How to prepare resume, Tips to attend interview successfully•Religion related –To understand basic religion, Do & Don’t tips•Health related - Be vegetarian, Be healthy, Manage health•Corporate - Project management, Assertiveness, Ownership, Effective communication, Leadership, Be entrepreneur•Finance - To avoid speculation in stock market•Social - Women empowerment, Choosing life partner, conflict resolution, stop ragging, stop alcohol, snakebite treatment

About us


• Organisation• Quality• Quality management• Quality Assurance• Quality Control• Processes• Case studies experience • Strategic Quality Planning

Index


Index


Quality Control


Project RAG Status

NOTE: Pl update the RAG status from spectrum for last 4 weeks. If any of the parameter is Red / Amber, pl update the action required and status for the current week.

Parameter Last Week -1

Last Week

Current Week

Action Required

Status Target Date

Budget

Scope

Effort

Schedule

Resource

Quality


• Functionality related gap between expectations of client and actual delivery

• Defects are injected during Requirement analysis, High & Low level Design and coding phase.

• Continuous reviews help to prevent/ remove defect injection

• Rigorous testing during Unit level, Integration and application level testing detect the defects.

RA R Design R Coding R UT IT AT

Achieving Quality


Reviews• Review helps • To confirm client’s requirements• To detect defects in early stages• To reduce rework at later phases


RA Design Coding UT IT ATDef

ects

in

syst

em

Various Phases of SDLC

To be at better quality after every phase

• Removing defects by reviewing & testing• From each deliverable• At each phase

Reviews & Testing


Team structure & reviews

PMO Reviews

PMO Reviews

Reviews of deliverables of each member at every level by supervisor help improving quality

Creating proposal Project Plan, ReportsFollowing Process & best practicesReview of DD, Coding conventions

Reviews of Proposal; Reports, EscalationsCompliances of the best practices, client management & business growth

Creating DD, Design Framework, Coding convention, Reviews of deliverables of Developers & testers

Creating Specification, Code, Unit & SI test cases, Testing, etcCompletion checklist, Self review


Reviews• Assuring:-• Quality of design document impacts quality of

the code & the test cases.• To identify the right resource for the reviews

depending on entity to be reviewed• Assuring incorporating review comments• Reviews are discontinued when progress is

delayed. To assure to have reviews as planned• Tracking review efforts & duration for each entity

(like code, unit test cases, etc) (e.g. review time per page)


Reviews• Deciding on review density before starting the

activity• Review comments expected to reduce on repeat

cycle• Cause for concerns• No. of review comments & No. of major

comments are not decreasing on progress as expected is indication of unstable quality

• Volume change of entity (e.g. no of pages for design doc, no. of test cases, etc) pre & post review


Code reviews• Build detailed code checks into the development

process• Make it impossible to release code that violates

these checks• Few errors would only be discovered by code

reviews or testing• Many errors only occurred on specific datasets


Performance testing• Execute to check performance of each query• Create database as appropriate• Compile & Check manually all the queries

individually for performance time, keep record• Every SQL statement is run against multiple real

world customer databases• Improve/ optimise the queries• Log the bugs for worst performing statements


Issue Management• Issues are restraints to accomplishing the

deliverables of the project. • Typically identified throughout the project and

logged and tracked through resolution.

Issues impact cost, time, quality, most important is reputation and branding!

Types of issue – Minor, Normal, Major, Critical, BlockerMinor issue is dangerous!


Issues/Risks (non-financial)

Highlight highest risks for management attention


Metrics-Development & Enhancement

NOTE: Pl highlight the summary from the metrics report

Work Packets

Effort Variance

On-time Delivery

Defect Density

Field Defect Density

Productivity COQ Review Effective

ness

Goal Actual G A G A G A G A G A G A


Metrics-Maintenance & Support

Provide the SLA compliance summary If you use any tool provide the snap shot of the same.

The number of tickets includes the new received in the reporting month as well as the backlog from previous months

Priority Number of Tickets/Faults

Response SLA Compliance%

Resolution SLA Compliance%

Remarks

Goal Actual Goal Actual

P1

P2

P3

P4


• The perceptions of the user determine the purpose. Quality therefore is a user perception.

• Correctness : Does it as per users agreed on? (Unit Level)

• Integrity : Does it maintain accurate data? (Integration Level)

• Reliability: Does it behave right all the time? (Application Level)

• Efficiency : Does it run as well as it could? (Application Level)

• Robustness: Does it handle invalid data? (Application Level)

Quality & User perceptions


Testing Improvements• PM decides on type of testing - automated or

manual or both• 400% increase in automated regression tests in

the same efforts • Tests created for all key business flows• Created by dedicated product QA teams• Used for

• Ongoing maintenance/bug fixes• Platform certification (e.g. Linux)• Technology certifications

• eBusiness Flows certified prior to release• Progress measured daily, and reported


Prerequisites of TestingTest Cases• Latest & Completed test cases documents from

CVS (or any Configuration management tool)Test Environments• Document that gives the details of H/W and S/W

required to start testing• Identical environment which client would have

while going live• Development environment• Testing environment


Flow of Testing• Test cases• Test Data• Unit Testing

• Screen Layout, look & feel• Screen Validations • Functionality

• Integration Testing• Application Testing• Users Acceptance Testing


Testing GuidelinesWhy the testing guidelines?• “Test cases may not cover 100% aspects thus

extend your thoughts beyond the written statement of the test case.”


Who should testThere are 2 possible actors• Developers to execute 100% unit testing• Reasons being they take the entire ownership of quality of the

delivery• Second possibility is dedicated testing team executing test cases• Testing being skill of expert!


Unit Testing - UITo use a check list & test for following• Consistency- Control sizes• Basic functionalities• Tables, navigation to next page• Search listing result• Validation error message• Layout (as per Prototype)• Spelling Mistakes• Font (Type & Size)• Icons display• Color Scheme• Screen resolution


Unit Testing - UI• Tab sequence & Text selection• Links (Check each link to ensure it works)• Current selection indication• Alignments (Text, Numbers, Row, Column)• Compatibility with various browsers• Duplicate display of information• Mention of Copyright


Unit Testing - Validations• Field type & Size (Numeric, Alphanumerical,

Database field width)• Formatted fields (e.g. Date, E-mail, Phone

Numbers, Security ID)• Mandatory Fields• Messaging (Alert, Warning, field message,

message post submissions, sequence, etc)• Database Records verification• Verify the values stored in database for Radio

buttons & Checkboxes• Test spaces trimming (leading & trailing spaces)• Update By & Update on fields


Unit Testing - FunctionalityTo use matrix of screens & functionalitiese.g. File Upload/Download•Try all types of files – image files, office files, animated GIF•Try downloading files – verify they don’t contain garbage•Search using special characters like (%,_,’ etc.)Sorting functionality - Check the number of records before and after the sorting are same• In case of mailing functionality, check for non-existing e-mail address. (The mail should be returned back to some preset mail address)


Unit Testing - FunctionalityTo use matrix of screens & functionalities•Test User login Case Sensitivity•Test the dependant behaviors of the controls•Enabling / Disabling based on some actions•Data shown in Popup Screens based on some actions (Refreshing)•Language Support Test (Wherever applicable Japanese / German/ French, etc Input)•Test readability by changing encoding type in the browser


Integration Testing• Refer Integration Matrix while testing• Enter boundary values (Min , Max, 0 , negative

etc)• Errors occur when values passed by once

screen are used in calculations on other screen


• Testing Business Transactions• Refer Screen Flow Diagrams while identifying

Business Transactions• Identify the User Roles to perform testing (Sales

Person, Purchase Manager, Dispatch Person OR User , Administrator)

• Ensure Test Data Variety & Test data Volume• Roles & Rights testing (Test for access rights)• Logging Mechanism testing (Errors and User

Actions)• Deployment Testing (Perform Deployment as

per the Deployment Document)

Application Testing


• Application Menu should be in sequence of the functionality• User Management->Master Data Mgt-> Order

Entry->Purchase Mgt->Dispatch• Navigation

• Using Browser ‘Back’ & ‘Forward’ buttons• Using ‘Back Button’ & ‘Links’ in the application

• Browser dependency • Test for all the browsers as per client request

(Min should be IE & Netscape)• Disable Cookies and try to login or perform

Transactions

Application Testing


• Perform transaction using multiple instances of application, Open other applications like (Word, Excel, Access, Others) • Quick Actions in the Application • Click Submit button twice• Switch between the screens quickly

• Enter data in one instance & try to see it immediately in another instance

• Random operations • Enter any number of characters and click

anywhere• Clicks on read only area

Application Testing


• Copy Paste of the URLs • Copy URL after Logging in and paste it in

another instance of browser• Same operation while in the middle of a

business Transactions• Session Timeout verification (test for < / = / >

defined timeout value)• Disconnection Test (to verify there is no

inconsistent data generated)• Disconnect Net connection in the middle of a

transaction• Stop Database service while transaction is in

process on the client

Application Testing


• Reports printing • Print Multiple Pages• Try to print without installing any printer driver• Verify Report Margins for the printed copy• Status related tests (Display only ‘Active’ / ‘Live’

records)• Multi-user Testing• Concurrent users (Different users / users with

different roles)• Concurrent Transactions (Add / Update /

Delete / File Upload)• Locking Issues Test• Common Tests…ref file

Application Testing


How Much Testing Is Enough?• There is always one more bug…• So how much testing is needed to find “all” ?• Cost• Time to market• Feasibility• Rate of detection of defects• Industry-specific standards• To predict how many bugs will be in a release?• In order to provide correct budget and schedule

estimates, QA managers need the means to predict fault levels.


Types of defects• Blocker• Severe/ Critical• Major• Normal• Minor• Trivial• To publish phase wise defect data for each

module for above types• Responsibility wise

Sample statistics


Update defect passbook at end of day and check status and trend

Defect passbook

Making earth little softerConfidential 41

QAG measures QualityQAG tracks and measures quality using following template

Organisation


Quality measurement

No. of defects 1effort in manmonths 3Loc per hr 10Lines of code 4800no. of defects 1% of defects to LoC 1*100/4800=2.1%Achieved quality 97.9167%

• 1 defect detected by customer for 3 man months of efforts is said to be fairly good quality

• Count each type of defect (minor, normal, major, critical or blocker) ; Minor defect is worst!


Measure where are you• To set a quality baseline & remain determined• To seek feedback• To measure and conclude current levels from

desired level• To draw action plan to be at set goal

0

1

2

3

4

5

6

7

8

Vertex NTTD

Desired level

Current Level

Current level Desired level


Aim for Defect Metrics (sample)

Phase Metrics UnitLower Limit

Average

Upper Limit

Requirement definition

Requirement documents

Review comments

Per 100 pages

9.8 15 24

Design

Detail design (DD)

Review comments

Per 100 pages

7.6 13 21

Program design (PD)

Review comments

Per 100 pages

4.9 12 30

Coding

Unit TestingTest Case

DensityKLOC 65 110 170

Defect density KLOC 3 6.4 9.7

Integration Testing (SI)

Test Case Density

KLOC 16 39 92

Defect density KLOC 0.7 2.2 3.6

TestingSystem Testing

(PT)

Testing density KLOC 2.3 9.5 39Bug detection

densityKLOC 0 0.7 1.4

Entity which is measured, can be managed well


Bug content analysisPhenomenon Action example

Module having major change

Suspend test workReview of design document, review of source code with the designerEnhanced test according to test items, bug contents

Test density < allowable range

Enhanced test according to test items, bug contents

Bug detection density is less than allowable

Enhanced test according to test item, bug contents

Bug detection density exceeds allowable range

As per 1 above


Bug content analysisPhenomenon Action example

Bug cause Problem of defective specification

Re-check specifications for client, business Design document review with an expertIndetermination of specifications of design doc. ambiguous part

Bug cause Problem of development from specification

Confirmation of specification to designerReview of design document & source code with the designerClarification of implementation, implementation method

Bug - Other than specification

Thoroughness of walk through and source code review

Occurrence type Environment

Specify reason according to occurrence period, contentsRe-examination of test environment, system


Bug content analysisOccurrence type Action example

Operational error, without reproduction, identification mistake)

Review of test item, processRe-adjustment of design document, test environmentEducational guidance to designer

Coding/ modification mistake

Thoroughness of source code review

Occurrence of important bug that impacts test progress

Suspend test workReview of design document & review of source codeEnhanced test according to the test item, bug contents

occurrence of similar bugs

Check the latency of a similar bug in another module


• No. of pages/ characters of design document & LOC of the code

• Check planned No. of test cases (a statistical index page for test cases shows the progress)

• Test cases planned & actually execution• To capture No. of test cases executed on daily

basis and cumulative basis• No. of defects detected on daily and

cumulative basis – check if volume is apt• Calculate defect density–check if volume is apt• Check if any delay in execution, understand &

remove the delay

Quantitative analysis


•Check the pending test cases on daily & cumulative basis – check if volume is apt

•Bug fixing on daily and cumulative basis - check if volume is less or more

•Check if any delay in fixing, understand & remove the delay


Matrix Analysis

Test density Low Target value Upper limit

Bug Density -Low Less Bugs detection

Retest adding few more test cases

Member skill level?

Target Value Write additional test cases

Judge verification of the result qualitatively

Judge verification of the result qualitatively

Upper Limit Multiple bugs occurred

Multiple bugs occurred Suspend testing, get into other measure like code review, etc


Quantitative analysis# Category Details Type1 Specification

InadequacyDescription Inadequacy Interface Error

Logical ErrorData definition errorTable definition errorFormat ErrorOthers

Wrong descriptionAmbiguousStandard ViolationDocument not correctedDocument not consistent

2 Issue while deploying

Slipping over the specificationLack of understanding the Lack of confirming the specificationInsufficient investigation of specs

3 Issues except the matters related to specification

Less knowledge for language usageHalfway publicize / communicationStandard violationNot checked while re-usingNot checked while modifyingUnforced mistake


• Root cause analysis – assign root cause to each bug & conduct analysis (fishbone diagram)

• Severity wise• Developer wise• Tester wise• Defect injection phase v/s defect detection• Defect injection phase v/s root cause• Daily status (like a bank passbook)


Defect detected RA-Review HLD, IT -Review LLD, UT-Review Code review UT IT AT

Defect injected

RA

HLD

LLD

Coding


Process• This PCB is released as Organization PCB• This PCB should be referred to when providing

Organizational Data to the customers• Each project depending on the size, complexity

and customer requirement shall define the metrics for the project.

• This would be done in Process Plan as part of the project initiation.

• In case the project targets are outside the UCL or LCL limits an approval would be required from the respective VP.


PCB Contents• Productivity Data• Phase wise effort distribution• Effort – Actual Vs. Estimated• Schedule – Actual Vs. Estimated• Defect Metrics


Productivity Data

LOC/ １MM

Productivity LCL Target UCL

Java 700 1000 1300

VB.Net 800 1500 2200

C# .net 700 950 1200

C 700 1000 1300

This data applies to all the efforts spent on all SDLC Phases – from RA to Acceptance testing

To finalise parameters for data to be capturedTo create PCB data for each projectThus organisational data will be createdAnalyse & use the data


Plan to Collect Data• PCB shall be updated every three months, End

of every quarter - Jun, Sep, Dec, and Mar• For development, testing & maintenance project• Finalise project metrics during kickoff of project• The project metrics collection plan shall be

finalized as part of the project initiation process (Process Plan)

• The projects shall contribute Phase End data or Milestone data every six months

• To share PCB data during project closure


Project ClosureTo conduct a meeting with all stake holdersTo share following information estimated & actual• Efforts• Schedule• Defect Matrices• Profitability• Client happiness IndexDetailing• Key learning• Best performance• What next?

Remember


Thank You

Author has worked at as Sr management professionalVertex Software, now NTTDataTech MahindraSoftBridge Solutions, India& extremely thankful to these organisation for offering exciting opportunity to work in related areas, gain experience & contribute& a gentleman having deep passion about quality..Mukand Bhagwat

2 earthsoft-quality control in software organidation

Business

Transcript of 2 earthsoft-quality control in software organidation