Testing Methodology Training Manual

8/14/2019 Testing Methodology Training Manual

1/71

Testing Methodology Training Manual


2/71

TABLE OF CONTENTS

1.0 INTRODUCTION TO TESTING..................................................................................................III

1.1 O VERVIEW ON TESTING ........................................................................................................................ III1.2 T ESTERS ............................................................................................................................................ VI1.3 T YPES OF TESTING ............................................................................................................................... VI

2.0 TEST PREPARATION PROCESS ................................................................................................X

2.1 B ASELINE DOCUMENTS ......................................................................................................................... X2.2 P ROTOTYPE ........................................................................................................................................ XII2.3 T EST STRATEGY ................................................................................................................................ XIV2.4 H IGH LEVEL TEST CONDITIONS .......................................................................................................... XVII

............................................................................................................................................................ XXII2.5 T RACEABILITY ................................................................................................................................ XXIII2.6 T ESTBED ......................................................................................................................................... XXV2.7 T EST CASE.......................................................................................................................................322.8 T EST SCRIPT .....................................................................................................................................352.10 B ACKEND TESTING ...........................................................................................................................382.11 N ON CERTIFIED TESTING ...................................................................................................................41

3.0 TEST EXECUTION PROCESS.................................................................................................... .44

3.1 S TAGES OF TESTING ............................................................................................................................443.2 P RE- R EQUIREMENTS FOR TESTING .......................................................................................................443.3 T EST PLAN .......................................................................................................................................453.4 A UTOMATION OF TEST CASES ..............................................................................................................483.5 D EFECT MANAGEMENT .......................................................................................................................48DEFECT R EPORT SAMPLE ..........................................................................................................................553.6 T EST DOWN TIMES ............................................................................................................................573.7 S OFTBASE TRANSFER ..........................................................................................................................593.8 P ERFORMANCE TESTING ......................................................................................................................61

4.0 POST TEST PROCESS...................................................................................................................63

4.1 S IGN OFF.........................................................................................................................................634.2 D ELIVERABLES ...................................................................................................................................634.3 M ETRICS ...........................................................................................................................................654.4 D EBRIEFS ..........................................................................................................................................65

..........................................................................................................................................................674.5 A RCHIVING ........................................................................................................................................69

..........................................................................................................................................................71


3/71

1.0 Introduction to Testing

1.1 Overview on Testing

There are many published definitions of software testing; however, all of thesedefinitions essentially convey the same statement:

Software testing is the process of executing software in a controlled manner, in order to answer the question Does the software comply to specifications.

Software testing is often used in association with the terms Verification andValidation. Verification is the checking or testing of items, including software, for conformance and consistency with an associated specification. Software testing is onekind of verification, which also uses techniques such as reviews, analysis, inspectionsand walkthroughs. Validation is the process of checking if what has been specified iswhat the user actually wanted.

Validation: Are we building the right product? Verification: Are we building the product right?

Software testing should not be confused with debugging. Debugging is the process of analyzing and locating the bugs when software does not behave as expected. Althoughthe identification of some bugs will be obvious from playing with the software,methodical approach to software testing is a much more thorough means of

identifying bugs. Debugging is therefore an activity that supports testing, but cannotreplace testing. However, no amount of testing can be guaranteed to discover all bugs.

What is testing?

Testing is the process of executing a program with the intent of finding an error. Agood test case is one that has a probability of finding an as-yet undiscovered error. Asuccessful test is one that uncovers an as-yet-undiscovered error.

Why testing?

The development of software systems involves a series of production activities whereopportunities for injection of human fallibilities are enormous. Errors may begin tooccur at the very inception of the process where the requirements may be erroneouslyor imperfectly specified. Because of human inability to perform and communicatewith perfection, software development is accompanied by a quality assurance activity.


4/71


5/71

1.1.2 Lifecycle of Testing

Requirements

FunctionalSpecification

Design

Code Code Review

Unit Test

Integration Test

Acceptance Test


6/71

1.2 Testers

What does a tester do -

Understand the Application Under Test Prepare test strategy Assist with preparation of test plan Design high level conditions Develop test scripts Understand the data involved Execute all assigned test cases Record defects in the defect tracking system Retest fixed defects Assist the test leader with his/her duties Provide feedback in defect triages Automate test scripts Understanding of SQL

1.3 Types of testing

Testing can be basically classified as

White Box Testing

Aims to establish that the code works as designed Examines the internal structure and implementation of the program Target specific paths through the program Needs accurate knowledge of the design, implementation and code

Black box testing

Aims to establish that the code meets the requirements Tends to be applied later in the lifecycle Mainly aimed at finding deviations in behavior from the specification or

requirement Causes are inputs, effects are observable outputs


7/71

Testing at inception was classified as below

Alpha Testing

A customer conducts the Alpha testing at the developers site. The software is used in

a natural setting with the developer recording errors and usage problems. Alpha testsare conducted in the controlled environment by the developer.

Beta Testing

The beta testing is conducted at one or more customer sites by the end user(s) of thesoftware. The developer will not be present in the customers place. So, the Beta testis a live application of the software in an environment that cannot be controlled by adeveloper. The customer records all the problems (real or apparent) that areencountered during the beta testing and reports to the developer at regular interval. Asa result of problems reported during beta test, the software developer makes themodifications and then prepares for release of the software product to the entirecustomer base.

Integrated Systems Testing

Integrated System Testing (IST) is a systematic technique for validating theconstruction of the overall Software structure while at the same time conducting teststo uncover errors associated with interfacing. The objective is to take unit testedmodules and test the overall Software structure that has been dictated by design. ISTcan be done either as Top down integration or Bottom up Integration.

Top Down: Approach is to check the integrity starting from the apex of the systemand moving down to check integrity of various modules

Main

Module 1 Module 2


8/71

Bottom up: Approach is to check the integrity starting from a module and checkingthe integrity to the apex.

User Acceptance Testing

User Acceptance Testing (UAT) is performed by Users or on behalf of the users toensure that the Software functions in accordance with the Business RequirementDocument. UAT focuses on the following aspects:

All functional requirements are satisfied All performance requirements are achieved Other requirements like transportability, compatibility, error recovery etc. are

satisfied Acceptance criteria specified by the user are met.

Difference between IST and UAT

Particulars IST UATBaseline Document Functional Specification Business RequirementData Simulated Live DataEnvironment Controlled Simulated LivePerspective Functionality User styleLocation Off Site On SiteTester Composition Tester Company Test company & Real

UsersPurpose Validation & Verification User Needs

Main

Section 1 Section 2

Module 1


9/71

Performance Testing

Performance testing is designed to test run time performance of software within thecontext of an integrated system. It is not until all systems elements are fully integratedand certified as free of defects the true performance of a system can be ascertained.

Performance tests are often coupled with stress testing and often require bothhardware and software infrastructure. That is, it is necessary to measure resourceutilization in an exacting fashion. External instrumentation can monitor intervals, logevents. By instrumenting the system, the tester can uncover situations that lead todegradations and possible system failure.


10/71

2.0 Test Preparation Process

2.1 Baseline Documents

Construction of an application and testing are done using certain documents. Thesedocuments are written in sequence, each of it derived from the previous document.

2.1.1 Business Requirement

This document describes users needs for the application. This is done over a period of time, and going through various levels of requirements. This should also portraysfunctionality that are technically feasible within the stipulated times frames for delivery of the application.

As this contains user perspective requirements, User Acceptance Test is based on thisdocument.

2.1.2 How to read a Business Requirement?

In case of the Integrated Test Process, this document is used to understand the user requirements and find the gaps between the User Requirement and FunctionalSpecification.

User Acceptance Test team should break the business requirement document intomodules depending on how the user will use the application. While reading thedocument, test team should put themselves as end users of the application. Thisdocument would serve as a base for UAT test preparation.

2.1.3 Functional Specification

This document describes the functional needs; design of the flow and user maintained parameters. These are primarily derived from Business Requirement document, whichspecifies the clients business needs.

The proposed application should adhere to the specifications specified in thisdocument. This is used henceforth to develop further documents for softwareconstruction and validation and verification of the software.

In order to achieve synchronisation between the software construction and testing process, Functional Specification (FS) serves as the Base document.

2.1.4 How to read a Functional Specification?


11/71


12/71

The tester must also bear in mind, the test type i.e. Integrated System Testing (IST) or User Acceptance Testing (UAT). Based on this, he should orient his testing approach.

2.1.6 Design Specification

This document is prepared based on the functional specification. It contains thesystem architecture, table structures and program specifications. This is ideally

prepared and used by the construction team. The Test Team should also have adetailed understanding of the design specification in order to understand the systemarchitecture.

2.1.7 System Specification

This document is a combination of functional specification and design specification.This is used in case of small applications or an enhancement to an application. Under such situations it may not be advisable make two documents.

2.2 Prototype

This is look and feel representation of the application that is proposed. This basicallyshows the placement of the fields, modules and generic flow of the application. Themain objective of the prototype is to demonstrate the understanding of the applicationto the users and obtain their buy-in before actual design and construction begins.

The development team also uses the prototype as a guide to build the application

This is usually done using HTML or MS PowerPoint with user interaction facility.

2.2.1 Scenarios in Prototype

The flow and positioning of the fields and modules are projected using several possible business scenarios derived from the application functionality.

Testers should not expect all possible scenarios to be covered in the prototype.

2.2.2 Flow of Prototype

The flow and positioning are derived from initial documentation on the project. A project is normally dynamic during initial stages, and hence tester should bear in mind


13/71

the changes to the specification, if any, while using the prototype to develop testconditions.

It is a value addition to the project when tester can identify mismatches between thespecifications and prototype, as the application can be rectified in the initial stages

itself.


14/71

2.3 Test Strategy

Actual writing of a strategy involves aspects, which define other issues between theTesting organization and the client. Testers must basically understand some of theissues that are discussed in the strategy document, which are outlined below.

2.3.1 Testing Approach

The testing process may take the form of an End-to-End approach or individualsegment testing using various values.

End-to-End: The test path uses the entire flow provided in the application for completion of a specified task. Within this process various test conditions and valuesare covered and results analyzed. There maybe a possibility of reporting severaldefects relating to the segments while covering the test path. The advantage of usingthis approach is to minimize combination and permutation of conditions/values andensure coverage and integration.

Individual Segment Testing: Several conditions and values are identified for testingat the unit level for testing. These are tested as separate cases.

2.3.2. Automation Strategy

Automation of testing process is done to reduce the effort during regression testing. Insome cases automating the entire testing process may not possible due to technicaland time constraints. The possible automation strategies that could be adopteddepending on the type of the project are

Selective: Critical and complex cases are identified. These test cases are generallyautomated to simplify the testing process and save time.

Complete: As the term suggests, all test cases technically possible are automated.

2.3.3 Performance Strategy

The client specifies the standards for the performance testing. It generally contains

Response time Number of Virtual Users

Using the above information, a Usage Pattern of the application is derived anddocumented in the strategy. Issues discussed in the performance strategy documentare

Please refer to Annexure One for complete details of a Test Strategy


15/71

Resources: Personnel trained in Performance testing tool identified. Datewiseutilization of the resources is laid down.

Infrastructure: Generation of virtual users require huge amount of RAM. The performance team should be given a machine, which is suitable for the performancetool.

Report: The type of report that will be generated after the tests are discussed. Reportsare ideally in the form of graphs. Reports generated are:

Detailed Transaction Report (By Virtual user) Throughput Graph Hits per second Graph Transaction per second Transaction Response Time Graph Transaction Performance Summary Graph Transaction Distribution Graph Transaction Performance Summary Graph

Performance Test Risk: This is similar to IST of UAT risks, which are dealt in 2.3.4

2.3.4 Risk Analysis

Risks associated with projects are analyzed and mitigations are documented in thisdocument. Types of risk that are associated are

Schedule Risk: Factors that may affect the schedule of testing are discussed.

Technology Risk: Risks on the hardware and software of the application are discussedhere

Resource Risk: Test team availability on slippage of the project schedule is discussed.

Support Risk: Clarifications required on the specification and availability of personnel for the same is discussed.

2.3.5 Effort Estimation

The function points in the Functional Specifications will be used, as the basis for the purpose of estimating the effort needed for the project. The average of the differentestimates from the Peers in the test team will be taken as the basis for calculation of the effort required.

There could be some variation in the planned to actual effort. An effort estimationreview will be done by a Senior Consultant to identify gaps, if any.

In case of the UAT, function points are taken from the Business Requirementdocument.


16/71

2.3.6 Infrastructure

Hardware and software requirements for the testing the application are documented.

Apart from this, any other requirement should also be documented. Infrastructure thathas to be provided by the client is also specified.

Test Strategy

Milestone


17/71

2.4 High Level Test Conditions

2.4.1 What is a High Level Test Condition

It represents the possible values that can be attributed to a particular specification. Theimportance of determining the conditions is:

Deciding on the architecture of testing approach Evolving design of the test scripts Ensuring coverage

2.4.2 Understanding the maximum conditions for a specification

At this point the tester will have a fair understanding of the application and hismodule. The functionality can be broken into

Field level rules Module level rules Business rules Integration rules Processing logic

It may not be possible to segment the specifications into the above categories in allapplications. It is left to the test team to decide on the applicable segmentation.

For the segments identified by the test team, the possible condition types that can be built are

Positive condition: Polarity of the value given for test is to comply with the conditionexistence.

Negative condition: Polarity of the value given for test is not to comply with thecondition existence.

Boundary condition: Polarity of the value given for test is to assess the extremevalues of the condition

User Perspective condition: Polarity of the value given for test is to analyse the practical usage of the condition

ExampleCondition Positive Negative Boundary User Perspective

Interest Percentagefor a deposit

10 -10 100, 101, 99 9.4325 (with four decimals)


18/71

2.4.3 Queries on Functional Specification

Preparation of test conditions would lead to certain queries arising because of

Gap between the understanding of the tester and specification Implied specification Prototype being contradictory Implementation issues Design restrictions Contradictions within the functional specification document Contradictions between other application documents Contradictions between functional specification and real time applicability

These queries can be clarified using the following resources

Domain Consultant: Normally the author of the FS, and an expert in the field of theapplication.

Test Manager: Person responsible for the management and co-ordination of the project.

Test Lead: Person responsible for the testing processes and team.

Peer Group: Other team members, who are generally in the test team.

2.4.4 Determining the critical path

All the possible values for the segment (in most cases the fields) should be listedhorizontally using a spreadsheet (in IST). Such columns for all the segments puttogether will form a matrix.

In the cases of an End-to-End test approach, each row of the spreadsheet willdetermine the test path provided there are no values that will hinder the progress of the test path. To determine the test paths, the following procedure needs to be adapted

1. Determine the segment with maximum number of values, which will give the total paths for the module.

ExampleFS Reference: 3.2.1.DepositAn order capture for deposit contains fields like Client Name, Amount, Tenor and interestfor the deposit.Business Rule: If tenor is great than 10 months interest rate should be greater than 10% else a

warning should be given by application.

If Tenor greater than 12 months, then the order should not proceed.


19/71

Test ScriptID

ClientName

Amount Tenor Interest Warning

Dep/01 123 >0 12 months 0


20/71

Test ScriptID

ClientName


Dep/01 123 >0 12 months 00 0 No Warning

4. In the row, understand the implication of combining the values into a single path

5. Attach the NOGO conditions in the appropriate segment and fill the gaps, like instep two

Test ScriptID

ClientName


Dep/01 123 >0 12 months 0


21/71

Test ScriptID

ClientName


Dep/06 abc >0 Invalid date >100 No WarningDep/07 abc >0 0 No Warning

Dep/08 abc >0 13 months Nogo

6. While filling the gaps in the NOGO path, it is advisable to use the values whichwhere previously used in a End-to-End path

In case of a single test case approach, each of the value pertaining to a segment willform a test case.

In the cases of UAT, the conditions are given to the clients for their review. The process of determining the conditions using the Business Requirement is the same.

Finally this tabular format is converted into UAT report format.The format with which conditions are delivered to the clients is given below with anexample taken from the first test path Dep/01

Scenario: Dep 01

Valid Client Name Amount greater than zero Tenor greater than 10 months Interest rate lesser than 10%

Conditions Checked

Warning when tenor is greater than 10months and interest rate less than 10%

Order should be successfully captured

Note that Dep/08 which has the NOGO condition does not have a value Interest fieldthe application should bail the user out of order capture.

Other fields filled in the NOGO have already been tested, and it is only a duplicate or repeated again.


22/71

2.4.5 Intelligent Testing

Most testers would tend to follow a combination and permutation method for arrivingat the test conditions. The above section clearly explains a way in which coverage of

the test values using a non combination and permutation would suffice completetesting.

This option may not be applicable for all the applications.

Another important part of intelligent testing would be to reduce the number of testvalues for a segment. Generally, more than one negative test value can beincorporated for one segment in one test path.

Test Conditions

Milestone


23/71

2.5 Traceability

2.5.1 BR and FS

The requirements specified by the users in the business requirement document maynot be exactly translated into a functional specification. Therefore, a trace onspecifications between functional specification and business requirements is done on aone to one basis. This helps finding the gap between the documents. These gaps arethen closed by the author of the FS, or deferred after discussions.

Testers should understand these gaps and use them as an addendum to the FS, after getting this signed off from the author of the FS. The final FS form may vary from theoriginal, as deferring or taking in a gap may have ripple effect on the application.Sometimes, these ripple effects may not be reflected in the FS. Addendums maysometime affect the entire system and the test case development. There the traceablitydiscussed in section 2.5.2 gains importance

2.5.2 FS and Test Conditions

Test conditions built by the tester are traced with the FS to ensure full coverage of the baseline document. If gaps between the same are obtained, tester must then buildconditions for the gaps. In this process, testers must keep in mind the rules specifiedin test condition writing (3.4.4).

2.5.3 Gap Analysis

This is the terminology used on finding the difference between what it should be andwhat it is. As explained, it is done on the Business requirement to FS and FS to testconditions. Mathematically, it becomes evident that Business requirements that areusers needs are tested, as Business Requirement and test conditions are matched.

Simplifying the above,

A = Business RequirementB = Functional SpecificationC = Test Conditions

A = B, B = C, Therefore A = C

Another way of looking at this process is to eliminate as many mismatches at everystage of the process, there by giving the customer an application, which will satisfytheir needs.

In the case of UAT, there is an direct translation of specification from the Business

Requirement to Test Conditions leaving lesser amount of understandability loss.


24/71

2.5.4 Tools Used

The entire process of traceability is a time consuming process. In order to simplify,Rational Software Incorporated has developed a tool, which will maintain the

specifications of the documents. Then these are mapped correspondingly. Thespecifications have to be loaded into the system by the user. Even though it is a timeconsuming process, it helps in finding the ripple effect on altering a specification.The impacts on test conditions can immediately be identified using the trace matrix.

Traceability

Milestone


25/71

2.6 Testbed

2.6.1 High Level Planing

In order to test the conditions and values that are to be tested, the application should be populated with data. There are two ways of populating the data into tables of theapplication.

Intelligent: Data is tailor-made for every condition and value, having reference to itscondition. These will aid in triggering certain action by the application. Byconstructing such intelligent data, few data records will suffice for the testing process.

Unintelligent: Data is populated in mass, corresponding to the table structures. Itsvalues are chosen at random, and not with reference to the conditions derived. Thistype of population can be used for testing the performance of the application and its

behavior to random data. It will be difficult for the tester to identify his requirementsfrom the mass data.

Having now understood the difference between intelligent and unintelligent data, andalso at this point having a good idea of the application, the tester should be able todesign intelligent data for his test conditions.

Application may have its own hierarchy of data structure, which are interconnected.

Example:

Business rule, if the Interest to be Paid is more than 8 % and the Tenor of the deposit exceeds one month,then the system should give a warning.

To populate an Interest to be Paid field of a deposit, we can give 9.5478 and make the Tenor as two monthsfor a particular deposit.

This will trigger the warning in the application.

Example:Using the above example, to find a suitable record with Interest exceeding 8% and the Tenor being morethan two months is difficult.


26/71

ExampleA client may have different accounts in different locations. Each of these locationsmay have multiple account types.

Based on the above structure, and the conditions, tester must carefully match the

individual or End-to-End scenarios for test to this data hierarchy.

In cases of an End-to-End testing, the same client can be used for multiple test paths,varying the location and/or accounts. Tester must also note the dependencies in thedata hierarchy to design the data.

Each condition in the test path will have a specific data associated with it. So, whenthe test path is executed, the tester will be sure of triggering the conditions he hasdesigned the test case for.

Below is a real time illustration of how high level data design is made.

Client

London Chicago

Loan Current Deposit Saving DepositSaving


27/71


28/71

MMSurplus/Shortfall After in MM 1.508

For a particular test script xx/001, the type of accounts required and clients types are mentioned clearly.Also, the values that will be entered during the order capture are also decided prior to the test execution

leaving no surprises to Tester.Validations and business rules that happen after some events are also calculated.


29/71

2.6.2 Feeds analysis

Most applications are fed with inputs at periodic intervals, like end of day or everyhour etc. Some applications may be stand alone i.e., all processes will happen within

its database and no external inputs of processed data is required.

In the case of applications having feeds, received from other machines, they are sentin a format, which are predesigned. These feeds, at the application end, will be

processed by local programs, and populated in respective tables.

It is therefore, essential for testers to understand the data mapping between the feedsand the database tables of the application. Usually, a document is published in thisregard.

Translation of the high level data designed previously should be converted into thefeed formats, in order to populate the applications database.

2.6.3 Feeds format

A feed format is translated or converted in a tabular format in Microsoft Word, withall the fields as the table headings. Testers then begin to convert the high level datainto application readable format through this document.

These are imported into excel and finally into MS Access database. They are thenconverted into actual feed files, depending on the platform on which the applicationworks.

Incases of, stand alone application a similar process should be adopted, convertinghigh level data into table structure data for populating the tables.

Below is a real time sample of feed format for cash accounts.


30/71

Asset class: Cash & Cash Equivalents

PortfolioNo.

AccountNo.

Ass.SubType

Status

OpeningBalance

ACA LineAmount

(Only for MB)

AccruedInterest

As of Date NominalCurrenc

y

Avl CashUp day 1 to

days

Avl Cashfrom ____ to

___ days

1/100001/001 1/100001/102

2 0 500000 0 USD 500000

Here, account 1/100001/102, which was described in the real time example of 2.6.1, is translated into feed format. Account number, Portfolio inwhich the account number is held, currency of the account, balances etc are explained clearly. There are also some application specific codesand status populated.


31/71

2.6.4 Final Set-up

The feed files are then uploaded into the application using a series of programs. Incase of unintelligent data, some tool would be used to generate mass data specific tothe application, by specifying the applications requirements to the tool. These will

then be uploaded in to application.

Once these are uploaded, data might have to be interconnected to the application business logic. This may be necessary for both types of applications, stand-alone andfeeds fed application.

In the case of UAT, the test team does not simulate test data. Live production datafrozen in a separate UAT environment is used for executing test cases.

ExampleLinking accounts to Location Singapore and then linking them to a client using some codenumbers, which may be unique to a client.

Test Data

Milestone


32/71

2.7 Test Case

2.7.1 Test Case Formation

At this stage, the Tester has clarity on how the application is to be tested. It now, becomes necessary to aid the actual test action with test cases. Test cases are written based on the test conditions. It is the phrased form of test conditions, which becomesreadable and understandable by all.

2.7.2 Explicit writing

There are three headings under which a test case is written. Namely,

Description: Here the details of the test on a specification or condition are written

Data and Pre-requirements: Here either the data for the test or specification ismentioned. Pre-requirements for the test to be executed should also be clearlymentioned.

Expected Results: The expected result on execution of the instruction in thedescription is mentioned. In general, it should reflect, in detail the result of the testexecution.

While writing a case, to make the test case explicit, the tester should include thefollowing

Reference to the rules and specifications under test in words with minimaltechnical jargons

Check on data shown by the application should refer to the table names if possible Location of the fields or if a new window displayed must be specified clearly Names of the fields and screens should also be explicit.

2.7.3 Expected Results

The out-come of executing an instruction would have a single or multiple impact onthe application. The resultant behavior of the application after test execution is theexpected result.

Single Expected Result: Has a single impact on the instruction executed

ExampleDescription: Click on the hyperlink New Deposit at the top left hand corner of the MainMenu ScreenExpected Result: New Time deposit Screen should be displayed


33/71

Multiple Expected Result: Has multiple impact on executing the instruction

Language used in the expected results should not have ambiguity. The resultsexpressed, should be clear and have only one interpretation possible. It is advisable touse the term Should in the expected results.

2.7.4 Pre-Requirements

Test cases cannot generally be executed with normal state of the application. Below isa list of possible pre-requirements that could be attached to the test case:

1. Enable or disable external interfaces

2. Time at which the test cases is to be executed

3. Dates that are to be maintained (pre-date or post-date) in the database beforetesting, as its sometimes not possible to predict dates of testing, and populatecertain date fields when they are to trigger certain actions in the application

ExampleDescription: Click on the hyperlink New Deposit at the top left hand corner of the Main Menu ScreenExpected Result: New Time deposit Screen should be displayed

Customer contact date should be prefilled with the system date

ExampleReuters, a Foreign exchange rate information service organization server to be connected to the application

Example Test to be executed after 2:30 p.m. in order to trigger a warning

ExampleMaturity date of a deposit should be the date of test. So, it is difficult to give the value of the maturity date while

data designing or preparing test cases.


34/71

4. Deletion of certain records to trigger an action by the application

5. Change values if required to trigger an action by the application

2.7.5 Data definition

Data for executing the test cases should be clearly defined in the test cases. Theyshould indicate the values that will be entered into the fields and also indicate thedefault values of the field.

In the cases of calculations involved, the test cases should indicate the calculatedvalue in the expected results of the test case.

In brief, the entire process should be within the control of the tester, and no action isoutside the testers anticipation.

Example An document availability indicator field to be made null, so as to trigger an warning from the application

ExampleChange the value of the interest for a deposit so as to trigger a warning by the application

ExampleDescription: Enter clients nameData: John SmithORDescription: Check the default value of the interest for the depositData: $100

ExampleDescription: Check the default value of the interest for the depositData: $100This value ($100) should be calculated using the formula specified well in advance while data design.


35/71

2.8 Test Script

2.8.1 Brief on Test Scripts

This will sequence the flow of an End-to-End test path or sequence of executing theindividual test condition.

Test case specifies the test to be performed on each segment. Though the sequences of a path are analyzed, navigations to test conditions are not available in the test cases.

Test scripts should ideally start from the login screen of the application. Doing thishelps in two ways

Start conditions are always the same, and uniformity can be achieved Automation of test scripts requires start and end conditions i.e. the automation tool

will look for the screen to be the same, as specified in its code. Then the tool willautomatically run the series of cases without intervention by the user. So, the testscripts must start and end in the same originating screen.

The test scripts must explain the navigation paths very clearly and explicitly. Theobjective of this is to have flexibility on the person who would execute the cases.

Test scripts sequences must also take into account the impacts the previous cases i.e.in cases of deletion of certain record, the test should not flow by searching for detailsof the same.

In short, the test cases in series will form the test script in case of an End-to-End testapproach. In individual test conditions, the navigation and the test instruction will be atest case and this will constitute a test script.

In practice, for End-to-End test approach, test scripts are written straightwayincorporating the test cases. It is only for explanation, these were categorized into twosteps.

Given below is the format used for writing the test script in IST and UAT.


36/71

IST Test Script Format

Test Script

Feature Ref.: Test Script ID:

TestCaseNo:

Test Case Description Data Expected ResultsResultPass 1

ResultPass 2

DefectLog No Comments


37/71

UAT Test Script Format

ACCEPTANCE TEST SCRIPT

PROJECTUSER DEPT. PREPARED BY:PASS NO: SCRIPT NO:FEATURE TESTED: REQUIREMENT ID:

CONDITIONS:

INSTRUCTIONS TO EXECUTE TEST:

1.2.

EXPECTED RESULTS:

1.2.

SUCESS/FAILURE TESTED BY: DATE:

Conditions that are tested for this test script i.e. from the corresponding scenario are entered here. Test description isentered in sequence as steps in the Instruction to execute test and the expected results are entered ExpectedResults section in sequence corresponding to the test description.

In cases of data requirement they are filled either in test description or expected result depending on the case.


38/71

2.8.2 Interaction with development team

Interaction between the testing team and development team should begin whilewriting the test scripts. Any interaction prior to test case writing would ideally bias

both the teams. Screen shots of the various screens should be obtained from the

development team as well as interact on the design of the application.

The tester should not make any changes to the test script at this point based on whatthe development team has presented. The contradictions between the Test Scripts andactual application are left to Project Managers decisions only. Recommendations fromthe test team on the contradiction would be a value addition.

2.8.3 Review of Test Scripts

Test cases are given to project leader and managers in think soft for review. The testscripts are then sent to the Client for review. Based on the review, changes have to bemade to the entire block that was built i.e. test conditions, test data and test scripts.The Client then marks their comments in the comments column in the test

preparation script. Testers should understand that, if a change is made in the test scriptthen it requires, changes in the test conditions and data.

2.9 Activity Report

Test lead should report to his/her test manager on day to day activity of the test team.The report should basically contain plan for the next day, activities pending, activityfor the day-completed etc.

2.10 Backend Testing

The process of testing also involves the management of data, which at times arerequired to flow as an input into the application referred to as feeds hence, fromexternal data stores / applications or data that is generated within the scope of thesame application. The required data is to be extracted and processed to produceinformation as per requirements specified by the business requirements document.

The process of absorbing data into an application could be varied, depending onfactors like:

Nature of data required Format of data received Limitations of the system supplying data, in presenting data in a required

pattern.


39/71

2.10.1 Understanding the application

The understanding of the application, as specified in the Business requirements andFunctional Specification, plays a key role in testing. The business requirements andthe functional specifications draw a parallel between the requirements and the

offering of the proposed system. This gives an understanding of the data requirementsfor the application.

2.10.2 Data Feeds Management

The process of Data Feeds Management involves the study of the data requirementsof the business, the gathering of data in a format most suited to handle the process of upload and presentation of data. The Project Management team does this study, andthe outcome is a document called the Data Acquisition Document, that identifies thedata required.

2.10.3 Data Requirements for Integrated System Testing (IST)

The data requirements expected of the application given by the functionalspecification may not be dealt in an exhaustive manner. Data requirements could berelated to both feeds and maintenance that are required to control program execution.

The Data Acquisition Document gives the input regarding the data feeds intended for the application, its format, the description of each field, also referred to as the dataelement, the data type and its size. This forms the basis for the data content of theapplication.

The data for use by IST team either can be in the form of live feeds from the ProductProcessors or simulated feeds. The possibility of receiving live feeds from ProductProcessors, for the purpose of testing cannot be exercised always due to variousreasons, which may vary from secrecy issues to problems in supplying the requireddata. To overcome this feeds are simulated to substitute for the live feeds. The feedsare generated after the study of the layouts and various data conditions that may berequired to test the functionality of programs that are used to upload them into theapplication.

The Data Generation Tool , developed indigenously is used to automate the task of generating simulated feeds. This is used to generate volumes of data, which at timescould be unintelligent data depending on then complexity of the data requirement.

The tool needs to be customised, to generate data as per the requirements of theapplication under test.

2.10.4 Data Upload Process

Data feeds are uploaded and stored in the application database, by executing asequence of programs. The Design Specification Document contains information


40/71

about tables used in the application database. The development team provides theinputs relating to the programs that are to be executed, their sequence of execution,the functionality of each program, the maintenances required to ensure proper

program execution and the tables acted upon by the program and changes caused to it.

The impact the programs have on the business process, their status and the resultstracking mechanism are to be understood. The representation of data in tables,correlation of data across tables and their representation in the user interface are vitalcomponents that help ensure the verification and conformance of the data displayedfor its preciseness.

2.10.5 Test Preparation

The test conditions are arrived using the inputs regarding the functionality of the process as provided. The process of deriving test conditions and test casessubsequently is the same as described in earlier sections. The results of the test casescould reflect in the application interface or may be required to be checked in thetables.

The program is run and the percentage contributions checked as displayed in the user interface. Also, the percentages could be calculated in the database itself and checkedwith the corresponding percentages displayed in the user interface.

2.10.6 Master and Parameter Tables

An application could contain various operation modes, support varied products andhave varying values as inputs depending on the product category serviced or theoperation modes. This entails the use of tables that contain the list of values that could

be in use. To control the application in performing differing actions in response to a particular input value, parameter tables are used to identify the possible permissiblevalues, which also could be used to control application flow.

2.10.7 Maintenance

ExampleConsider the case of a program that is supposed to calculate percentage of contribution of each accountheld to the assets or liabilities of the customers holding. The program in this case has to group thecustomers total assets and liabilities and calculate the percent contribution of each accounts asset or liability in the totality of the customers assets or liabilities.


41/71

Before the process of testing is commenced, verification is to be carried out to check that the tables in the database are populated with the required data. This specificationwill be available as part of the program functionality and the task on checking itscorrectness and ensuring that it adheres to the specifications lies with the tester.

2.11 Non Certified Testing

Testing may sometimes have to be performed on application, which do not have baseline documents. This situation may arise when

Application is already in use and not accepted by the users Enhancement to the application is proposed

Under these situations, conventional process explained may not be possible. Under those circumstances the following preparation methodology is used

2.11.1 Documents

All possible documents should be collected from the client regarding the application.Testers must go through these documents, understand and extract as many possibleinformations as possible.

Most important of it would be the data base layout document. This document would

explain data structure and layout.

2.11.2 Screen Shots

Screen shots from the application should be captured. These screen shots shouldrepresent the application screen by screen and maintaining the flow pattern of theapplication.

2.11.3 Mapping

Tester at this point will have both the database and front-end screen shots. Carefullydata base should be mapped by understanding the entries made in front end (input)and values displayed in front end (output).The purpose of each field, screens andfunctionality should also be understood. The tester should arrive at clarity on the inputand output of the application.

In these cases, tester should use his discretion to decide the validations required atfield, Module and application level depending on the application purpose.

Once these are done then the test team can start building test conditions for theapplication and from then on proceed with the normal test preparation style.


42/71

Form the above, one can infer that the process only verifies what the application cando and not validate its integrity. Hence it is not possible to certify and guarantee theapplication.


43/71

Test Scripts &Preparation

Milestone


44/71

3.0 Test Execution Process

The preparation to test the application is now over. The test team should next plan theexecution of the test on the application. In this section, we will see how test executionis performed.

3.1 Stages of Testing

3.1.1 Three passes

Tests on the application are done on stages. Test execution takes place in three passesor sometimes four passes depending on the state of the application. They are:

Pre IST or Pass 0: This is done to check the health of the system before the start of the test process. This stage may not be applicable to most test process. Free formtesting will be adopted in this stage.

Comprehensive or Pass 1: All the test scripts developed for testing are executed.Some cases the application may not have certain module(s) ready for test, hence theywill be covered comprehensively in the next pass. The testing here should not onlycover all test cases but also business cycles as defined in the application.

Discrepancy or Pass 2: All Test scripts that have resulted in a defect during thecomprehensive pass should executed. In other words, all defects that have been fixedshould be retested. Function points that may be affected by the defect should also betaken up for testing. Automated test scripts captured during the pass one are used here.This type of testing is called as Regression testing. Defects that are not fixed will beexecuted only after they are fixed.

Sanity or Pass 3: This is the final round in the test process. This is done either at theclients site or at test lab depending on the strategy adopted. This is done in order tocheck if the system is sane enough for the next stage i.e. UAT or production as thecase may be under a isolated environment. Ideally the defects that are fixed from the

previous pass are checked and freeform testing done to ensure integrity is conducted.

These categories apply for both UAT and IST.

3.2 Pre- Requirements for Testing

3.2.1 Version Identification Values

The application would contain several program files for it to function. The version of these files and an unique identification number for these files is a must for changemanagement.


45/71

These numbers will be generated for every program file on transfer from thedevelopment machine to the test environment. The number attributed to each programfile is unique and if any change is made to the program file between the time it istransferred to the test environment and the time when it is transferred back to thedevelopment for correction, it can be detected by using these numbers. These

identification methods vary from one client to another.

These values have to be obtained from the development team by the test team. Thishelps in identifying unauthorized transfers or usage of application files by both partiesinvolved.

The responsibility of acquiring, comparing and tracking before and after softbasetransfer lies with the test team.

3.2.2 Interfaces for the application

In some applications external interfaces may have to connected or disconnected. In both cases the development team should certify that the application would function inan integrated fashion. Actual navigation to and from an interface may not be coveredin black box testing.

3.4.3 Unit and Module test plan sign off

To begin an Integrated test on the application, development team should havecompleted tests on the software at Unit and module levels.

Unit and Module Testing: Unit testing focuses verification effort on the smallest unitof software design. Using the Design specification as a guide, important control pathsand field validations are tested. This is normally a white box testing.

Clients and the development team must sign off this stage, and hand over the test planand defect report for the test to the testing team.

In cases of the UAT, IST sign off report must be handed over to the UAT beforecommencement of UAT.

3.3 Test Plan

This document is a deliverable to client. It contains actual plan for test execution withdetails to the minute.

3.3.1 Test Execution Sequence

Test scripts can either be executed in a random format or in an sequential fashion.

Some applications have concepts that would require sequencing of the test cases before actual execution. The details of the execution are documented in the test plan.


46/71

Sequencing can also be done on the modules of the application, as one module would populate or formulate information required for another.


47/71

3.3.2 Allocation of test cases among the team

The Test team should decide on the resources that would execute the test scripts.Ideally, the tester who designed the test script for the module executes the test. Insome cases, due to shortage of time or resource at that point of time, additional test

scripts might have to be executed by some members of the team.

Clear documentation of responsibilities is done in the test plan.

3.3.3 Allocation of test cases on different passes

All test scripts may not be possibly executed in the first passes. Some of the reasonsfor this could be

Functionality may some-times be introduced at a later stage and application maynot support it, or the test team may not be ready with the preparation

External interfaces to the application may not be ready The client might choose to deliver some part of the application for testing and rest

may be delivered during other passes

3.3.4 Targets for completion of Phases

Time frames for the passes have to decided and committed to the clients well inadvance to the start of test. Some of the factors consider for doings so are

Number of cases/scripts: Depending on the number of test scripts and the resourceavailable, completion dates are prepared

Complexity of Testing: In some cases the number of test cases may be less but thecomplexity of the test may be a factor. The testing may involve time consumingcalculations or responses from external interfaces etc

Number of Errors: This is done very exceptionally, Pre-IST testing is done to check the health of the application soon after the preparations are done. The number of errors that were reported should be taken as a benchmark.

Milestone

Annexure Two shows the Test Plan Template normally used for preparing a Test Plan for a project.


48/71

3.4 Automation of Test Cases

3.4.2 Capturing During First pass

As, we now understand, the tool has to capture the test sequence, inputs, outputs and

calculations involved in the test cases. This is done during the first pass of the test.While executing the test cases, testers should capture them using the tool. Automationexperts in the team will provide guidance.

3.4.3 Intelligent Automation

Like intelligent testing, automation can also be chosen carefully. Depending on thestrategy i.e. Complete or selective the test cases are automated.

In cases of selective, only critical and complex test cases are automated. Time-consuming test cases are also automated, as it would take less time for regressiontesting.

3.5 Defect Management

3.5.1 What is a defect?

A Defect is a product anomaly or flaw. Defects include such things as omissions andimperfections found during testing phases. Symptoms (flaws) of faults contained insoftware that is sufficiently mature for production will be considered as defects.Deviations from expectation that is to be tracked and resolved is also termed a defect.

An evaluation of defects discovered during testing provides the best indication of software quality. Quality is the indication of how well the system meets therequirements. So in this context defects are identified as any failure to meet thesystem requirements.

Defect evaluation is based on methods that range from simple number count to

rigorous statistical modelling.

Rigorous evaluation uses assumptions about the arrival or discovery rates of defectsduring the testing process. The actual data about defect rates are then fit to the model.Such an evaluation estimates the current system reliability and predicts how thereliability will grow if testing and defect removal continue. This evaluation isdescribed as system reliability growth modeling.

Life cycle of a defect is explained diagrammatically below,


49/71

Authorised

Raised

Fixed

Closed

Deferred

Reraised

Du licate


50/71

3.5.2 Types of Defects

Defects that are detected by the tester are classified into categories by the nature of thedefect. The following are the classification

Showstopper (X): The impact of the defect is severe and the system cannot go into the production environment without resolving the defect since an interim solution may not beavailable.

Critical (C): The impact of the defect is severe, however an interim solution is available.The defect should not hinder the test process in any way.

Non critical (N): All defects that are not in the X or C category are deemed to be in the Ncategory. These are also the defects that could potentially be resolved via documentationand user training. These can be Graphic User Interface (GUI) defects are some minor

field level observations.

3.5.3 Defect reporting by tester

Defects or Bugs when detected in the application by the tester must be duly reportedthrough an automated tool. Particulars that have to be filled by a tester are

Defect Id: Number associated with a particular defect, and henceforth referred by its ID

Date of execution: The date on which the test case which resulted in a defect wasexecuted

Defect Category: These are explained in the next section, ideally decided by the testleader

Severity: As explained, it can be Critical, Non-Critical and Showstopper

Module ID: Module in which the defect occurred

Status: Raised, Authorised, Deferred, Fixed, Re-raised, Closed and Duplicate.

Defect description: Description as to how the defect was found, the exact steps thatshould be taken to simulate the defect, other notes and attachments if any.

Test Case Reference No : The number of the test case and script in combination whichresulted in the defect

Owner: The name of the tester who executed the test case

50


51/71

Test case description: The instructions in the test cases for the step in which the error occurred

Expected Result: The expected result after the execution of the instructions in the testcase descriptions

History of the defect: Normally taken care of the automated tool used for defect trackingand reporting.

Attachments: The screen shot showing the defect should be captured and attached

Responsibility : Identified team member of the development team for fixing the defect.

3.5.4 Defect Tracking by Test Lead

The test lead, categorizes the defects after meetings with the clients as, Modify Cases: Test cases to be modified. This may arise when the testers understandingmay be incorrect.

Discussion Items: Arises when there is a difference of opinion between the test and thedevelopment team. This is marked to the Domain consultant for final verdict.

Change Technology: Arises when the development team has to fix the bug.

Data Related: Arises when the defect is due to data and not coding.

User Training: Arises when the defect is not severe or technically not feasible to fix, it isdecided to train the user on the defect. This should ideally not be critical.

New Requirement: Inclusion of functionality after discussion

User Maintenance: Masters and Parameter maintained by the user causing the defect.

Observation: Any other observation, which is not classified in the above categories like auser perspective GUI defect.

Reporting is done for defect evaluation and also to ensure that the development team isaware of the defects found and is in the process of resolving the defects. A detailed reportof the defects is generated everyday and given to the development team for their feedback on defect resolution. A summary report is generated for every report to evaluate the rateat which new defects are found and the rate at which the defects are tracked to closure.

Defect counts are reported as a function of time, creating a Defect Trend diagram or report, and as a function of one or more defect parameters like category or status, creating

51


52/71

a Defect Density report. These types of analysis provide a perspective on the trends or distribution of defects that reveal the systems reliability, respectively.

It is expected that defect discovery rates will eventually diminish as the testing and fixing progresses. A threshold can be established below which the system can be deployed.

Defect counts can also be reported based on the origin in the implementation model,allowing detection of weak modules, hot spots, parts of the system that are fixedagain and again, indicating some fundamental design flaw.

Defects included in an analysis of this kind are confirmed defects. Not all reporteddefects report an actual flaw, as some may be enhancement requests, out of the scope of the system, or describe an already reported defect. However, there is a value to looking atand analysing why there are many defects being reported that are either duplicates or notconfirmed defects.

3.5.5 Tools Used

Tools that are used to track and report defects are,

ClearQuest (CQ): It belongs to the Rational Test Suite and it is an effective tool inDefects Management. CQ functions on a native access database and it maintains acommon database of defects. With CQ the entire Defect Process can be customized. For e.g., a process can be designed in such a manner that a defect once raised needs to bedefinitely authorized and then fixed for it to attain the status of retesting. Such asystematic defect flow process can be established and the history for the same can bemaintained. Graphs and reports can be customized and metrics can be derived out of themaintained defect repository.

Test Director (TD): TestDirector is an Automated Test Management Tool developed byMercury Interactive for Test Management to help to organize and manage all phases of the software testing process, including planning, creating tests, executing tests, andtracking defects. TestDirector enables us to manage user access to a project by creating alist of Authorized users and assigning each user a password and a user group such that a

perfect control can be exercised on the kinds of additions and modifications an user canmake to the project. Apart from Manual Test Execution, the WinRunner automated testscripts of the project can also be executed directly from TestDirector. TestDirector activates WinRunner, runs the tests, and displays the results. Apart from the above, it isused for

To report defects detected in the software. As a sophisticated system for tracking software defects. To monitor defects closely from initial detection until resolution. To analyze our Testing Process by means of various graphs and reports.

52


53/71

3.5.6 Defects Meetings

Meetings are conducted at the end of everyday between the test team and developmentteam to discuss test execution and defects. Here, defect categorizations are done whichwere explained in section 4.5.4

Before meetings with the development team, test team should have internal discussionswith the test lead on the defects reported to the test lead. This process ensures that alldefects are accurate and authentic to the best knowledge of the test team.

53


54/71

3.5.7 Defects Publishing

Defects that are authorized are published in a mutually accepted media like Internet,Intranet, email etc. These are published in the Intranet and depending on the clientsrequirements, defects are published either in their Intranet or Internet.

Reports that are published are

Daily defect report Summarized defect report for the individual passes Final defect report

Format used for publishing the defects are given below with some examples.

54


55/71

Defect Report SampleDefe

ctID

Dateof

Testing

Mod

TestScriptID

Test CaseDescription

Defect Description Expected Results Severity

Status

Defect

Cat

Resp. Comments

1 28-Jun

xx xx/002/0

03

Navigate to NewItem screen through

Account detail SKr, book xx

Order Template is notdisplayed, when New

item screen is reachedthrough 'Action' in theaccount details screen.

Order Templateshould be displayed

to select the sourceaccount

Critical

Close

d

CT Sundar Tested and closed on22nd ju

2 28-Jun

xy xy/005/0

31

Memo BalanceIncorrect

The available balancein the security account

decreased by no of units * price entered in

the Sell Securitytemplate

Settlement account isdecreasing by the sell

order amount (no of units * price entered

in the sell Securitytemplate). Security

account, the amountshould decrease by no

of units * price(original price).

Critical

Close

d

UT Raj Reraised on 15th JulyClosed/UT by CI20th July). As pe

specification .[19-072000] Rajaram ha

clarified that thamount should b

calculated on the inpu price. This has bee

closed last week itself3 28-

Jun

xz xz/0

10/007

Sell Security with

account status =900.

After the NOGO

message the systemwas still in the sellsecurity template.

Sell security with

security status = 900is a NOGO status.After the NOGO

message the systemshould go the security

search screen.

Non

Critical

C

losed

CT Sundar The error message is

not right 'Error iobtaining templatReraised on 15th July

[19-07] Message ha been change

Transferred. RetesTested and reraised on

27th july. Tested andclosed on 8th Aug

55


56/71


57/71

3.6 Test Down Times

During the execution of the test, schedules prepared earlier may slip based on certainfactors. Time lost due to these should be recorded duly by the test team.

3.6.1 Server problems

Test team may come across problems with the server, on which the application is planted.Possible causes for the problems are

Main server on which the application may have problems with number of instances onit slowly down the system

Networking to the main server or internal network may get down Software compatibility with application and middleware if any may cause concerns

delaying the test start New version of databases or middleware may not be fully compatible with the

application Improper installation of system applications may cause delays Interfaces with applications may not be compatible with the existing hardware setup

3.6.2 Problems on Testing side / Development side

Delays can also be from the test or development teams like

Data designed may not be sufficient or compatible with the application (missing some parameters of the data)

Maintenance of the parameters may not be sufficient for the application to function Version transferred for testing may not be the right one Delay on transfer technique i.e. FATS may have some technical problems

3.6.3 Show Stopper

Schedule may not only slip because of the above-mentioned reason. Health of the software

may not be appropriate for testing, Like

Module Show Stopper: Testing on some components of the application may not be possible due to fundamental errors in it, stopping further testing

Application Show Stopper: Testing the application may not be possible due tofundamental errors in it, stopping further testing


58/71

Given below is a sample of a test down time log with examples

58


59/71

Downtime Log

DateTime

From To HrsApplication Machine Description Referredto Action taken

04-Jul-

00

9:45 12:20 2.35 Unable to

upload DGTdata, SMSServer Down

All

machines

K1 values

received 34files withoutmatchingK1 values.

Clients IST could

not bestarted.

05-Jul-00

10.23

12:41

4.45

11:02

1:07

5:15

.39

.26

.30

RM Modulefailure,deletion of filesfrom controlmachine,Server down,SMS Error

Allmachines

Clients IST - couldnot proceedwithexecution of test cases

3.7 Softbase Transfer

3.7.1 Between Passes

Softbase is the term used for describing the application software in the test andconstruction process. Control of softbase should be with either the development team or the test team depending on the process time frames i.e. whether construction or testing is

in progress. There should also be control on the version that is released for either construction or testing.

Softbase is transferred to the test environment after which the first pass of testing starts.During the first pass, the defects discovered are fixed in the development environment onthe same version of the softbase, which is being tested. At the end of the first pass, after the test team has completed the execution of all the test cases, the fixed version of thesoftbase is transferred into the test environment, to commence the second pass of testing.

The same process is continued till the end completion of all passes.

3.7.2 FTP

The Acronym of FTP is File Transfer Protocol. File Transfer is used to transfer one or multiple files form one system to another. Files can also be transferred one system toanother immaterial of the operating system there are functioning on. FTP supports twodifferent modes of transfer: Binary and ASCII

59


60/71

60


61/71

3.7.3 FATS

The Acronym of FATS is Fully Automated Transfer System. FATS is an Automatedversion control mechanism adopted by Citibank India Technologies for source codetransfer from Development server to Test Control Machine. In case of UAT source code is

transferred from Development server to UAT machine. FATS Uses SCCS (Source codecontrol system) of Unix for Version Control. Source code transfer will be transferred fromDevelopment server to FATS Server. FATS server generates a unique Identification for each Program file that is to be transferred. For completion of transfer there are threesecurity levels namely

Project Manager password User password Quality Assurance (QA) password.

Ideally, first the Project managers from the client side checks the file out of FATS transfer

to check its integrity. Then the user acknowledges the file for user requirements. Finally,QA clears the file on quality standards.

3.7.4 Revision of Test Cases

At times, transfer of softbase could take time from release for correction to transfer back into test environment. During the period testers should

Include test scripts for new functionality incorporated during execution Enrich data and test cases based on happenings during the previous pass of testing Complete documentation for the previous passes Modify test cases which are classified as MC Free form testing on the softbase available

3.8 Performance Testing

3.8.1 Execution

To execute the Performance Test, test cases are planned and developed for the application.The test cases constitute simulation of series of activities carried out by the user. These arecalled virtual scripts. Virtual scripts are generated by a Virtual Generator that runs in the

background creating test code using the specific navigation steps that the user performs. Ascript-recording device marks the test cases as transactions. Several such sets of scripts(scenarios) running simultaneously at different load conditions form the basis for Performance testing.

61


62/71

The analysis of response times provide a result on the behavior of application and of specific transactions, under different load conditions. A successful transaction passes thecriteria outlined for it in the User Acceptance defined in the specifications. A stabletransaction responds to the loading of extra concurrent users in a predictable manner with

regard to its operational norm of success as established in the User Acceptance criteria. Asuccessful and stable transaction fulfills both the above conditions.

3.8.2 Tools

LoadRunner enables you to test your system under controlled and peak load Conditions.To load your system, LoadRunner simulates an environment where Multiple users work concurrently. To generate load, LoadRunner runs multiple of Virtual Users that aredistributed over a network. Using a minimum of hardware resources, these Virtual Users

provide consistent, repeatable, and measurable load to exercise the system just as the real

users would. While the system is under load, LoadRunner accurately measures, monitors,and analyzes the systems Performance. LoadRunners in- depth reports and graphs provide the information that needed to evaluate the performance of the system.

Test Execution

Milestone

62


63/71

4.0 Post Test Process

4.1 Sign Off

4.1.1 Sign off Criteria

In order to acknowledge the completion of the test process and certify the application, thefollowing has to completed

All passes have been completed All test cases should have been executed All defects raised during the test execution have either been closed or deferred Show stoppers in the last pass of the test have been rectified

4.2.2 Authorities

The following personnel have the authority to sign off the test execution process

Client: The owners of the application under test.

Project Manager: Personnel who managed the project

Project Lead: Personnel who managed the test process

4.2 Deliverables

4.2.1 Internal

The following are the internal deliverable

Test Preparation Scripts: Test script that was sent to clients and the correction made

Conditions Matrix: High Level test conditions

Data Sheets: Sheets used for designing the data for test

Minutes of meetings/discussions: Team meetings, meetings with clients etc

Project archives: These are explained further in the document

63


64/71

4.2.2 External

The following are the deliverables to Clients

Test Strategy Effort Estimation Test Plan (includes Test scripts) Test Results Traceability matrix Pass defect report Final Defect report Final Summary Report Defect Analysis Metrics

4.2.3 Software Quality Assurance (SQA) Deliverable

The following are SQA deliverables

The Test plan is a document, which needs SQA approval and sign off Test results though do not require sign off by SQA authority, need to be delivered for

SQA perusal. The Traceability document, Test Scripts with review comments (without the result

status), defect Report format, Defect analysis and tool evaluation document (for selection of tools for the project), should be part of Test plan.

Test Scripts bearing module name Mails requesting release, Risk Assessment Memo, Effort Estimation, Project & Defect

Metrics Test results including the Final Defect Report & Executive Summary, Test Results

(Test Results with Pass/Fail status)

64


65/71


66/71

66


67/71

Bad: Problems faced during the test process are highlighted. Like

Communication issues with project management team Lack of functional/ technical knowledge and guidance Lack of automation experience Sign off done verbally Baseline documents not frozen Baseline documents not signed off Baseline document not made available Incomplete softbase transferred for testing No Process to analyze risks Status of unit testing may not be available No proper version control by the development team

Ugly: Low points and Mistakes made during the test process are highlighted. Like,

Estimation of effort not accurate or deviation not within permissible limits Profiles of members not matching to application under test Introduction of new tools without understanding its complexity and compatibility to

our methods and approaches Lack of proper feel of the application Risk mitigation plan

Good: High points during the test process are highlighted. Like,

New test approaches used Completion of project within projected time Preparing datewise test schedule Defects publishing on time Domain knowledge acquired through project and domain consultant Track of revisions in the test deliverables

67


68/71

Sign Off & ProjectCompletion

Milestone

68


69/71

4.5 Archiving

4.5.1 Tree

All documents and paper work should be archived. An archive tree is created which willhelp in easy retrieval of information. The diagram given below explains the tree and isquite self-explanatory.

69


70/71

BaselineDocuments

Test Strategy

Environment andControl

EnrichmentProcess

Reviews

Reports

IST Test Process

TestPlanning/Data

Preparation

Test Execution Test Results

Pass #0

Pass #1

Pass #2

Pass #3

Performance

Metrics

Summary

AB

Business Requirement Functional Specification Design Specification Prototype E Mails

Minutes of Meetings

Test Strategy Automation/Performance

Strategy Prepare Test Plan High Level Conditions Prepare Test Cases/Test

Scripts Prepare Test Data Setup Test Environment Setup Test Bed Receive Executable Executable Pass 1

Comprehensive Testing Prepare Defect Log Release Executable for Fixing Execute Pass II Discrepancy

Pass

C

Daily Reports

Executive Summary Progress Report Defect Report Defect Log with Screen

Shots Downtime Log Pass I Consolidated

Executive Summary

Final Reports

Defect Report Traceability Matrix Test

Execution Functionalities not tested Issues Related to

Maintenance

70


71/71

4.5.2 Collection of Details

Details that have to be collected and personnel to be contacted are

Sl

No

Documents Domain

Consultant

Client Test

Manager

Test

Lead

Test

Team

Development

Team1 Test Strategy

2 Test Plan

3 High Level TestConditions

4 Test Data5 Test Scripts6 Mails on the

project

7 Defect Metrics

8 Project Metrics

9 Defect Reports

10 FunctionalSpecification

11 BusinessRequirements

12 DesignSpecification

13 Prototype

14 Minutes of themeetings

4.5.3 Document Warehouse

External deliverables that were discussed earlier are then put into clients and Intranet.Hence these become published documents. This can be used for future references.

4.5.4 CD Recording

All the documents collected above are then recorded into a CD-ROM using the achievingtree described above. This becomes reference material for all further clarifications, whichmay arise in future.

Milestone

Testing Methodology Training Manual

Documents

Transcript of Testing Methodology Training Manual