8/10/2019 Test Case Execution in Software Testing

1/20

Test Case Execution

Major testing activities:

Test planning and preparation Execution (testing)

Analysis and followup

Test execution:

!ant a s"oot# transition fro" one test run to anot#er

Execution planning and "anage"ent

$elated activities: i"portant part

failure identification and "easure"ent

ot#er "easure"ent

http://www.apextgi.in/

8/10/2019 Test Case Execution in Software Testing

2/20

Test Execution

%eneral steps

Allocating test ti"e (& resources)

'nvoing tests (& collecting execution info)

'dentifying syste" failures

(& gat#ering info for followup actions)

Allocating test ti"e

planned* +ut needs "onitoring & adjusting

,-+ased or Coverage+ased

Alternative: +otto"up approac#

individual test cases (test ti"e)

(su"up) overall allocation

+y ,- or coverage areas

8/10/2019 Test Case Execution in Software Testing

3/20

Test Execution

'nvoing tests follow plan* provide input varia+le values over w#oleexecution duration following preplanned se.uence

'nvoing test (,-+ased)

,- /0 input varia+les (test points) 1ollow pro+a+ilistic distri+utions (could +e dyna"ically deter"ined)

2e.uence (w#at to test first3): C,T2* product* super syste"

'nvoing test (coverage+ased)

,rgani4e sensiti4ed test cases

2e.uence (coverage #ierarc#ies)

Co""on part: $etest due to 5efect fix /0 verify fix

Code or feature c#ange

%eneral regression test

8/10/2019 Test Case Execution in Software Testing

4/20

Test Execution

Identifying system failures (oracle problem):

Similar for OP-/coverage-based

naly!e test output for deviations

"etermine: deviation # failure $

%andling normal vs& failed runs non-bloc'ing failure andling

Solving oracle problem:

eoretically undecidable&

Some cases obvious: cras* ang* etc&

Practically based on euristics:

product domain 'no+ledge

cross-cec'ing +it oter products

implementation 'no+ledge , internals

limited dynamic consistency cec'ing

8/10/2019 Test Case Execution in Software Testing

5/20

Test Execution

ailure observation and measurement:

.en determining deviation # failure

stablis +en failure occurred

used in reliability and oter analysis ailure information (e&g&* O"0):

+at/+ere/+en/severity/etc&

"efect andling and test measurement:

"efect status and cange (controlled)

Information gatering during testing: e1ample template: able 2&3 (p&45)

ollo+-up activities:

fi1-verification cycle

oter possibilities (defer* invalid* etc&)

8/10/2019 Test Case Execution in Software Testing

6/20

8/10/2019 Test Case Execution in Software Testing

7/20

Testing Analysis and 1ollow up

6a7or testing activities:

est planning and preparation

1ecution (testing)

nalysis and follo+ up est analysis and follo+ up:

1ecution/oter measurement analy!ed

nalysis results as basis for follo+ up

eedbac' and follo+ up:

defect fi1ing

decision ma'ing (e1it testing$ etc&)

ad7ustment and improvement&

8/10/2019 Test Case Execution in Software Testing

8/20

Testing Analysis and 1ollowup

Input to analysis

est e1ecution information

Particularly failure cases

iming and caracteristics data

nalysis and output

8asic individual (failure) case

problem identification/reporting

repeatable problem setup Overall reliability and oter analysis$ (6odule 9)

ollo+-up activities

"efect analysis and removal (, re-test)&

"ecision ma'ing and management&

est process and uality improvement&

8/10/2019 Test Case Execution in Software Testing

9/20

8/10/2019 Test Case Execution in Software Testing

10/20

Testing Analysis and 1ollow up

nalysis and follo+-up for overall testing:

determine if goal as beenacieved/release> steps to acieve if not> id lo+ reliability areas

0overage analysis and follo+-up& surrogate for reliability> input for stopping criterion

"efect analysis and follo+-up&

defect distribution> id ig-defect areas

ocus of Part I9&

ollo+ up activities: Similar& "ecision ma'ing and management&

est process and uality improvement&

8/10/2019 Test Case Execution in Software Testing

11/20

Test Manage"ent

People?s roles/responsibilities in formal and informal testing&

In informal testing:

@plug-and-playA by novice users&

@run-and-observeA by testers (or automation tools)& Informal testing +it ad-oc 'no+ledge for easy* simple systems&

"eceptively @easyA* but not all failures or problems easy to recogni!e&

In formal testing:

8/10/2019 Test Case Execution in Software Testing

12/20

Test Manage"ent

est team organi!ation:

9ertical: Pro7ect oriented

product domain 'no+ledge*

staffing/resource management issues (personnel reassigned)&

%ori!ontal: as' oriented

teams perform one 'ind of testing on many different products

even distribution of staff/resources

lac' of internal 'no+ledge/e1pertise

6i1ed models migt +or' better&

;sers and 5rd party testers: ;ser involvement in beta-testing and oter variations

I9,9 +it 5rd party testing/B ("O" e1tensively uses tis)

Impact of ne+ tecnologies:

08S* 0OS impact

security* dependability reuirements&

8/10/2019 Test Case Execution in Software Testing

13/20

Test Auto"ation

8asic understanding:

utomation needed for large systems&

ully automated: eoretically impossible&

ocus on specific needs/areas&

Cey issues to consider:

Specific needs and potential for automation&

1isting tools available/suitable$

related: cost/training/etc&

0onstructing specific tools$

dditional cost in usage , support&

8/10/2019 Test Case Execution in Software Testing

14/20

Test Auto"ation

utomation by test activity areas:

utomated test planning , preparation&

Automated test execution.

utomated test measurement* analysis* and follo+-up&

Sligtly different grouping due to tigt coupling for measurement ,analysis&

utomation for test e1ecution&

6any debuggers: semi-automatic (testers may intervene)&

as' seuencing/sceduling tools&

=oad/test generator: script #D runs Eenerally easier to obtain test scripts&

8/10/2019 Test Case Execution in Software Testing

15/20

Test Auto"ation

utomation for test planning/preparation:

est planning: %uman intensive not muc can be done (inspection and9)&

est model construction: similar to above&

automation possible at a small scale&

est case generation: focus&

est case generation:

rom test model to test cases&

Specific to individual tecniues

e&g&* cover cec'list items* pats* etc& 9arious specific tools&

Cey: +ic specific testing tecniue supported by te specific tool$

8/10/2019 Test Case Execution in Software Testing

16/20

Test Auto"ation

est measurement* analysis* and follo+-up&

nalyses dictate measurements needed&

6ost common: reliability/coverage&

need timing data for individual test runs

"efect measurement needed in most cases:

defect trac'ing tools collect data on personnel* component

can identify problematic areas for improvement

8/10/2019 Test Case Execution in Software Testing

17/20

Test Auto"ation

Coverage+ased testing

"easuring coverage and co"pare to preset goals

usually re.uires "ore detailed infor"ation t#an relia+ility analysis

Test coverage tools:

5ifferent levels6definitions of coverage/0 different tools

Exa"ple tools:

McCa+e: execution (control flow) pat#

2TCAT: functional coverage7 callpair infor"ation

ATAC: data flow coverage

Test coverage steps:

-reparation: progra" instru"entation Measure"ent step: run and collect data

Analysis step: analysis for coverage

8/10/2019 Test Case Execution in Software Testing

18/20

8/10/2019 Test Case Execution in Software Testing

19/20

2u""ary

est activities:

Planning , preparation: focus of Part II&

1ecution , measurement: common&

nalysis , follo+ up: focus of Part I9&

est management:

"ifferent roles and responsibilities&

Eood management reuired&

est automation:

Set realistic e1pectations&

Specific areas for automation* esp& in e1ecution* measurement* and

analysis&

8/10/2019 Test Case Execution in Software Testing

20/20