Microsofts Protocol Documentation Program: A Success Story for Model-Based Testing Yiming Cao, Lead...

Microsoft’s Protocol Documentation Program:

A Success Story for Model-Based TestingYiming Cao, Lead Software Design Engineer Windows Interoperability Engineering Team, Microsoft Corporation

Dagstuhl Seminar, October 2010, Leibniz-Zentrum für Informatik, Germany.

http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=10421

http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=10421

Part 1: Background

Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2

Microsoft Open Protocols

Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation

http://www.microsoft.com/protocols

9/4/2010 3



http://www.microsoft.com/protocols/


“BlueLine” Technical Document Testing Program of Windows (as of 03/09) 222 protocols/technical documents tested

22,847 pages studied and converted into requirements

36,875 testable requirements identified and converted into test assertions 69% tested using MBT 31% tested using traditional test automation

66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers

9/4/2010 4



Formalization

Technical Document Test Suite

EnglishStatements(Requirements)

AssertionsChecked against Windows(Accuracy)

• Validate consistency

• Simulate implementation process

Test-Driven Document Quality Assurance

9/4/2010 5


Study

• Document study• Test strategy

(modeled or traditional)

• Requirements spec (initial)

• Estimates of Effort

Plan

• Requirements spec (final)

• Test plan• Architecture• Scenarios

Design

• Test design• Model, or• Test case

descriptions• Adapter

• Design

Final

• Test case code• Generated, or• Manual

• Adapter• Implementatio

n• Test logs and

report• Network

captures• Test setup

artifacts

Process: Managing a Project with 350 Test Engineers

Review after every phase• High-profile reviewer team consisting of industry experts and Microsoft

senior engineers, build along the model of a program committee• Review based on process report and test suite artifacts• Verdict accept, conditional accept, reject

9/4/2010 6

Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation 7

Requirements TrackingTechnical Document

Requirements Spec

Model or Test Design

Test Suite

Logs

Network Captures

Statements identified in document and tracked with requirement ID

IDs referenced from model or test assertion

IDs logged at test execution time

IDs send to network as beacon packages for network monitor capturing

manual

automated

9/4/2010

Typical Requirements Specification


Verification Comment

Use this column to explain the reason why the requirement is non-verifiable or unverified (for Normative, Server requirements).

Req ID Doc Sect Description Po

sNeg

Derived

Scenarios Behavior ScopePriorit

y

Informative/

NormativeVerificatio

n

Verification

Comment

MS-CHAP_R14

3.2.5When a peer receives a Failure-Request message, if the embedded MSCHAPv2 packet indicates a password expiration error, the peer MAY reply with a Change-Password-Response message.

Protocol Client p2 Normative Unverified Client behavior not verified.

MS-CHAP_R15

3.3.1In addition, the authentication phase is terminated by sending an EAP-Success/EAP-Failure packet to the peer.

S1,S2,S3,S4,S5,S6

Protocol Server p2 Informative

Test Case

MS-CHAP_R16

3.3.5An EAP server MUST transmit a Challenge-Request message to initiate an Extensible Authentication Protocol Method for Microsoft CHAP authentication procedure.

S1,S2,S3,S4,S5,S6,S7,S8

Protocol Server p0 Normative Test Case

9/4/2010 8

Part 2: Model-Based Test Suite Development



Model-Based Document Analysis

Model

Implementation

Expected Outputs (Test

Oracle)

Inputs(Test

Sequences)

Control

Document

Generate

Observe

Feedback

Feedback

Author

Issue

Feedback

Test Suite

Verdict

Feedback

9/4/2010

Process Drilldown


Gather Requirements

Define Actions Define Adapters

Model & Explore

Generate Tests Implement Adapters

Run Tests

PartiallyGenerated from Doc

Technical Document Issues (TDIs) filed in all phases

9/4/2010 11

Gathering Requirements


Gather Requirements


Model & Explore


Run Tests

9/4/2010 12

Developing and Implementing Adapters


Gather Requirements


Model & Explore


Run Tests

9/4/2010 13


Adapter Interface

TCTest Cases(traditional or MBT)

TC TC

TC TC TC

Adapter Implementation

Test Development

Adapter Development

SUT(System under test)

• Abstracts SUT functionality

• Contract between teams• Test case team• Adapter team

• Pluggable• Different server

setups• Different transports

Test Adapters

9/4/2010

Develop Model and Generate Tests


Gather Requirements


Model & Explore


Run Tests

Potential dependencyon data types

9/4/2010 15

Spec Explorer 2010 Technology Breakdown Model programs

Guarded state update rules Rich object-oriented model state (collections, object

graphs) Language agnostic (Based on .Net intermediate

language interpretation)

Trace patterns Regular style language to represent scenarios Slicing of model program by composition

Symbolic state exploration and test generation Expands parameters using combinatorial interaction

testing Extracts a finite interface automaton (IA) from composed

model Traverses IA to generate standalone test code –or- Runs on-the-fly tests from IA

Integrated into Visual Studio 2010Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 16



Spec Explorer 2010 Look & Feel

Explore

Analyze

Generate

Execute

C# Model (or other .Net Language)

Model Graph

Test Suite

VSTT Result

9/4/2010

Spec Explorer Walkthrough: Chat Example

Users can Enter the session Exit the session List all session users Broadcast a

message Received by all session

users

18

Chat Room Server

User1

SUT

User2 User3

What happens if two users broadcast a message at virtually the same time?

Chat Requirements

19

R1: User MUST receive response for logon request

R2: User MUST receive response for logoff request

R3: User MUST receive response for list request

R4: List response MUST contain the list of logged-on users if successful

R5: All logged-on users MUST receive broadcast message

R6: Messages from one sender MUST be received in order

20

Chat traces

Is the trace correct?

(Assume two users (user1, user2) are logged on)

T1Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user2,”1”)BroadcastAck(user1,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”2”)



If each user sends one message, any receiving order is correct!


21

Chat traces

Is the trace correct?

(Assume two users (user1, user2) are logged on)

T4Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user2,”1a”

)BroadcastAck(user1,”1a”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1b”)

T5Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user1,”1a”)BroadcastAck(user2,”1b”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1a”)

Local order consistency: messages sent by one user must be received in order


22

Chat Slice where local consistency does not matter

23

Chat Slice where local consistencydoes matter

24

Spec Explorer’s Conformance Notion

Alternating simulation: SUT must “simulate” all stimuli of model Model must “simulate” all responses of SUT (responses are

buffered)

Broadcast(1,”1”)


Model SUT



BroadcastAck(1,”1”)




BroadcastAck(2,”1”) BroadcastAck(1,”

2”)

Spec Explorer’s Approach to State ExplosionSlicing techniques:

Parameter selection (Constraints, Pairwise, etc.)

State filtering Trace patterns Requirement coverage

Slicing requires human intervention Smart slicing is an art

Spec Explorer tailored for slicing design25

26

Where a Trace can end: Accepting State Condition

Is the following a valid word?Micr

Does the following trace represents a useful test?Broadcast(1,”1”); Broadcast(1,”1”)

An Accepting state condition characterizes those states in which a trace can end Used to ensure that a trace does not stop at arbitrary

points Used to ensure that a test leaves the system in a good

state

Accepting state condition for Chat: All messages have been delivered to recipients

Demo: Chat Model

What the demo shows OO model state Using helper methods in models Modeling with events Slicing


Demo: Server Message Block Protocol v2What the demo shows: Modeling a key Microsoft product Using C# to model state transition

systems Using trace patterns to slice test

purposes from models Generating and executing tests

9/4/2010 28

Part 3: Evaluation



Comparison MBT vs Traditional

Vendor 1 Vendor 20

10

20

30

40

50

60

70

80

90

100

5866

100 100

Model-BasedTraditional

• In % of total effort per requirement, normalizing individual vendor performance

• Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 1

9/4/2010


Remarks on Comparison

Numbers backed up by correlation analysis conducted by empirics expert

Engineers applying MBT were lay folks, most of them freshly hired from college

Few power modelers achieve much higher efficiency rates

Though model authoring takes only a smaller part of overall effort per requirement, modeling approach seems to determine efficiency

9/4/2010 31


Factors for Success Tool support

Modeling in mainstream languages and environments

Visualization of exploration and actual generated test suite (no magic as in online testing)

Dealing with state explosion by scenario slicing Training and Guidance

One week class for every new hire (including both general methodology and model based testing)

Size of project generated critical mass for community help

Reviewing process added additional guidance

9/4/2010 32


Pain Points Observed

Delta Testing Support A new set of test cases generated based on a

new version of model: Which old cases can be re-used? Which old cases are not valid anymore? Which new cases need to be added?

By-passing assertion at test execution time Push-button technology for requirement

coverage with minimum transitions Good heuristics is needed when behavior is

infinite

9/4/2010


Pain Points Observed (continued) Ambiguity in Requirement Coverage

What does it mean to cover a requirement in model? Covering a transition once/multi-times? Or

even path coverage or pairwise coverage?

Rule Debugging Support C# provides rich programming

experience But rules can get so complex that a

debugger is needed to analyze what’s going wrong

9/4/2010


Conclusions

Biggest application of behavioral model-based testing known so far

Probe big enough for relevant empiric results, proving efficiency gain of 42%

Will lead to more momentum on modeling in general inside of Microsoft

Model-based testing rocks!9/4/2010 35


Advertisement…

Microsoft offers Spec Explorer 2010 as preview technology free of charge via Visual Studio Gallery:

http://visualstudiogallery.msdn.microsoft.com/en-us/271d0904-f178-4ce9-956b-d9bfa4902745 Licensing allows for commercial use Academic institutions can join MSDN

Academic Alliance for access to Visual Studio 20109/4/2010 36

http://visualstudiogallery.msdn.microsoft.com/en-us/271d0904-f178-4ce9-956b-d9bfa4902745






© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9/4/2010 37

Microsofts Protocol Documentation Program: A Success Story for Model-Based Testing Yiming Cao, Lead...

Documents

Transcript of Microsofts Protocol Documentation Program: A Success Story for Model-Based Testing Yiming Cao, Lead...