Windows Protocol Test Suites: Overview and Demo › events › 2017 › redmon… · •Visual...
Transcript of Windows Protocol Test Suites: Overview and Demo › events › 2017 › redmon… · •Visual...
Windows Protocol Test Suites: Overview and Demo
Diane Larsen
Jiajun Wang
What is a protocol test suite?
Evaluates whether a protocol implementation meets certain interoperability requirements.
Originally developed for in-house testing of Microsoft Open Specifications.
Used to test/verify Windows behavior.
Also used to test 3rd-party implementations.
Our test suites do not cover every protocol requirement, and do not certify an implementation, but can be a useful indication of interoperability.
Windows Test Suite Types
Individual protocols: MS-SMBD, MS-SMB, MS-ADFSPIP
Protocol families: AD, File Sharing (MS-SMB2 and related protocols), Remote Desktop, Kerberos, Branch Cache
Overview documents: MS-ADOD, MS-AZOD
2. Develop protocol test suite (Synthetic Client) according to Open Specification
3. Run against Windows Server to verify Open Specification & OS behavior
4. Run against partner’s Server Implementation to help identify & debug issues
1. Open Specification defines messages, sequences, behaviors
5. Success! Interop between partner’s Server and Windows Client
Example: Client/Server Side Partner Implementation
Open Specification
Windows Client(desktop , surface,
phone etc)
Windows Server
Partner’s Server(storage, AD…)
Protocol Test Suite
Synthetic Client
4
5
Message Over the Wire1
2
3
Test Case 1:1. Establish connection;2. Client.Send(msgA);3. Expect server respond with msgB, and
• Assert (msgB.Field1 == valid1)• Assert (msgB.Field2 == valid2)• …• Assert (msgB.FieldN == validN)
Test Case 2:1. Establish connection;2. Client.Send(Invalid msgA);3. Expect server respond with msgC, and
• Assert (msgC.Status == ERROR)
Protocol Specification:If SMB2 client sends message A to SMB2 server, SMB2 server should respond with message B. If message A is invalid, server should return error message C.
Where are they?
Windows protocol test suites are on GitHub.
Two formats: source code and MSI files
Both formats have been updated in the past week for this event.
Support is available via GitHub (or talk to test team members this week).
https://github.com/Microsoft/WindowsProtocolTestSuites
Required and optional components
Required Software
• Windows 8, Windows Server 2012 (and later)
• Visual Studio 2012• Visual Studio 2013 for MS-SMBD test suite
• Spec Explorer Power Tool • For model-based testing used by most test suites
• Microsoft Protocol Test Framework
• Simplifies configuration, logging, checking
• Based on Visual Studio Unit Test framework
• Test Suite(s)
• Full list per test suite is available on GitHub
Optional Software
• Protocol Test Manager
• Simplifies test suite configuration and execution.
• Detects system capability, selects test cases, displays status.
• Demo in a few minutes!
• Microsoft Message Analyzer
• Capture, display, and analyze protocol messaging traffic.
Other sessions this week
We’ll have a demo next (as part of this session)
After this session, Vivian will talk about open source test suites.
At 3:00 in building 20, Helen will introduce the test suites environment and show how to run a test suite.
Also at 3:00 is a session in this room about Office tests and tools followed by a session about SQL Server test and tools.
Later in the week will be a session about Message Analyzer, the Microsoft network (and more) analysis tool.
Protocol Test ManagerJiajun Wang
Overview of Protocol Test Manager
A unified UI wizard to execute protocol test suites:
• Embed deployment guide
• Detect SUT capability automatically
• Select test cases automatically
• Configure test suite easily
• Run test cases friendly
Options to Execute Test Suites
Option 1: Using batch files bundled with Test Suites installer
Pros: Less dependency.
Cons: Manually modifying the
ptfconfig file is time-consumingand error-prone.
Options to Execute Test Suites
Option 2: Using Visual Studio
Pros: Easy to control every test step and debug.
Cons: Need to install Visual Studio.
Need to download source code and build the test suite yourself.
Need to manually modify ptfconfig file (difficult).
Visual Studio
Options to Execute Test Suites
Option 3 (recommended): Using Protocol Test Manager (PTM)
Pros: Easy to use.
Auto detects your test environment and modifies the ptfconfig file.
Import/Export test profiles to save your current configurations.
Cons: Need to download & install PTM separately.
Protocol Test Manager
Protocol Test Manager
Detect
Detection summary
Filter cases
All configurations
Run test cases
Protocol Test Manager
Detect
Detection summary
Filter cases
All configurations
Run test cases
Protocol Test Manager
Detect
Detection summary
Filter cases
All configurations
Run test cases
Protocol Test Manager
Detect
Detection summary
Filter cases
All configurations
Run test cases
Protocol Test Manager
Detect
Detection summary
Filter cases
All configurations
Run test cases
Command Line Interface
PtmCli.exe -p adfamily.ptm -s -r report.txt
cat report.txtFailed APDS_S1_TC23_NTLM_NETWORK_VALIDATE_A2AF_RESTRICTIONPassed APDS_S3_TC04_DIGEST_DIFF_RESERVED4_SAME_REPLYPassed APDS_S3_TC05_DIGEST_REJECT_ALGTYPE_MD5Passed APDS_S3_TC06_DIGEST_REJECT_ALGTYPE_MD5_ASSUMEDPassed DRSR_DRSBind_UnBind……
Handy to use when doing automatic testing and analysis the results.
Recent Updates to Protocol Test Manager
Added status icons in the result page so that users can more easily track the status of test cases.
Added more error handlers and improved stability.
Demo
Using Protocol Test Manager to run ADFamily test cases.
Thank You!Questions?