Microsoft Australia Security Summit Tools for Quality Code Nigel Watson, Microsoft Australia Sean...

Microsoft Australia Security Summit

Tools for Quality CodeTools for Quality Code

Nigel Watson, Microsoft AustraliaSean Salisbury, Compuware CorpNigel Watson, Microsoft Australia

Sean Salisbury, Compuware Corp


AgendaAgenda

Testing – so what?

Testing in Visual Studio Team System

Extending VSTS – Compuware DevPartner

Summary

Testing – so what?

Testing in Visual Studio Team System

Extending VSTS – Compuware DevPartner

Summary


Projects and TestingProjects and Testing

Often an expensive afterthought

Strategies for minimising impact

Often an expensive afterthought

Strategies for minimising impact

RequirementsCoding

IntegrationBeta Test

Post-Release

5

10

15

20

25

30

Relative CostTo Fix Bugs...


Problems...Problems...

It is expensive to find and fix bugs that get past daily development practices

Potential security flaws need to be caught early

It is hard to diagnose errors at runtime

Why does an application run slowly?

Individual Developers and Testers need to know if they are on track

Test and development are often out of synch

Final test phase for shipping is often ad-hoc

How much testing is enough?

It is expensive to find and fix bugs that get past daily development practices

Potential security flaws need to be caught early

It is hard to diagnose errors at runtime

Why does an application run slowly?

Individual Developers and Testers need to know if they are on track

Test and development are often out of synch

Final test phase for shipping is often ad-hoc

How much testing is enough?


Defense In DepthDefense In Depth

Microsoft uses a 'defense in depth' strategyUnit testing

Code reviews

Frequent builds

Catch bugs earlyStatic checks

Runtime checks

Microsoft uses a 'defense in depth' strategyUnit testing

Code reviews

Frequent builds

Catch bugs earlyStatic checks

Runtime checks


Testing in VSTSTesting in VSTS

Change Management

Work Item Tracking

Reporting

Project Site

Visual Studio

Team Foundation Project Management

Visual Studio

Team Architect

Visio and UML Modeling

VS Pro

Class Modeling

Application Modeling

Logical Infra. Modeling

Deployment Modeling

Visual Studio

Team DeveloperVisual Studio

Team Test

Project SiteWork Item Tracking

Reporting

Project Management

Integration Services

Load Testing

Manual Testing

Test Case Management

Unit Testing

Code Coverage

Dynamic Code Analyzer

Static Code Analyzer

Code Profiler

Team Foundation Client


Test-Driven DevelopmentTest-Driven Development

Integrate testing into the development process

Tests define what code will doTests come from specifications

Write code to pass tests

Don't write code that doesn't contribute to passing a test...

Integrate testing into the development process

Tests define what code will doTests come from specifications

Write code to pass tests

Don't write code that doesn't contribute to passing a test...

CodeCode


VSTS Unit TestingVSTS Unit Testing

Integrated into VS

Automatic generation of test classes

Comprehensive test management

Code coverage testing

Integrated into VS

Automatic generation of test classes

Comprehensive test management

Code coverage testing[TestMethod()][TestMethod()]public void public void GetValueTestGetValueTest()() {{ double d = myObject.double d = myObject.getValuegetValue();(); if (d < 10.0)if (d < 10.0) Assert.Fail("Bad return value");Assert.Fail("Bad return value"); }}


Unit TestingUnit Testing


Code ReviewsCode Reviews

For the Visual Studio 7.0 product cycle86% of bugs occurred in reviewed code

60% of all bugs were coding errors

Static analysis helps catch bugsSource code analysis

PREfast for C and C++

FxCop for .NET

For the Visual Studio 7.0 product cycle86% of bugs occurred in reviewed code

60% of all bugs were coding errors

Static analysis helps catch bugsSource code analysis

PREfast for C and C++

FxCop for .NET


PREFastPREFast

Static analysis for C/C++ codeManaged and unmanaged C++

Catches common bugsBuffer overruns, uninitialized memory

Memory leaks, null pointer dereference

Reported as compiler warningsDisplay path to problem

Use #pragma to turn off

Static analysis for C/C++ codeManaged and unmanaged C++

Catches common bugsBuffer overruns, uninitialized memory

Memory leaks, null pointer dereference

Reported as compiler warningsDisplay path to problem

Use #pragma to turn off


FxCopFxCop

Static analysis for .NET assembliesNot just C++

Uses design guidelines(including many in the .NET Class Design Guidelines)

CustomizableWhich checks to include

Whether to report as error or warning

Create custom rules

Static analysis for .NET assembliesNot just C++

Uses design guidelines(including many in the .NET Class Design Guidelines)

CustomizableWhich checks to include

Whether to report as error or warning

Create custom rules


Static code analysisStatic code analysis


Integrating Dev and TestIntegrating Dev and Test

Tests are just another form of source code:Stored in source code control

Versioned with the product

“Test Complete”Test writing is scheduled along with development work

Tracked by work items

Testers are notified when bugs are fixed

Tests are just another form of source code:Stored in source code control

Versioned with the product

“Test Complete”Test writing is scheduled along with development work

Tracked by work items

Testers are notified when bugs are fixed


VSTS Test TypesVSTS Test Types

Unit TestsTest class methods

Web TestsRecord and playback interactions

Load TestsSimulate multiple users

Manual TestsProvide scripts for manual tasks

Third-party TestsIntegrated into VSTS

Unit TestsTest class methods

Web TestsRecord and playback interactions

Load TestsSimulate multiple users

Manual TestsProvide scripts for manual tasks

Third-party TestsIntegrated into VSTS


Application QualityApplication Quality

Best Practices and ToolsBest Practices and Tools

Sean SalisburySenior Regional Tech SpecialistCompuware [email protected]


Integrated development and test automation tools

Rich process management Detailed and relevant

project information

Microsoft and CompuwareMicrosoft and Compuware

Production Readiness

Automated Software Quality

Development & Integration

Performance & Availability

Management

QACenterExtends quality assurance testing

DevPartnerExtends quality in development

VS & Team Systemintegration platform, base tools


Compuware DevPartner Studioenhance and extend Visual Studio

Compuware DevPartner Studioenhance and extend Visual StudioNative and Managed Code Analysis

Local and Remote Data Collection:Performance Analysis

.NET Memory Analysis

Code Coverage Analysis

Distributed Application Analysis

VB, VB.NET, ASP.Net and C# Source Code Review with >600 Rules

C/C++ Memory Error & Thread Deadlock Detection

Native and Managed Code Analysis

Local and Remote Data Collection:Performance Analysis

.NET Memory Analysis

Code Coverage Analysis

Distributed Application Analysis

VB, VB.NET, ASP.Net and C# Source Code Review with >600 Rules

C/C++ Memory Error & Thread Deadlock Detection


Code AnalysisCode Analysis

600+ Rules enhance problem resolution

Supports VS6/2002/2003/2005

Accelerates learning curves

Improves code quality and maintainability

Supports Visual Basic, VB.NET, C#, ASP.Net

600+ Rules enhance problem resolution

Supports VS6/2002/2003/2005

Accelerates learning curves

Improves code quality and maintainability

Supports Visual Basic, VB.NET, C#, ASP.Net


Memory AnalysisMemory Analysis

Optimize Local or Remote Memory Use

View allocations/deallocations over time: get an overall feel for memory use

Identify Objects That:

Consume a lot of memory

Create a lot of temporary objects

Stay around longer than they need to, including leaks

Compare Runs- Did Code Changes Help?

Tune Garbage Collection

Optimize Local or Remote Memory Use

View allocations/deallocations over time: get an overall feel for memory use

Identify Objects That:

Consume a lot of memory

Create a lot of temporary objects

Stay around longer than they need to, including leaks

Compare Runs- Did Code Changes Help?

Tune Garbage Collection


Memory Analysis at Run TimeMemory Analysis at Run Time

Real-Time

trace of memory usage

System Allocations

Your Code

RAM usage

Time


Memory AnalysisMemory Analysis

Many Different Data Views with Details Available

Many Different Data Views with Details Available


Automatic Error DetectionAutomatic Error Detection

Memory/Resource/ Interface Leaks

API Errors

Threading Issues

Event Debugging

C/C++/VC++

Memory/Resource/ Interface Leaks

API Errors

Threading Issues

Event Debugging

C/C++/VC++


Thread Deadlock Detection Thread Deadlock Detection

Locate Actual or Potential Thread Deadlocks or Other Synchronization Issues

Deadlock: 2 or more code paths running at the same time, contending for the same resource(s)

BenefitsThread deadlock are difficult to detect: automating detection is very useful

Locate Actual or Potential Thread Deadlocks or Other Synchronization Issues

Deadlock: 2 or more code paths running at the same time, contending for the same resource(s)

BenefitsThread deadlock are difficult to detect: automating detection is very useful


Performance ProfilingPerformance Profiling

Pinpoint bottlenecks across app Tiers/Versions

Optimize application performance

Increase usability

Pinpoint bottlenecks across app Tiers/Versions

Optimize application performance

Increase usability


Compare Performance Runs Compare Performance Runs


Code CoverageCode Coverage

Quickly identify untested code across tiers & VS6/02/03/05

Ensure test coverage during unit testing

More reliable components and applications

Quickly identify untested code across tiers & VS6/02/03/05

Ensure test coverage during unit testing

More reliable components and applications


Distributed AnalysisDistributed Analysis


What’s New….What’s New….


IT ChallengesIT Challenges

Identifying what errors can occur & when

Tools lacking for error simulation and analysis

Errors corrupt the debugging environment

Impossible to trace error handling execution

Difficult to create repeatable tests

Time-consuming, manual process

Identifying what errors can occur & when

Tools lacking for error simulation and analysis

Errors corrupt the debugging environment

Impossible to trace error handling execution

Difficult to create repeatable tests

Time-consuming, manual process


What If You Could…What If You Could…

Quickly determine what errors could occur at any point in your application?

Ensure you have error handlers in place to cope

Simulate errors safely and efficiently?With no impact on the OS, .NET framework or any other running application

Observe and debug your error handlers

Build reusable fault test libraries?Create repeatable tests that are reusable by development & QA

Quickly determine what errors could occur at any point in your application?

Ensure you have error handlers in place to cope

Simulate errors safely and efficiently?With no impact on the OS, .NET framework or any other running application

Observe and debug your error handlers

Build reusable fault test libraries?Create repeatable tests that are reusable by development & QA


DevPartner Fault SimulatorDevPartner Fault Simulator

Developer Insight

What errors can occur at what point in the code

Integrated with Visual Studio debugging features to monitor error handling execution

Break at fault occurrence

Developer Insight

What errors can occur at what point in the code

Integrated with Visual Studio debugging features to monitor error handling execution

Break at fault occurrence



Error handling validation

Simulate Environmental and .NET Framework faults

Simple method of selection of errors to validate, with user defined conditions

Reusable Fault Sets for repeat and QA testing

VS 2003/05 IDE integrated, standalone and command line operation

Error handling validation

Simulate Environmental and .NET Framework faults

Simple method of selection of errors to validate, with user defined conditions

Reusable Fault Sets for repeat and QA testing

VS 2003/05 IDE integrated, standalone and command line operation



Results analysis

Simulate Stack tracing & error details

“Go to source” linking for detailed analysis

Live view and summary of fault execution

Saved Results files for later review

Results analysis

Simulate Stack tracing & error details

“Go to source” linking for detailed analysis

Live view and summary of fault execution

Saved Results files for later review



DemonstrationDemonstration


Securing ASP.Net ApplicationsSecuring ASP.Net Applications


Security VulnerabilitySecurity Vulnerability

“Today over 70% of attacks against a company’s network come at the Application Layer, not the Network or System Layer”

John Pescatore, Gartner chief security analyst

The responsibility for application security is shifting to the development organization

How do they address this aspect of application quality?

How do they gain the skills they need to assess and correct security vulnerabilities?

“Today over 70% of attacks against a company’s network come at the Application Layer, not the Network or System Layer”

John Pescatore, Gartner chief security analyst

The responsibility for application security is shifting to the development organization

How do they address this aspect of application quality?

How do they gain the skills they need to assess and correct security vulnerabilities?


What If You Could…What If You Could…

Quickly locate security vulnerabilities in your application during development?

Minimize the cost and mean-time-to-repair

Improve the quality/reliability of your application

Have a wealth of security expertise and advice at your fingertips?

Have the information you need, when you need it

Quickly locate security vulnerabilities in your application during development?

Minimize the cost and mean-time-to-repair

Improve the quality/reliability of your application

Have a wealth of security expertise and advice at your fingertips?

Have the information you need, when you need it


DevPartner SecurityCheckerDevPartner SecurityChecker

A vulnerability assessment scanner that locates security vulnerabilities in ASP.NET (C# or VB.NET)

Locates complex & hard-to-find security problems

Organizes results by priority and category Pinpoints vulnerabilities to the line of source code

Explains why it is an issue

Suggests steps to repair each vulnerability

Provides links to additional technical information

A vulnerability assessment scanner that locates security vulnerabilities in ASP.NET (C# or VB.NET)

Locates complex & hard-to-find security problems

Organizes results by priority and category Pinpoints vulnerabilities to the line of source code

Explains why it is an issue

Suggests steps to repair each vulnerability

Provides links to additional technical information



Integrity Analysis(attach simulation)

Replays a series of known security attacks against the application

Secures the interface to the application

Compile-time Analysis

Scans source code for known security problems

Test while coding

Run-time Analysis Monitors execution of the application

Observes interior/hidden facets, beyond the external interface

Expert Advisor

Go to line of source code

Detailed assistance

Allows the developer to quickly: Find & fix the vulnerability

Become more knowledgeable about security

Accelerates secure application development



DemonstrationDemonstration


Quality Continues in TestingQuality Continues in Testing

Automate functional testing and validationManage test plans and execution

Comparison of complex data results

Seamlessly capture defect information

Simulate application under loadSimulate load conditions ‘000,000’s of users

Determine application scalability

Compuware QACenter Enterprise Wide

Compuware Vantage - Network and Server monitoring

Automate functional testing and validationManage test plans and execution

Comparison of complex data results

Seamlessly capture defect information

Simulate application under loadSimulate load conditions ‘000,000’s of users

Determine application scalability

Compuware QACenter Enterprise Wide

Compuware Vantage - Network and Server monitoring


Microsoft & CompuwareMicrosoft & Compuware

Tools to:

Improve application reliability & performance

Increase team productivity

Lower costs

Deliver better applications to the market faster

Tools to:

Improve application reliability & performance

Increase team productivity

Lower costs

Deliver better applications to the market faster

Production Readiness

Automated Software Quality

Development & Integration

Performance & Availability

Management


SummarySummary

Appreciated the importance of testing to the development process

Had a quick look at some of the testing tools in Visual Studio Team System

Sean showed us how Compuware DevPartner Studio uses the integration capabilities of Visual Studio to extend the power of the IDE

Appreciated the importance of testing to the development process

Had a quick look at some of the testing tools in Visual Studio Team System

Sean showed us how Compuware DevPartner Studio uses the integration capabilities of Visual Studio to extend the power of the IDE

Microsoft Australia Security Summit Tools for Quality Code Nigel Watson, Microsoft Australia Sean...

Documents

Transcript of Microsoft Australia Security Summit Tools for Quality Code Nigel Watson, Microsoft Australia Sean...