Black Box Testing Methodology SANS.ppt Box Testing... · Why Black Box Test? ! ... Why Black Box...

Black Box Testing Methodologies

Joe Cummins, PCIP, OPST Jonathan Pollet, CISSP, CAP, PCIP January 24, 2011 SANS SCADA Webinar, SCADA Summit Series 2011

welcome

Outline

� Why Black Box Test?

�  Layered approach

�  Black Box vs. White Box

�  Components of an Assessment

�  Process

�  Reports and metrics

…Why Black Box testing?

� Know what you are putting out on the network…

� How does a device respond to protocols it does not recognize?

� What happens when it gets a confusing message?

… are you sure?

Phased Approach to Device / Application Testing

�  Protocol �  RFC’s �  Proper communications

�  Software �  DOS, Overflow, Etc… �  Kernel

�  Firmware �  Assembler

�  Hardware �  Components �  Monitoring

Firmware

Hardware

• Applications • Kernel

• Assembler

Layered Defence

6. Embedded Device

5. Communication Method

4. Servers / Workstations

3. DMZ

2. Infrastructure

Software / Middleware

� Exceptions �  Failures �  Null Pointers �  Access Violations

� Memory Corruption �  Buffer Overflow �  Stack Overflow

Hardware

� Components �  NIC (wired, wireless) �  Ports

� Monitoring �  CPU �  Temperature �  Cycles �  Processes �  Stack

Tools of the Trade

Manual Code Review

� Automated tools �  Highlights errors / changes �  Known common application faults �  Verification of Syntax

� Viewers �  Import / Export Source �  Render �  Analyze

Analysis Engine

� Core Fuzzing Process �  Reliance on the Tools and plugins to generate proper data

� Manual Code Review �  Line by line review

� Blended Analysis

Blended Analysis

� Device Testing Methodology

� Combination of both aspects �  Code review + Fuzzing = closer examination

� Benefits of both forms of Analysis

Anatomy of the Analysis

Model to Mayhem

White Box vs. Black Box Testing

Delivery

Application Implementation Protocol Specification

Function

Design Abstraction Dissection

Analysis

Code Review Input Testing

Testing

Verification Validation

Analysis Engine

Final Deliverable

Input Modules

Protocol Template

Target

Seed File Session

Assembler

Sessions

Collection Method

Outputs

Core Fuzzing Process

Input Generation Methods (Invalid)

Error Collection

Isolated Element

Invalid Data

Input Generation Methods (Valid)

Valid Output

Isolated Element

Valid Data

Device EKG / ECG

ICMP •  Echo •  Reply •  Config

SNMP •  Status •  Agent • Manager

TCP • HTTP (S) • SSH (22) • TELNET (23)

Device EKG / ECG

� ICMP �  ICMP Echo / Reply �  Dropped Config, Delayed Response, etc…

� TCP �  Active Session, keep-alive, timeouts �  HTTPS, SSH, Telnet,

� SNMP �  Monitoring �  Statistics

Comparison and Contrast

� What does an error look like? �  How do you work with this information? �  What can be determined about the program / device? �  Can this lead to cascading errors?

� What can you do with an error? �  POC? �  Weaponization / Exploit Development

Exploit Weaponization (Stages)

Staged Attack Binary

Exploit Payload

Socket

Packaged Exploit

Exploit

Output Collection

� Comparison and contrast �  Characteristics of an error �  Scale of vulnerability

� “Weaponization” �  Malicious code �  Payloads �  Repeatable

� Hardware EKG �  Health of the device �  “State” of the device

Reports and Metrics

� Black box testing Report: �  Spreadsheet of tests and outputs, �  Tools used, �  Findings, �  Recommendations, �  Remediation steps,

� Include: �  Packet Captures (in pcap) for replay �  Screen captures �  Outputs for future analysis

Wrap-up

� Devices need to be tested �  Vendors continue to “push” product to market �  Consumers need to be aware of the hazards

� Small investment / Resilient Devices

� Testing is CRITICAL

� Does not need to be resource intensive �  Complex task, automated and facilitated �  Part of the internal Testbed

Contact info:

Jonathan Pollet, CISSP, CAP, PCIP Red Tiger Security - USA office: +1.877.387.7733 web: www.redtigersecurity.com jpollet@redtigersecurity.com Check out our Industry Forum and sign up for RSS feed:

Forum: http://www.redtigersecurity.com/forum/

Joe Cummins, PCIP, OPST Founder, Principal Consultant Red Tiger Security - Canada office: +1.877.387.7733 web: www.redtigersecurity.ca jcummins@redtigersecurity.ca

Black Box Testing Methodology SANS.ppt Box Testing... · Why Black Box Test? ! ... Why Black Box...

Documents

Transcript of Black Box Testing Methodology SANS.ppt Box Testing... · Why Black Box Test? ! ... Why Black Box...

Testing software: black box and white box testing

04 black box testing

ISTQB: Black Box testing Strategies used in Financial ...830538/...There are two classes of software testing; Black box testing and white box testing, Williams [10]. The black box

The Design Processpeople.tamu.edu/~rob.light/spring2019/csce315... · TESTING FOCUS •Broadly, there are 2 types of testing: •Black-box testing •White-box testing •Black-box

Black Box and White Box Testing

BLACK BOX TESTING K.KARTHIKEYAN. Black box testing technique Random testing Equivalence and partitioning testing Boundary value analysis State transition.

Software Testing & Quality Assurance- Black-Box Testing

Black Box Testing

Formal Black Box Testing

Unit Testing Methods White-box Testing Black-box Testing Extensions.

White Box and Black Box Testing

w 5 Black Box Testing

Black & White Box testing

Black box & white-box testing technique

Black-Box Testing Techniques II

Black Box Software Testing

Black Box Unit Testing

Finding bugs from failed verification attemptsi12 · White-box testing through black-box testing + speci cation mining (Beckert, Gladisch) ... White-box testing through black-box

Black-box Testing Black-box Testing Categories Types of Black box testing Equivalence class Partitioning Equivalence class Partitioning Boundary Value.

Techniques in black box testing