Test Design Optimization in Systems Engineering€¦ · Test Design Optimization helps minimize...

Test Design Optimization

in Systems Engineering

ITEA’s SOS Conference

28 Jan 2016

El Paso, TX

16-TDOSE-1A

Mark J. Kiemele, Ph.D.President and Co-Founder

Air Academy Associates

Office: 719-531-0777

Cell: 719-337-0357

[email protected]

www.airacad.com

©2016 Air Academy Associates, LLC. Do Not Reproduce.

Simplify, Perfect, Innovate

Testing has always been with us …...



Medieval times saw the proliferation of launching devices.



Medieval Cannon (combining gunpowder, molten metal, and wheels)



1903: First Flight Test by Wright Brothers

Why Kitty Hawk, North Carolina?

• The land was flat and they thought it a perfect place to fly and land.

• There were sand dunes there so they had a better chance to survive in the event of a crash.

• There were no trees in the area to crash into.

• They needed a steady wind.



1947: Bell X-1, first airplane to break the sound barrier

(flown by Chuck Yeager)



First wheels on suitcases: 1970 (long handle in 1987)



The Ultimate in Test: Invention of the Light Bulb by Thomas Edison

“I have not failed. I’ve just found 10,000 ways that won’t work.”

Edison, 1879



Goals

What is the Motivation for Test Design Optimization?

What is Test Design Optimization?

• Three Primary Reasons for Testing

• What Influences a Test Design?

• Examples of Optimal Test Design

• What is different about DOE?

• Simple Rules of Thumb for Selecting a Test Design

Where does it fit in with Systems Engineering?



Motivation for Test Design Optimization

Technical Debt

• Work which has been done but which has not been shown to

add value

• Associated with “batch” production in manufacturing

Examples:

• Getting far into design without concept validation by the customer

• Getting far into development without proper hardware testing

• Test Debt: the difference between testing everything and what

we can test most effectively and efficiently and still generate the

same knowledge gain as if we had tested everything

Technical debt has to be paid, and it can be expensive.

Test Design Optimization helps minimize technical debt by

minimizing test debt.



What is Test Design Optimization?

More effective testing

• Performing the right test

• For the right purpose

• At the right time

More efficient testing

• Using resources (time, people, and test materials) wisely

• Removing non-value activities

Minimizing the test debt



Definition of a Testing Process

Y1

X3

X4

X5

X6

X7

X2

X1

Y2

Y3



Inputs

Input Variable

Input Factor

Input Parameter

Indicator Variable

Independent Variable

Outputs

Output Variable

Response Variable

Process Performance Measure

Critical to Customer (CTC)

Critical to Quality (CTQ)

Dependent Variable

x y

IPO Terminology



A test matrix where each row (run) specifies a test case,

namely a combination of the levels of each of the factors

(inputs) that are being tested.

Run

1

2

3

.

.

X1 X2 X3 X4 Y1 Y2 . . . . . . Y SY

Inputs

A = X1

B = X2

D = X4

C = X3

YOutputs

.

.

.

.

.

.

PROCESS

What is a Test Design?



The Test Design is Dependent on:

• The purpose of the test

• The number of factors to be tested

• The number of levels to be tested for each factor

• The number of test resources available

• Other considerations and constraints unique to

the test scenario



Three Major Reasons for Testing

• Modeling

- For building functions that can be used to predict

outcomes, assess risk, and optimize performance. These

include the ability to evaluate interaction and higher order

effects.

• Performance Verification and Validation

- For confirming that a system performs in accordance with

its specifications/requirements and to get great test coverage

at low cost. Detecting and isolating bugs fall in this category.

• Screening

- For testing many factors in order to separate the vital few

factors from the trivial many.



Motivation for DOE from Dr. Gilmore (DOT&E)(from his 26 June 2013 memo on Flawed Applications of DOE)

1. One of the most important goals of operational testing is to

characterize a system’s effectiveness over the operational envelope.

2. I advocate the use of DOE to ensure that test programs are able to

determine the effect of factors on a comprehensive set of

operational mission-focused and quantitative response variables.

3. Future test plans must state clearly that data are being collected to

measure a particular response variable (possibly more than one) in

order to characterize the system’s performance by examining the

effects of multiple factors … and clearly delineating what statistical

model (e.g., main effects and interactions) is motivating … the

variation of the test.

4. Confounding factors must be avoided.

5. Another pitfall to avoid is relying on binary metrics as the primary

response variable.



Inputs

Input Variable

Input Factor

Input Parameter

Indicator Variable

Independent Variable

Outputs

Output Variable

Response Variable

Process Performance Measure

Critical to Customer (CTC)

Critical to Quality (CTQ)

Dependent VariableProcess

Characterization:

y = f(x)

y is a function of x (a transfer function)

y is related to x

True Scenario: y = f(x) + error

x y

IPO Terminology



Performance

(# home page loads/sec)

CPU

RAM Amount

HD Size

VM

Cost

($)

Performance

Tuning

Web-Based Application Testing

OS



Performance Tuning Terminology

Factors/Inputs

(X’s)

Levels

(Choices)

Performance/Outputs

(Y’s)

CPU Type

CPU Speed

RAM Amount

HD Size

VM

OS

Itanium, Xeon

1 GHz, 2.5 GHz

256 MB, 1.5 GB

50 GB, 500 GB

J2EE, .NET

Windows, Linux

# home page loads/sec

Cost

Which factors are important? Which are not?

Which combination of factor choices will optimize performance?

What Test Design would you choose to help you answer these questions?



Famous Quote

“All experiments (tests) are

designed experiments;

some are poorly designed,

some are well designed.”

George Box (1919-2013), Professor of Statistics, DOE Guru



Design of Experiments (DOE): A Subset of All Possible Test Design Methodologies

The Set of All Possible Test Design

Methodologies (Combinatorial Tests)

Orthogonal or

Nearly

Orthogonal

Test Designs

(DOEs)

One

Factor

At a

Time

(OFAT)

Best Guess

(Oracle)

Boundary Value Analysis

(BVA)

Equivalence Partitioning (EP)

Decision

Tables



Design of Experiments (DOE)

• “Interrogates” the process

• Changes “I think” to “I know” (with some level of confidence)

• It is the science of test and the key link between test &

evaluation.

• Used to identify important relationships between inputs and

outputs

• Identifies important interactions between process variables

• Can be used to optimize a process

• An optimal data collection methodology



Modeling Flight

Characteristics

of New 3-Wing

Aircraft

Pitch )

Roll )

W1F )

W2F )

W3F )

INPUT OUTPUT

(-15, 0, 15)

(-15, 0, 15)

(-15, 0, 15)

(0, 15, 30)

(0, 15, 30)

Six Aero-

Characteristics

Value Delivery: Reducing Time to Develop New Technologies

Patent Holder: Dr. Bert Silich

What test design would you choose to help you build the models on the next page?



CL = .233 + .008(P)2 + .255(P) + .012(R) - .043(WD1) - .117(WD2) + .185(WD3) + .010(P)(WD3) -

.042(R)(WD1) + .035(R)(WD2) + .016(R)(WD3) + .010(P)(R) - .003(WD1)(WD2) -

.006(WD1)(WD3)

CD = .058 + .016(P)2 + .028(P) - .004(WD1) - .013(WD2) + .013(WD3) + .002(P)(R) - .004(P)(WD1)

- .009(P)(WD2) + .016(P)(WD3) - .004(R)(WD1) + .003(R)(WD2) + .020(WD1)2 + .017(WD2)2

+ .021(WD3)2

CY = -.006(P) - .006(R) + .169(WD1) - .121(WD2) - .063(WD3) - .004(P)(R) + .008(P)(WD1) -

.006(P)(WD2) - .008(P)(WD3) - .012(R)(WD1) - .029(R)(WD2) + .048(R)(WD3) - .008(WD1)2

CM = .023 - .008(P)2 + .004(P) - .007(R) + .024(WD1) + .066(WD2) - .099(WD3) - .006(P)(R) +

.002(P)(WD2) - .005(P)(WD3) + .023(R)(WD1) - .019(R)(WD2) - .007(R)(WD3) + .007(WD1)2

- .008(WD2)2 + .002(WD1)(WD2) + .002(WD1)(WD3)

CYM= .001(P) + .001(R) - .050(WD1) + .029(WD2) + .012(WD3) + .001(P)(R) - .005(P)(WD1) -

.004(P)(WD2) - .004(P)(WD3) + .003(R)(WD1) + .008(R)(WD2) - .013(R)(WD3) + .004(WD1)2

+ .003(WD2)2 - .005(WD3)2

Ce = .003(P) + .035(WD1) + .048(WD2) + .051(WD3) - .003(R)(WD3) + .003(P)(R) - .005(P)(WD1)

+ .005(P)(WD2) + .006(P)(WD3) + .002(R)(WD1)

Aircraft Equations



EW Test Example with 6 Factors(18 orthogonal test cases)



EW Test Example (cont.)(Marginal Means Analysis)

WHAT

IS THE

GOAL?

SCREENINGVALIDATE PERFORMANCE

(TEST COVERAGE)

Rules of Thumb for Selecting a Test Design(based on the objective of the test)

MODELING

Notes:

1. “Mixed” factors means a combination of quantitative and qualitative (categorical)

2. “Mixed” levels means that not all factors have the same number of levels (settings)

3. “K” = Number of Factors and “L” = Number of Levels

4. “OA” = Orthogonal Array; “PAVO” = Pairwise Value Ordering

5. Software such as DOE Pro™, HD ToolsTM, rdExpertTM Lite, Pro-TestTM and

Quantum XLTM generate some or all of these designs

* DS and LHS are sampling techniques to generate representative samples

according to a specified distribution and a specified sample size

* Representative samples do not give orthogonal designs. They are often

used for getting test coverage, validating performance/ determining

capability, or creating noise combinations for test

© Copyright Air Academy Associates, LLC.

DOE Pro™ software is copyright Air Academy Associates, LLC and Digital Computations, Inc.

HD ToolsTM is a trademark of Air Academy Associates, LLC and software is copyright SigmaXL.rdExpertTM Lite software is copyright Phadke Associates, Inc.Pro-TestTM software is copyright Digital Computations, Inc.Quantum XLTM software is copyright SigmaZone.com.

2-Level Designs:

L12(6 ≤ K ≤11)

3-Level Designs:

L18(4 ≤ K ≤ 8)

High Factor/High LevelDesigns (K ≥ 9 and L ≥ 5):

Nearly Orthogonal Latin

Hypercube Designs

(NOLHDs) with K*L runs

Mixed Factor/Mixed Level Designs:HTT (OA or PAVO)

2-Level Designs:Full Factorial(K ≤ 4)

Fractional

Factorial

(K = 5)

3-Level Designs:Full Factorial(K ≤ 3)

CCD (3 ≤ K ≤ 5)

BB (3 ≤ K ≤ 4)

Mixed Factor/Mixed Level Designs:Full FactorialHTT (OA or PAVO, with select interactions only)

Fixed Number of Samples:Descriptive Sample (DS)*Latin Hypercube Sample (LHS)*

Not a Fixed Number of Samples:HTT (OA or PAVO)

©2016 Air Academy Associates Do Not Reproduce.

Integrating Test Design into Systems Engineering

FL

OW

ING

RE

QU

IRE

ME

NT

S D

OW

N

Customer Needs

System

Requirements

Sub-system

Requirements

Module

Requirements

Parts

Requirements

Parts

Performance

Module

Performance

Sub-system

Performance

System

Performance

Customer Acceptance

FL

OW

ING

CA

PA

BIL

ITY

UP

Design & Development



Critical Parameter Management and COIs

– A Critical Operational Issue (COI) is linked to operational effectiveness and

suitability.

– It is typically phrased as a question, e.g.,

Will the system detect the threat in a combat environment at

adequate range to allow for successful engagement?

y2 (engagement)

y1 (detect)

x1 x2 x3 (ranges) x4 (threat type) x5 x6



Good Test Design Enables Critical Parameter Management (CPM)

CPM is a systems engineering best practice that is extremely useful in managing, analyzing, and reporting technical product performance.



Where can Big Data help?

We already have DT&E and OT&E, but we also need continuous monitoring of

the system throughout its entire lifecyle to detect, predict, and prevent emergent

behavior. Big Data analytics can help us continuously test & evaluate (CT&E).



Guidelines for More Effective and Efficient Testing

Use continuous response variables whenever possible.

Eliminate confounding and aliasing of factors by using orthogonal or

nearly orthogonal designs.

Orthogonality means balance, both vertical and horizontal balance,

in the test design matrix or covering array.

Orthogonality provides the following capabilities:

• Can evaluate each of the factors independently, which implies a

cause-and-effect relationship can be established

• Can build linear and non-linear models, including interaction effects,

for prediction and risk assessment

• Provides better test domain coverage, due to the balance in the

designs, than non-orthogonal designs

• Not only detects defects but can also isolate them

Namely, use DOE.

Great data is never an accident. It happens by design.



Thank You

Questions

Colorado Springs, Colorado



Air Academy Associates, LLC1650 Telstar Drive, Ste 110

Colorado Springs, CO 80920

Toll Free: (800) 748-1277 or (719) 531-0777

Facsimile: (719) 531-0778

Email: [email protected]

Website: www.airacad.com

For More Information, Please Contact

Test Design Optimization in Systems Engineering€¦ · Test Design Optimization helps minimize...

Documents

Transcript of Test Design Optimization in Systems Engineering€¦ · Test Design Optimization helps minimize...