Approaches to Large, Unstructured, Complex Problems

The Big PictureGeoff Vining

Professor of Statistics

Virginia TechTÍTULO

Outline

• Big Picture: How to Provide Value to an Organization

• Large, Unstructured, Complex Problems

• Scientific Method

• The Science for Solving Problems with Data

• NASA Carbon Overwrapped Pressure Vessel Example

• The Future

Big Picture

• What Is Ultimately Valuable to an Organization?

• Understand Constancy of Purpose

• Why Does the Organization Exist

– Whale Oiling Companies No Longer Exist (from Deming)

– Did Not Understand the Real Reason They Existed!

• Modern Example – Kodak

Big Picture

• Leaders Concentrate on the Major Opportunities for the Organization

• The Sexy – New Markets

– The Next I-Phone

– Disruptive Innovation

Big Picture

• Not so Sexy – Improving What the Organization Does

– More than Lean Six Sigma, Continuous Improvement

– Requires Critical Focus on What the Organization Does and Why

– Often (Not Always!) Incremental Innovation

Big Picture

• Both Involve Problems that “Keep You Awake at Night”

• Value Comes from Opportunities!

Big Picture: Major Opportunities

• Are:

– Large

– Unstructured

– Complex

• Cut Across the Organization

Big Picture: Major Opportunities

• Require the Proper Data!

– Properly Collected

– Properly Analyzed

– Properly Interpreted

• Proper Solutions Must Be Sustainable

Large, Unstructured, Complex Problems

• Examples:

– DuPont

– Scott Paper

• Solutions Combine

– Academic – Basic Tools

– Practical Experience – How to Apply the Tools

Large, Unstructured, Complex Problems

• Currently, No Text Discusses Large, Unstructured, Complex Problems!

• Emerging New Discipline: Statistical Engineering

Solutions

• Interdisciplinary Teams

• Proper Blend of Talents

– Technical – Engineering/Scientific

– Statistical/Analytical

– Organizational Psychology/Anthropology

– Project Management

Solutions

• Strong Leadership!

– Form the Right Team

– Provide the Appropriate Resources

– Engaged in the Process

Scientific Method

• The heart of Statistical Engineering is the scientific method.

• Most theories underlying statistical engineering involve strategic application of the scientific method.

– Deming-Shewhart PDCA (Plan, Do, Check, Act)

– DMAIC (Define, Measure, Analyze, Improve, Control)

Scientific Method

• The Scientific Method Is

– Fundamental Approach for Discovery and Problem Solving

– Inductive/deductive problem solving process

Scientific Method: Basic Process

1. Define the problem (deductive)

2. Propose an educated theory, idea or model (deductive)

3. Collect data to test the theory (inductive)

4. Analyze the results (inductive)

5. Interpret the data and draw conclusions (inductive)

Scientific Method: Basic Process

• This process continues until a reasonable solution emerges.

• The scientific method is a sequential learning strategy!

Summary of the Scientific Method

• Understand the Real Problem at Hand

• Define the Problem

Summary of the Scientific Method

• Discover Solutions

– Abstract from the concrete to the abstract

– Develop a theory

– Test the theory using data

– Modify the theory as necessary

• Strong Need for Interdisciplinary Collaboration

Scientific Method and Data

• Data are the keys to the successful application of the scientific method

– Data collection

– Data analysis

– Data Interpretation

Scientific Method and Data

• Quality Engineering/Industrial Statistics are the handmaiden.

• Very important role in solving large, unstructured, complex problems.

Strategy of Statistical Engineering

• Identify Problem

• Provide Structure

• Understand Context

Strategy of Statistical Engineering

• Develop the Solution Strategy

• Develop and Execute Tactics

• Deploy Final Solution

Tactics of Statistical Engineering

• Data Acquisition

• Data Exploration

• Analysis

– Traditional Statistical Methodologies

– Modern Analytics (Big Data)

Tactics of Statistical Engineering

• Inference to the Process/Problem

• Deployment of Tentative Solution

– Does It Work?

– Is It Sustainable?

Overarching Methodologies

• Data Visualization

• Project Management

• Teamwork

• Organizational Culture

More on Tactics

• The Basic Tactical Tools Are Taught in Statistics/Industrial Engineering

• Data Acquisition:

– Planning the Data Collection

– Designing Experiments

– Data Cleansing

More on Tactics

• Data Exploration:

– Exploratory Data Analysis

– Analytics!

– Preliminary Models

– Follow Up

More on Tactics

• Analysis

– Statistical Inference: Estimation and Testing

– Formal Model Building

– Predictive Analytics

– Critical Point:

• Right Tool

• Properly Applied

More on Tactics

• Inference Back to the Problem – What Did We Discover?

• Propose, Evaluate, Deploy Solutions

• Statistical Engineering: The Science for Deploying the Tools!

NASA Example - COPVs

• Relatively Small Statistical Engineering Project

• COPV – Carbon Overwrapped Pressure Vessel

• Overarching Question of Interest: Reliability of COPVs at Use Conditions for Expected Life of Mission

NASA Example - COPVs

• Issues:

– Many different types of COPVs used in spacecraft

– Vessel tests are very expensive: money and time

• NASA Engineering Safety Center (NESC) Project

Core NESC Analytics Team:

• Reliability Engineers:

– JPL

– Langley Research Center

– Glenn Research Center

• Statisticians:

– Marshall Space Flight Center

– Virginia Tech

NASA Example - COPVs

• NASA Team’s Approach: Focus on Strands Used to Wrap Vessels

– Less expensive

– Can have many more experimental units than for vessels

• Still Issue with Time to Test

• Problem: How Do Strands Predict Vessel Behavior?

Previous Strand and Vessel Tests

• Relevant strand study conducted at a national lab:

– 57 strands at high loads for 10 years

– Net information learned: Strands either fail very early or last more than 10 years

• Vessel studies:

– Also 10 years

– Weibull model parameters similar to strands

Team’s Initial Concept

• Much larger study

• Censor very early

– Reduces time

– Allows the larger study in a practical amount of time

• Proceed in phases

• Have detailed data records to track any problems

NASA Example - COPVs

• Phase A: Conducted During Shake-Out of Equipment

• Small study (bigger than the national lab study!)

• Statistical goal: Determine if the parameters from the national lab study are valid as the basis for planning the larger study!

• Note: Phase A gave the team an opportunity to re-plan the larger experiment, if necessary!

NASA Example - COPVs

• Phase B: “Gold Standard” Experiment

• Planned time required: 1 year

• Used 4 “blocks” of equal numbers of strands

– Allowed the team to correct for time effects

– Allowed the team to mitigate problems

• Size assured ability to assess model

NASA Example - COPVs

• Total Size of the Database: Huge

• Kept data from start of specific strand test to failure on the second

• Kept the last 2 minutes at the .01 second from buffer

• Buffer allowed team to investigate unusual phenomena at failure

• Essential for proper data cleansing

NASA Example - COPVs

• Parallel Vessel Study

• Reasonably large ISS study targeted to end early (< 10 yrs)

• Opportunity to step up loads to mimic strands

• Censored but longer censor time than strands

Results to Date

• Phase A: Surprisingly similar to national lab study

• Phase B:

– Serious problem occurred in the first block

– Serious conversations with possibility of replacing!

– Other three blocks produced better than the planned precision for the estimates

– Residual analysis confirmed the Weibull model

Why is COPVs Statistical Engineering?

• Application of Scientific Method to a Complex Problem

• Sequential Data Collection/Experimentation

• Each Phase Targeted Different Questions

• Clearly Documented Assumptions, Assessed via Data

• Took Proper Steps to Cleanse Data

• Real Research Question Involves System of Systems

The Future

• Success Requires Providing Value to Organizations

• Solving Large, Unstructured, Complex Problems Provides Great Value

• Current State: People Know Tools, Often Limited Number of Tools

The Future

• Future State: How Do We Use the Tools Most Effectively

– Most Understand the Full Range of Tools Required

– Most Understand Best Practices for Using the Tools

– Must Understand Leadership in the Broad Sense

• The Future Is a Journey!