A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment Atlee...
-
Upload
salma-gregory -
Category
Documents
-
view
213 -
download
1
Transcript of A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment Atlee...
A System Analysis Study Comparing Reverse Engineered
Combinatorial Testing to Expert Judgment
Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. HolmanLockheed Martin
2
Agenda
Introduce the trade study space: the F16 and Combinatorial Test (CT) problem
Define the F16 Failure
Present the steps of the CT study
Cover the results
Conclusions
3
INTRODUCTION: F16 VENTRAL FIN STUDY AND APPLYING COMBINATORIAL TESTING
Evaluate the use of Combinatorial Testing (CT) to a real “problem”
–Used a historic F-16 problem and data–See if CT could be used in place of or to support an
expert• Save time and/or people• Mixed example, between what was done and what
could have been done
Problem space–Interacting factors (good for CT)–Outside of the software testing–System-Hardware failure resolution and design evaluation –Demonstrate CT is viable
4
F16 Failure Case Study
During production and maintenance of the F-16 fighter aircraft a structural problem immerged
– Buffeting of the F-16 ventral fins has provided a classic example of structural fatigue
of such aerodynamic surfaces by an upstream source of severe turbulent wakes
– These fins are very thin surfaces, about 5 ft. chord and 2 ft. span, composed of three
wedge like surfaces that taper down to edge thicknesses of 0.05 inches, all of which
makes the fins susceptible to turbulence buffeting
– Examples of possibly interacting turbulence sources include: various centerline
stores, side slips and inlet lip spillage during rapid decelerations
The historic work done by Atlee M. Cunningham, Jr. and Ryan J. Holman
The Combinatorial analysis and case study was primarily done Jon Hagar with support from Atlee Cunningham as the “expert”
5
ORIGINAL F-16 PROBLEM DETAILS
Added 2 avionics LANTIRN pods on the F-16 just aft of the inlet on the lower fuselage directly upstream of the ventral fins
Avionics pods in general are often not very aerodynamic in shape and hence can produce very turbulent wakes
The damage to the right hand ventral fin on first flight with LANTIRNS
– Originally, the primary source of the fin’s fatigue and loss was high speed throttle chops
that produced severe turbulence from inlet lip spillage during rapid decelerations where
the throttle was suddenly pushed to idle position
A comparison of the ventral fin response to LANTIRN and throttle chop turbulence was done
– The response levels are about the same; however, constant buffeting by the LANTIRNS
produced much higher fatigue damage per flight hour as compared to that due to the
transient throttle chop
– As a result, several major structural re-designs of the fins and other associated structures
followed over the following years that incrementally improved the fatigue life of these
components
6
F16 with LANTIRN Pod and Ventral Fin
7
Failure: Damaged Ventral Fin
But Why (what parameters and interactions)?
8
ORIGINAL ANALYSIS HISTORY
After a number of years more the problem continued. As a result, a detailed analysis of the flight data was performed by
Atlee Cunningham , yielding
–Showed that the most severe buffeting of the ventrals consistently occurred with only LANTIRN pods on the aircraft and with high speed throttle chops at Mach 0.95 on the clean aircraft
–Anomalous trends were also seen in throttle chop data with LANTIRNs where response levels were 3-to-4 times as high as level flight with LANTIRN
–Recognizing that the very thin ventrals (leading edge thickness is 0.05 in.) would probably be subject to leading edge separation at small angles of side slip
–Flow change resulted in a large increase in the slope of side force with side slip angle, which would have a significant impact on dynamic loads due to large amplitude turbulence
9
Do More Testing and Analysis (by experts)
A low speed small scale wind tunnel test was conducted to
– Explore various aspects of ventral aerodynamics and effects of modifications
– Data were obtained with 1/5 scale models of the fin mounted on the wind tunnel wall and
rotated for incidence effects
– Testing was to determine “sensitivities” (variables and values) but had to be designed by
expert
Flight Tests were conducted
– Three of these four fins, plus several early block ventral fins, were tested on an early Block 15
F-16. The fins consisted of:
(1) the baseline fin, “BSLN,” the standard Block 40 ventral fin;
(2) the “MMC” fin, the Block 40 fin with 40% stiffer skins of MMC aluminum material
(3) the “MMCNC” fin, the MMC fin with an added rounded “nose cap” glove with a NACA 0012
airfoil section of 5 inch chord
(4) the “NACA” fin, the Block 40 modified to have a full span, full chord airfoil section that
eliminated the sharp leading edge and sharp tip section of the fin
– An expert had to define test program (combinations) for these too = hundreds of hours
10
DEFINING CONDITIONS FOR CT
What was the situation(s) that brought the failure on? Factors considered include:
– Aircraft (AC)
– Maneuver
– Speed (Mach)
– Altitude
– Aircraft add on structures (tanks, pods, etc.)
Which design solutions (4 fin designs) might solve the problem with different aircraft configurations:
– Block 15
– Block 40
11
IDEALIZED ANALYSIS STEPS USING CT (HYPOTHETICAL RECONSTRUCTION)
NIST ACTS Combinatorial Tool was used to “reverse” engineer a test program
–Other tools were considered
–Open source nature was deciding factor
This can be viewed as a “reverse” or “Re” engineering case study
– We were trying to see if the tool would replicate the historic test program without an system expert
– Test planning using a CT tool (not the expert)
A series of idealized steps were done using the tool
12
CT Step 1
Historic “first” test program - clean baseline configuration, which in the example are F16s in block 15 and 40 in “clean” configuration, and apply “testing” to points associated
Input to tool (equivalence classes):
Tool produced: 90 test cases (similar to actual effort) with 2 way
Aircraft 15 , 40
Altitude (s) 5k, 10k, 15k, 20k, 30k, 40k, 50k
Maneuvers hi-speed throttle , slow accel/dwell , L/R 5deg side slip , L/R 360 roll , R/L 360 roll, R/L 5deg side slip, Med accel/dwell, R-L-R-L banking, Hi-speed to Low, 360 nose roll
Mach(100th) 40, 50, 60, 70, 80, 90, 100, 110, 120
Parameters: Variable:
13
Step 2: Refinement of the test program
Step 2 a more refined set of analyses would have been done based on information from:
– Step 1
– Historic databases
– More detailed wind tunnel analyses
– Supplemental water tunnel analysis
– Flight data and constraints were available
This effort confirmed design work
– Produced 30 test cases from 2 way coverage
Parameters: Variable:Aircraft 15 , 40Alt 5, 10, 15Mach(100th) 60, 80, 85, 90, 95LANTIRN on, off
14
Step 3: Final Design Flight Test Program
Parameters: Variable:
AC-BLK&Ventral-Fin-Config
Blk15-Blk15 ventral, Blk15-Blk40 ventral, Blk15-MMC ventral, Blk15- MMC ventral + cap, Blk40-Blk40 ventral, Blk40-MMC ventral, Blk40-MMC Ventral+ nonosecap, Blk40-NACA
LANTIRN [on, off]Alt [5, 10, 15]Mach (100ths) [60, 70, 80, 85, 90, 95]Maneuver Block [basic, basic +]
Tests for a flight test program
Number of test (cases) generated with 2 way coverage: 72
15
Conclusions An example where combinatorial test could have aided
– Provided another test design method for teams to use
– Reduce the "shotgun” approach and expert judgment needed for situations dealing with “many” parameters
–Showed CT can support a system failure (fault isolation) analysis Historic data useful in a CT proof on concept (case study example) Lockheed Martin will continue advocating CT as a technique
–Looking for pilots and more data points
–Would be interesting to compare to DOE approaches
–How to get Engineers to start using Other items noticed
–Tool interchange (operability), particularly into a test automation framework
–Constraints were “tricky”
– Interface to/from Model based testing would be useful
16
Summary
Demonstrated Combinatorial Test tool could have supported the F16 problem (or other hardware, software, system test/analysis)
–Expert felt results would have been similar
–Approach could support other programs
Open source tool worked
–Commercial tools worked too
–Supports move from “theory” to real use
Supported an “non” software area
– System Design/Failure Evaluation