Objective In which source files are undetected bugs most dangerous?
1
How Well does MM5 Resist Software Defects? How Well does MM5 Resist Software Defects? Dongping Xu 1 , Daniel Berleant 1,3 , Gene Takle 1,4 , Zaitao Pan 2 1 Iowa State University, 2 St. Louis University, 3 [email protected], 4 [email protected] Objective • In which source files are undetected bugs most dangerous? • Which files are most important to debug carefully? Motivating Question: How well can MM5 run despite software defects? Study #1 Study #1 : Effects of Bugs : Effects of Bugs on Forecasts on Forecasts Method Use mutation Analysis to statistically understand the effects of bugs on MM5. Tested 12 common bug types, 13 source files, and 10,893 1-bug mutations. Procedure for applying bugs to source code: Classify mutations into 3 categories: Calculating dependability of source file c: R c =# of mutations in the “results affected” category F c =# of mutations in the “Fail to complete” category. Dependability metric D c =F c /(R c +F c ) Low D c for a file suggests a need for extra care in testing and debugging that file. Results not affected Results affected (R c ) Given one bug in one file Runs successful ly Fail to complete (F c ) Value Description Unit U U-wind m/s V V-wind m/s T Temperature K Q Water Vapor mixing ratio Kg/kg CLW Cloud water mixing ratio Kg/kg RNW Rain water mixing ratio Kg/kg RAD Atmospheric radiation tendency K/DAY W Vertical velocity m/s PP Perturbation pressure Pa PSTARCRS Pstar cb GROUND Ground temperature K Results Values of D c for a number of important files in MM5: Of the files tested, bugs in exmoiss.F, hadv.F, init.f, mrfpbl.F, exmoiss.F, hadv.F, init.f, mrfpbl.F, param.f, and vadv.F param.f, and vadv.F are more likely than bugs in the others to have insidious, rather than obvious effects. 1.Failed to C om plete multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 86 103 128 3 7 8 17 22 45 2 8 10 hadv.F 30 30 32 1 9 8 7 1 17 0 3 2 init.F 7 0 0 14 9 2 4 11 9 0 0 2 initts.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara2.F 296 12 12 0 2 4 15 4 18 0 3 4 kfpara.F 189 21 21 0 9 16 11 1 9 2 0 0 lexmoiss.F 46 150 94 2 4 1 4 1 7 0 1 0 m rfpbl.F 148 53 60 8 23 26 44 22 30 2 14 17 param.F 0 0 0 47 29 9 26 9 16 0 5 4 param r.F 0 0 0 0 6 2 1 1 2 0 0 0 solve.F 99 438 444 3 0 23 29 8 48 1 27 28 vadv.F 15 29 30 1 0 5 5 3 3 0 4 6 tridi2.F 12 0 0 0 0 2 6 2 12 0 1 4 outts.F 2 0 0 0 0 0 1 1 0 0 0 0 2.Final ResultChanged multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 0 0 0 14 39 26 33 44 36 13 4 12 hadv.F 0 33 31 0 16 18 10 0 10 0 3 9 init.F 12 0 0 4 44 17 21 4 12 0 1 23 initts.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara2.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara.F 0 0 0 0 0 0 0 0 0 0 0 0 lexmoiss.F 0 0 0 0 0 0 0 0 0 0 0 0 m rfpbl.F 0 0 0 17 49 44 70 40 73 7 7 50 param.F 0 0 0 15 47 10 11 6 20 0 3 3 param r.F 0 0 0 0 0 0 0 0 0 0 0 0 solve.F 0 0 0 19 24 32 38 0 20 0 9 44 vadv.F 0 11 11 0 7 4 7 4 2 0 3 12 tridi2.F 0 0 0 0 0 0 4 1 0 0 1 2 outts.F 0 0 0 0 0 0 0 0 0 0 0 0 3.Final R esultU nchanged multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 0 0 0 27 292 21 1 2 2 0 11 1 hadv.F 5 24 24 0 115 18 12 0 0 0 8 3 init.F 17 90 90 43 108 23 19 13 19 0 74 48 initts.F 12 0 0 0 48 6 20 2 12 0 0 0 kfpara2.F 0 0 0 182 1017 227 225 178 255 10 48 53 kfpara.F 100 0 0 152 1181 204 212 181 269 18 49 51 lexmoiss.F 0 0 0 39 320 52 44 66 71 0 22 23 m rfpbl.F 0 0 0 26 547 67 57 21 50 1 63 19 param.F 34 0 0 321 253 103 86 43 48 0 41 44 param r.F 34 0 0 0 127 23 12 21 34 3 0 0 solve.F 0 0 0 83 412 208 95 13 41 1 145 97 vadv.F 4 17 16 1 69 28 22 2 0 3 19 8 tridi2.F 0 0 0 0 48 0 2 0 0 0 4 0 outts.F 24 0 0 0 56 8 11 3 14 0 0 0 Discussion and Future Work • The amount of change that a bug causes (not just if it changed or not) should also be analyzed. • Could differences observed across source code files be due in part to differences in the bugs applied rather than the files themselves? • How well do the effects of the tested bugs reflect the effects of real bugs caused by programmers? • A limitation of the study is its reliance on a single weather forecasting scenario. How general are the results across other scenarios? Motivation Sensitivity Analysis: How much perturbing initial conditions perturbs outputs. • Sensitivity analyses answer questions like “if we change CO 2 output by ∆, how much will that affect global warming?” • Ensemble forecasting provides understanding by perturbing initial conditions. Study #2 Study #2 : Comparing Effects on : Comparing Effects on Forecasts to Effects on Sensitivity Forecasts to Effects on Sensitivity Analyses Analyses Objective Answer the question, “Does sensitivity analysis resist bugs better than point prediction?” • sensitivity of original, unmutated software: S=∆O/∆I (O is output, I is input) • sensitivity of software as modified by mutation m: S m =∆O m /∆I Magnitude of mutation m’s Magnitude of mutation m’s effect on forecasting (F m ) effect on Sensitivity (S m ) Definition F m =|O m -O|/O S m =|S m -S|/S Relation F m >S m F m <S m Suggests the opposite. Method • The set of 24-hour forecasts produced by mutated variants of MM5 for a region of the U.S. midwest with a time step of 4 minutes forms a typical usage scenario. • 10,893 mutations*8 output parameters=87,144 data points. • Perturbation to input conditions: some variables (prognostic 3D variables: UA, UB, VA, VB, TA, TB, QVA, QVB) in file init.f were changed by 0.0001%. Results • Most of the 87,144 data points were unaffected by the mutation. • Sensitivity analysis was more affected than forecasting (i.e. F m <S m ) for 12,512 data points. • Sensitivity analysis was less affected (i.e. F m >S m ) for 323 data points. Hence – • MM5 resists bugs better for forecasting MM5 resists bugs better for forecasting than for sensitivity analyses. than for sensitivity analyses. Discussion and Future Work • What are the wider implications? • Would different input perturbations lead to significantly different results? • Would different input scenarios lead to significantly different results? • Would results generalize to climate change forecasting vs. prediction of the effects of changes in CO 2 production? Evaluation: we picked a representative subset of final outputs for analysis. Raw data for the effects of several thousand bugs on different source code files: A change in any of these counts as a change in simulation results. References: [1] Berleant, D. and B. Liu. Is sensitivity analysis more fault tolerant than point prediction? Simulation in the Medical Sciences: Proceedings of the 1997 Western MultiConference, The Society for Computer Simulation International, pp. 196-199. [2] Voas, J. and J. Payne (2000). Dependability certification of software components, Journal of Systems and Software 52 (2-3): 165-172. K Ground temperature GROUND cb Pstar PSTARCRS Pa Perturbation pressure PP m/s V ertical velocity W K/D AY Atm ospheric radiation tendency RAD Kg/kg Rain w ater mixing ratio RNW Kg/kg Cloud w ater mixing ratio CLW Kg/kg W ater Vapor mixing ratio Q K Tem perature T m/s V-wind V m/s U-wind U Unit Description Value K Ground temperature GROUND cb Pstar PSTARCRS Pa Perturbation pressure PP m/s V ertical velocity W K/D AY Atm ospheric radiation tendency RAD Kg/kg Rain w ater mixing ratio RNW Kg/kg Cloud w ater mixing ratio CLW Kg/kg W ater Vapor mixing ratio Q K Tem perature T m/s V-wind V m/s U-wind U Unit Description Value 1.Failed to C om plete multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 86 103 128 3 7 8 17 22 45 2 8 10 hadv.F 30 30 32 1 9 8 7 1 17 0 3 2 init.F 7 0 0 14 9 2 4 11 9 0 0 2 initts.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara2.F 296 12 12 0 2 4 15 4 18 0 3 4 kfpara.F 189 21 21 0 9 16 11 1 9 2 0 0 lexmoiss.F 46 150 94 2 4 1 4 1 7 0 1 0 m rfpbl.F 148 53 60 8 23 26 44 22 30 2 14 17 param.F 0 0 0 47 29 9 26 9 16 0 5 4 param r.F 0 0 0 0 6 2 1 1 2 0 0 0 solve.F 99 438 444 3 0 23 29 8 48 1 27 28 vadv.F 15 29 30 1 0 5 5 3 3 0 4 6 tridi2.F 12 0 0 0 0 2 6 2 12 0 1 4 outts.F 2 0 0 0 0 0 1 1 0 0 0 0 2.Final ResultChanged multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 0 0 0 14 39 26 33 44 36 13 4 12 hadv.F 0 33 31 0 16 18 10 0 10 0 3 9 init.F 12 0 0 4 44 17 21 4 12 0 1 23 initts.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara2.F 0 0 0 0 0 0 0 0 0 0 0 0 kfpara.F 0 0 0 0 0 0 0 0 0 0 0 0 lexmoiss.F 0 0 0 0 0 0 0 0 0 0 0 0 m rfpbl.F 0 0 0 17 49 44 70 40 73 7 7 50 param.F 0 0 0 15 47 10 11 6 20 0 3 3 param r.F 0 0 0 0 0 0 0 0 0 0 0 0 solve.F 0 0 0 19 24 32 38 0 20 0 9 44 vadv.F 0 11 11 0 7 4 7 4 2 0 3 12 tridi2.F 0 0 0 0 0 0 4 1 0 0 1 2 outts.F 0 0 0 0 0 0 0 0 0 0 0 0 3.Final R esultU nchanged multiPower switchIJK sw itchIJK-2 changeO P changeC oe plusToMinu MinusToPlu divToMul mulToDiv powerToM u offByO ne offByMinus exmoiss.F 0 0 0 27 292 21 1 2 2 0 11 1 hadv.F 5 24 24 0 115 18 12 0 0 0 8 3 init.F 17 90 90 43 108 23 19 13 19 0 74 48 initts.F 12 0 0 0 48 6 20 2 12 0 0 0 kfpara2.F 0 0 0 182 1017 227 225 178 255 10 48 53 kfpara.F 100 0 0 152 1181 204 212 181 269 18 49 51 lexmoiss.F 0 0 0 39 320 52 44 66 71 0 22 23 m rfpbl.F 0 0 0 26 547 67 57 21 50 1 63 19 param.F 34 0 0 321 253 103 86 43 48 0 41 44 param r.F 34 0 0 0 127 23 12 21 34 3 0 0 solve.F 0 0 0 83 412 208 95 13 41 1 145 97 vadv.F 4 17 16 1 69 28 22 2 0 3 19 8 tridi2.F 0 0 0 0 48 0 2 0 0 0 4 0 outts.F 24 0 0 0 56 8 11 3 14 0 0 0 Suggests MM5 resists the deleterious effects of bugs on sensitivity analysis better than it resists their effects on forecasting. (Graphic from: www.stalbans.gov.uk/ living/energy/global- warming.gif) Introduction: We investigate the impact of bugs in MM5. In Study #1, different source files were compared to see which are most susceptible to undetected bugs. In Study #2, we compare the effects of bugs on sensitivity analysis to their effects on forecasting. The findings help fill a gap in knowledge about the dependability of MM5, leading to both new understanding and further questions.
description
Runs successfully. Results affected (R c ). Results not affected. Given one bug in one file. Fail to complete (F c ). - PowerPoint PPT Presentation
Transcript of Objective In which source files are undetected bugs most dangerous?
How Well does MM5 Resist Software Defects?How Well does MM5 Resist Software Defects?Dongping Xu1, Daniel Berleant1,3, Gene Takle1,4, Zaitao Pan2
1Iowa State University, 2St. Louis University, [email protected], [email protected]
Objective
• In which source files are undetected bugs most dangerous?
• Which files are most important to debug carefully?
Motivating Question:
How well can MM5 run despite software defects?
Study #1Study #1: Effects of Bugs on : Effects of Bugs on ForecastsForecasts
Method
Use mutation Analysis to statistically
understand the effects of bugs on MM5.
Tested 12 common bug types, 13 source files,
and 10,893 1-bug mutations.
Procedure for applying bugs to source code:
Classify mutations into 3 categories:
Calculating dependability of source file c:
Rc=# of mutations in the “results affected” category
Fc=# of mutations in the “Fail to complete” category.
Dependability metric Dc=Fc/(Rc+Fc)
Low Dc for a file suggests a need for extra care in testing and debugging that file.
Results not affected
Results affected (Rc)
Given one bug in one file
Runs successfully
Fail to complete (Fc)
Value Description Unit
U U-wind m/s
V V-wind m/s
T Temperature K
Q Water Vapor mixing ratio Kg/kg
CLW Cloud water mixing ratio Kg/kg
RNW Rain water mixing ratio Kg/kg
RAD Atmospheric radiation tendency
K/DAY
W Vertical velocity m/s
PP Perturbation pressure Pa
PSTARCRS Pstar cb
GROUND Ground temperature K
Results
Values of Dc for a number of important files in
MM5:
Of the files tested, bugs in exmoiss.F, hadv.F, exmoiss.F, hadv.F, init.f, mrfpbl.F, param.f, and vadv.F init.f, mrfpbl.F, param.f, and vadv.F are
more likely than bugs in the others to have insidious, rather than obvious effects.
1. Failed to CompletemultiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOne
exmoiss.F 86 103 128 3 7 8 17 22 45 2 8 10hadv.F 30 30 32 1 9 8 7 1 17 0 3 2init.F 7 0 0 14 9 2 4 11 9 0 0 2initts.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara2.F 296 12 12 0 2 4 15 4 18 0 3 4kfpara.F 189 21 21 0 9 16 11 1 9 2 0 0lexmoiss.F 46 150 94 2 4 1 4 1 7 0 1 0mrfpbl.F 148 53 60 8 23 26 44 22 30 2 14 17param.F 0 0 0 47 29 9 26 9 16 0 5 4paramr.F 0 0 0 0 6 2 1 1 2 0 0 0solve.F 99 438 444 3 0 23 29 8 48 1 27 28vadv.F 15 29 30 1 0 5 5 3 3 0 4 6tridi2.F 12 0 0 0 0 2 6 2 12 0 1 4outts.F 2 0 0 0 0 0 1 1 0 0 0 0
2. Final Result ChangedmultiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOne
exmoiss.F 0 0 0 14 39 26 33 44 36 13 4 12hadv.F 0 33 31 0 16 18 10 0 10 0 3 9init.F 12 0 0 4 44 17 21 4 12 0 1 23initts.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara2.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara.F 0 0 0 0 0 0 0 0 0 0 0 0lexmoiss.F 0 0 0 0 0 0 0 0 0 0 0 0mrfpbl.F 0 0 0 17 49 44 70 40 73 7 7 50param.F 0 0 0 15 47 10 11 6 20 0 3 3paramr.F 0 0 0 0 0 0 0 0 0 0 0 0solve.F 0 0 0 19 24 32 38 0 20 0 9 44vadv.F 0 11 11 0 7 4 7 4 2 0 3 12tridi2.F 0 0 0 0 0 0 4 1 0 0 1 2outts.F 0 0 0 0 0 0 0 0 0 0 0 0
3. Final Result UnchangedmultiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOne
exmoiss.F 0 0 0 27 292 21 1 2 2 0 11 1hadv.F 5 24 24 0 115 18 12 0 0 0 8 3init.F 17 90 90 43 108 23 19 13 19 0 74 48initts.F 12 0 0 0 48 6 20 2 12 0 0 0kfpara2.F 0 0 0 182 1017 227 225 178 255 10 48 53kfpara.F 100 0 0 152 1181 204 212 181 269 18 49 51lexmoiss.F 0 0 0 39 320 52 44 66 71 0 22 23mrfpbl.F 0 0 0 26 547 67 57 21 50 1 63 19param.F 34 0 0 321 253 103 86 43 48 0 41 44paramr.F 34 0 0 0 127 23 12 21 34 3 0 0solve.F 0 0 0 83 412 208 95 13 41 1 145 97vadv.F 4 17 16 1 69 28 22 2 0 3 19 8tridi2.F 0 0 0 0 48 0 2 0 0 0 4 0outts.F 24 0 0 0 56 8 11 3 14 0 0 0
Discussion and Future Work
• The amount of change that a bug causes (not just if it changed or not) should also be analyzed.
• Could differences observed across source code files be due in part to differences in the bugs applied rather than the files themselves?
• How well do the effects of the tested bugs reflect the effects of real bugs caused by programmers?
• A limitation of the study is its reliance on a single weather forecasting scenario. How general are the results across other scenarios?
Motivation
Sensitivity Analysis: How much perturbing initial conditions perturbs outputs.
• Sensitivity analyses answer questions like “if we change
CO2 output by ∆, how much will that affect global warming?”
• Ensemble forecasting provides understanding by
perturbing initial conditions.
Study #2Study #2: Comparing Effects on: Comparing Effects on
Forecasts to Effects on SensitivityForecasts to Effects on Sensitivity
AnalysesAnalyses
Objective
Answer the question, “Does sensitivity analysis resist bugs
better than point prediction?”
• sensitivity of original, unmutated software:
S=∆O/∆I (O is output, I is input)
• sensitivity of software as modified by mutation m:
Sm=∆Om/∆I
Magnitude of mutation m’s Magnitude of mutation m’s
effect on forecasting (Fm) effect on Sensitivity (Sm)
Definition Fm=|Om-O|/O Sm=|Sm-S|/S
Relation Fm>Sm
Fm<Sm Suggests the opposite.
Method
• The set of 24-hour forecasts produced by mutated variants of MM5 for a region of the U.S. midwest with a time step of 4 minutes forms a typical usage scenario.
• 10,893 mutations*8 output parameters=87,144 data points.
• Perturbation to input conditions: some variables (prognostic 3D variables: UA, UB, VA, VB, TA, TB, QVA, QVB) in file init.f were changed by 0.0001%.
Results
• Most of the 87,144 data points were unaffected by the mutation.
• Sensitivity analysis was more affected than forecasting (i.e. Fm<Sm) for 12,512 data points.
• Sensitivity analysis was less affected (i.e. Fm>Sm) for 323 data points. Hence –
• MM5 resists bugs better for forecasting than MM5 resists bugs better for forecasting than for sensitivity analyses.for sensitivity analyses.
Discussion and Future Work
• What are the wider implications?
• Would different input perturbations lead to significantly different results?
• Would different input scenarios lead to significantly different results?
• Would results generalize to climate change forecasting vs. prediction of the effects of changes in CO2 production?
Evaluation: we picked a representative
subset of final outputs for analysis.Raw data for the effects of several thousand bugs on
different source code files:
A change in any of these counts as a change in simulation results.
References:[1] Berleant, D. and B. Liu. Is sensitivity analysis more
fault tolerant than point prediction? Simulation in the Medical Sciences: Proceedings of the 1997 Western MultiConference, The Society for Computer Simulation International, pp. 196-199.
[2] Voas, J. and J. Payne (2000). Dependability certification of software components, Journal of Systems and Software 52 (2-3): 165-172.
KGround temperatureGROUND
cbPstarPSTARCRS
PaPerturbation pressurePP
m/sVertical velocityW
K/DAYAtmospheric radiation tendency
RAD
Kg/kgRain water mixing ratioRNW
Kg/kgCloud water mixing ratioCLW
Kg/kgWater Vapor mixing ratioQ
KTemperatureT
m/sV-windV
m/sU-windU
UnitDescriptionValue
KGround temperatureGROUND
cbPstarPSTARCRS
PaPerturbation pressurePP
m/sVertical velocityW
K/DAYAtmospheric radiation tendency
RAD
Kg/kgRain water mixing ratioRNW
Kg/kgCloud water mixing ratioCLW
Kg/kgWater Vapor mixing ratioQ
KTemperatureT
m/sV-windV
m/sU-windU
UnitDescriptionValue1. Failed to Complete
multiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOneexmoiss.F 86 103 128 3 7 8 17 22 45 2 8 10hadv.F 30 30 32 1 9 8 7 1 17 0 3 2init.F 7 0 0 14 9 2 4 11 9 0 0 2initts.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara2.F 296 12 12 0 2 4 15 4 18 0 3 4kfpara.F 189 21 21 0 9 16 11 1 9 2 0 0lexmoiss.F 46 150 94 2 4 1 4 1 7 0 1 0mrfpbl.F 148 53 60 8 23 26 44 22 30 2 14 17param.F 0 0 0 47 29 9 26 9 16 0 5 4paramr.F 0 0 0 0 6 2 1 1 2 0 0 0solve.F 99 438 444 3 0 23 29 8 48 1 27 28vadv.F 15 29 30 1 0 5 5 3 3 0 4 6tridi2.F 12 0 0 0 0 2 6 2 12 0 1 4outts.F 2 0 0 0 0 0 1 1 0 0 0 0
2. Final Result ChangedmultiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOne
exmoiss.F 0 0 0 14 39 26 33 44 36 13 4 12hadv.F 0 33 31 0 16 18 10 0 10 0 3 9init.F 12 0 0 4 44 17 21 4 12 0 1 23initts.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara2.F 0 0 0 0 0 0 0 0 0 0 0 0kfpara.F 0 0 0 0 0 0 0 0 0 0 0 0lexmoiss.F 0 0 0 0 0 0 0 0 0 0 0 0mrfpbl.F 0 0 0 17 49 44 70 40 73 7 7 50param.F 0 0 0 15 47 10 11 6 20 0 3 3paramr.F 0 0 0 0 0 0 0 0 0 0 0 0solve.F 0 0 0 19 24 32 38 0 20 0 9 44vadv.F 0 11 11 0 7 4 7 4 2 0 3 12tridi2.F 0 0 0 0 0 0 4 1 0 0 1 2outts.F 0 0 0 0 0 0 0 0 0 0 0 0
3. Final Result UnchangedmultiPowerswitchIJK switchIJK-2changeOP changeCoefplusToMinusMinusToPlusdivToMul mulToDiv powerToMuloffByOne offByMinusOne
exmoiss.F 0 0 0 27 292 21 1 2 2 0 11 1hadv.F 5 24 24 0 115 18 12 0 0 0 8 3init.F 17 90 90 43 108 23 19 13 19 0 74 48initts.F 12 0 0 0 48 6 20 2 12 0 0 0kfpara2.F 0 0 0 182 1017 227 225 178 255 10 48 53kfpara.F 100 0 0 152 1181 204 212 181 269 18 49 51lexmoiss.F 0 0 0 39 320 52 44 66 71 0 22 23mrfpbl.F 0 0 0 26 547 67 57 21 50 1 63 19param.F 34 0 0 321 253 103 86 43 48 0 41 44paramr.F 34 0 0 0 127 23 12 21 34 3 0 0solve.F 0 0 0 83 412 208 95 13 41 1 145 97vadv.F 4 17 16 1 69 28 22 2 0 3 19 8tridi2.F 0 0 0 0 48 0 2 0 0 0 4 0outts.F 24 0 0 0 56 8 11 3 14 0 0 0
Suggests MM5 resists the deleterious effects of bugs on sensitivity analysis better than it resists their effects on forecasting.
(Graphic from: www.stalbans.gov.uk/living/energy/global-warming.gif)
Introduction: We investigate the impact of bugs in MM5. In Study #1, different source files were compared to see which are most susceptible to undetected bugs. In Study #2, we compare the effects of bugs on sensitivity analysis to their effects on forecasting. The findings help fill a gap in knowledge about the dependability of MM5, leading to both new understanding and further questions.
