doing, learning, being: some lessons learned from malaysia's ...
“Lessons learned” not being learned
3
An Example of “Lessons Learned” Not Being Learned NASA was extremely interested in the lessons learned during the Space Shuttle program development, operations, and retirement be carefully captured. The Original Source quote below is from a paper I co- authored during the last months of the operational phase of the Space Shuttle program. Recently, in looking for reference material, I once again came across Nancy Leveson’s Software and the Challenge of Flight Control chapter. In a quick scan of the material, my eye caught the passage below labeled Used As Reference . While the situation is quoted almost verbatim, the “lesson learned” (explanation) is totally incorrect. The statement by Nancy Leveson that “nervousness about the patching led to the use of much more extensive verification for the patches than for the high-level language changes” is 100 % incorrect. All Space Shuttle flight software verification used exactly the same process regardless if the change was via machine level patches or high order language source updates. In fact, many of the errors found on the STS-2 system were found by the identical test case ran against the machine language patches on STS-1. The “lesson learned” (explanation) was that for the STS-1 machine language patches, the process required the software verification analyst to participate in the pre-release formal inspection of the patch. During this same period, source updated for STS-2 required a formal inspection but participation was limited to development personnel only. Verification did an independent code inspection using its own formal process after the source solution was tested by development and promoted to the baseline source library. Once this “lesson learned” was observed by IBM management, there was an immediate action to develop a new high order language source inspection process used to the end of the program were the verification analyst was a required participant in the pre-build joint development/verification formal software inspection. I was one of two people assigned this action since I was a software verifier at the
Transcript of “Lessons learned” not being learned
An Example of “Lessons Learned” Not Being Learned
NASA was extremely interested in the lessons learned during the Space Shuttle program development, operations, and retirement be carefully captured. The Original Source quote below is from a paper I co-authored during the last months of the operational phase of the Space Shuttle program. Recently, in looking for reference material, I once again came across Nancy Leveson’s Software and the Challenge of Flight Control chapter. In a quick scan of the material, my eye caught the passage below labeled Used As Reference. While the situation is quoted almost verbatim, the “lesson learned” (explanation) is totally incorrect.
The statement by Nancy Leveson that “nervousness about the patching led to the use of much more extensive verification for the patches than for the high-level language changes” is 100 % incorrect. All Space Shuttle flight software verification used exactly the same process regardless if the change was via machine level patches or high order language source updates. In fact, many of the errors found on the STS-2 system were found by the identical test case ran against the machine language patches on STS-1.
The “lesson learned” (explanation) was that for the STS-1 machine language patches, the process required the software verification analyst to participate in the pre-release formal inspection of the patch. During this same period, source updated for STS-2 required a formal inspection but participation was limited to development personnel only. Verification did an independent code inspection using its own formal process after the source solution was tested by development and promoted to the baseline source library.
Once this “lesson learned” was observed by IBM management, there was an immediate action to develop a new high order language source inspection process used to the end of the program were the verification analyst was a required participant in the pre-build joint development/verification formal software inspection. I was one of two people assigned this action since I was a software verifier at the time. Within a week, we combined the well document development inspection process (previously pre-build) with the well documented verification inspection process (previously post build) into the organization’s only formal inspection process which was performed pre-build (nominally prior to development testing) whose inspection participants included moderator, developer, development peer, requirement analyst, and verification analyst.
The resulting process served the program well over the next quarter century.
Original Source
J. Christopher. Hickey, James B. Loveall, James K. Orr, and Andres L. Klausman, “The Legacy of Space Shuttle Flight Software,” AIAA Space 2011 Conference, Sept. 2-29, 2011, Long Beach California, 2011.
Also available for free at http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf
“For over one year prior to STS-1, the compiled and linked system was frozen and all mandatory changes were made using machine language patches. In parallel, these same mandatory changes were implemented on the STS-2 system as high order language source updates. In an analysis of the quality of STS-1 versus STS-2, it was determined that the quality of the machine languages patches for STS-1 were higher than the corresponding source changes on STS-2. Causal analysis determined that the significant difference was that Verification Analysts were added to the patch inspections due to the perceived high risk of making machine language patches. This immediately led to a process change where the Verification Analyst was a required Design/Code inspection participant on all changes effective with the release for STS-5 and subsequent.”
Used As Reference
Software and the Challenge of Flight Control by Nancy Leveson. To appear as a chapter in Space Shuttle Legacy: How We Did It/What We Learned edited by Roger Launius, James Craig, and John Krige and to
be published in AIAA in 2013. Note: Book has been published.
“For a year prior to STS-1, the software was frozen and all mandatory changes were made using machine language patches. In parallel, the same changes were made in the STS-2 software. Later it was determined that the quality of the machine language patches for STS-1 was better than the corresponding high-level language (HAL/S) changes in STS-2.23 This result seemed to defy common beliefs about the danger of patching software. Later the difference was explained by the fact that nervousness about the patching led to the use of much more extensive verification for the patches than for the high-level language changes.”
