INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw...
-
Upload
mervyn-harper -
Category
Documents
-
view
216 -
download
2
Transcript of INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw...
![Page 1: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/1.jpg)
INCOSE Proprietary InformationLimited Distribution 1
Automated Specification Analysis
Walter Sobkiw
INCOSE Spring 09
April 2, 2009
![Page 2: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/2.jpg)
Problem Statement
How Can you Create a Clean Useful Specification?
Buzz Words are:Clear, Complete, Consistent, Testable, etc
![Page 3: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/3.jpg)
Typical Tools
Requirement Management• Automated database oriented tools
Modeling and Simulation• Automated visualization and calculation tools
Requirement Text Authoring• Manual mentors and check lists
![Page 4: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/4.jpg)
The Idea
Prelim Spec Doc
Spec Review Final Spec
ReportsUpdatesAuthors
PreviouslyManual
Inspections
![Page 5: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/5.jpg)
Why Do It
Specifications written in natural language• Initial text is rarely perfect
Everyone relies on specification text• Users, designers, testers, vendors, policy makers
Inspections used for surfacing defects, but• Time consuming, costly, only some defects found
Inspections may not even be performed• Fear of findings, not sure how to proceed once
surfaced
![Page 6: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/6.jpg)
Why Do It
Over 50% software defects are reqs problems• Source: CMU SEI QuARS Presentation and James
Martin, INCOSE 21 June 05
Over 80% rework spent on reqs related defects• Source: CMU SEI QuARS Presentation and Dean
Leffingwell, INCOSE 21 June 05
NASA IVV Entry in 2006• Studied the effects of introducing an automated tool
into the specification review process
![Page 7: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/7.jpg)
Findings
Review time not shortened• Time usually arbitrarily set
• People hunt & peck until bored or exhausted
Tool findings more consistent• Humans tend to miss categories
Tool finds all problems of certain type• Humans tend to miss full sets
![Page 8: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/8.jpg)
Findings
Humans better finding domain problems• So give humans time to do so
• And let tool excel in its area
More findings when tool is used• Humans + machines complement each other
![Page 9: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/9.jpg)
The Idea
Prelim Spec Doc
SAT Final Spec
ReportsUpdatesAuthors
PreviouslyManual
Inspections
Lets machines do what they do well• search, count, filter, categorize, profile, visualize
Lets humans do what they do well• creativity, critical thinking, inspiration, intuition
![Page 10: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/10.jpg)
Introduction toNatural Language Analysis
Lexical Analysis• Uses dictionary words and phrases
• Vague, subjective, imply choice or option
Syntactical Analysis • Relates to the syntax or grammar of the language
• Weak phrases, multiplicity, implicit, under-spec
Statistical Analysis• Statistical properties of language structure and usage
Consistency check• Areas such as units of measure
![Page 11: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/11.jpg)
Examples
Lexical examples• Ambiguous words: low, bad, clear, easy, efficient, etc.
Syntactical examples• Multiple requirements: use of and / or
• Under-specification: e.g. ‘report’, what kind of report
Statistical Analysis• Count frequency of words , such as ‘strip’
• If 50 times indicates important concept (domain term)
Consistency Check• check units e.g. 5 Hz and 5kHz, 10 ft and 10 meters
![Page 12: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/12.jpg)
SAT Key Requirements
Open rules visible to everyone• Rules for finding bad requirements text should
NOT be proprietary
Rules should be easy to change and coincident with analysis run• To allow users to quickly tune tool to their
domain and adjust for potential false finding
![Page 13: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/13.jpg)
SAT Key Requirements
It must be fast, NO overnight runs Encourage on-the-fly analysis Let users tweak rules, look at results,
and press submit button every 30 - 60 seconds
![Page 14: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/14.jpg)
SAT Key Requirements
Load specs anywhere off network Save results anywhere on network Duplicate previous analysis runs Modify previous runs Convert previous analysis into templates
![Page 15: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/15.jpg)
SAT Key Requirements
Do not discount power of machine to• Consistently search, count, organize, and
duplicate search-based findings
It is better than using word processor or requirements database tool• To search ad hoc during specification review
![Page 16: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/16.jpg)
SAT Key Requirements
Let tool reside and run on user computer Offer simple install remove mechanism Let user peer into where tool is located Let user view tool "libraries" using
standard operating system services Use Internet interface and web search
engine experience as stepping off point• permeated our collective experience
![Page 17: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/17.jpg)
SAT Architecture
Templates
PreviousAnalysis
SAT Engine
Apache Server
WebBrowser
UserDocuments
Services & Rules
SATExports
- Metrics & Results- Excel & HTML
Help
Runs on your computerReturns document text blocksSearch criteria many attributesSearch saved as templatesGrouped by rules & servicesFast 150 pages in 60 seconds
Works Like Internet Search Engine But
Nothing is hiddenUses predefined directories
Encourages directory accessAdd your own directories
![Page 18: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/18.jpg)
SAT Architecture
Rules grouped into services Services grouped into templates Templates grouped into domain Domains exist
![Page 19: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/19.jpg)
Services & Rules
Enable and disable services and rules using these display
filters to fit your situation
![Page 20: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/20.jpg)
Services & Rules
Modify, Delete, Add services and
rules
![Page 21: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/21.jpg)
Reports
Text Blocks
Metrics
Search Counts
All Words Counts
Settings
Shape
Reading Levels
![Page 22: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/22.jpg)
SAT Operations
SAT LeadStake Holders
Updated Specs
Reports
Official Reports
1.
2.
![Page 23: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/23.jpg)
SAT SAT SAT SAT SAT
spec writers
Designers QA Test IV&V
RelatedDocuments
Small 1-10 page
Problem StatementsProject Summaries
Architects
Uncommitted unclear position, do not begin modeling decomposing, designing, implementing until you understand & commit your stakeholders
Surface key Reqs
Consistency, completeness, testability, metrics
Evaluate docs, outline compliance, quality reqs
Specifications
Non Engineering
DomainsStartedHere
NASAStudy
Evolution
![Page 24: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/24.jpg)
SAT Application
While Writing a Specification During a Peer Review (original purpose) Quality Assurance As Part of Delivery Preparing for Test Ad Hoc Analysis
• Analyze and compare previous systems
• Compare Spec with architecture statements
• Measure requirement coverage
![Page 26: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/26.jpg)
Back Up Slides
If you have the time...
![Page 27: INCOSE Proprietary Information Limited Distribution 1 Automated Specification Analysis Walter Sobkiw INCOSE Spring 09 April 2, 2009.](https://reader030.fdocuments.us/reader030/viewer/2022032803/56649e405503460f94b320eb/html5/thumbnails/27.jpg)
SAT Big Picture
3. TEMPLATES
1. RULES
2. SERVICES
5. DOMAINS
COMMUNICATIONS
DEFENCE
ATC
6. PRECISE LANGUAGE INDUSTRIES
ENGINEERING
GOV REGULATION
LEGAL4. APPLICATION
.