Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools
-
Upload
promise07 -
Category
Technology
-
view
1.255 -
download
0
description
Transcript of Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools
Make the Most of Your Time: How Should the Analyst Work with
Automated Traceability Tools?*
Alex DekhtyarJane Huffman HayesJody Larsen*Funded by NASA
IntroductionIntroduction
• What is requirements traceability?• Information retrieval and traceability• What’s different about traceability?• Research questions• Experimental design & validity• Results• Analysis• Future Work• Questions
What is Requirements Traceability?What is Requirements Traceability?
“Requirements traceability is the ability to describe and follow the life of a requirement, in both a forward and backward direction, i.e., from its origins, through its development and specification, to its subsequent deployment and use, and through periods of ongoing refinement and iteration in any of these phases.” Gotel and Finkelstein, 1994.
Why is Traceability Important?Why is Traceability Important?
• Requirements verification and validation
• Impact analysis
• Validating contracts and bids
• Testing Coverage
RepresentationDesign Document
Information Retrieval and TraceabilityInformation Retrieval and Traceability
Requirements Document
Matching algorithm
Analyst
What’s Different about Traceability?What’s Different about Traceability?
• Typical Information Retrieval Problem:– Search Engines: Google, Yahoo– Large and small documents– Document indexes must support arbitrary
search queries– High accuracy needed
• Requirements Traceability– Queries are known prior to indexing– Documents are all very small– High coverage needed
IR Based Traceability FrameworksIR Based Traceability Frameworks
• ADAMS Re-Trace– Part of an artifact management suite– Latent Semantic Indexing (LSI) Based– Web Application
• Poirot: Tracemaker– Probabilistic Information Retrieval– Web Application
• RETRO– Vector Space Model and LSI– Windows Application
RETRO.NETRETRO.NET
• Candidate Link• Top Link• Feedback• Confirmed Links
Research QuestionsResearch Questions
• What is the optimal upper bound on analyst efficiency using Information Retrieval based tools?
• What analyst behaviors produce accurate and complete RTMs most efficiently?
Experimental DesignExperimental Design
• Subject– A simulated analyst engaging in a
specified behavior
• Variables– Independent
• Analyst Behavior– Dependent
• Confirmed True Links• Observed Links• Confirmed Recall• Precision• Selectivity
N - number of low-level requirements;M - number of high-level requirements;
Hits - number of confirmed candidate linksStrikes - number of false positivesMisses - number of missed links
Precision =Hits
Hits + Strikes Recall =Hits
Hits +Misses
Selectivity =Hits + StrikesM * N
MeasuresMeasures
Global Ordering Without FeedbackGlobal Ordering Without Feedback
• All candidate link lists merged
• Links processed one at time in descending order
• Results in analyst looking at links that have highest global similarity ratings first
Local Ordering Without FeedbackLocal Ordering Without Feedback
• Each candidate link list sorted in descending order
• Top link for each requirement element is processed
• Results in analyst looking at the links for each requirement by highest similarity rating
Global Ordering With FeedbackGlobal Ordering With Feedback
• All candidate link lists merged
• Links processed one at time in descending order
• After each link is processed, feedback is executed
• Results in analyst looking at links that have highest global similarity ratings first
Local Ordering With FeedbackLocal Ordering With Feedback
• Each candidate link list sorted in descending order
• Top link for each requirement element is processed
• After all top links have been processed, feedback is executed
• Results in analyst looking at the links for each requirement by highest similarity rating
DatasetDataset
• CM-1• http://mdp.ivv.nasa.gov/mdp_glossary.html#CM1
# of elements in # of elements in requirements (high-level)requirements (high-level)
235
# of elements in design # of elements in design specification (low-level)specification (low-level)
220
# of correct links# of correct links 361
Total # of retrieved Total # of retrieved candidate linkscandidate links
36,556
Total # of correct links Total # of correct links retrieved by basic Vector retrieved by basic Vector SpaceSpace
358
RecallRecall 99%
PrecisionPrecision 0.1%
SelectivitySelectivity 70.7%
• In PROMISE repository• Sanitized dataset for a
NASA scientific instrument • Goal is to help analyst
avoid examining all 36,556 links
Experimental Design DetailExperimental Design Detail
• Two Studies– Impact of Behavior on Analyst Effort when
Recall is fixed• Recall set to 89%• Included all four structured behaviors
– Impact of Behavior on Precision and Recall when Analyst Effort is fixed
• Effort fixed to 1595 observed links• Added two additional behaviors
– Global, Random Order– Local, Random Order
Random?Random?
• Menzies et al. note that software analysis should “start with random methods because they are so cheap, moving to the more complex methods only when random methods fail.”
• Source: Menzies, T., Owen, D., and Richardson, J. The Strangest Thing about Software. IEEE Computer, January 2007, pp. 54 – 60.
Random MethodsRandom Methods• Global Filtered Random Selection
– All Candidate Link Lists Merged– List Sorted By Similarity– Bottom 75% (Least Relevant) Removed– 1595 Links Selected Randomly
• Local Filtered Random Selection– All Candidate Link Lists Sorted By
Similarity– Bottom 75% (Least Relevant) Removed– 7 or 8 Links Selected Randomly For Each
Requirement– 1595 Links Selected
• Executed 1000 times• Median values were used
Experiment OperationExperiment Operation
• Input:– Requirement and Specification Documents– Desired Analyst Behavior
• Operation:– Performs multiple rounds of interaction– Provides link status determination for each
observed link
• Output:– Candidate RTM– Dependent Variable Data
Experiment ValidityExperiment Validity• Internal Threats
– Accuracy of Requirements Traceability Matrix
– Suitability of text for Information Retrieval methods
• External Threats– Can Results Based on CM-1 Dataset be
Generalized?– Realism of Simulated Analyst Behavior
Study 1 Results, Fixed RecallStudy 1 Results, Fixed Recall
Observed LinksObserved Links PrecisionPrecision SelectivitySelectivity
Local, FeedbackLocal, Feedback 1595 20% 3%
Local, No FeedbackLocal, No Feedback 1713 19% 3%
Global, FeedbackGlobal, Feedback 5399 6% 10%
Global, No FeedbackGlobal, No Feedback 6149 5% 12%
Study 2 Results, Fixed EffortStudy 2 Results, Fixed Effort
Confirmed LinksConfirmed Links PrecisionPrecision RecallRecall
Local, FeedbackLocal, Feedback 321 20% 89%
Local, No FeedbackLocal, No Feedback 326 20% 87.5%
Global, FeedbackGlobal, Feedback 236 15% 65%
Global, No FeedbackGlobal, No Feedback 227 14% 63%
Local, RandomLocal, Random 65 4% 18%
Global, RandomGlobal, Random 58 3.6% 16%
Study 2 Intermediate ResultsStudy 2 Intermediate Results
0
50
100
150
200
250
300
350
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Observed Candidate Links
Ob
serv
ed C
orr
ect
Lin
ks
Global, No Feedback Global, feedback Local, Feedback Local No Feedback
AnalysisAnalysis• Knowing when to stop is crucial
– Accounts for largest difference between local versus global
• Feedback helps– Feedback resulted in a 7% to 13%
reduction of effort
• Have a system– Unstructured analyst behavior is not
efficient
Future WorkFuture Work
• User Interface Design
• Larger Datasets
• Live Analysts
Questions?Questions?