Dynamic Queries cs5984: Information Visualization Chris North.
Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey
description
Transcript of Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey
![Page 1: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/1.jpg)
1
Program Comprehension through Dynamic Analysis
Visualization, evaluation, and a survey
Bas Cornelissen (et al.)Delft University of Technology
IPA Herfstdagen, Nunspeet, The NetherlandsNovember 26, 2008
![Page 2: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/2.jpg)
Context
• Software maintenance– e.g., feature requests, debugging– requires understanding of the program at hand– up to 70% of effort spent on comprehension process
Support program comprehension
2
![Page 3: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/3.jpg)
Definitions
Program Comprehension• “A person understands a program when he or she is able
to – explain the program, its structure, its behavior, its
effects on its operation context, and its relationships to its application domain
– in terms that are qualitatively different from the tokens used to construct the source code of the program.”
3
![Page 4: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/4.jpg)
Definitions (cont’d)
Dynamic analysis• The analysis of the properties
of a running software systemUnknown system
Instrumentation
Execution Scenario
e.g., open source
e.g., using AspectJ
(too) much data
• Advantages• preciseness• goal-oriented
• Limitations• incompleteness• scenario-dependence• scalability issues
4
![Page 5: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/5.jpg)
Outline
1. Literature survey2. Visualization I: UML sequence diagrams3. Comparing reduction techniques4. Visualization II: Extravis5. Current work: Human factor6. Concluding remarks
5
![Page 6: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/6.jpg)
Literature survey
6
![Page 7: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/7.jpg)
Why a literature survey?
• Numerous papers and subfields– last decade: many papers annually
• Need for a broad overview– keep track of current and past developments– identify future directions
• Existing surveys (4) do not suffice– scopes restricted– approaches not systematic– collective outcomes difficult to structure
7
![Page 8: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/8.jpg)
Characterizing the literature
• Four facets– Activity: what is being performed/contributed?
• e.g., architecture reconstruction
8
– Target: to which languages/platforms is the approach applicable?
• e.g., web applications
– Method: which methods are used in conducting the activity?
• e.g., formal concept analysis
– Evaluation: how is the approach validated?• e.g., industrial study
![Page 9: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/9.jpg)
Attribute framework
9
![Page 10: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/10.jpg)
Characterization
10Etc.
![Page 11: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/11.jpg)
Attribute frequencies
11
![Page 12: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/12.jpg)
Survey results
• Least common activities– surveys, architecture reconstruction
12
• Least common target systems– multithreaded, distributed, legacy, web
• Least common evaluations– industrial studies, controlled experiments,
comparisons
![Page 13: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/13.jpg)
Visualization I: Sequence Diagrams
13
![Page 14: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/14.jpg)
UML sequence diagrams
• Goal– visualize testcase executions as sequence diagrams– provides insight in functionalities– accurate, up-to-date documentation
• Method1. instrument system and testsuite2.execute testsuite3.abstract from “irrelevant” details4.visualize as sequence diagrams
14
![Page 15: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/15.jpg)
Evaluation
• JPacman– Small program for educational purposes– 3 KLOC– 25 classes
• Task– Change requests
• addition of “undo” functionality• addition of “multi-level” functionality
15
![Page 16: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/16.jpg)
Evaluation (cont’d)
• Checkstyle– code validation tool– 57 KLOC– 275 classes
• Task– Addition of a new check
• which types of checks exist?• what is the difference in terms of implementation?
16
![Page 17: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/17.jpg)
Results
• Sequence diagrams are easily readable– intuitive due to chronological ordering
• Sequence diagrams aid in program comprehension– supports maintenance tasks
• Proper reductions/abstractions are difficult– reduce 10,000 events to 100 events, but at what cost?
17
![Page 18: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/18.jpg)
Results (cont’d)
• Reduction techniques: issues– which one is “best”?
• which are most likely to lead to significant reductions?• which are the fastest?• which actually abstract from irrelevant details?
18
![Page 19: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/19.jpg)
Comparing reduction techniques
19
![Page 20: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/20.jpg)
Trace reduction techniques
• Input 1: large execution trace– up to millions of events
• Input 2: maximum output size– e.g., 100 for visualiz. through UML sequence diagrams
• Output: reduced trace– was reduction successful?– how fast was the reduction performed?– has relevant data been preserved?
20
![Page 21: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/21.jpg)
Example technique
Stack depth limitation [metrics-based filtering]• requires two passes
discard events above maximum depth
determine depth frequenciesTrace
0 28,4501 13,9022 58,4443 29,9334 10,004...
determinemaximum depth
maximum outputsize (threshold)
Trace
200,000events
50,000events
42,352events
> depth 1
21
![Page 22: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/22.jpg)
How can we compare the techniques?
• Use:– common context– common evaluation criteria– common test set
Ensures fair comparison
22
![Page 23: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/23.jpg)
Approach
• Assessment methodology1. Context2. Criteria3. Metrics4. Test set5. Application6. Interpretation
23
need for high level knowledge
reduction success rate; performance; info preservation
output size; time spent; preservation % per type
five open source systems, one industrial
apply reductions using thresholds 1,000 thru 1,000,000
compare side-by-side
![Page 24: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/24.jpg)
Techniques under assessment
• Subsequence summarization [summarization]
• Stack depth limitation [metrics-based]
• Language-based filtering [filtering]
• Sampling [ad hoc]
24
![Page 25: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/25.jpg)
Assessment summary
25
Subseq. summ.
Stack depth limitation
Lang.-based filterings
Sampling
Reduction success rate o o -- +
Performance-- o o --
Information preservation + o o --
![Page 26: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/26.jpg)
Visualization II: Extravis
26
![Page 27: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/27.jpg)
Extravis
• Execution Trace Visualizer– joint collaboration with TU/e
• Goal– program comprehension through trace visualization
• trace exploration, feature location, ...
– address scalability issues• millions of events sequence diagrams not adequate
27
![Page 28: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/28.jpg)
28
![Page 29: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/29.jpg)
Evaluation: Cromod
• Industrial system– Regulates greenhouse conditions– 51 KLOC– 145 classes
• Trace– 270,000 events
• Task– Analysis of fan-in/fan-out characteristics
29
![Page 30: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/30.jpg)
Evaluation: Cromod (cont’d)
30
![Page 31: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/31.jpg)
Evaluation: JHotDraw
• Medium-size open source application– Java framework for graphics editing– 73 KLOC– 344 classes
• Trace– 180,000 events
• Task– feature location
• i.e., relate functionality to source code or trace fragment
31
![Page 32: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/32.jpg)
Evaluation: JHotDraw (cont’d)
32
![Page 33: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/33.jpg)
Evaluation: Checkstyle
• Medium-size open source system– code validation tool– 73 KLOC– 344 classes– Trace: 200,000 events
• Task– formulate hypothesis
• “typical scenario comprises four main phases”• initialization; AST construction; AST traversal; termination
– validate hypothesis through trace analysis33
![Page 34: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/34.jpg)
Evaluation: Checkstyle (cont’d)
34
![Page 35: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/35.jpg)
Current work: Human factor
35
![Page 36: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/36.jpg)
Motivation
• Need for controlled experiments in general– measure impact of (novel) visualizations
• Need for empirical validation of Extravis in particular– only anecdotal evidence thus far
36
Measure usefulness of Extravis in software maintenance
• does runtime information from Extravis help?
![Page 37: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/37.jpg)
Experimental design
• Series of maintenance tasks– from high level to low level– e.g., overview, refactoring, detailed understanding
• Experimental group– ±10 subjects– Eclipse IDE + Extravis
• Control group– ±10 subjects– Eclipse IDE
37
![Page 38: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/38.jpg)
Concluding remarks
38
![Page 39: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/39.jpg)
Concluding remarks
• Program comprehension: important subject– make software maintenance more efficient
• Difficult to evaluate and compare– due to human factor
• Many future directions– several of which have been addressed by this research
39
![Page 40: Program Comprehension through Dynamic Analysis Visualization, evaluation, and a survey](https://reader031.fdocuments.us/reader031/viewer/2022020500/56814c74550346895db994bb/html5/thumbnails/40.jpg)
Want to participate in the controlled experiment..?
• Prerequisites– at least two persons– knowledge of Java– (some) experience with Eclipse– no implementation knowledge of Checkstyle– two hours to spare between December 1 and 19
Contact me:– during lunch, or– through email: [email protected]
40