Inferring Specifications
description
Transcript of Inferring Specifications
1
Inferring Specifications
A kind of review
2
The Problem
Most programs do not have specifications Those that do often fail to preserve the
consistency between specification and implementation
Specification are needed for verification, testing and maintenance
3
Suggested Solution
Automatic discovery of specifications
4
Our Playground
Purpose Verification, testing, promote understanding
Specification representation Contracts, properties, automaton, …
Inference technique Static, dynamic, combination
Human intervention
5
Restrictions and Assumptions
Learning automata from positive traces is undecidable [Gold67]
An executing program is usually “almost” correct
If a miner can identify the common behavior, it can produce a correct specification, even from programs that contain errors
6
Perracota: Mining Temporal API Rules From Imperfect Traces
Jinlin Yang, David EvansDepartment of Computer Science, University Of VirginiaDeepali Bhardwai, Thirumalesh Bhat, Manuvir DasCenter for Software Excellence, Microsoft Corp.
ICSE ‘06
7
Key Contribution
Addressing the problem of imperfect traces Techniques for incorporating contextual
information into the inference algorithm Heuristics for automatically identifying
interesting properties
8
Perracota
A dynamic analysis tool for automatically inferring temporal properties
Takes the program's execution traces as input and outputs a set of temporal properties it likely has
ProgramInstrumented
Program
instru
men
tation Test Suite
ExecutionTraces
PropertyTemplates
InferredProperties
Testin
g
Infe
rence
9
Property Templates
Name QRE Valid Example
Invalid Examples
Response S*(PP*SS*)* SPPSS SPPSSP
Alternating (PS)* PSPS PSS, PPS, SPS
MultiEffect (PSS*)* PSS PPS, SPS
MultiCause (PP*S)* PPS PSS, SPS
EffectFirst S*(PS)* SPS PSS, PPS
CauseFirst (PP*SS*)* PPSS SPSS, SPPS
OneCause S*(PSS*)* SPSS PPSS, SPPS
OneEffect S*(PP*S)* SPPS PPSS, SPSS
10
Initial approach
Algorithm is developed for inferring two-event properties (scalability)
Complexity O(nL) time, O(n2) space n – number of distinct events L – length of trace
Each cell in the matrix holds the current state of a state machine that represent the alternating pattern between the pair of events
Require perfect traces
11
Approximate Inference
Partition trace into sub-traces For example: PSPSPSPSPSPPPP
PS|PS|PS|PS|PS|PPP Compute satisfaction rate of each template
The ratio between partitions satisfying the alternate property and the total number of partitions
Set a satisfaction threshold
12
Contextual Properties
lock1.acqlock2.acqlock2.rellock1.rel
neutral
sensitive
No property Inferred
lock1.acq→lock2.acq lock1.acq→lock2.rellock1.acq→lock1.rel lock2.acq→lock2.rellock2.acq→lock1.rel lock2.rel→lock1.rel
slicing
//lock1acqrel
//lock2acqrel
lock1.acq→lock2.acq
13
Selecting Interesting Properties Reachablity
Mark a property P→S as probably uninteresting if S is reachable from P in the call graph
For example:
The relationship between C and D is not obvious from inspecting either C or D
X() { … C(); … D(); …}
A() { … B(); …}
14
Selecting Interesting Properties Name Similarity
A property is more interesting if it involves similarly named events
For Example:ExAcquireFastMutexUnsafe
ExReleaseFastMutexUnsafe Compute word similarity as )/(2 sp www
15
Chaining
Connect related Alternating properties into chains A→B, B→C and A→C implies A→B→C
Provide a way to compose complex state machines out of many small state machines
Identification of complex multi-event properties without suffering a high computational cost
16
SMArTIC: Towards Building an Accurate, Robust and Scalable Specification MinerDavid Lo and Siau-Cheng Khoo
Department of Computer Science, National University of Singapore
FSE ‘06
17
Hypotheses
Mined specifications will be more accurate when: erroneous behavior is removed before learning they are obtained by merging the specifications
learned from clusters of related traces than when they are obtained from learning the entire traces
18
Filterin
g
Merg
ing
Traces
Clu
stering
Learn
ing
Merged AutomatonFiltered
Traces
Clusters ofFilteredTraces
Automatons
Structure
19
Filtering
How can you tell what’s wrong if you don’t know what’s right?
Filter out erroneous traces based on common behavior
Common behavior is represented by “statistically significant” temporal rules
20
Pre → Post Rules
Look for rules of the form a→bcwhen a occurs b must eventually occur after a, and c must also eventually occur after b
Rules exhibiting high confidence and reasonable support can be considered as “statistical” invariants Support – Number of traces exhibiting the property
pre→post Confidence – the ratio of traces exhibiting the property
pre→post to those exhibiting the property pre
21
Clustering
Convert a set of traces into groups of related traces Localize inaccuracies Scalability
22
Clustering Algorithm
Variant of the k-medoid algorithm Compute the distance between a pair of data
items (traces) based on a similarity metric k is the number of clusters to create
Algorithm:Repeatedly increase k until a local maximum is reached
23
Similarity Metric
Use global sequence alignment algorithm
Problem: doesn’t work well in the presence of loops
Solution: compare the regular expression representation
FTFTALILLAVAVF--TAL-LLA-AV
ABCBCDABCBCBCDABCD
(A(BC)+D)+ABCD
24
Learning
Learn PFSAs from clusters of filtered traces PFSA per cluster
A “place holder” In current experiment – sk-strings learner
25
Merging
Merge multiple PFSAs into one The merged PFSA accepts exactly the union
of sentences accepted by the multiple PFSAs Ensures probability integrity
Probability for transition in output PFSA
)()(1
δpwδpiA
k
iiA
26
From Uncertainty to Belief: Inferring the Specifications Within
Ted Kremenek, Paul Twohey, Andrew Y. Ng, Dawson EnglerComputer Science Dep., Stanford University
Godmar BackComputer Science Dep., Virginia Tech
OSDI ‘06
27
Motivating Example
Problem: Inferring ownership roles Ownership idiom: a resource has at any time
exactly one owning pointer Infer annotation
ro – returns ownership co – claims ownership
Is fopen a ro? fread/fclose a co?
FILE* fp = fopen(“myfile.txt”, r);fread(buffer, n, 1000, fp);fclose(fp);
28
Basic Ownership Rules
fp = ro(); ¬co(fp); co(fp); fp = ¬ro(); ¬co(fp); ¬co(fp);
¬Owned
Owned
Uninit
Claimed OK
Bug
¬co
¬ro
ro
co
co
end-of-path
anyuse
¬co
29
Goal
Provide a framework that:
1. Allows users to easily express every intuition and domain-specific observation they have that is useful for inferring annotations
2. Reduce such knowledge in a sound way to meaningful probabilities (“common currency”)
30
Annotation Inference
1. Define the set of possible annotations to infer
2. Model domain-specific knowledge and intuitions in the probabilistic model
3. Compute annotations probabilitis
31
Factors – Modeling Beliefs
Relations mapping the possible values of one or more annotations variables to non-negative real numbers
For example: CheckFactor
belief: any random place might have a bug 10% of times; set <bug>= 0.1, <ok>= 0.9
<ok> : if DFA = OK{ <bug> : if DFA = Bugf<check>=
32
Factors
Other factors bias toward specifications with ro without co based on naming conventions …
33
Annotation Factor Graph
fopen:ret fread:4 fclose:1 fdopen:ret fwrite:4
f<check> f<check>
f<ro> f<co> f<co> f<ro> f<co>
annotationvariables
prior beliefs
behavioral tests
34
Results
fopen:ret fread:4 fclose:1 DFA f<check> f<co>(fread:4) P(A)
ro ¬co co + 0.9 0.7 0.483
¬ro ¬co co + 0.9 0.7 0.282
ro ¬co co - 0.1 0.7 0.125
ro co ¬co - 0.1 0.3 0.054
ro co ¬co - 0.1 0.3 0.023
¬ro ¬co co - 0.1 0.7 0.013
¬ro co ¬co - 0.1 0.3 0.013
¬ro co ¬co - 0.1 0.3 0.006
35
QUARK: Empirical Assessment of Automaton-based Specification MinersDavid Lo and Siau-Cheng Khoo
Department of Computer Science, National University of Singapore
WCRE ‘06
36
QUARK Framework
Assessing the quality of specification miners Measure performance along multiple
dimensions Accuracy – extent of inferred specification being
representative of the actual specification Scalability – ability to infer large specifications Robustness – sensitivity to errors
37
QualityAssessment
QUARK Framework
Simulator Model (PFSA)
TraceGenerator
User Defined Miner
Measurements
38
Accuracy (Trace Similarity)
Absence of error Metrics:
Recall – correct information that can be recollected by the mined model
Precision – correct information that can be produced by the mined model
Co-emission – probability similarity
39
Robustness
Sensitivity to errors “Inject” error nodes and error transition to the
PFSA model
1 2 3 4
start
end
AB C
D
E
E F
H
error
Z Z
Z
Z
40
Scalability
Use synthetic models Build a tree from a pre-determined number of
nodes Add loops based on ‘locality of reference’ Assign equal probabilities to transition from the
same node Vary the size of the model (nodes,
transitions)