Software Verification and Synthesis using Constraints and ...
Quantitative Verification and Synthesis of Systems · Quantitative Verification and Synthesis of...
26
Quantitative Verification and Synthesis of Systems Quantitative Verification and Synthesis of Systems Sanjit A. Seshia Sanjit A. Seshia Assistant Professor Assistant Professor EECS, UC Berkeley EECS, UC Berkeley Software-at-Scale Workshop August 2010
Transcript of Quantitative Verification and Synthesis of Systems · Quantitative Verification and Synthesis of...
Quantitative Verification and Synthesis of Systems
Quantitative Verification and Synthesis of Systems
Sanjit A. SeshiaSanjit A. Seshia
Assistant ProfessorAssistant ProfessorEECS, UC BerkeleyEECS, UC Berkeley
Software-at-Scale WorkshopAugust 2010
– 2 –S. A. Seshia
Quantitative Analysis / VerificationQuantitative Analysis / Verification
Does the brake-by-wire software always actuate the brakes within 1 ms?Safety-critical embedded systems
Can this new app drain my iPhone battery in an hour?Consumer devices
How much energy must the sensor node harvest for RSA encryption?Energy-limited sensor nets, bio-medical apps, etc.
– 3 –S. A. Seshia
OutlineOutline
Boolean vs. Quantitative VerificationBoolean vs. Quantitative Verification
An Example: Timing AnalysisAn Example: Timing Analysis
Challenges: Environment ModelingChallenges: Environment Modeling
Approach: Learning ProgramApproach: Learning Program--Specific Specific Environment ModelEnvironment Model
Conclusion and QuestionsConclusion and Questions
– 4 –S. A. Seshia
Traditional “Boolean” VerificationTraditional “Boolean” Verification
SystemS
EnvironmentE
Specification
VERIFIERDoes S || E ²
?
YES / NO
– 5 –S. A. Seshia
Boolean Verification on ModelsBoolean Verification on Models
SystemS
EnvironmentE
Specification
VERIFIERDoes S || E ²
?
YES / NO
Finite Automata Logical Formula
SatisfiabilityProblem
– 6 –S. A. Seshia
Boolean Verification on SoftwareBoolean Verification on Software
SystemS
EnvironmentE
Specification
VERIFIERDoes S || E ²
?
YES / NO
Program MODEL
Env model typically over-approximate
Logical Formula
e.g. Finite Automata
– 7 –S. A. Seshia
Success of Boolean Software VerificationSuccess of Boolean Software Verification
From theoretical ideas to industrial practice in From theoretical ideas to industrial practice in ~ 15 yrs~ 15 yrs
Some Reasons:Some Reasons:
Availability of open source softwareAvailability of open source software
WellWell--defined problems: Device drivers, memory defined problems: Device drivers, memory safety, security vulnerabilities, concurrency, safety, security vulnerabilities, concurrency, ……
Value of bug findingValue of bug finding
Less sensitive to imprecise environment modelLess sensitive to imprecise environment model
– 8 –S. A. Seshia
Quantitative VerificationQuantitative Verification
SystemS
EnvironmentE
Specification
VERIFIER
YES / NO
Quantitative Quantitative Quantitative
R
At least oneof these holds
– 9 –S. A. Seshia
Quantitative Verification on ModelsQuantitative Verification on Models
SystemS
EnvironmentE
Specification
VERIFIER
Timed Automata
R
G(request F·5 response)
Worst-case latency request response
– 10 –S. A. Seshia
Quantitative Verification on SoftwareQuantitative Verification on Software
SystemS
EnvironmentE
Specification
VERIFIER
Program
R
G(request F·5 response)
Worst-case latency request response
MODEL?
Timed Automaton
?
– 11 –S. A. Seshia
Challenge: Environment Modeling (Quantity = Time) Challenge: Environment Modeling (Quantity = Time)
Timing properties of the Program depend heavily Timing properties of the Program depend heavily on its environmenton its environment
Environment =Environment =Processor & Memory HierarchyProcessor & Memory Hierarchy
++ Operating System, other processes/threads, Operating System, other processes/threads, ……++ NetworkNetwork++ I/O DevicesI/O Devices++ ……
Cannot construct a Program model independent Cannot construct a Program model independent of its environment!of its environment!–– Unlike Boolean version of the Verification problemUnlike Boolean version of the Verification problem
– 12 –S. A. Seshia
Simplifying the ProblemSimplifying the Problem
Program = Sequential, Program = Sequential, terminating programterminating program
Runs uninterruptedRuns uninterrupted
Environment = Environment = SingleSingle--core Processor + core Processor + Memory HierarchyMemory Hierarchy
– 13 –S. A. Seshia
Simple Illustrative ProgramSimple Illustrative Program
Control-flow graph(loop bound = 1)
flag==1
flag=1; *x++;
*x += 2;
flag==1
flag==1
flag=1; *x++;
CFG unrolled to a DAG
*x += 2;
while(!flag) {
flag = 1;*x++;
}*x += 2;
– 14 –S. A. Seshia
Simple Illustrative ProgramSimple Illustrative Program
flag==1
flag==1
flag=1; *x++;
CFG unrolled to a DAG
*x += 2;
On a processor with a data cache
x
– 15 –S. A. Seshia
Simple Illustrative ProgramSimple Illustrative Program
flag==1
flag==1
flag=1; *x++;
CFG unrolled to a DAG
*x += 2;
On a processor with a data cache
x
Case 1:x is originally in cache
x
x x
– 16 –S. A. Seshia
Simple Illustrative ProgramSimple Illustrative Program
flag==1
flag==1
flag=1; *x++;
CFG unrolled to a DAG
*x += 2;
On a processor with a data cache
Case 2:x is NOT originally in cache
cache miss
x
x
cache miss
– 17 –S. A. Seshia
Summary of ExampleSummary of Example
Edge timing depends onEdge timing depends on1.1. Initial Initial environment stateenvironment state2.2. Program pathProgram path executedexecuted
Challenge: Challenge: –– ExponentiallyExponentially--many many envenv states!states!–– ExponentiallyExponentially--many program paths!many program paths!
E.g. Finding WorstE.g. Finding Worst--case execution time boundcase execution time bound–– Very loose bound somewhat easy Very loose bound somewhat easy
(need to consider timing anomalies)(need to consider timing anomalies)–– Reasonably tight bound Reasonably tight bound
VERY HARDVERY HARD
– 18 –S. A. Seshia
Current State-of-the-art for Timing Analysis Current State-of-the-art for Timing Analysis
Program = Sequential, Program = Sequential, terminating programterminating program
Runs uninterruptedRuns uninterrupted
Environment = Environment = SingleSingle--core Processor + core Processor + Memory HierarchyMemory Hierarchy
Timing Model
PROBLEM:Takes several man-
months to construct!Also: limited to
extreme-case analysis
– 19 –S. A. Seshia
One-size-fits-all Solution?One-size-fits-all Solution?
Why aim to Why aim to construct a SINGLE construct a SINGLE timing model for timing model for ALL programs?ALL programs?
We are only We are only interested in interested in verifying a specific verifying a specific program!program!
Why not construct a Why not construct a programprogram--specificspecific timing model?timing model?–– Even done in Even done in
Boolean verificationBoolean verification
– 20 –S. A. Seshia
The GameTime ApproachThe GameTime Approach
Environment is an Environment is an ‘‘adversaryadversary’’ who picks weights who picks weights on the edges of the controlon the edges of the control--flow graphflow graph
Weights selected in twoWeights selected in two--phase approachphase approach– w ∈
Rm: : pathpath--independentindependent weightsweights
––
∈
Rm: : pathpath--dependentdependent weightsweights
GameTimeGameTime operation operation –– Runs systematically generated tests Runs systematically generated tests –– Automatically learns Automatically learns programprogram--specific specific
environment modelenvironment model from measurementsfrom measurements–– Predicts execution times (worstPredicts execution times (worst--case, distribution, case, distribution,
etc.)etc.)–– Theoretical guarantees rely on statistical Theoretical guarantees rely on statistical
assumptions about environmentassumptions about environment
[Seshia & Rakhlin, ICCAD’08, ACM TECS’10]
(m = #edges)
– 21 –S. A. Seshia
Estimating Distribution of Execution Times for a Program (using GameTime) Estimating Distribution of Execution Times for a Program (using GameTime)
For StrongARMprocessor
– 22 –S. A. Seshia
Timing Predictability/RepeatabilityTiming Predictability/Repeatability
Verification made easier if timing behavior of Verification made easier if timing behavior of environment is (more) predictableenvironment is (more) predictable
How do we formalize this?How do we formalize this?
– 23 –S. A. Seshia
Formalizing Timing PredictabilityFormalizing Timing Predictability
An Attempt in the An Attempt in the w+w+
model:model:1. 1. Path predictabilityPath predictability
Fix initial Fix initial envenv statestateTiming of an edge (basic block) Timing of an edge (basic block) varies little as a function of the varies little as a function of the path it lies onpath it lies on
2. 2. EnvEnv state predictabilitystate predictabilityFix program pathFix program pathTiming of an edge (or path) Timing of an edge (or path) varies little as a function of varies little as a function of initial platform stateinitial platform state
X
– 24 –S. A. Seshia
Expected Payoff / MetricsExpected Payoff / Metrics
Software increasingly integrated with the Software increasingly integrated with the physical world physical world –– Physical properties will be importantPhysical properties will be important–– Needs quantitative verification Needs quantitative verification
Metrics: How to assess progress?Metrics: How to assess progress?–– Lines of code? Lines of code? –– Complexity of properties?Complexity of properties?–– Complexity of hardware?Complexity of hardware?–– Type of software: Type of software:
sequential sequential
multithreaded multithreaded
distributeddistributed
– 25 –S. A. Seshia
Risk FactorsRisk Factors
Lack of OpenLack of Open--Source BenchmarksSource Benchmarks–– Progress in Boolean software verification was Progress in Boolean software verification was
driven by wide availability of opendriven by wide availability of open--source source softwaresoftware
–– More challenging for quantitative verification!More challenging for quantitative verification!–– Heavy dependence on platform makes it more Heavy dependence on platform makes it more
challenging to experiment on same problemschallenging to experiment on same problems
Hardware + Software SkillsHardware + Software Skills–– Students need crossStudents need cross--cutting skills (or willingness cutting skills (or willingness
to learn) to work in this area to learn) to work in this area
– 26 –S. A. Seshia
Questions for DiscussionQuestions for Discussion
1.1. Relevance beyond embedded:Relevance beyond embedded: What proportion of What proportion of general software problems arise from unexpected general software problems arise from unexpected timing effects?timing effects?
2.2. Timing predictability contract:Timing predictability contract: What does it mean for a What does it mean for a hardware platform to have predictable timing? hardware platform to have predictable timing?
3.3. Compositionality: Compositionality: How to do compositional timing How to do compositional timing (quantitative) analysis?(quantitative) analysis?
4.4. Beyond timing:Beyond timing: What other quantitative properties are What other quantitative properties are relevant for software? Energy consumption? relevant for software? Energy consumption? Reliability?Reliability?
5.5. Formalisms:Formalisms: What modeling formalisms and/or What modeling formalisms and/or techniques are best suited to quantitative verification? techniques are best suited to quantitative verification? E.g. weighted automata?E.g. weighted automata?
6.6. Synthesis:Synthesis: Program synthesis for quantitative Program synthesis for quantitative requirements? Challenge problems?requirements? Challenge problems?
