Post on 28-Dec-2015
PINCETTE project:Validation of changes and upgrades
in large software systemsUnique challenges and suggested solutions
Hana Chockler
2
Motivation:Challenges of validation of evolving software Large software systems are usually built incrementally:
Maintenance (fixing errors and flaws, hardware changes, etc.) Enhancements (new functionality, improved efficiency,
extension, new regulations, etc.) Changes are done frequently during the lifetime of most systems
and can introduce new software errors or expose old errors Upgrades are done gradually, so the old and new versions have to
co-exist in the same system Changes often require re-certification of the system, especially
for mission-critical systems
3
"Upgrading a networked system is similar to upgrading software of a car while the car's engine is running, and the car is moving on a highway. Unfortunately, in networked systems we don't have the option of shutting the whole system down while we upgrade and verify a part of it.“
source:
software update
4
PINCETTE Project – Validating Changes and Upgradesin Networked Software
Checking for crushes
Static Analysis Component
Black box testing
Dynamic AnalysisComponent
Front end
Methodologybook
Using function
summaries
Verifying only the change
White box testing
5
What does it mean to validate a change in a software system?
• Equivalence checking – when the new version should be equivalent to the previous version in terms of functionality• Changes in the underlying hardware• Optimizations
• No crashes – when several versions need to co-exist in the same system, and we need to ensure that the update will not crash the system• When there is no correctness specification, this is often the
only thing we can check• Checking that a specific bug was fixed
• A counterexample trace can be viewed as a specification of a behavior that needs to be eliminated in the new version
• Validation of the new functionality• If a correctness specification for the change exists, we can
check whether the new (or changed) behaviors satisfy this specification
6
Why is it validation of evolving software different from standard software validation?
• Software systems are too large to be formally verified or exhaustively tested at once
• Even if it is feasible to validate the whole system, often the process is too long and expensive and does not fit into the schedule of small frequent changes
• When validating the whole system, there is a danger of overlooking the change
How can we use the fact that we are validating evolving software?
• If the previous version was validated in some way, we can assume that it is correct and not re-validate the parts that were not changed
• If the results of previous validation exist, we can use them as a basis for the current validation – especially useful when there are many versions that differ from each other only slightly
• The previous version can be used as a specification
7
After more than two years of PINCETTE:
1. How to verify an evolving system that does not have assertions?
2. How to verify only the change without relying on results of previous verification?
8
1. How to verify an evolving system that does not have assertions?
9
Dynamically Discovering Assertionsto Support Formal Verification
Motivation: •“Gray-box” components (such as OTS components) – poor specifications, partial view of internal details•Lack of specification complicates validation and debugging•Lack of description of the correct behavior complicates integrationIdea: Analyze gray-box components by dynamic analysis techniques:•Monitor system executions by observing interactions at the component interface level and inside components• Derive models of the expected behavior from the observed events• Mark the model violations as symptoms of faults
©Leonardo Mariani, UniMiB
10
Dynamically Discovering Assertions at BCT (UniMiB tool)
• Combining dynamic analysis and model-based monitoring• Combining classic dynamic analysis techniques (Daikon) with
incremental finite state generation techniques (kBehavior) to produce I/O models and interaction models• FSA are produced and refined based on subsequent executions
• Extracting information about likely causes of failures by automatically relating the detected anomalies
• Filtering false positives in two steps:• Identify and eliminate false positives by comparing failing and
successful executions with heuristics already experienced in other contexts
• Rank the remaining anomalies according to their mutual correlation and use this information to push the related likely false positives far from the top anomalies
©Leonardo Mariani, UniMiB
11
Extracting a set of valid assertions for regression
Program V.0 Program V.1
Testing
set of traces
Analysis
Set of candidateassertions
Formalmethods
Set of assertionsthat hold in V.0
Set of invalidassertions in V.1
Formalmethods
Set of assertionsthat hold in V.0
+
Efficient and automatic regression verification
user
UniMiB+Oxford+USI
12
Parameters of the assertions discovery
• Scope of monitoring• All lines• Function invocations• Loops• Particular variables
• Output refinement• Many violations – maybe leave only those in the slice of
the change• Look for a minimal set
• How to design the search algorithm? • Can we check properties of the type “if P_1, P_2,…P_n
are true then Q_1, Q_2,..Q_m” where each property may hold at a different code location?
13
1. how2. How to verify only the change without
relying on results of previous verification?
The main idea: Verify only the changeignoring the rest of the program
Using ExpliSAT
Joint work with Sitvanit Ruah
14
What is ExpliSAT?
• A model-checker that combines explicit traversal of CFG with symbolic representation of data:• Traverses CFG• Ensure that we traverse only real paths by computing a
representative path with real data values• Builds a SAT formula for each path on the control flow graph and
invokes a SAT solver• Bugs-hunting heuristics can be added to change the order of
traversal of the graph, so that the places where bugs are more likely to occur are traversed first
The concolic approach is
widely used in testing
concrete+symbolic
Heuristics direct the search towards areas in which bugs are more likely to occur
15
The path representative – checking that the symbolic path is real
• ExpliSAT traverse the CFG and create all possible control paths
• Symbolically represent data variables of all executions that follow the same control path
• A path representative for a certain control path is a valuation of every variable in the control path that holds the path constraint.
• For every legal control path there exists a path representative
• The representative is computed iteratively
AUpdate the representative until A and look at command in A
Use the representative prefix until A to decide which branch to take
a control path with a representative is a legal control path
The other branch may be
infeasible – checked by
trying to construct a
representative
16
Ordering - Heuristically deciding in which order to traverse the control paths
11: x=input();
2: if (x>N) then
else
3: y=x;
4: y=N;
1: x=input();
2: if (x>N) then
else4: y=N;
5: assert(y>=N);
2
43
5
1
2
4
55: assert(y>=N);
The particular heuristic depends on the verification goals – for updates, we use
the update-checking heuristic
17
Extracting the part of the control-flow graph affected by a change
change
subgraph induced by the changed node
initial state
Backward reachability of the initial state from the changed node
18
ExpliSAT with update-checking
ExpliSAT with
update checking
goto-cc SAT solversyntactic
marking of changes
Oldversion
------------------
Newversion
------------------
only the new version with marking is
passed
Checks paths that go through a
changed node first
result: goto-program with the change
markingParser + translationto a goto-program
Cprover framework
19
Update-checking heuristics in ExpliSAT
Turns the model-checker to a bug
hunter
Two options:
Check only the change Check the change first, then continue checking the whole program
Very efficient New bugs will
be discovered
quickly
The whole process is complete
20
Effect of a change on functionality of the program
• Claim: in sequential programs, the algorithm from the previous slide finds all paths affected by a change
straightforward
• Observation: in concurrent programs, a change can also affect other paths – future work
Changes in global
variables
21
Experimental results
• Compared performance of ExpliSAT on the whole program with the performance of ExpliSAT on the change only
Not surprisingly, ExpliSAT on
changes only is much faster
• Real-life example: a C++ program supplied by VTT Technical Research Center of Finland, computing the velocity and acceleration of a robot used in the European ITER project – a new type of a reactor based on energy of nuclear fusion
ExpliSAT found a bug in update in several seconds
On the whole program ExpliSAT doesn’t terminate
22
VTT (Finland) - the ITER EU Project: the thermo-nuclear reactor
ITER is an international magnetic confinement fusion experimental project participated by the EU, India, Japan, China, Russia, South Korea, and the USA. The expected budget of ITER is €5B with 50% of funding provided by the EU. The reactor will be constructed in Cadarache, France, around 2016. ITER is one of the most important and complex research projects in the history of the EU.
divertor:goes at the bottom of the reactor;
should be changed periodicallyRobot that changes
the divertor cassette
23