May 1, 2013

Metric Driven Formal Verification

Raik Brinkmann

OneSpin Solutions

May 1, 2013

OneSpin Solutions - Who we are…

OneSpin provides enduring solutions that

enable the most thorough & easiest to use logic verification available

OneSpin Solutions was spun out of Infineon in 2005

Technologies include Assertion-based

verification (ABV) and equivalence checking (EC)

Production-proven tools used in

thousands of designs

The company is a completely independent entity owned by

Azini Capital and OneSpin’s employees

The original technologies have been augmented & enhanced to provide the most thorough formal capabilities available

Engineering HQ based in Munich, Germany

Sales offices in US, France, Germany, Israel, Japan, and

Turkey

May 1, 2013

Formal Verification with OneSpin 360

• Unique push button observation coverage analysis for verification progress

• Faster assertion development in timing diagram style using transaction level assertions

• 100% functional coverage using unique gap free verification

• Unique structural assertion debugger and active value/driver tracing through RTL

• Incremental compilation of assertions for quick turn around

• Automatic clock and reset detection for easy design setup

Unique productivity features

• HDL-linting

• Structural assertion synthesis

• Coverage closure for simulation

• SoC connectivity verification

• Register verification

• Formal score boarding

• Protocol verification IP

• X-propagation verification

Large selection of push-button design verification solutions, included

• Verification of logic, sequential and power optimizations

• Supporting RTL and gate level for FPGA and ASIC

• Integrated with major vendors and many synthesis flows

Push-button equivalence checking

May 1, 2013

Outline Importance of Progress Metrics

Coverage Metrics

Control vs. Observation Coverage

Practical Observation Coverage

Combining Control and Observation Coverage

Examples

Summary

May 1, 2013

Why do we need a metric?

DUV

Tests / Scenarios Checkers

Bus Functional

Model (BFM)

Test #1

Test #2

Stimulus / Scenario Generation

Constraints Formal

Engine

Stimulus / Scenario Exclusion

Check #1

Check #2

Check #3

Check #4

Simulation

Formal

Verification Environment (simplified):

Verification Process:

• Plan: Verification planning

• Do: Write tests / properties and verify and fix DUV, tests, properties

• Check: Are we done, (still) making progress?

• Act: Adapt planning or get ready for tape-out

Progress Metric : Key Component of Verification Flow

Plan Do

Check Act

May 1, 2013

Standard Formal ABV

Progress of Formal Verification?

• Unclear how many assertions to write and where to put them

• Unclear verification progress

• Unclear how formal verification affects overall verification quality

Integration with Simulation?

• Weak integration into verification flow and sign-off

• Additional effort to existing simulation effort

• Formal used as “point tool” for verifying specific aspects (e.g. hard to test corner cases)

Low acceptance of formal ABV

We need a practical progress metric!

May 1, 2013

Progress Metric & Coverage Taxonomy

Coverage

Structural Functional

Progress Metric

Bug Rate

Code Circuit

• Line/Statement/Block • Expression/Branch/Condition • FSM

• Toggle • Latch

Scenarios

Implementation artefacts

Analytical

Bugs Fixed

100%

time

ok #

Assertion Complete

• Gap Free Analysis • Verification Plan

Anecdotal

Have I written enough tests / properties? How much has been verified? Where are the gaps in my verification? Are we done and ready for tape-out?

if (…)

…

if (…)

else …

May 1, 2013

Control / Simulation Coverage:

• Focused on quality of stimuli

• How to judge quality of checkers?

Observation & Control Coverage

coverage metrics

stimulus generation checkers/assertions

plan

DUV

report

How good are my test vectors & constraints?

How good are my checkers and assertions?

How much of my DUV is verified? When is my verification finished?

Observation Coverage:

• Focused on quality of checkers

• Exposes unverified DUV parts

May 1, 2013

Been there? Done what?

• Has the statement been reached?

• Idea:

– If a statement has not been reached during verification, it can’t break a check.

– If a statement has been reached, would some check fail?

• Can measure quality of stimuli.

case (state)

…

burst:

if (cancel_i)

done_o <= 1

…

active

case (state)

…

burst:

if (cancel_i)

done_o <= X

…

mutate

• Has the statement been verified?

• Idea:

If a statement is modified, some check should fail.

But would some check fail, if the statement cannot be reached?

• Can measure quality of checkers.

Been there! Done that!

Example: Statement Coverage

Coverage

Control Observation

Been there! Done that!

May 1, 2013

Control Coverage for Formal No explicit stimulus generation for formal

Formal considers all possible input stimuli by default (exhaustive verification)

However: Control coverage still an issue because

• Formal requires constraints excluding illegal environment behavior

• Constraints may accidentally be too strong and exclude vital functionality

• Constrained functionality can neither be controlled nor observed

(Structural) control coverage quantifies degree to which locations of a design have been activated during verification.

control

controllable

reached

unreached

unknown

uncontrollable

dead

constrained

May 1, 2013

(Formal) Observation Coverage Verification tries to answer the question:

• “Does a design satisfy a specification?”

(Observation) coverage tries to answer the question:

• “What causes a design to satisfy a specification?”

Causality:

• “When do we say A is a cause of B?”

Common approach: Counterfactual causality:

• “A is a cause of B if, had A not happened, then B would not have happened.”

• A code line is a cause of a design satisfying an assertion. Source: H. Chockler, IBM

(Structural) observation coverage quantifies degree to which locations of a design have been responsible to make checks pass.

observation

unknown

observable

observed

unobserved

unobservable

May 1, 2013

Practical Observation Coverage

Mutation coverage

• Particular changes depending on error model, usually static

• Replacing for example “a <= b;” with “a <= 1´b1;”

• Problems:

• Can lead to vacuously holding assertions

• Requires several modifications at each location, high run time

“Quantify MDV” metric

• Use abstraction by free variables, allow dynamic behavioral change

• Replacing for example “a <= b;” with “a <= free_b;”

• Advantages:

• No unintended vacuity

• One modification for each location leads to faster results

• Multiple locations can be checked at the same time to prove “unobserved”

May 1, 2013

always @(posedge clk or posedge reset)

if (reset)

z <= 1’b0;

else

begin

case (i)

3'b001: z <= a;

3'b010: z <= b;

3'b100: z <= c;

default: z <= <input>;

endcase

end

M5

always @(posedge clk or posedge reset)

if (reset)

z <= 1’b0;

else

begin

case (i)

3'b001: z <= a;

3'b010: z <= b;

3'b100: z <= <input>;

default: z <= 1'b1;

endcase

end

M4

always @(posedge clk or posedge reset)

if (reset)

z <= 1’b0;

else

begin

case (i)

3'b001: z <= a;

3'b010: z <= <input>;

3'b100: z <= c;

default: z <= 1'b1;

endcase

end

M3

Formal Observation Coverage Example module select1(onehot, a, b, c, z, clk, reset); input clk; input reset; input [2:0] i; input a; input b; input c; output reg z; always @(posedge clk or posedge reset) if (reset) z <= 1'b0; // L1: not covered (reset case) else begin case (i) 3'b001: z <= a; // L2: covered by assertion 3'b010: z <= b; // L3: not covered 3'b100: z <= c; // L4: not covered default: z <= 1'b1; // L5: not covered endcase end // if there is no reset, then 'a' is stored in 'z' if ‘i' is 3'b001 A: assert property ( @(posedge clk) disable iff (reset) i == 3'b001 |=> z == $past(a) ); endmodule

Q: Which assignment locations Lx in design M are observed by proven assertion A?

2. Re-Check property A for each M1..M5

always @(posedge clk or posedge reset)

if (reset)

z <= <input>;

else

begin

case (i)

3'b001: z <= a;

3'b010: z <= b;

3'b100: z <= c;

default: z <= 1'b1;

endcase

end

M1

always @(posedge clk or posedge reset)

if (reset)

z <= 1’b0;

else

begin

case (i)

3'b001: z <= <input>;

3'b010: z <= b;

3'b100: z <= c;

default: z <= 1'b1;

endcase

end

M2

Assertion A holds on M1: L1 not observed

Assertion A fails on M2: L2 is observed

M

A

L3 not observed

L4 not observed

L5 not observed

1. Modify each location L1..L5

of M: Producing M1..M5

A: The locations Lx for which A fails after replacing the assignment with a free input.

May 1, 2013

Merging Observation and Control Observation Coverage Control Coverage Merged Result

observed reached1) covered

unknown reached reached

unobserved reached unobserved

unknown unreached unreached

unobserved unreached uncovered

unobservable2) constrained constrained

unobservable2) dead dead

1. If a location is uncontrollable, it is also unobservable. 2. If a location is observed, it is also reached and thus controllable.

observation

unknown

observable

observed

unobserved

unobservable

control

controllable

reached

unreached

unknown

uncontrollable

dead

constrained 2.

1.

Been there! Done that!

May 1, 2013

Interfacing with UCIS

UCIS

• Coverage standard evolved from Mentor Graphics‘ UCDB

• Only supports control coverage, but not observation coverage

How to interface?

• Use user defined structures to store Quantify MDV results

• Use result merging to enhance standard UCIS coverage

Result Merging from UCIS to Quantify MDV:

• Covered bins -> Reached locations

Merging Quantify MDV to UCIS:

• Dead/Constrained -> Coverage Excludes

• Reached locations -> Covered bins

• Covered locations -> Covered bins

Conclusions on UCIS interfacing

• Merging results with UCIS is possible and useful now

• Direct support for observation coverage by UCIS is desirable

May 1, 2013

Simple Quantify MDV Example • Mixed control and observation

coverage results indicate:

– holes indicating missing assertions

– constrained code indicating over-constraining verification

hole

verified code

constrained code

dead code

Management overview

May 1, 2013

Industry Examples

Design LOCs #Assertions #Locations Runtime

FIFO 321 30 21 100s

FSM-DDR2-Read 839 6 93 106s

vCore-Processor 295 8 86 204s

SQRT 383 2 35 257s

IFX-Aurix-1*) 25563 85 2316 4d

IFX-Aurix-2 27374 157 1993 5d

IFX-Aurix-3 57253 253 5309 7d**)

*) “Quantification of Formal Properties for Productive Automotive Microcontroller Verification”, Holger Busch, DVCon, San Jose - February 26, 2013 **) interrupted at 80% completion

May 1, 2013

Summary

Coverage is an important aspect of verification

New metric introduced: “Quantify MDV”

• Combines observation and control coverage

• Agnostic to assertion style, assertion language, design style and design language

• Scales to relevant big industry designs

• Useful for

• Quick feedback on formal verification quality, as well as

• Criterion for (formal) verification closure

• Can be combined with simulation based verification and metrics

May 1, 2013

Thank You!

May 1, 2013

What about redundancy?

Redundant code cannot be observed!

• Redundant code will be marked uncovered if not treated specially.

What are sources of redundancy?

• Redundant logic not driving any outputs

• Safety logic meant to implement fail safe devices

How to mitigate?

• Identify redundant code and mark it

• Remove redundant code before running coverage

• Selectively activate/de-activate redundant parts

May 1, 2013

Functional vs. Structural Coverage Type Functional Structural

Control How much of the specified behavior has been activated?

How much of the DUV implementation has been activated?

Observation How much of the specified behavior has been verified?

How much of the DUV implementation has been verified?

Functional vs. Structural

• Functional - relates to specification

– Measures completeness of requirements verification

– Can identify gaps in verification plan

• Structural - relates to DUV

– Measures completeness of design intent verification

– Can identify unverified parts of DUV

Control vs. Observation

• Control - relates to activation

– Measures quality of stimuli

– Can identify unreachable code / over-constraining

• Observation - relates to causality

– Measures quality of checkers

– Can identify verification gaps

• Control coverage doesn’t imply any observation coverage.

• Observation coverage doesn’t imply any functional coverage.

• 100% observation coverage implies 100% control coverage.

• 100% Functional coverage + all assertions proven implies 100% observation coverage (if DUV contains no dead code, no constrained code, no redundant code)

May 1, 2013

Verification Environment with Exhaustive Coverage Analysis

DUV

Tests / Scenarios Checkers

Spec

Verification Environment

Verification

Plan

Coverage

Analysis

Control

Coverage

Observation

Coverage

Functional

Coverage

Have I written

enough stimuli?

Which parts of

my DUV have

been exercised?

Which parts of my

DUV have been

checked?

Did I write

enough checks?

Are all specified

functions

implemented?

Are all specified

functions verified?

Functional

Structural

May 1, 2013

What does block level coverage mean in the larger context?

System-level design

Block-level design (RTL)

Architecture-level design

Design start Verification closure

System-level simulation

Block-level simulation

Sub-system-level simulation

Formal ABV with Covage Analysis

UCDB

Aggregation of module / block level coverage results with higher level context

High-Level/RTL Design & Verification Flow