16th A i P ifi D i A t ti C f

Analog Circuit Verification by 16th Asian Pacific Design Automation Conference

g yStatistical Model Checking

Author Ying-Chih Wang (Presenter)Anvesh Komuravelli Paolo ZulianiEdmund M. Clarke

Date 2011/1/26

Thi k i d b th Gi S l S t R h C tThis work is sponsored by the GigaScale Systems Research Center, Semiconductor Research Corporation, National Science Foundation, General Motors, Air Force and the Office of Naval Research

OverviewIntroductionBackgroundBackground

AssumptionsBayesian StatisticsBayesian Statistics

AlgorithmApplication to Analog CircuitsApplication to Analog Circuits

Bounded Linear Temporal LogicExperimental ResultsExperimental ResultsDiscussionConclusion and Future WorkConclusion and Future Work

2

Introduction – MotivationAnalog circuits’ behaviors vary dramatically with changes in prametersg p

MOSFET parameters, temperature, etc.Performance, correctness

Circuits’ resiliency under uncertaintiesyProcess variation, noise, etc.

How to deal with uncertainties?

3

Introduction – Problem“Does the design satisfy the property with probability p ≥ threshold θ?”probability p ≥ threshold θ?

Monte Carlo: estimate the probability pHow good is the estimate?g

Property : Bounded Linear Temporal Logic (BLTL)Statistical Model Checking (SMC)

Recently applied to Stateflow/Simulink models[1]

Probabilistic GuaranteesBayesian Hypothesis Testing Bayesian Estimation

4

[1] P. Zuliani, A. Platzer and E. M. Clarke, “Bayesian Statistical Model Checking with Application to Stateflow/Simulink Verification,” in HSCC 2010.

Background – Assumptions

Monte Carlo SimulationDraw independent and identically distributed (i i d )Draw independent and identically distributed (i.i.d.) sample tracesCharacterize each simulation trace as Pass/Fail

Property ФBiased coin

+ =Stochastic system M

Bernoulli Random Variable Y P(Y=1) = pM ⊨ Ф with probability p

5

( ) pP(Y=0) = 1-p

M ⊨ Ф with probability p

Background – Bayesian Statistics

1.5

2

2.5

Perform Experiments Sequentially

0.5

1

p q y

Experimental Results{Pass Fail Pass Pass }

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

Prior density of p(Beta Distribution)

{Pass,Fail,Pass,Pass,…}

Bayesian

30

35

40

B F t

Bayesian EstimationBayesian

Hypothesis Testing

10

15

20

25 Bayes Factor

60 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

5

Posterior density of p

Overview –Analog Circuit Verification

Analog Ci it

SPICECircuit

Si l

Circuit Descriptions

P

Execution Traces σSimulator

BLTLBLTL M ⊨ P ( )

Process Variation

Descriptions

Traces σ

Bayes Factor > T

Bayes Test

BLTL Model

Checker

BLTLFormula

φ

M ⊨ P≥θ(φ)

M ⊨ P<θ(φ)B F t < 1/TBayes Factor < 1/T

7

Background – Bayesian StatisticsNull hypothesis H0(p≥θ)Alternative hypothesis H1(p<θ)yp 1(p )

P(H0), P(H1) sum to 1X: experimental resultspBayes Factor

Jefferys’ Test [1960s]Fixed Sample SizepBayes Factor > 100 : Strongly supports H0

Sequential Version (Fixed Bayes Factor Threshold)

8

Statistical Model Checking – AlgorithmRequire: Property P≥θ(Φ), Threshold T ≥ 1, Prior density gn := 0 {number of traces drawn so far}

0 { b f t ti f i Φ f }x := 0 {number of traces satisfying Φ so far}repeat

σ := draw a sample trace of the system (iid)n := n + 1n := n + 1if σ Φ then

x := x + 1endifendifB := BayesFactor(n, x, θ, g)

until (B > T v B < 1/T )if (B > T ) then Theorem (Error bounds) When theif (B > T ) then

return “H0 accepted”else

return “H0 rejected”

Theorem (Error bounds). When the Bayesian algorithm – using threshold T – stops, the following holds:return H0 rejected

endif9

Prob (“accept H0” | H1) ≤ 1/TProb (“reject H0” | H0) ≤ 1/T

Bounded Linear Temporal Logic

States: State VariablesAt i P iti (AP)Atomic Propositions (AP): e1 ~ e2

e1, e2: arithmetic expressions over state variables, ~ ∈ {<,≤,>,≥,=} ∈ { , , , , }

SyntaxLet σ = (s0,t0), (s1,t1), . . . be a simulation trace of

1 2:: | | | |t tAP F Gφ φ φ φ φ φ= ∨ ¬

the modelstays in si for duration ti.

σk: suffix trace of σ starting at state k

… …s0 s1 skt0 t1 tk

σ : suffix trace of σ starting at state kσk = (sk,tk), (sk+1,tk+1),…

10

Bounded Linear Temporal LogicThe semantics of BLTL for suffix trace σk

(trace starting at state k):( g )σk ⊨ iff atomic proposition AP is true in state sk

σk ⊨ iff σk ⊨ Ф1 or σk ⊨Φ21 2φ φ∨AP

σk ⊨ iff σk ⊨ Φ does not holdσk ⊨ iff exists i ≥0 s.t. and σk+i ⊨ Φ

φ¬tF φ

1

0

ik ll

t t−+=

≤∑Ф

… …sk sk+itk tk+1 tk+i

Ф

time bound tF[10ns] (Vout>1)Within 10ns, Vout should eventually be larger than 1

time bound t

Within 10ns, Vout should eventually be larger than 1 volt.

11

Bounded Linear Temporal LogicThe semantics of BLTL for suffix trace σk

(trace starting at state k):1i−∑σk ⊨ iff for all i ≥0 s.t. , σk+i ⊨ ΦtG φ1

0

ik ll

t t−+=

≤∑… …sk sk+it t t

Ф ФФФ

G[10ns] (Vout>1)

k k+itk tk+1 tk+i

time bound tG[10ns] (Vout>1)In the next 10ns, Vout should always be greater than 1 volt.

Nesting OperatorsNesting OperatorsF[100ns]G[10ns] (Vout>1)Within 100ns, Vout will be greater than 1 volt and stay greater than 1 volt for at least 10 ns.

12

An Example : OP amp

An Example : OP amp – Specifications

SpecificationsSpecifications1 Input Offset Voltage < 1 mV

2 Output Swing Range 0 2 V to 1 V2 Output Swing Range 0.2 V to 1 V

3 Slew Rate > 25 V/μSec

4 Open-Loop Voltage > 8000 V/VGain

5 Loop-Gain Unit-gain Frequency

> 10 MHzq y

6 Phase Margin > 60°

14

Example : OP amp – Specifications

Specifications : Quantities computed from simulation tracessimulation traces

in most cases, can be translated to BLTL directly from definitionsde t o s

e.g. Swing Range: Max(Vout) > 1V and Min(Vout) < 0.2V( ) ( )F[100μs](Vout < 0.2) ∧ F[100μs](Vout > 1)“Within entire trace, Vout will eventually be greater than 1V and smaller than 0.2V”100μs : the end time of transient simulation

15

Example : OP amp – Specifications

Frequency-domain properties?Transient response σ = (s t ) (s t ) (s t )Transient response σ = (s0,t0), (s1,t1),(s2,t2),…Frequency response σ = (s0,f0), (s1,f1),(s2,f2),…

Substitute frequency-stamps for time-stampsSubstitute frequency stamps for time stampsSpecify frequency domain properties in BLTLCheck with the same BLTL checkerSemantics Changed

G[1GHz]φ :φ holds for all frequencies from current to 1GHz largerF[1GHz]φ :φ holds for some frequency from current to 1GHz larger1GHz larger

16

An Example : OP amp – Specifications Specifications ( Transient ) BLTL Specifications1 Input Offset Voltage < 1 mV F[100μs](Vout = 0.6) ∧

G[100μs]((Vout = 0.6) → (|Vin+ − Vin | <G ((Vout 0.6) (|Vin+ Vin−| 0.001))

2 Output Swing Range 0.2 V to 1 V F[100μs](Vout < 0.2) ∧ F[100μs](Vout > 1)3 Slew Rate > 25 V/μSec3 Slew Rate > 25 V/μSec

G[100μs]( ((Vout = 1 ∧ Vin > 0.65) → F[0.008μs](Vout = 0.8)) ∧((Vout = 0.2 ∧ Vin < 0.55) → F[0.008μs](Vout = 0.4)) )

Specifications ( Frequency-domain ) BLTL Specifications4 Open-Loop Voltage Gain > 8000 V/V G[1KHz](V magout > 8000)5 Loop Gain Unit gain > 10 MHz G[10MHz](V mag > 1)5 Loop-Gain Unit-gain

Frequency> 10 MHz G[10MHz](V magout > 1)

6 Phase Margin > 60° F[10GHz](V magout = 1) ∧G[10GHz]( (V mag = 1) →

17

G[10GHz]( (V magout = 1) →(Vphaseout >60°) )

Experimental SetupPlatform: Linux virtual machine running on a 2.26GHz i3-350M, 4GB RAM computerWe model-check the BLTL formulae on previous slides

e.g. (Swing Range) H0:M ⊨ P≥θ[F[100μs](Vout < 0.2) ∧ F[100μs](Vout > 1)]

18

Experimental ResultsTarget yield: 0.95Monte Carlo analysis: y

do not satisfy /satisfy yield thresholdMonte Carlo (1000 samples) — Measured ValueSpecifications Mean Stddev Yield1 Input Offset Voltage (V) .436 .597 .8262 Swing Range Min (V) 104 006 1 002 Swing Range Min (V) .104 .006 1.00

Swing Range Max (V) 1.08 .005 1.003 Negative Slew Rate (V/μSec) -40.2 1.17 1.00

Positive Slew Rate (V/μSec) 56.4 2.54 1.00 4 Open-Loop Voltage Gain (V/V) 8768. 448 .975 5 Loop-Gain UGF (MHz) 19 9 30 1 00

19

5 Loop Gain UGF (MHz) 19.9 .30 1.006 Phase Margin (°) 64.1 .44 1.00

Experimental ResultsStatistical Model Checking

Null Hypothesis (H0): From Specs,e.g. H0:M ⊨ P≥θ[F[100μs](Vout < 0.2) ∧ F[100μs](Vout > 1)]

Probability Threshold θ: Set to Target Yield

Bayes Factor Threshold T: Test Strength Needed

T = 1000 : Probability of error is less than 0.001y

Prior Distribution of p:

Uniform : No ideaUniform : No idea

20

Experimental ResultsMonte Carlo : do not satisfy /satisfy yield threshold 0.95SMC testing result : Prior: Uniform, T = 1000, θ = 0.95

j t / t ll h th ireject / accept null hypothesisMonte Carlo (1000 samples) — Measured Value SMC(θ = 0.95 )Specifications Mean Stddev Yield Samples/RuntimeSpecifications Mean Stddev Yield Samples/Runtime1 Input Offset Voltage (V) .436 .597 .826 31/39s2 Swing Range Min (V) .104 .006 1.00 77/98s

Swing Range Max (V) 1.08 .005 1.00 77/98s3 Negative Slew Rate (V/μSec) -40.2 1.17 1.00 77/98s

Positive Slew Rate (V/μSec) 56.4 2.54 1.00 77/98s4 Open-Loop Voltage Gain (V/V) 8768. 448 .975 239/303s5 Loop-Gain UGF (MHz) 19 9 30 1 00 77/98s

21

5 Loop Gain UGF (MHz) 19.9 .30 1.00 77/98s6 Phase Margin (°) 64.1 .44 1.00 77/98s

Experimental ResultsPrior: uniform, T = 1000, θ in [0.7, 0.999]

SMC testing result : reject / accept null hypothesisSpec Null Hypothesis

1 M╞═ P≥θ[F[100μs](Vout = 0.6) ∧ G[100μs]((Vout = 0.6) → (|Vin+ − Vin−|<0.001))]2 M╞ P [F[100μs](V < 0 2) F[100μs](V > 1)]2 M╞═ P≥θ[F[100μs](Vout < 0.2) ∧ F[100μs](Vout > 1)]3 M╞═ P≥θ[G[100μs]( ((Vout = 1 ∧ Vin > 0.65) → F[0.008μs](Vout < 0.8)) ∧

((Vout < 0.2 ∧ Vin < 0.55) → F[0.008μs](Vout < 0.4)) )]Samples/Runtime

Probability threshold θSpec 0 7 0 8 0 9 0 99 0 999Spec 0.7 0.8 0.9 0.99 0.999

1 77/105s 9933/12161s 201/275s 10/13s 7/9s2 16/18s 24/27s 44/51s 239/280s 693/813s

22

3 16/23s 24/31s 44/57s 239/316s 693/916s

Experimental ResultsPrior: uniform, T = 1000, θ in [0.7, 0.999]

SMC testing result : reject / accept null hypothesisSpec Null Hypothesis

4 M╞═ P≥θ[G[1KHz](Vmagout > 8000)]5 M╞ P [G[10MHz](Vmag > 1)]5 M╞═ P≥θ[G[10MHz](Vmagout > 1)]6 M╞═ P≥θ[F[10GHz](Vmagout = 1)∧G[10GHz]((Vmagout=1)→ (Vphaseout>60°))]

Samples/RuntimeProbability threshold θ

Spec 0.7 0.8 0.9 0.99 0.9994 23/26 43/49 98/114 1103/1309 50/574 23/26s 43/49s 98/114s 1103/1309s 50/57s5 16/18s 24/28s 44/51s 239/279s 693/807s6 16/20s 24/30s 44/55s 239/303s 693/882s

23

DiscussionStatistical Model Checking is faster when

The threshold probability θ is away from theThe threshold probability θ is away from the unknown probability p,

e.g. p ≈ 1 θ 0.7 0.8 0.9 0.99 0.999

Easily integrated in the validation flowSamples 16 24 44 239 693

Relies only on Monte Carlo sampling and SPICEAdd-ons: Online BLTL monitoring, Bayes factor calculationcalculation

Low computation overhead

R ntime dominated b SPICE sim lationRuntime dominated by SPICE simulation

24

ConclusionIntroduced SMC to analog circuit verification

Avoid Monte Carlo simulation drawing unnecessary g ysamples when evidence is enough.

Demonstrated the feasibility of using BLTL specifications for a simple analog circuit

BLTL can specify complex interactions between signalsFuture Works

More experimentsl l ith li t dlarger examples with more complicated specifications

Introducing the technique of importance samplingIntroducing the technique of importance sampling

25

Thank you.

26

Bayesian Statistical Model Checking Suppose satisfies with (unknown) probability p

p is given by a random variable (defined on [0,1]) with density gg represents the prior belief that satisfiesg represents the prior belief that satisfies

Generate independent and identically distributed (iid) sample traces σ1 σ(iid) sample traces σ1,…,σn

X = {x1,…,xn}, xi: the ith sample trace σi satisfies xi = 1 iff xi = 0 iffxi = 1 iff xi = 0 iff

Then, xi will be a Bernoulli trial with conditional density (likelihood function) : f(xi|u) = uxi(1 − u)1-xidensity (likelihood function) : f(xi|u) u (1 u)

27

Statistical Model Checking

Calculate Bayes Factor using posterior and i b bilitprior probability

Posterior density (Bayes Theorem) (cond. iid Bernoulli’s)

28Likelihood function

Statistical Model Checking

H0: p ≥ θ, H1: p < θ,Prior Probability (g: prior density of p)

1

0 0( ) ( )P H g u duθ

π= = ∫ 1 00( ) ( ) 1P H g u du

θπ= = −∫

Bayes Factor of sample X={x1,…,xn}, and hypotheses H0, H1 is

1

10 01( | ) ( | ) ( )( | ) 1( ) nf x u f x u g u duP H X P H θ π⋅ −∫0 01

1 0 010

( | ) ( )( | ) ( ) ( | ) ( | ) ( )n

P H X P H f x u f x u g u duθθ π

⋅ = ⋅⋅

∫∫

29

Beta Prior

Prior g is Beta of parameters α>0, β>0g p , β

F(·,·)(·) is the Beta distribution function (i.e., Prob(X ≤ u))

Why Beta priors?

Defined over [0,1][ , ]

Beta distributions are conjugate to Binomial distributions:

If prior g is Beta and likelihood function is BinomialIf prior g is Beta and likelihood function is Binomial then posterior is Beta

Suppose likelihood Binomial(n,x), prior Beta(α,β): posteriorpp ( ) p ( β) p

f(u | x1,…,xn) ≈ f(x1|u) · · · f(xn|u) · g(u)

= ux(1 − u)n-x · uα-1(1 − u)β-1 u (1 u) u (1 u)

= ux+α -1(1 − u)n-x+β-1

where x = Σi xi

07/16/0907/16/0907/16/0907/16/0907/16/0907/16/09

where x Σi xi

Posterior is Beta of parameters x+α and n-x+β

Beta Density Shapes

3

3.5

7

8

2

2.5

3

4

5

6

0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 10.5

1

1.5

0

1

2

3

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

α=.5 β=.5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

α=2 β=.5 3

3.5

1.5

2

2.5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.5

1 α=4 β=10

Computing the Bayes Factor

PropositionPropositionThe Bayes factor of H0:M╞═ P≥θ (Φ) vs H1:M╞═P<θ (Φ) for n Bernoulli samples (with x≤n<θ (Φ) p (successes) and prior Beta(α,β)

where F(·,·)(·) is the Beta distribution function.

Bayesian Interval Estimation - IV

40

y

2

2.5

width 2δ

30

35

40

1.5

2

20

25

0.5

1

10

150 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

prior is beta(α=4,β=5)

0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 10

5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

posterior density after 1000 samples and 900 “successes” is beta(α=904,β=105) posterior mean = 0.8959

Related Work – Monte Carlo Simulation

A random variable X with unknown di t ib ti f( ) ( )P X t>distribution f(x)Monte Carlo is a statistical method to estimate probability of an event (such as X

( )p P X t= >

estimate probability of an event (such as X > t) under unknown distribution.The goal is to find { }( }p E I X t>The goal is to find

1

1ˆ ( )N

MC ii

p I X t pN

= > →∑ 1,( )

0x t

I x totherwise

>⎧> = ⎨

⎩

{ }( }p E I X t= >

The Monte Carlo method generates N independent samples of X (X X ) to

1iN = 0,otherwise⎩N → ∞

independent samples of X (X1, …,XN) to form a estimate of p

35

Related Work – Importance SamplingImportance Sampling samples from a biased distribution that the rare event happens more

foften. Let f(x) is the density of X, f*(x) is the density of X*{ }( }

( ) ( )

p E I X t

f d

= >

∫*

( )( )( )

f xW xf x

=*

*

( ) ( )

( )( ) ( )( )

I x t f x dx

f xI x t f x dxf

= >

= >

∫

∫

Sampling from a biased random variable X*{ }* *

( )

( ) ( )

f x

E I X t W X= >

∫

Sampling from a biased random variable X , IS estimator:

* *1ˆ ( ) ( )N

p W X I X t= >∑36

1( ) ( )IS i i

ip W X I X t

N =

= >∑

Bayesian Interval Estimation - V

Require: BLTL property Φ, interval-width δ, coverage c, prior beta parameters α βprior beta parameters α,βn := 0 {number of traces drawn so far}x := 0 {number of traces satisfying so far}

trepeatσ := draw a sample trace of the system (iid)n := n + 1if Φ thif σ Φ then

x := x + 1endif

( )/( β)mean = (x+α)/(n+α+β)(t0,t1) = (mean-δ, mean+δ)I := PosteriorProbability (t0,t1,n,x,α,β)

until (I > c)return (t0,t1), mean

Bayesian Interval Estimation - VI

Recall the algorithm outputs the interval (t t )(t0,t1)Define the null hypothesisH0: t0 < p < t1

Theorem (Error bound). When the BayesianTheorem (Error bound). When the Bayesian estimation algorithm (using coverage ½< c < 1) stops – we haveProb (“accept H0” | H1) ≤ (1/c -1)π0/(1-π0)Prob (“reject H0” | H0) ≤ (1/c -1)π0/(1-π0)Prob ( reject H0 | H0) ≤ (1/c 1)π0/(1 π0)π0 is the prior probability of H0