Metrics for real time probabilistic processes

Radha Jagadeesan, DePaul University

Vineet Gupta, Google Inc

Prakash Panangaden, McGill University

Josee Desharnais, Univ Laval

Outline of talk

Models for real-time probabilistic processes

Approximate reasoning for real-time probabilistic processes

Discrete Time Probabilistic processes Labelled Markov Processes

For each state sFor each label a K(s, a, U)

Each state labelledwith propositional information

0.50.3

0.2

Discrete Time Probabilistic processes Markov Decision Processes

For each state sFor each label a K(s, a, U)

Each state labelledwith numerical rewards

0.50.3

0.2

Discrete time probabilistic proceses

+ nondeterminism : label does not determine probability distribution uniquely.

Real-time probabilistic processes

Add clocks to Markov processes

Each clock runs down at fixed rate r c(t) = c(0) – r t

Different clocks can have different rates

Generalized SemiMarkov Processes Probabilistic multi-rate timed automata

Generalized semi-Markov processes.

Each state labelledwith propositional Information

Each state has a setof clocks associated with it.

{c,d}

{d,e} {c}

s

tu

Generalized semi-Markov processes.

Evolution determined bygeneralized states <state, clock-valuation>

<s,c=2, d=1>

Transition enabled when a clockbecomes zero

{c,d}

{d,e} {c}

s

tu

Generalized semi-Markov processes.

<s,c=2, d=1> Transition enabled in 1 time unit

<s,c=0.5,d=1> Transition enabled in 0.5 time unit

{c,d}

{d,e} {c}

s

tu

Clock c

Clock d

Generalized semi-Markov processes.

Transition determines:

a. Probability distribution on next states

b. Probability distribution on clock values for new clocks

{c,d}

{d,e} {c}

s

tu

Clock c

Clock d

0.2 0.8

Generalized semi Markov proceses

If distributions are continuous and states are finite:

Zeno traces have measure 0

Continuity results. If stochastic processes from <s, > converge to the stochastic process at <s, >

Equational reasoning

Establishing equality: Coinduction Distinguishing states: Modal logics Equational and logical views coincide Compositional reasoning: ``bisimulation is

a congruence’’

Labelled Markov Processes

PCTL Bisimulation [Larsen-Skou,

Desharnais-Panangaden-Edalat]

Markov Decision Processes

Bisimulation [Givan-Dean-Grieg]

Labelled Concurrent Markov Chains

PCTL [Hansson-Johnsson]

Labelled Concurrent Markov chains (with tau)

PCTLCompleteness: [Desharnais-

Gupta-Jagadeesan-Panangaden]

Weak bisimulation [Philippou-Lee-Sokolsky,

Lynch-Segala]

With continuous time

Continuous time Markov chains

CSL [Aziz-Balarin-Brayton-

Sanwal-Singhal-S.Vincentelli]

Bisimulation,Lumpability

[Hillston, Baier-Katoen-Hermanns]

Generalized Semi-Markov processes

Stochastic hybrid systems

CSL

Bisimulation:?????

Composition:?????

Alas!

Instability of exact equivalence

Vs

Vs

Problem!

Numbers viewed as coming with an error estimate.

(eg) Stochastic noise as abstraction Statistical methods for estimating

numbers

Problem!

Numbers viewed as coming with an error estimate.

Reasoning in continuous time and continuous space is often via discrete approximations.

eg. Monte-Carlo methods to approximate probability distributions by a sample.

Idea: Equivalence metrics

Jou-Smolka, Lincoln-Scedrov-Mitchell-Mitchell

Replace equality of processes by (pseudo)metric distances between processes

Quantitative measurement of the distinction between processes.

Criteria on approximate reasoning

Soundness Usability Robustness

Criteria on metrics for approximate reasoning Soundness

Stability of distance under temporal evolution: ``Nearby states stay close '‘ through temporal evolution.

``Usability’’ criteria on metrics

Establishing closeness of states: Coinduction.

Distinguishing states: Real-valued modal logics.

Equational and logical views coincide: Metrics yield same distances as real-valued modal logics.

``Robustness’’ criterion on approximate reasoning The actual numerical values of the

metrics should not matter --- ``upto uniformities’’.

Uniformities (same)

m(x,y) = |x-y| m(x,y) = |2x + sinx -2y – siny|

Uniformities (different)

m(x,y) = |x-y|

Our results

Our results

For Discrete time models: Labelled Markov processes Labelled Concurrent Markov chains Markov decision processes

For continuous time: Generalized semi-Markov processes

Results for discrete time models

Bisimulation Metrics

Logic (P)CTL(*) Real-valued modal logic

Compositionality Congruence Non-expansivity

Proofs Coinduction Coinduction

Results for continuous time models

Bisimulation Metrics

Logic CSL Real-valued modal logic

Compositionality ??? ???

Proofs Coinduction Coinduction

Metrics for discrete time probablistic processes

Bisimulation

Fix a Markov chain. Define monotone F on equivalence relations:

Defining metric: An attempt

Define functional F on metrics.

Metrics on probability measures

Wasserstein-Kantorovich

A way to lift distances from states to a distances on distributions of states.

Metrics on probability measures

Metrics on probability measures

Example 1: Metrics on probability measures

Unit measure concentrated at x

Unit measure concentrated at y

x y

m(x,y)

Example 1: Metrics on probability measures

Unit measure concentrated at x

Unit measure concentrated at y

x y

m(x,y)

Example 2: Metrics on probability measures

Example 2: Metrics on probability measures

THEN:

Lattice of (pseudo)metrics

Defining metric coinductively

Define functional F on metrics

Desired metric is maximum fixed point of F

Real-valued modal logic

Real-valued modal logic

Tests:

Real-valued modal logic (Boolean)

q

q

Real-valued modal logic

Results

Modal-logic yields the same distance

as the coinductive definition However, not upto uniformities since glbs

in lattice of uniformities is not determined by glbs in lattice of pseudometrics.

Variant definition that works upto uniformities

Fix c<1. Define functional F on metrics

Desired metric is maximum fixed point of F

Reasoning upto uniformities

For all c<1, get same uniformity

[see Breugel/Mislove/Ouaknine/Worrell]

Variant of earlier real-valued modal logic incorporating discount factor c characterizes the metrics

Metrics for real-time probabilistic processes

Generalized semi-Markov processes.

{c,d}

{d,e} {c}

s

tu

Clock c

Clock d

Evolution determined bygeneralized states <state, clock-valuation>

: Set of generalized states

Generalized semi-Markov processes.

{c,d}

{d,e} {c}

s

tu

Clock c

Clock d

Path:

Traces((s,c)): Probability distribution on a set of paths.

Accomodating discontinuities: cadlag functions

(M,m) a pseudometric space. cadlag if:

Countably many jumps, in general

Defining metric: An attempt

Define functional F on metrics. (c <1)

traces((s,c)), traces((t,d)) are distributions on sets of cadlag functions.

What is a metric on cadlag functions???

Metrics on cadlag functions

Not separable!

are at distance 1 for unequal x,y

x y

Skorohod metrics (J2)

(M,m) a pseudometric space. f,g cadlag with range M.

Graph(f) = { (t,f(t)) | t \in R+}

t

fg

(t,f(t))

Skorohod J2 metric: Hausdorff distance between graphs of f,g

f(t)g(t)

Skorohod J2 metric

(M,m) a pseudometric space. f,g cadlag

Examples of convergence to

Example of convergence

1/2

Example of convergence

1/2

Examples of convergence

1/2

Examples of convergence

1/2

Examples of non-convergence

Jumps are detected!

Non-convergence

Non-convergence

Non-convergence

Non-convergence

Summary of Skorohod J2

A separable metric space on cadlag functions

Defining metric coinductively

Define functional on 1-bounded pseudometrics (c <1)

Desired metric: maximum fixpoint of F

a. s, t agree on all propositions

b.

Real-valued modal logic

Real-valued modal logic

Real-valued modal logic

h: Lipschitz operator on unit interval

Real-valued modal logic

Real-valued modal logic

Base case for path formulas??

Base case for path formulas

First attempt:

Evaluate state formula F on stateat time t

Problem: Not smooth enough wrt time sincepaths have discontinuities

Base case for path formulas

Next attempt:

``Time-smooth’’ evaluation of state formula F at time t on path

Upper Lipschitz approximation to evaluatedat t

Real-valued modal logic

Non-convergence

Illustrating Non-convergence

1/2

1/2

Results

For each c<1, modal-logic yields the same uniformity as the coinductive definition

All c<1 yield the same uniformity. Thus, construction can be carried out in lattice of uniformities.

Proof steps

Continuity theorems (Whitt) of GSMPs yield separable basis

Finite separability arguments yield closure ordinal of functional F is omega.

Duality theory of LP for calculating metric distances

Results

Approximating quantitative observables:

Expectations of continuous functions are continuous

Continuous mapping theorems for establishing continuity of quantitative observables

Summary

Approximate reasoning for real-time probabilistic processes

Results for discrete time models

Bisimulation Metrics

Logic (P)CTL(*) Real-valued modal logic

Compositionality Congruence Non-expansivity

Proofs Coinduction Coinduction

Results for continuous time models

Bisimulation Metrics

Logic CSL Real-valued modal logic

Compositionality ??? ???

Proofs Coinduction Coinduction

Questions?