Backward Thinking

Confessions of a Numerical Analyst

Keith Evan Schubert

Simple Problem

Consider the problem ax=b

The resulting x value is

A1.50 1.50

1.01 0.990

b

3.00

2.00

x 1.00

1.00

Simple Problem 2

Consider the problem ax=b

The resulting x value is

A1.5 1.5

1.0 0.99

b

3.0

2.0

x 2.0

0.0

What’s Up?

The condition number (sensitivity to perturbations) is about 400.

A condition number of 1 is perfect. Perturbation is 0.01, so 0.01*400=4. Components of x can vary by this much!

What Can We Do?

Rather than solve it the standard way• X=a\b

• X=(ATA)-1atb

Consider the following:• X=(ATA+i)-1atb =.01

Then:x

1.0

1.0

Lucky Guess?

-1 -0.5 0 0.5 10

0.5

1

1.5

2

x 2

-1 -0.5 0 0.5 10

0.5

1

1.5

2

x 1

Does It Always Work?

No Consider X0 Consider i

2 (i is singular value of A)

X± Picking the wrong value can get junk

Skyline

Consider a 1 dimensional picture Use height instead of color Result looks like the silhouette of a city’s

skyline Have smog which blurs and softens Don’t know exactly how much blur Want to get sharp edges

Getting Garbage

Getting Improvement

Why Backward? Forward errors

• Explicitly account for each error source• (X+1)(y+2)=xy+(y1+x2+12)

Backward errors• Check that my algorithm acting on data will give me

a solution that is “near” to the actual system acting on a nearby set of data

• I.E. My algorithm with good data should do about as well as a perfect calculation on ok data

•

error Ax bA x b

Picture Please!

ActualData (x)

NearbyData (x*)

Perfect Calculations

My Algorithm

Inherent errorsin A

b

Errors due toalgorithm

b*

best

Least Squares

Usually we don’t have an invertible matrix Need to find an estimated solution Criterion: minimize ||ax-b|| Normal equation

• ATA x = ATb

Solution• X = (ATA)-1atb

Backward Error

Criterion: minimize ||Ax-b||/(||A|| ||x||+||b||) Normal Equations

•

Solution:•

ATA I xATb

A Ax b

x A x b

x ATA I 1ATb

Non Convex

Finding The Root

g A Ax b

x A x b

Look At Critical Region

Informal Algorithm

Get (A,b) svd(A) [u1 u2],,v

U1b b1

Use rootfinder (bisection, Newton, etc.) to get in [-n

2,0]

vT(2- I)-1 b1 x

What You GetBlurred Image

Least SquaresLeast Squares Solution

Total Least SquaresTotal Least Squares Solution

TikhonovTikhonov Solution

Backward ErrorMin Max Backward Error Solution

OriginalActual Image

ComparisonActual Image Blurred Image

Least Squares Solution Tikhonov Solution

Degenerate Min Min Solution Min Max Solution

Min Max Backward Error Solution Total Least Squares Solution

Final Thoughts

BE is always optimistic in that it presumes that the real system is “better”

Even with this it is “robust” There is a perturbed version of this

algorithm which can be either optimistic or pessimistic

That version is not fully proven