Numerical Methods for Evolutionary Systems – Lecture 3 C. W. Gear Celaya, Mexico, January 2007

Copyright 2006, C. W. GearL3-1

Numerical Methods for Evolutionary Systems – Lecture 3C. W. Gear

Celaya, Mexico, January 2007

Differential-Algebraic Equations (DAEs) and Low Dimensional Manifolds

Differential-Algebraic Equations (DAEs) have been a separate topic for about 45 years. They refer to combinations of differential equations and coupled (non-linear) equations. One place familiar to chemical engineers in which they arise is in reaction kinetics. Suppose we had the system:

that represents some reaction kinetics and we knew that, say, the reaction for the 7-th species, y7, was very fast. For that case it has sometimes been recommended that the corresponding differential equation be deleted and the right hand side equated to zero.


If we do this we get

0

These are now a set of DAEs. In this simple case we could compute y7 from the last equation using y7 = μk5y1y2 /(k4fy3) and then eliminate it from the other 6 equations to get a system of 6 ODEs in 6 unknowns that presumably require less computation for their solution than the original larger system.

Today DAEs arise in many contexts and in some cases they are very easy to solve, but in other cases they are effectively unsolvable by computational methods until the equations have been significantly modified. Furthermore, some DAEs can give rise to significantly larger numerical errors in their solution than others with only slightly different properties, so it is important for anyone using an off-the-shelf code for DAEs to be aware of the issues.


We noted in the last lecture that stiff equations did not become a computational issue until the digital computer was introduced and problems large enough to have stiff components could be considered for solution.

DAEs did not become an issue until computer had enough memory and compute capacity that one could start trying to model systems so large that it became difficult, if not impossible to write down a system of ordinary differential equations to describe them. This first happened in Network Analysis which is still one of the largest sources of DAE problems. In Network Analysis one is presented with an interconnected system of discrete components such as resistors, capacitors and transistors (for an Electrical Engineering example) or beams and columns and other structural elements (for a Civil Engineering example) and one wants to compute various characteristics. In Transient Analysis one wants to determine the time-dependent behavior under applied driving conditions (electrical inputs or forces due to changing loads). In fact, the analysis of these two cases is very similar. Until about 1970 these problems were handled by generating a set of ODEs that described the circuit. Let us look at a very simple electrical network to see what is involved.

We are given the graph topology and parameters of the network and we want to determine the output waveform from the input waveform. This means we need to find the differential equation that it satisfies.This can be done in various ways that will be discussed on the next slide.

Vout = ?Vin = g(t)

R1 R2

C1C2

R3


1. We start with the network:

2. Next we label all the voltages and currents in the network – in this case there are only two unknown voltages, and one is the output voltage.

3. Now we write down all of the equations satisfied by the system. (a) First for the components in the system:

1 1 1

1 2 2

3 6

11 4

2 5

in

out

out

out

V V R I

V V R I

V R I

dVC IdtdV

C Idt

3 (b) … then for the circuit topology – called the Kirchoff current laws

1 7 1 2 4

4 7 8 2 5 3

8 5 9

3 6 6 9

0;

;

;

0; 0

I I I I I

I I I I I I

I I I

I I I I

4. Now eliminate variables until we only have differential equations –which in this case is fairly easy

I1I2

I3

I4I5

I7 I8 I9

V1

I6

1 111

1 2

12

2 3

in out

out out out

V V V VdVCdt R R

dV V V VC

dt R R

Vout = ?Vin = g(t)

R1 R2

C1C2

R3


Obviously, any competent engineer could have written this down much more rapidly without introducing all those unnecessary variables. However, when the network is very large and the equations have to be produced by a computer we need an algorithm that is guaranteed to generate the equations.

Throughout the 1960’s, automatic codes generated sets of ODEs from such networks by using techniques related to what we just did, although they used some important short cuts to avoid generating all of those current variables. (The primary technique was to build a spanning tree of the graph by removing edges in the graph until there were no cycles - if you don’t know what this means, it’s not important to what follows - the point is that such techniques then permitted a minimum number of current variables to be defined, knowing that there were direct methods to express all other current in terms of those variables.) Some other techniques were used that could be guaranteed to find the set of ODEs satisfied by the system.

HOWEVER, as computers grew in capacity, more and more complex networks were analyzed (today they may have millions of elements) and more complex elements (not just resistors, capacitors and simple transistors) were introduced it became increasingly difficult to generate ODEs. At the same time, because the networks were increasing in size, they were tending to become more stiff so implicit methods for stiff equations were being used.

If we look back at the equations on the previous slide we see that they can be written as

F(y’,y,t) = 0

where y is the vector of all variables (the voltages and the currents in this example) and F is the function that represents all of the equations. This could be called an implicit ordinary differential equation.


Suppose we have the system of equations

F(y’,y,t) = 0 (1)

for an s-dimensional vector function F operating on two s-dimensional vectors, y’ and y, and

on t. If this equation can be solved for y’ it can be rewritten as

y’ = f(y,t) (2)

so it really is equivalent to an ordinary differential equation, and it is properly called an implicit ODE. Note that a condition for the existence of a solution (2) to (1) is that the Jacobian

is non singular. However, even if we can, in principle, solve to get (2) it might be computationally more efficient to deal with the system (1) directly when we are dealing with stiff equations because with stiff equations we often need to use implicit methods such as the Backward Differentiation formulae (BDFs). Recall that these take the form

F

y

1 10

q

i n i niy hy

Suppose we substitute this directly into (1). we get:

1 1 1 0 1 10 11 1( / , , ) ( / / , , ) 0n

q q

i n i n n i i nnni iF y h y t F hy yy h t

This is a (non-linear) equation for yn+1 that we could solve at each time step, thus effectively

solving (2) with a BDF method without ever actually finding equation (2) explicitly. If the

Jacobians of F with respect to y’ and y are sparse, this may be a much more efficient calculation.


Can we apply this method to the electrical network problem we discussed in slides 3 & 4 and avoid the step of reducing it to an ODE system? Obviously sometimes the answer is “yes” (or we wouldn’t be discussing it here!) For this example it works fine after we handle a minor difficulty. However, there are networks (and other systems) that can’t be solved with such a simple approach.

Looking back at the electrical network problem earlier we see that there are 12 equations (five describing the behavior of the circuit elements, in this case resistors and capacitors, and 7 from the Kirchoff current law). However, there are only 11 variables (2 voltages and 9 currents). If we studied the current equations carefully we would see that one of them is redundant. In fact, that will always be true for a fully connected circuit and there are well known ways to avoid generating the unnecessary equation. Since we are not especially concerned with the network problem – we are only using it as an illustration – we won’t bother with the details and just drop the first current equation.

Now we have 11 equations in 11 unknowns that look like

F(y’,y,t) = 0 (3)

but is very clearly singular (since only two of the variables even appear with a derivative). Hence (3) is not an implicit ODE, it is a DAE. When we solve (3)

for yn+1 we only need that

be non singular – which happens to be true in this example.

1 1 10

( / , , ) 0q

i n i n ni

F y h y t

1 1 10 0

1

( / ,(4)

, )

q

i n i n ni

n

F y h y tF F

y y h y

/F y


In fact, most (but not all) electrical network problems can be solved by such a method. Handling problems directly in this way has several advantages:

1. It avoids having to find the regular ODE (which may not always be possible).2. If the Jacobian of F is sparse, it frequently avoids a lot of fill in and so is

computationally more efficient.3. It handles automatically generated systems of equations directly – it is fairly easy to

set up a “rule” for a computer program to generate the equations from a circuit description – and this is important when the “circuit” of an integrated circuit on a chip may involve millions of elements.

However, being able to solve equation (4) on the last slide is no guarantee that we will solve the DAE correctly. To see that, let us look at a very simple example of a DAE, namely

1 0 0 0 1 0 0

( , , ) 0 1 0 0 0 1 0 0,

0 0 0 1 0 0 ( )

u u u

F y y t v v y v

w w g t w

For this equation we have

0 0

1 0 0 0 1 0

0 1 0 0 0 1

0 0 0 1 0 0

F F

y h y h

which is clearly non-singular - its determinant is -1. However, let’s use Backward Euler on it with variable step sizes.


Note that if we write out the equations in a more direct manner, we actually have

( )u g t

v u

w v

In other words, it is not a differential equation at all -- and needs no initial values!

This points out a problem with the DAE formulation – it may not be what it looks like! There may be far fewer degrees of freedom (initial values) that we might at first think.

To see what might happen we will go ahead and use a Backward Differentiation (BDF) method on the problem – we will use Backward Euler to simplify the discussion.

When we use Backward Euler we need initial values. Let’s assume that these are u0, v0, w0 and for the moment assume that we have the correct values. (Finding them is one of the challenges of DAEs that we will discuss later.)


Using Backward Euler starting from t = 0 with step sizes h0, h1, … we get the following solution at t1 :

1 0 0

1 1 0 0 0 0

1 1 0 0 1 0

( ) ( )

( ) / [ ( ) (0)] /

( ) / [ '(0)] /

u g h g h

v u u h g h g h

w v v h v g h

u1 is obviously correct. v1 is a first order approximation to g’(t1) which seems like all we can

expect from a first order method. For v1 we get

2 2 1 1 2 1 2 1 0 1 1

02 12

1 1

( ) / [{ '( ) ( ) / 2} { '( ) ( ) / 2}] /

'( ) '( )(1 ) ( ) / 2

w v v h g t h g t g t h g t h

hg t g tg t

h h

L

21 1 1 0 1 0 1 0

1 0 1

( ( ) [ ( ) ( ) ( ) / 2 ]) /

( ) ( ) / 2

v g t g t h g t h g t h

g t h g t

L

Lwhich is order h correct, but from this we find that w1 is

which is half of what it should be. We could wonder if this is a problem with initial values (although we started with the correct values). (Indeed, if we went one more step with the same step size we would get a reasonable approximation to w1.) Let us take one more step with a different step size to see what happens. As before, u and v will have reasonable solutions. For w we get

1 1 0 1 0

1 0 1 1 0 1 0

1

[ ( ) ( ) / 2 (0)] /

[ ( ) ( ) / 2 { ( ) ( )}] /

( ) / 2

w g t h g t g h

g t h g t g t h g t h

g t

L

L

L


02 2

1

( )[1 ] / 2h

w g th

L

If h0 ≠ h1 then this is not an approximation to the true answer. In this case, we can get the correct

answer by using equal step sizes, but that is not necessarily true for non-linear problems (which most interesting problems are!) This difficulty was not an artifact of the particular problem we chose (which wasn’t really a differential equation at all) as we will see now.

We can rewrite w2 as

There is another important source of DAEs that exhibit very similar characteristics. they come from MECHANICAL SYSTEMS consisting of many rigid components interconnected at a variety of joints which can be of many kinds (freely rotating, sliding, hinges {which can rotate only in one direction}, etc.) They are really networks of elements, and, as with electrical networks, it can be very difficult to write down the governing differential equations. We will illustrate the class of problems with a very simple example – a classicexample – of a pendulum consisting of a point of mass M swinging on a inextensible, weightless rod of length L. (Note that most examples we construct are idealizations of reality. There is no such thing as an inextensible, weightless rod in real life. However, when we model problems we often ignore issues that we do not believe affect the solution to the level of accuracy of interest.)

This problem is simple enough that we can write down an ODE without difficulty. If we parameterize by the angle from the vertical, θ, it is not difficult to write down the ODE

θ

Mg

M

L

sin( ) / L


However, writing down the ODE, as we did on the last slide, required a little bit of geometry/trigonometry – that was easy in this case, but could be very tedious in a complex case. Suppose, instead we had written down Newton’s equations of motion for the point mass and added the constraint that it was connected to the rigid pendulum rod. This is an example of a general approach that leads to the Euler-Lagrange equations of motion.

Mg

M

L

x

-y

0

Instead of using the angle as the variable, we use the usual 2-D Cartesian coordinates, x and y and the velocities u and v. Newton’s equation of motion (ignoring the constraint imposed by the pendulum) are

x’ = uy’ = vMu’ = 0Mv’= -Mgr

where gr is the acceleration due to gravity – which we will take as 1. However, we also need to take account of the constraint which is

g(x,y) = x2 + y2 - L2 = 0

This is done by adding a term to the acceleration equations that provides a force in the direction the constraint acts whose purpose is to keep the rod at its fixed length. The resulting system of equations is

x’ = uy’ = vMu’ = + λgx

Mv’= -M + λgy

g(x,y) = 0

where gx and gy are the partials of g with respect to x and y respectively.

In this case, g(x,y) = x2 + y2 - L2 . The advantage of this statement of the problem is that is it easy to generate the equations for a complex, interconnected system – one just writes done Newton’s equations of motion for each separate piece, and then adds the constraints.


For a general system we can write the Euler-Lagrange equations of motion in the form

x’ = vv’ = f(x,v) + GTλg(x) = 0

where x is a vector of all spatial coordinates, v is a vector of all velocities, g(x) is a vector

of all of the constraints, G is its Jacobian with respect to x, and λ (called a Lagrange multiplier) is a vector whose dimension is the number of constraints and which can be interpreted as the force provided to maintain the constraint (e.g., the force in the pendulum

rod in our earlier example). GT λ is a force that is perpendicular to the constraint surface – just as the force in the pendulum rod is perpendicular to the path of the pendulum ball.

The advantage of this representation is that it is relatively easy for a computer program to generate the equations for a complex mechanical system. The downside to the technique is that it is difficult to solve the problems automatically in this form with an automatic, variable step size code. It is easy to solve them with fixed step size BDF methods (but usually that is inadequate for large problems). The next two slides show a Matlab code and the results for BDF methods of order 1 through 6 for the pendulum problem. As can be seen, the correct order is obtained. The different colored lines give the errors in the different components of the solution. (This code uses a regular Matlab integrator to determine starting values for the method from an ODE form. Getting correct starting values is, in general, another difficulty with DAEs.)


% Ex10 Attempted solution of 2-D pendulum problem by BDF, several fixed h's.% Length of pendulum is L = 1, Mass is 1 (Mass scales out except in lambda).% We will take initial values that we know are correct.clear% alpha(p,j) are the BDF coefficients for order p. \alpha_i is in Alpha(p,i+1)Alpha = [[1 -1 0 0 0 0 0]; [3/2 -2 1/2 0 0 0 0]; [11/6 -3 3/2 -1/3 0 0 0]; [25/12 -4 3 -4/3 1/4 0 0]; [137/60 -5 5 -10/3 5/4 -1/5 0]; [147/60 -6 7.5 -20/3 3.75 -6/5 1/6]];

Lt = 5; %Length of integration intervalx0 = 0.6; % initial xy0 = -sqrt(1-x0^2); % initial yu0 = 0; v0 = 0;

% Parameters for computing very accurate solutiontheta0 = [asin(.6); 0]; %Initial angular position & velocitytol = 1E-12; for order = 3:6 alpha_0 = Alpha(order,1); alpha = Alpha(order,order+1:-1:2); % Reverse order %Storage for errors: x_err = []; y_err = []; u_err = []; v_err = []; lambda_err = []; H = []; % and step sizes used for j = 1:10-order N = 2^(j+3); % Number of steps h = Lt/N; % Fixed step size % Compute very accurate solution for starting and end point options = odeset('RelTol',tol,'AbsTol',tol,'Stats','off'); [Time,Theta] = ode113(@fun10,[h*(-order+1:0) Lt],theta0,options); x_end = sin(Theta(:,1)); y_end = -cos(Theta(:,1)); u_end = -Theta(:,2).*y_end; v_end = Theta(:,2).*x_end; lambda_end = -(Theta(:,2).^2 - y_end)/2; x_old = x_end(1:order); y_old = y_end(1:order); % Starting values for order u_old = u_end(1:order); v_old = v_end(1:order); lambda_old = lambda_end(1:order); t = h*(order - 1); X = x_old; Y = y_old; T = (0:order-1)'*h; % Saved values for plots

for i = 1:N % integrate N steps % We will "predict" the next value by using the current one x = x_old(end); y = y_old(end); lambda = lambda_old(end); t = t+h; Del = 1; % Del will be a measure of the change in the Newton iteration % When small, stop iterating. Forces the first iteration while Del > 1E-11 % test for Newton convergence % These are the residuals that we wish to zero f = [(alpha_0*(alpha_0*x + alpha*x_old)/h + alpha*u_old - 2*lambda*x*h); (alpha_0*(alpha_0*y + alpha*y_old)/h + alpha*v_old - 2*lambda*y*h + h); (x^2 + y^2 - 1)]; % Derivative for Newton iteration J = [[(alpha_0^2/h-2*lambda*h) 0 -2*x*h]; [0 (alpha_0^2/h-2*lambda*h) -2*y*h]; [2*x 2*y 0 ]]; Dx = -J\f; %Newton change -- J\f means J^(-1)*f x = x + Dx(1); y = y + Dx(2); lambda = lambda + Dx(3); % Update iterate Del = sum(abs(Dx)); % Measure of change for convergence test end % Compute final u an v for error comparison u = (alpha_0*x + alpha*x_old)/h; v = (alpha_0*y + alpha*y_old)/h; x_old = [x_old(2:end); x]; y_old = [y_old(2:end); y]; u_old = [u_old(2:end); u]; v_old = [v_old(2:end); v]; lambda_old = [lambda_old(2:end); lambda]; % Save past values X = [X;x]; Y = [Y;y]; T = [T;t]; end x_err = [x_err; x-x_end(end)]; y_err = [y_err; y-y_end(end)]; u_err = [u_err; u-u_end(end)]; v_err = [v_err; v-v_end(end)]; lambda_err = [lambda_err; lambda-lambda_end(end)]; H = [H;h]; end figure(10+order); loglog(H,abs(x_err),'-b',H,abs(y_err),'-g',H,abs(u_err),'-r',... H,abs(v_err),'-m',H,abs(lambda_err),'-k') title(['Order ' num2str(order) ' Method']) eval(['print -dpsc Ex10order' num2str(order)]) pause(0.1)end

function derivative = fun10(t,y)% Pendulum in angular coordinatesderivative(1,1) = y(2);derivative(2,1) = -sin(y(1));

Matlab code for fixed step BDF methods for Pendulum problem


Errors for BDF fixed step methods, order 1 to 6. X-axis is step size, h, y-axis is error


In some cases we can even manage to get a variable step size method to work, but we have to realize that the problem we saw on slide L3-10 where the incorrect value was calculated for one variable when the step size changed can cause problems. The plot below was produced by the variable step size Backward Euler code on the next slide for the pendulum problem. (That code is very similar to the one two slides back, so many comments have been left out to fit the code on one slide.) (Because of the linear relation between x and u and y and v, u and v have been eliminated to simplify the calculation – that step does not effect the results.) We plot the errors in x, y, and λ as the requested tolerance is reduced. (Because we are controlling the local h2

error by tol, h is proportional to √(tol) so the global error, which is proportional to h is also

proportional to √(tol).)


% Ex10auto Solution of 2-D pendulum problem by B. Euler with error control.% Length of pendulum is L = 1, Mass is 1 (Mass scales out except in lambda).% We will take initial values that we know are correct.x0 = 0.6; y0 = -0.8; u0 = 0; v0 = 0; h = 0.001; % Initial step sizeLt = 10;% Compute very accurate solution for starting and end pointoptions = odeset('RelTol',1E-11,'AbsTol',1E-11,'Stats','off');[Time,Theta] = ode113(@fun10,[0 Lt],[asin(.6); 0],options);x_end = sin(Theta(end,1)); y_end = -cos(Theta(end,1));u_end = -Theta(end,2).*y_end; v_end = Theta(end,2).*x_end;lambda_end = -(Theta(end,2).^2 - y_end)/2;Xer = []; Yer = []; Lambdaer = []; Tol = [];for itol = 1:9 %Max itol of 9 tol = 2E-4*2^(-itol); % Our error tolerance Tol = [Tol; tol]; xn = x0; yn = y0; un = u0; vn = v0; lambda_n = 0; t = 0; %Past values at last t first = 1; while t < Lt % integrate past Lt t = t+h; fail = 1; % Flag indicating next step not yet successful while fail % Loop until we have a successful step if first x = xn; y = yn; lambda = lambda_n; else % We will "predict" the next value using linear extrapolation xp = xn +(xn-x_old)*h/h_old; yp = yn +(yn-y_old)*h/h_old; lambda_p = lambda_n +(lambda_n-lambda_old)*h/h_old; end Del = 1; while Del > 1E-10 % test for Newton convergence f = [((x - xn)/h - un - 2*lambda*x*h); ((y - yn)/h - vn +h - 2*lambda*y*h); (x^2 + y^2 - 1)]; J = [[(1/h-2*lambda*h) 0 -2*x*h]; [0 (1/h-2*lambda*h) -2*y*h]; [2*x 2*y 0 ]]; Dx = -J\f; x = x + Dx(1); y = y + Dx(2); lambda = lambda + Dx(3); Del = sum(abs(Dx)); end if first h_new = h; fail = 0; % There will be no error test for the first step else % Not first step, so we have information to test error Error_est = sqrt((x-xp)^2 + (y - yp)^2); h_ratio = sqrt(Error_est/tol); % Estimated corrector error

if h_ratio >= 1 h = 0.9*h/h_ratio; % Reduce h for failed step else if h_ratio < 0.9 h_new = 0.9*h/h_ratio; % Increase h if possible else h_new = h; end fail = 0; end end end u = (x-xn)/h; v = (y-yn)/h; % Compute new u an v x_old = xn; y_old = yn; lambda_old = lambda_n; % Save past values xn = x; yn = y; un = u; vn = v; lambda_n = lambda; h_old = h; h = h_new; H = [H;h_old]; first = 0; end % Linearly interpolate to end point r = (T(end) - Lt)/(T(end) - T(end-1)); x_e = X(end)*(1-r) + X(end-1)*r; y_e = Y(end)*(1-r) + Y(end-1)*r; lambda_e = Lambda(end)*(1-r) + Lambda(end-1)*r; Xer = [Xer; (x_e - x_end)]; Yer = [Yer; (y_e - y_end)]; Lambdaer = [Lambdaer; (lambda_e - lambda_end)];endfigure(10)loglog(Tol,abs(Xer),'-b',Tol,abs(Yer),'-g',Tol,abs(Lambdaer),'-k')xlabel('Tolerance requested'); ylabel('Error'); title('Variable step size Backward Euler for Pendulum')print -dpsc Ex10

function derivative = fun10(t,y)% Pendulum in angular coordinatesderivative(1,1) = y(2);derivative(2,1) = -sin(y(1));


Note that in the code above we are only controlling the errors in x and y, NOT in λ. (See red code line.) If we tried to control errors in λ, the code would fail to make headway because the variable λ is like the variable w on Slide L3-10 and has big errors when h changes in size. Since λ is calculated “from scratch” at each step (it is not given by a differential equation) these errors do not propagate to future steps so the final error is still reasonable. HOWEVER, if the user is interested in the values of λ along the trajectory then these errors are an issue. Since the values of λ represent the forces at the connections, the modeler is very often interested in them – one must be certain that the physical system is strong enough to stand up to the forces.

How do we know when a system is likely to cause trouble? The most important characteristic of a DAE problem that determines whether we can compute a solution easily is its index. There are many definitions of index in the literature. Perhaps the first, and one of easiest to understand, is the differentiation index. It is defined as the number of times that one has to differentiate a DAE in order to be able to find a system of ODEs that have a solution equal to the solution of the original DAE. Let us illustrate with our first example:

1 0 0 0 1 0 0

( , , ) 0 1 0 0 0 1 0 0,

0 0 0 1 0 0 ( )

u u u

F y y t v v y v

w w g t w

Writing this is the simpler form u = g(t); v = u’; w = v’ we see that we have to differentiate 3 times, first getting u’ = g’(t), v’ = u”, w’ = v”; next getting u” = g”(t), v” = u’’’, w’’ = v’’’; and finally getting u’’’ = g’’’(t), v’’’ = u””, w’’’ = v””. From this system we can extract the ODEu’ = v, v’ = w, w’ = g’’’(t). Hence the index of this system is 3.


The Euler-Lagrange equations with constraints also have index 3. if we start with

x’ = vv’ = f(x,v) + GT λg(x) = 0

and differentiate the last equation we get

Gx’ = 0since . This leads to the equation Gv = 0 and we still don’t have a differential

equation for λ . Hence we differentiate again to get

One more differentiation yields

2 2

1 3

( , ) 0

[ ] [ ( , ) { ( )}]

Tx x

T Txx x x x x x x v

Gv G v Gf x v GG G v

GG G v G v v G vv G f x v G f v f f G

There are a number of reasons for not wanting to generate this differential equation. One obvious one is that it looks like we would get a real mess to evaluate! However, there is a more important reason, and that is that the three differentiations have introduced three (or more) arbitrary constants of integration. Going back to our earlier trivial example, we turned

u = g(t); v = u’; w = v’ which has no constants of integration because it is not a differential equation into

u’ = v, v’ = w, w’ = g’’’(t) which has three since its solution is

u = g(t) + c1 + c2t +c3t2/2, v = g’(t) + c2 + cct, w = g”(t) + c3

If we were to integrate this system numerically, the computed solution would wander off the manifold of the solution to the original problem.

/G g x


Meaning of the differential indexLet us consider a restricted form of DAEs called semi-explicit. (Generally, a DAE could be brought into this form by a transformation of variables, although it is often not practical. However, this form arises a lot in practice, and makes it easier to understand what is going on. Suppose we have the system:

y’ = f(y,z)0 = g(y,z)

If we differentiate the second equation we get

0 = gyf(y,z) + gzz’

If gz is non singular we can solve this for z’ to get

z’ = -gz-1f(y,z)

to get a differential equation for z, thus indicating that the differential index is 1. Thus, the non

singularity of gz is a necessary and sufficient condition for the problem to have index 1. Index 1

problems occur quite frequently and fortunately are very easy to solve. The BDF methods work

for index one problems (replace y’ with the linear combination of past y’s and solve for the most

recent y and z).

An index-one problem specifies a manifold in the y-z space given by the solution of g(y,z) = 0 on which the solution of the DAE must lie. In principle, we could just integrate for y with a

standard ODE package, and each time we need to evaluate y’ for a value of y we first solve

g(y,z) = 0 for z(y) and substitute into f(y,z) to find y’. (In practice, this may be inefficient, so direct substitution with the BDF formula may be better.)

Note that finding suitable initial values is not difficult. As long as we are only given y0 we do

not have to do anything special.


Suppose gz is singular, so the index is greater than one. Let us take the extreme case

y’ = f(y,z)0 = g(y)

In this case, the first differentiation gives us

gyf(y,z) = 0 (1)

and we will have to differentiate at least one more time to get

gyfzz’ + gyfyf(y,z) = 0

Clearly, if gyfz is non singular we can now solve for z’ to get an ODE for z and the index is

two. However, note that equation (1) specifies a second manifold (in addition to g(y,z) = 0) on which the solution must lie. We call this a hidden manifold and a large part of the difficulty with DAEs is the restriction that the solution must lie on these hidden manifolds although they can only be revealed by differentiation (which we would prefer not to do).

In general, for an index m problem there is one visible manifold and m-1 hidden manifolds on which the solution must lie.

There are no good direct computational methods for the most general problems of high index. Index 1 problems are easy. Many index 2 problems are OK. Special methods are needed for higher index problems.

For a good reference on DAEs, use the book K.E. Brenan, S.L. Campbell, and L.R. Petzold, Numerical solution of initial-value problems in differential-algebraic equations, SIAM, Philadelphia, 1996


What can we do if we have a high-index problem?

(assuming that it is not feasible to manipulate them symbolically into a simpler form)

1. We could reduce the index by differentiation, but that introduces additional constants of integration so we could wander off the constraint manifolds, so we must …

a) add additional equations as constraints to force the solution to lie in the manifolds of the problem

b) project the solution back onto the constraints after each integration step.

2. "Regularize" the equations – that means convert to a differential equation (or DAE of lower index) which is stiff and whose slow solution is (approximately) the solution of the DAE.

In either of these methods we have to be able to "get at the structure" of the DAE system to modify it in some way, and this almost always has to be done symbolically. Generally we can only deal with DAEs with restricted structures because of the near impossibility of performing the necessary manipulations on the most general form of equations.

In fact, many DAEs have a special structure because of the way they arise. These are described on the next slide.


The three most common special cases of DAEs are:

Semi-explicit: y' = f(y,z)` 0 = g(y,z)

Linear in derivatives: Ay' = f(y,t) where matrix A is constant (but singular)

"Hessenberg" y1' = f1(y1,y2)y2' = f2(y1,y2,y3)…yk' = fk(y1,y2,…,yk,z)0 = g(y1)

The first form, although it looks simpler than the general case F(y',y,t) = 0 really isn't any help without additional information since we can always convert the latter to the former by introducing another variable, z, and writing

y' = z0 = F(z,y,t)

(although this transformation does increase the index by 1.) Hence, the only case of interest here

is the case when gz is nonsingular and we have an index 1 problem.

The second form arises in, for example, finite element modeling where A is the Mass matrix. A

could also depend on t but in that case it would be important that its structure did not change (e.g., it's rank should not change during the calculation).

The third form occurs frequently – for example, the Euler-Lagrange equations have this form, and, fortunately, Hessenberg DAEs are one type of index 2 DAEs that can be solved without to much difficulty.


Hessenberg DAEs

y1' = f1(y1,y2)y2' = f2(y1,y2,y3)…yk' = fk(y1,y2,…,yk,z)0 = g(y1)

One important fact about Hessenberg DAEs is that their index is at least k+1. This follows by

differentiating first g and then f1, f2, … , fk in turn. Only after these k+1 differentiations does

z' appear in the result. By that time we will have1 2

1 2 3

0kfg f fz

y y y z

L L

where the other terms are functions of all of the y variables. Hence, if the matrix

1 2

1 2 3

kfg f f

y y y z

L

is nonsingular, the problem has index k+1. Fortunately, in many problems the partial derivatives in this product are either constant or have a simple structure than enables us to tell whether it is non-singular or not fairly easily. For example, in the simple Euler-Lagrange equations

x’ = vv’ = f(x,v) + GTλg(x) = 0

we see that the matrix product is GGT. G indicates specifies the directions in which the different constraint forces act. These have to be linearly independent or one or more of the constraints would be redundant (like using two pieces of string to hold a single pendulum). This linear independence of the rows of G means that GGT is nonsingular. Thus, this guarantees that the index is 3.


Because of the structure, it is very obvious how to reduce the index of a Hessenberg system of DAEs. If we want to reduce the index by 1, we simply replace the "algebraic" equation g(y1) = 0 with its derivative to get

where we have renamed the set of variables (y1,y2) as w1. If we now rename y3 through yk as w2 through wk-1 we will see that we have the reduced set of equations

w1' = f2(w1,w2)…wk-1' = fk(w1,w2,…,wk-1,z)0 = g1(w1)

where the f's have been suitably redefined. This system has index k-1.

This reduction is particularly simple for the Euler-Lagrange equations because we already have the partial of g, namely G, so the new algebraic equation is simply

G [f(x,v) + GTλ] = 0This is called the velocity constraint since it says that any velocities must be tangential to the constraints (so that there is no motion away from the constraint). However, when we discard the equation g(x) = 0 we no longer have a system that stays on the position

constraint g(x) = 0. Small errors in the numerical integration could cause us to wander off the constraint, and, for some systems, this could be disastrous, leading to physically unrealistic systems or unstable systems.

1 1 2 1 11

( , ) 0 ( )gf y y g w

y

@


One approach is to add this equation back to the system, but we would now have an apparently overdetermined system – more equations that there are unknowns (although we know that the equations are consistent). If we do this we get the system

( )

0

0 ( )

T

x v

v f x G

Gv

g x

and if we tried to solve this by replacing the derivatives with their backward differentiation approximations we would not only have an overdetermined system, but it would probably not have a solution because the numerical approximations to the derivatives introduce errors. In the pendulum example with backward Euler we get the system:

2 2

02

1

0

0 1

n

n

n

n

x x uh

y y v

u u x

v v y

xu yv

x y

6 equations to solve for the 5 variables x, y, u, v, and λ. (Here we have dropped the subscript n+1 on the new values that have to be calculated to reduce clutter. The n subscript refers to the previous time point.) In general, they will not have a solution.


How can we enforce an extra constraint that we know is satisfied by the differential system but will probably not be satisfied by the difference equation? We can ask this questiopn about an ODE system also. Suppose we have an ODE y’ = f(y) which we know has an invariant (that is, a function of the variables that doesn’t change). For example, such an invariant could be the total energy of a closed system, the total mass, or other quantities. (For chemical kinetics, the mass balance equations which essentially state that the total number of each type of atom present is constant, are linear relations and it turns out that virtually all numerical methods automatically preserve these within rounding error so they are not an issue. For other systems, the energy, which is a non-linear function, is not normally conserved by a numerical method.)

So we suppose that there is a function, g(y), that is conserved by the differential equation (say, at the value 0), meaning that dg/dt = gyf(y) = 0, and we want to ensure that g = 0 is conserved

by the numerical method. We can do this by appending the constraint with its own Lagrange multiplier in the form

( ) (1)

0 ( ) (2)

Ty f y G

g y

where G = gy. Here μ is an additional variable (actually, a number of variables equal to the

dimension of g) that gives us the same number of variables as equations. However, note that this modification does not change the solution of the differential equation because since we have the relation Gf = 0 we get by differentiating (2) and substituting for y’ from (1)

0 '

0 ' ( )or so that

T TGy Gf GG GG

y f y


Equations (1) and (2) on the preceding slide can be seen to be an index 2 DAE. It may not seem very useful to convert an ODE (which is easy to solve) into an index 2 DAE which is often not so easy to solve. However, note that this DAE is in Hessenberg form, which is relatively easy to solve numerically.

We can use the same approach to DAEs when we reduce their index by differentiation. Going back to our pendulum example, we can write the enhanced system after one differentiation as

( ) (1)

0

0 ( )

T

T

x v G

v f x G

Gv

g x

which makes it an index-2 Hessenberg DAE. We can use this process to reduce the index of any high-index system, but not below index 2. (However, if the system is not already in Hessenberg form so it is clear which are the constraints that have to be differentiated, this process will involve a lot of symbolic manipulation which may not be practical.)

In making this transformation we added a term to the derivative of x in the first equation above. It modifies x in the direction GT. This direction is perpendicular to the constraint, g = 0. Thus we can view this as nothing more than a projection back onto the constraint g = 0. This we could consider simply integrating the reduced equations one step and then projecting the solution back onto the original constraint. There is, however, a subtle difference between this and the process given by equations (1) above. In the equations above we require the projected result to satisfy all the other difference equations whereas if we project x after the integration step we will no longer exactly satisfy the other equations such as Gv = 0.


Initial Conditions for DAEs

When we have an initial value problem for an ODE, we know exactly what initial conditions we need – the value of each variable at the initial time. When we have a DAE the issue is much more complex. We need an initial value that lies on each of the manifolds implied by the

constraint – both the visible manifold, g(y) = 0, and the hidden manifolds. This is the problem of consistent initial conditions.

There is one hidden manifold for each index in excess of 1, so if we have an index one problem there are no hidden manifolds so it is not difficult to determine if a set of initial conditions are consistent, although it may not be trivial to determine them in the first place. In the case of an index one problem written in semi-explicit form

' ( , )

0 ( , )

y f y z

g y z

the calculation is straightforward. We need an initial condition for y and then can calculate z from the constraint equation.

Of course, it is possible that the user has some of the y values and some of the z values and

wishes to find the remaining y and z values. If the dimension of y is n and the dimension of

z is m then we need n initial conditions altogether and we could work with any unknowns for

which the Jacobian of g (which has to consist of m equations) with respect to the unknown values is non-singular.


When we have a higher index problem (or an index 1 problem not in semi-explicit form) the consistent initial value issue is much more difficult since it essentially requires that we determine the hidden manifolds. Since these are only revealed by differentiation we either have to differentiate the system symbolically or use some numerical technique to achieve something similar.

In practice, the appropriate technique will depend on the type of problem so it is difficult to give general methods. In the case of, for example, Euler-Lagrange equations, it is not difficult to find the first hidden manifold by differentiation, and this

serves to ensure that x and v are mutually consistent. In many cases it is possible to differentiate one more time which is enough to develop an equation for the Lagrange

multiplier, λ. Specifically, let us suppose that the dimension of x (and v) is n and the

number of visible constraints – the dimension of g is m. Then we need to know n – m components of x. With these and the constraint g(x) = 0 we can compute the

remaining m initial values of x. Next we need to know n – m components of v. With

these and the first hidden constraint equation Gv = 0 we can compute the remaining m initial values for v. We could then differentiate again to find λ but if we are going to reduce the index of the problem to 2 by one differentiation (and then add back the

constraint with its own Lagrange multiplier μ as we did earlier) and solve by a method

such as BDF, we do not need initial values for λ or μ.


We have discussed how to reduce the index of DAEs by differentiation and then ways to force the solution back onto the constraint manifold(s). An alternative approach to reducing the index is by what are known as a regularization technique. Regularization refers to the process of taking an ill-posed problem – that is, a problem which is very sensitive to perturbations – and turning it into a well-posed problem – one whose answer changes only by an amount more or less of the same size as the perturbation. (This is a very loose definition since our purpose here is not to delve into complex mathematical analysis but to get an understanding of issues facing the user of integration methods.)

There is a sense in which DAEs are ill-posed problems. The reason is that, as we have seen with our first trivial example on slide L3-8, their solution may involve the derivative of given functions. Hence any perturbation in a derivative will show up in the solution. If we specify a function numerically and then approximate its derivative by numerical differentiation, the answer is very sensitive because we have to civide by h, the spacing between the numerical values, to get the derivative. h has to be small to get a good approximation, so we multiply by a large number, 1/h, and amplify any errors significantly.

There are several indices defined for DAEs. We have been using the most common. Another is called the perturbation index and is essentially defined as the number of times that any numerical errors get multiplied by 1/h (the actual definition is much more technical than this simplifed one, but it will serve to give you the general idea). It is this sense in which a DAE is ill-defined, and regularization attempts to remove some of these problems (although one can’t really remove them without making too many approximations).


In regularization, we essentially try to change the DAE back to a stiff system of ODEs. Recall that on the first slide of this lecture we started with a system of ODEs representing a chemical kinetics problem and on the next slide used the assumption that one of species reacted so rapidly that it was essentially in equilibrium the whole time so that we could simply set its derivative to zero, thus getting a DAE. In regularization we try to undo this process and get back to the ODE because we know how to solve ODEs! (Of course, if we had gotten the DAE from an ODE in the first place, going back would require little thought.)

Let us illustrate this for the simple index-1 semi-explicit case (although there is no need to do it in this case because we can solve it directly).

change

' ( , ); 0 ( , )

'

s to

( , ); ( , )z

y f y z g y z

y f y z g z g y z

The algebraic equation has been replaced by a differential equation. We have introduced a small parameter, ε, so that the z changes very rapidly whenever g(y,z) ≠ 0 (i.e., the solution is not on the constraint manifold). We have also chosen a scaling of z’ by gz so that the eigenvalues of the Jacobian of z’ with respect to z are all -1/ε. This means that the solution of the second differential equation very rapidly approaches a value of z that satisfies the constraints.


Applying regularization to higher index DAEs can either reduce then to ODEs or simply reduce the index to something that can be solved. One of the earliest proposed regularizations for Euler-Lagrange index 3 problems is due to Baumgarte. It replaces the constraint equation g(x) = 0 with the differential equation

2

(1)

which i

( ) ( ) ( ) 0

s

(2)

0 x

g x g x g x

Gv G v Gv g

If we substitute the equation for v’ into the equation (2), we get an equation that can be solved explicitly for λ so while we have actually reduced the index to one, we can obtain a system of differential equations with λ eliminated. This system of ODEs has dimension 2n (if x and y have dimension n) so we have a system that does not necessarily lie on the constraint manifolds. However, if we look at equation (1) and think of it as an equation for g we see that it has a stationary value at g(x) = 0. If we chose α and β so that this is a stable equilibrium, then if the solution is off the constraint g(x) = 0 it will return to the constraint. It is often suggested that we should choose α = 2η, β = η2 because then g returns to zero as exp(-ηt). In this case, our final system is:

2 2

1 2 2

or we could replace the last two equations with

'

0 2

' [ ] [ 2 ]

T

Tx

T Tx

x v

v f G

GG Gf G v Gv g

v f G GG Gf G v Gv g


The Baumgarte regularization, which leads to the system of equations on the bottom of the last slide, has the interesting property that it gives us a system of equations which has a solution that is a solution of the original Euler-Lagrange DAE, and furthermore, all other solutions of the Baumgarte regularization are damped towards the solution of the original DAE. Even if we start with the correct initial values, small numerical errors in the reduced index problem will cause us to wander off the DAE solution. That suggests that we should choose a large value of η so that the solution decays rapidly to the solution of the DAE (where g and g' are zero.)

Choosing a large η leads to a stiff system of equations. Since we can reduce the system completely to an ODE (or a semi-explicit index one DAE which is equally easy to solve), why not just use a stiff solver. If we do use a stiff solver we may find ourselves running into some numerical difficulties. The problem can arise because a stiff solver will try to take a large step (that, after all, is the point of a stiff solver). When it does this, the very fast components look as if they obey the original DAE and we come back to all of the difficulties of DAEs. We can see this by considering the very simple problem

εy' = F(y,t)

when ε is very small, we will essentially be solving the implicit equation F(y,t) = 0 at each step, and our method will have to deal with all of the problem with the structure of F. Because of this it is usually recommended that only a modest size be used for η.


There are regularization techniques that are very useful but that do not have the nice property of the Baumgarte technique that the solution of the original problem is a solution of the new problem.

Recall that a DAE restricts the solution to a manifold in the space of all of its variables. This manifold is actually the intersection of the visible and all hidden manifolds. In the case of the Euler-Lagrange equations there are 2n+m variables, but there are only 2(n-m) degrees of freedom in the solution – that is, the solution lies in a 2(n-m)-dimension manifold. What a regularization method that reduces the system to an ODE does is to extend the solution to the whole space such that any solutions that are not close to the manifold of the solutions of the original problem decay rapidly to it. That is, the regularized problem has a slow manifold to which all other solution are attracted. We don't necessarily have to make this slow manifold a solution of the original problem, it is enough that it be close to the solution of the original problem. We are going to make some errors in our numerical solution, so we only need for the slow solution of the regularized problem to introduce no larger errors than we are already making.

One approach to regularization is to realize that a DAE is often an approximation to the real world. (For example on Slide L3-2 we created a DAE by assuming that a chemical species came to equilibrium "infinitely fast.") Consider the pendulum problem. There is no such thing as a pendulum rod that won't stretch. The pendulum rod stretches a small amount proportional to the force on it (which is proportional to λ). Also there will be a small energy loss due to a less than perfect material as it stretches and relaxes. Hence, rather than the constraint

we could model the problem more accurately using where

is the strain in the pendulum rod (that is, the amount it has stretched). The first term says that the force in the rod is proportional to the extension. The second term involving the derivative of S says that the energy absorbing factor in the rod (friction) is working to slow down any extending or contracting of the rod. (This serves to damp out any oscillations.) Since we are not particularly interested in modeling this behavior accurately (it is too small to be of real consequence to the problem) we are free to choose α and β to get the best behavior of the regularized system. Choosing α = -η2 and β = -2η adds a pair of eigenvalues that are approximately –η.

2 21 2 1 0x x

S S 2 21 2 1S x x


1

1

[ ] ( )

[ ] ( )

T T

T T

x v G GG g x

v f G GG g x

In the previous regularization, we used a physical basis for choosing the modification – essentially putting back a physical phenomenon that had been ignored in the mathematical model (very few problems are really DAEs, they mainly occur because of approximations to physical reality). However, we don't have to base a modification on physical phenomena as long as our modification does not change the slow solution very much. For example, we could change the Euler-Lagrange equations to

This change effectively introduces new eigenvalues of -α and -β. If these are sufficiently negative, their components die away rapidly until the components g and g' are almost zero.

We have illustrated just three possible regularizations – the "classical" Baumgarte one and a couple of others. What one might use depends very much on the nature of the problem, so it is difficult to give a general rule. Sometimes physical intuition can provide good starting points, other times one must simply look for changes that do not change the slow manifold too much.


THE CONSISTENT INITIAlZATION PROBLEM

One interesting advantage of using regularization is that now any initial values are valid – the regularized system will rapidly damp any "initial condition errors" since they just mean that the initial point is away from the slow manifold. This means that we no longer have to face the difficult challenge of finding consistent initial values. Even is one is able to use a code directly on a DAE (as is possible for an index 2 semi-explicit DAE, for example) , it may be worth using a regularization technique initially to get correct initial values.

The problem with this is that one now has to choose initial values for all variables, and then let the regularized equations drop rapidly onto the slow manifold. If one knew the dimension of the slow manifold (which one certainly does in the Euler-Lagrange case) and had that number of initial values, we might want to find initial values for all the variables such that the prescribed initial values were satisfied.

Typically we do want to start with specific values of some of the variables – for example, we may know very precisely the concentration of some of our species in the inflow to a chemical reactor, but not know others. Unfortunately the method just suggested will "fall down" onto the slow manifold in some place that may not be a suitable starting point for our integration.

In Lecture 4 we will discuss a way in which can handle this, but before we can look at that, we have to look at a more fundamental issue – singularly perturbed equations.


We see that when ε = 0 we have a semi-explicit DAE. Singularly perturbed problems have been studied for a long time. When ε is non-zero but small, the differential equation may have rapidly changing solutions because z' is large unless g(y,z) is almost zero. If we have a boundary value problem so that the boundary values constrain the solution to be finite at both ends of an interval we may find that we get rapidly changing solutions at the boundaries (or possibly in the middle) in what are called "boundary layers." These arise, for example, in the modeling of slightly viscous fluid flow near a boundary where the fluid velocity changes rapidly from zero (the boundary condition) to its value in the interior. If we have an initial value problem, ε must be such that the fast solutions rapidly decay until g(y,z) is mearly zero, that is, they decay down onto a slow manifold very close to the manifold defined by g(y,z) = 0.

An important feature of initial value problems for singularly perturbed problems is that if they do decay rapidly to the slow manifold, their solution can be expressed as an asymptotic expansion in ε of the form

' ( , )

' ( , )

y f y z

z g y z

SINGULARLY PERTURBED PROBLEMS.

Many stiff equations are examples of singularly perturbed problems, and DAEs are really examples of the limiting case. The "classical" singularly perturbed problem is the system

0

0

( ) ( )

( ) ( )

N nnn

N nnn

y t Y t

z t Z t

where Y0(t) and Z0(t)are solutions of the DAE obtained by setting ε = 0 and Yj(t) and Zj(t) are solutions of ODEs involving the Yk and Zk for smaller k.


When we regularize a DAE we convert it back into a stiff equation – which we have just seen can be viewed as a singularly perturbed equation. The solution of this singularly perturbed equation is only ε distant from the solution of the original DAE since it is the first term in the asymptotic expansion in ε. Hence if we can use a very small ε in the regularization we will know that we have not made too much of a perturbation to the solution of the DAE.

On the other hand, we have seen that we cannot use too small an ε or a stiff method will run into difficulties because it can't tell the difference between a very stiff equations and a DAE at large step sizes

We will discuss one possible solution to this in lecture 4.


ONE-SIDED CONSTRAINTS

One final topic we want to look at in DAEs is that of one-sided constraints. These are more likely to arise in mechanical problems, but could arise in other engineering areas. In the typical DAE, such as the Euler-Lagrange formulation of mechanical systems, we express the system ignoring any constraints caused by items being linked together in some way, and then add the constraints in with Lagrange multipliers – which turn out to the be forces in the linkages that are exerted to maintain the constraint. Thus, when we simulated a pendulum with a rigid rod, the length of the rod was a constraint. Suppose, instead, we had a pendulum suspended on a piece of string (which for now we will assume is inextensible and unbreakable!) Now the constraint is that the distance of the pendulum ball from the point of suspension is no more than the length of the string. As long as the pendulum only swings below the suspension point, there will be no difference, but if we started it moving sufficiently fast that it swung above the suspension point, but not fast enough to keep the string taut while it passed overhead, we know what would happen – the pendulum ball would no longer stay on a circle but go into free fall until the string once again became taut. Thus, the model would change from a system with one constraint to a system with no constraints when the pendulum ball no longer had the velocity to kepe the string taut. When the ball reached a position that the string was taut again it would be subject to a sudden impulse from the string to force it back onto the circular arc.

If we wish to simulate something like this we will have to utilize the techniques discussed in the first lecture for handling discontinuities. If we are initial in a state where there is an active constraint, we handle it like the DAE it is. When the force in the string changes from positive (tension) to negative (compression – which a string can't handle) we must detect the time of the change and change the model. The same must happen in the other diection.

Numerical Methods for Evolutionary Systems – Lecture 3 C. W. Gear Celaya, Mexico, January 2007

Documents

Transcript of Numerical Methods for Evolutionary Systems – Lecture 3 C. W. Gear Celaya, Mexico, January 2007