UNT Digital LibraryThe facilities of Argonne National Laboratory are owned by the United States...
157
ANL-77-XX-73 Updating the Symtric Indefinite Factorization with Applications in a Modified Newton's Method by A* twoa* .NM W" 0..4kS 1MIF - *6.0 .pMM "4 p u A uM *NgM ~ * . wW A.1l pM JA 8"m ARGONNE NATIONAL LABORATORY, ARGONNE, ILLINOIS U. S. ENERGY RESEARCH AND DEVELOPMENT operated under contract W-31-109-Eng-38 for the UI;411Wr I1S 41 11U.4 ulV 06 .41 f (a ADMINISTRATION
Transcript of UNT Digital LibraryThe facilities of Argonne National Laboratory are owned by the United States...
ANL-77-XX-73
Updating the Symtric Indefinite
Factorization with Applications
in a Modified Newton's Method
by
A* twoa* .NM W" 0..4kS1MIF - *6.0 .pMM "4 p u
A uM *NgM ~ * . wW A.1l pM
JA8"m
ARGONNE NATIONAL LABORATORY, ARGONNE, ILLINOIS
U. S. ENERGY RESEARCH AND DEVELOPMENT
operated under contract W-31-109-Eng-38 for the
UI;411Wr I1S 41 11U.4 ulV 06 .41 f (a
ADMINISTRATION
The facilities of Argonne National Laboratory are owned by the United States Govern-ment. Under the terms of a contract(W-31-109-Eng-38)between the U. S. Energy Research andDevelopment Adm'nistration. Argonne Universities Association and The University of Chicago.the University employs the staff and operates the Laboratory in accordance with policies andprograms formulated, approved and reviewed by the Association.
MEMBERS OF ARGONNE UNIVERSITIES ASSOCIATION
The University of ArisonaCarnegie-Mellon UniversityCase Western Reserve UniversityThe University of ChicagoUniversity of CincinnatiIllinois Institute of TechnologyUniversity of IllinoisIndiana UniversityIowa State UniversityThe University of Iowa
Kansas State UniversityThe University of KansasLoyola UniversityMarquette UniversityMichigan State UniversityThe University of MichiganUniversity of MinnesotaUniversity of MissouriNorthwestern UniversityUniversity of Notre Dame
The Ohio State UniversityOhio UniversityThe Pennsylvania State UniversityPurdue UniversitySaint Louis UniversitySouthern Illinois UniversityThe University of Texas at AustinWashington UniversityWayne State UniversityThe University of Wisconsin
NOTICE
This report was prepared as an account of work sponsoredby the United States Government Neither the United State.nor the United States Energy Research and DevelopmentAd-ministration. nor any of their employees. nor any of theircontractors, subcontractors, or their employees. makes anywarranty. express or implied. or assumes any legal liabil-ityor responsibility for the accuracy. completeness or use-fulness of any information, apparatus. product or processdisclosed, or represents that its use would not infringeprivately-owned rights. Mention of commercial products.their manufacturers, or their suppliers in thiu publicationdoes not imply or connote approval or disapproval of theproduct by Argonne National Laboratory or the U. S. EnergyRn.,arch and Dckdlupment Admniitratiun
UNIVERSITY OF CALIFORNIA
San Diego
Updating the Symmetric Indefinite
Pactorization with Applications
in a Modified Newton's Method
A dissertation submitted in partial satisfaction of the
requirements for the degree Doctor of Philosphy
in Mathematics
by
Danny 5hris-sorensen
Committee in charge:Professor James R. Buich, ChaintuuProfessor John W. Evans, M.D.Professor John A. TrangensteinProfessor Walter A. BurkhardProfessor Michael L. Predman
1977
TABLE OF. CONTENTS
Page
List of Symbols. . . . . . . . . . ivList of Figures. . . . . . . . . . . . . . . . . . . . . . . . VList of Tables... ... . .. . . . . . .. .. . . viAcknowledgments. . . . . . . . . . . . . .. . . . -. . . . . viiVita, Publications and Fields of Study . . . . . . . . . . . . viiiAbstract ......... 00 . . ., ., ... . .. . .. ix
I. An Overview. . . . . . . . b . . . . . . . . . . . . . . . . . 1
1. Introduction. .. . "". ".,."."."."."..".".".".".".".. 1
2. Newton-type Methods for Unconstrained Optimization. . . . 23. The Symmetric Indefinite Decomposition. . . . . . . . . . 94. Computational Results and Conclusions . . . . . . . . . . 11
II. Updating Factorizations of Symmetric Matrices. . . . . .. . . 13
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . 13
2. Description of the Algorithm. . .. .. . .. . 163. A Pictorial Description of tie Algorithm. . . . . . . . . 41
III. Error Analysis of the Updating Algorithm . . . . . . . . . . . 46
1. Introduction. . . . . . . . . . . . . . . . . 0 0 . . . . 46
2. A Detailed Description of the Updating Algorithma. . . . . 483. Floating PointnAnalysis . . . ."... ...... .. ... 56
IV. The Use of Directions of Negative Curvaturein a Modified Newton Iteration. . ". . .. "."..... .".. 86
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . 862. Descent Directions. .0.0.0.0. . 0.0. . .. . "0.0. .0.0.. 883. A Nodification of the Armijo Steplength Procedure . . . . 914. Determining Directions of Negative Curvature. . . . . . . 945. A Steplength Algorithm. .t.".". ....... . . . . 986. Convergence of the Modified Newton Iteration. . . . . . . 1047. Conclusions . . . . . . . . . . . . . . . . . . . . . . 110
V. ComputationaltResults.".. 0. 0.0.00.0.0.0..0.0. . . . .. 112
1. Introduction. . . . . . . . . . . . . . . . . . . . . 112* 2. Testing the Updating Algorithm. . . . . .. .0..00 . 112
3. Testing the Modified Newton's Method. . . . . . . . . . . 117
References .*.0.0.0.".0.0.0.".0.0. ... 0.0.0.0.0.0.0.0.0.0. . . 130
Appendix Al. . . . . . . . . . . . . . . . . . . . . . . . . . 132
Appendix A2. . . . . . . . . . . . . . . . . . . . . . . . . . 134
List of Symbols
R The set of .real Numbers.
Rn The n-dimensional real vector space.
Rmxn The space of mxn real matrices.
{ ) The set of.
E Element inclusion.
c Set inclusion.
x ,At The transpose of a vector x or matrix A..
x y The inner product of two vectors x,y E Rn.
10 The Euclidean norm; |lx|| =I-
L L The infinity norm; JAL - max I i a|.1<iisaj-l
I I The absolute value function.
sgn(*) The algebraic sign function.
f: V + X Aumapping f from a domain P into the set X.
kf(x) The first partial derivative of f at x with respect to the
k-th argument.
* f(x) The second partial derivative of f at x E vinwith respect to
the i-th and then the j-th argument.
A "> E A implies B.
ak + b ak approaches b.
Lv
List of Figures
1 Pivoting in the Updating Algorithm. . . . .*-. . . . . . 41
2 A Search Along x+as . . . . . . . . . . . . . . . . . .100
3 A Search Along x+ 2as+ad . . . . . . . . . . . . . . . .101
4 The Curve x+a +ud. . . . . . . . . . . . . . . . . . .102
v
List of Tables
1
2
3
4
5
6
7
8
9
. . . . . . . .Operations Required at Step k . . . .
Results for Increasing Order. . . . .
Results of a Long Range of Updates. .
Results of Tests with Standard Starts .
Results from Using Finite Differences .
Box's Function. . . . . . . . . . . .
Gottfried's Function. . . . . . . . .
Brown's Badly Scaled Problem. . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . ." .
. . . . ." . .
. . . . . . . .
. . . . . . . .
. . . . . . ..
38
115
116
123
124
125
126
127
128
Acknowledgments
First of all, I wish to express my deep appreciation to
Professor James R. Bunch. He has generously given his time, energy, and
valuable advice to help me with this dissertation. He has also given me
many fine Qpportunities that have enriched both my personal and academic
lives. Working under his guidance has been a very rewarding experience.
I would also like to thank the other members of the doctoral
comittee: Professors John W. Evans, John A. Trangenstein,
Walter A. Burkhard, and Michael L. Fredman. A special note of thanks
goes to Dr. Jorge J. More for stimulating my interest in optimization
and for many discussions and helpful suggestions.
There are many people who have not been directly involved in the
preparation of this thesis, and yet who have influenced my thoughts.
They have been my teachers along the way, and my gratitude for their
gift of knowledge is endless. A few people deserve special mention.
Professor John W. Evans and Professor Murray Rosenblatt at University of
California, San Diego gave me the opportunity to gain computing exper-
ience by working on their research projects. Dr. Gary Leaf and
Dr. Mike Minkoff provided a stimulating summer project at Argonne
National Laboratory in 1975.
In addition, I would like to express my appreciation to the Na-
tional Science Foundation for financial support under NSF grants MCS
76-03139 and MCS 76-17548. I also appreciate the financial support
from the Applied Mathematics Division at Argonne National Laboratory.
Finally, I want to thank Ms. Judy Beumer for an excellent job
of typing and for her infinite patience.
qt{
VITA
March 6, 1946 - Born - Concord, California
1972-
1972-1974
1974-1975
1975-
1975 (Summer)
1975-1976
1976 (Summer)
1976 (Fall)
1976-1977
1977-
B.S., University of California, DavisTeaching Assistant, Department of Mathematics,University of California, San Diego
Research Assistant, University of California, San Diego
M.A., University of California, San Diego
Participant in Graduate Summer Research Institute,Argonne National Laboratory, Argonne, Illinois
Research Assistant, University of California, San Diego
Participant in Graduate Sumer Research Institute,Argonne National Laboratory, Argonne, Illinois
Visited Moscow State University Computing Center underN.S.F. scientific exchange, Moscow, U.S.S.R.
Thesis Appointee, Argonne National Laboratory, Argonne,Illinois
Doctor of Philosophy
PUBLICATIONS
"A Survey of Some Methods for Modifying .Matrix Factorizations,"Nuerical Analysis in FORTRAN, Vol. 17, pp. 1942, Moscow StateUniversity, 1976.
FIELDS OF STUDY
Major Field: MathematicsStudies in Numerical Linear Algebra.Professor Jams I. lunch
Studies in Optimisation.Dr. Jorge aord, Argonne National Laboratory
viii
ABSTRACT OF THE DISSERTATION
Updating the Symmetric Indefinite
Factorization with Applications
in a Modified Newton's Method
by
Danny Chris Sorensen
Doctor of Philosophy in Mathematics
University of California, San Diego, 1977
Professor James R. Bunch
In recent years the use of quasi-Newton methods in optimization
algorithms has inspired much of the research in an area of numerical
linear algebra called updating matrix factorizations. Previous research
in this area has been concerned with updating the factorization of a
symmetric positive definite matrix. Here, a numerical algorithm is
presented for updating the Symetric Indefinite Factorization of Bunch
and Parlett. The algorithm requires only 0(n2) arithmetic operations
to update the factorization of a symmetric matrix when modified by a
rank one matrix. An error analysis of this algorithm is given. Compu-
tational results are presented that investigate the timing and accuracy
of this algorithm.
Another algorithm is presented for the unconstrained minimiza-
tion of a nonlinear functional. The algorithm is a modification of
is
Newton's method. At points where the Hessian is indefinite the search
for the next iterate is conducted along a quadratic curve in the plane
spanned by a direction of negative curvature and a gradient related
descent direction. The stopping criteria for this search take into
account the second order derivative information. The result is that the
iterates are shown to coverage globally to a critical point at which the
Hessian is positive semid'finite. Computational results are presented
which indicate that the method is promising.
x
Chapter I
An Overview
1. Introduction
In recent years the use of matrix methods in optimization algo-
rithms has received an increasing amount of attention. Interesting
problems in numerical linear algebra have been generated by advances in
optimization methods. Similarly, new approaches to optimization methods
are sometimes made possible or even suggested by advances in numerical
linear algebra. Here the Bunch-Parlett factorisation of a symetric
indefinite matrix is used in a Newton-type method which is based on the
use of directions of negative curvature. In anticipation of the exten-
sion of these ideas for use in a quasi-Newton method, we present and
analyze a method for updating this matrix factorisation.
In this chapter the problems which shall be considered are
introduced and motivated. Chapters II and III are concerned with the
updating algorithm and should be considered as a unit. On the other
hand, Chapter IV is meant to be self-contained. For this reason some of
the same concepts are introduced in both places. The numbering of equa-
tions is done separately in each chapter. For ezssle, a reference
within a chapter to equation (2.1) means to refer to the equation
numbered (2.1) which will be found in Section 2 of that chapter. When-
ever there is a cross reference between chapters it will be explicitly
mentioned.
I
2. Newton-type Methods for Unconstrained Optimization
One of the major problem areas of numerical analysis is the
minimization of a non-linear functional. If we denote the n-dimensional
real vector space by Rl and the real numbers by R, the problem is:
given a domain V c R and a functional
f: P+R
find a* sP such that
f(z*) < f(x)
for all x P.
Usually the task of trying to find a global miniamn of f is too
difficult numerically, and we eust be content with finding a local
maim. for f. That is, w seek-
sach that
f(W) I f(z*)
for all a a N(s*) c D where N(z*) is som neigbborbood of z*.
Let
*1 f)
S() *2 f * f()
be tbspgdiet of f at s, sod.let
3
a f (x) 812f ()...' in f (x)
*2 1 f(x) 322 f(x)....a 2of(x)
o f(x) *,gf(x)...8f(x)
-Vf(x)
be the Hessian of f at x. for a sequence {xk} we shall write
fk ()k
Sk-S(xk) .
0 k " a(xk *
Assume that f has two continuous derivatives on D. Then the
Hessian matrix is symmetric, and for any x, s D we have
f() - f(x) + S(x)t(u-x) +Zi-x)taG(x)(i-x) + o(I-x1 2) .
We write h(e) " o(s) if hs.O0. Ths f is modeledvwellccally
by the quadratic form defined by the first three turns of its Taylor
expansion about x. If the Hessian G(x) is positive definite then the
quadratic form
f(z) + t(x)(%-) +1 t(x)(i-x)
has a minimum at
(2.1) i x - 0"1N)g().
7oranla (2.1) suggests the iteration
(2.2) Given x 0 sV
for b0,l,2...
xk+1 0 "k+ c
4
This is, of course, the well known Newton's method for finding a zero of
the gradient g(x). Thus Newton's method can be viewed as minimizing the
local quadratic model of f and also as attempting to find a point x*
which satisfies g(x*) = 0. This is important since
(2.3) f has a local minimum at x* only if g(x*) = 0.
This method has two important properties that make it a very
powerft'.L tool for the solution of unconstrained minimization problems.
The first of these is the basic simplicity of the iteration (2.2). The
second and most important property of Newton's method is the local
quadratic rate of convergence of the iterates. Loosely stated this
means that when the iterates xk of (2.2) converge to a point x* with
G(x*) nonsingular, then eventually the number of significant digits in
the approximant xk doubles at each iteration. The more precise mathe-
matical statement is contained in the following theorem. Before the
theorem is stated it will be necessary to introduce the notion of a
point of attraction. A point x* is a point of attraction for the itera-
tion (2.2) if there is an open neighborhood N(x*) c P such that when
x0 t N(x*), the iterates defined by (2.2) all lie in V and converge to x*.
Theorem (2.1)
Assume that g: V c R + R is continuously differentiable on an
open neighborhood N(x*) c P of a point x* cP for which g(x*) - 0, and
G(x*) is nonsingular. Then x* is a point of attraction of the iteration
(2.2). If, in addition, there exists a positive constant L such that
10(x) - G(x*)I < LIx-x*I for all xt N(x*), then there exists a positive
constant C and a positive integer K such that k > K implies that
5
Nxk+l - x*I < cIxk - x*112
A proof of Theorem (2.1) can be found in [17, p. 312].
There are some major difficulties in implementing Newton's
method in its basic form. The first of these difficulties is that there
is no reason for the Hessian to be positive definite at an iterate xk
which is far from a local minimum. Another difficulty is that the step
sk predicted by the quadratic model at xk may be too large or too small.
These difficulties have led to several modifications of Newton's
method. Many of the modifications have taken the form
(2.4) Given x0 V
for k 0,1,2,...
AGk " Gk + Ek
AA
xk+l ="k+ aksk*
The symmetric matrix Ek in (2.4) is chosen to insure that Gk is positive
definite. This implies that the direction sk satisfies
(2.5) tSsk < 0
Thus the directional derivative of f at xk in the direction'sk is nega-
tive and the function must decrease initially in the direction sk. A
direction sk that satisfies (2.5) is called a descent direction. Once a
descent direction sk has been specified it is possible to determine a
positive number 5k such that f(xk+ sk) < fk'
Of course, the particular way in which the matrix Ek and the
6
scalar 0k are determined are crucial in analyzing the convergence of the
iteration (2.4). Some success has been achieved with iterations of type
(2.4) in specifying Ekok in such a way that the iterates xk are globally
convergent to a critical point x* (i.e. a point x* with g(x*) = 0).
Whenever possible these algorithms reduce naturally to Newton's method
so that the local quadratic rate of convergence is retained.
However, work in this area is not yet complete. In particular,
no algorithm has yet been given which can guarantee global convergence
to a local minimum. How can Newton's method be modified so that the
resulting iterates converge globally to a local minimum for f? In
attempting to answer this question, we have developed an algorithm which
is different from iterations of type (2.4). This algorithm is presented
and analyzed in Chapter IV. The algorithm is based more explicitly on
the local quadratic model for f in that the Hessian is not modified.
Instead, directions of negative curvature are used in combination with
the more usual descent directions. The resulting iterates {xk} are
shown to be globally convergent to a point x* such that g(x*) = 0, and
G(x*) is positive semi-definite. Thus by basing the iteration more
closely on the quadratic model we obtain an iteration which converges to
a point x* that satisfies the second order necessary conditions
(2.6) f has a local minimum at x* only if g(x*) = 0,
and G(x*) is positive semi-definite.
Yet another drawback to a modified Newton's method is the
expense in terms of both computation and programming effort associated
with calculating the Hessian at each step of the iteration (2.4).
Attempts to overcome this undesireable feature have led to a great deal
7
of research in a class of methods called quasi-Newton methods. These
methods replace the Hessian Gk with an approximation Bk. A quasi-
Newton iteration has the form
(2.7) Given x0 c V, and B0
for k=0,1,2,...
Bksk - k
xk+link + aksk
Bk+1 k + Uk
In iteration (2.7)
Uk - U (Bk, aksk' gk+l1gk)
is usually a rank one or rank two matrix with
(2.8) Bk+lsk - Ek+1 - gk
Equation (2.8) is called the quasi-Newton equation. The advantage of
iteration (2.7) over (2.4) is that the only new information required to
obtain Bk+l from Bk is the calculation of the gradient gk+1. The
computational savings is that only n instead ofi2 scalar function
evaluations are required to obtain an approximate Hessian at step k.
Moreover, the task of programing the Hessian is avoided.
The price one pays for the computational savings obtained
through the use of a quasi-Newton method is that the local quadratic
rate of convergence that is enjoyed by iteration (2.4) is no longer
guaranteed. Instead, if the iterates {xkl defined by (2.7) converge to
a point x* where g(x*) - 0 and G(x*) is nonsingular, then
(2.10) hia - 0k'. k
8
under suitable restrictions on (Bk} and G. A sequence 1xkj that satis-
fies (2.10) is said to converge Q-superlinearly to x*. A thorough
account of iterations of type (2.7) can be found in the excellent survey
by Dennis and Mord [7].
Evidently, the linear systems
Gksk = "k
that must be solved at each step are central to the implementation of
these methods. Solving linear systems Ax = b using matrix factoriza-
tions costs 1/3 as much as computing A1 and has been shown to be
numerically more stable than computing A 1. Since the linear systems
arising in the context of non-linear optimization have symmetric coeffi-
cient matrices it- is of great interest to obtain efficient and stable
methods for factoring symmetric matrices.
The advent of quasi-Newton methods has inspired a large portion
of the research in an area of numerical linear algebra called updating
matrix factorizations. Since the matrix Bk+l in (2.7) differs from Bk
by at most a rank two matrix, one might expect that the factorization of
5k+l could be obtained with less computational effort if the information
contained in the factorization of Bk were used. This has indeed been
found to be the case.
The types of quasi-Newton updating formulas that have been found
to be most successful so far have satisfied
(2.11a) Bk symmetric =>k &+i symmetric,
(2.llb) Bk positive definite => B+1 positive definite.
For this reason, there has been much work concerned with updating
9
variants of the Cholesky factorization [9,13,14] of a symmetric positive
definite matrix. No algorithm has been given for maintaining and updat-
ing the factorization of a symmetric (possibly indefinite) matrix. How-
ever, there is at least one promising updating formula that does not
satisfy (2.llb): Powell's symmetric form of Broyden's update [18].
3. The Symmetric Indefinite Decomposition
The modified Newton method that is to be presented in Chapter IV
relies heavily on the factorization of a symmetric matrix given by Bunch
and Parlett [5] and later improved upon by Bunch and Kaufman [4]. One
would hope that the techniques developed for the modified Newton method
could be extended to a quasi-Newton method. As a step towards realizing
this extension, the updating problem for the symmetric indefinite fac-
torization has been studied. A numerical method for updating the
factorization of a symetric matrix when followed by a rank one change
is presented in Chapter II. A detailed error analysis of this algorithm
is given in Chapter III.
As noted above, most of the work in quasi-Newton methods has
been concerned with maintaining positive definite approximations to the
Hessian. Hence the work in numerical linear algebra generated by these
methods has been primarily concerned with updating some form of
Cholesky's method for factoring a symmetric positive definite matrix.
The factorization of Bunch and Parlett does not require that the
matrix be positive definite. Given any symmetric matrix A c Rnxn this
algorithm produces a permutation matrix Q, a unit lower triangular
matrix 1, and a block diagonal matrix D such that
10
QAQt = MDM.
The diagonal blocks of D are order one or two. If we call an arithmetic
operation a multiplication followed by an addition, then the number of
arithmetic operations required to obtain this decomposition isk
3+ 0(n2). (If x - ajn with ak #00 we write x = 0(n2) and say x
k j=1is of order n .)
Another algorithm for factoring a symmetric indefinite matrix
was given by Aasen [1]. In that algorithm one obtains
QAQ - LTL t
where Q is a permutation matrix, L is unit lower triangular, and T is
tridiagonal. This factorization requires n3 + 0(n2) arithmetic opera-
tions also.
Since these factorizations both require n3 + 0(n2 ) operations,
an updating algorithm for obtaining the factorization of a symmetric
matrix A - A+U when the factorization of A is known should require at
most 0(n2) arithmetic operations. Otherwise, there would be no compu-
tational advantage over the alternative of actually computing the matrix
A and factoring the result. The updating algorithm presented in
chapter II is concerned with the following problem:
Given A c Rxn, A= At, z c Rn, a cR, let
QAQ t K be the Bunch-Parlett factorization
of A; let
A - A + azz.
Find an algorithm to compute
QAQ * DM
11
which requires at most 0(n2) arithmetic operations.
This algorithm makes use of the block structure of the matrix D.
We found no similar way to take advantage of the corresponding tridiago-
nal matrix T in Aasen's factorization. At present we do not know of an
algorithm for updating the factorization of Aasen. The updating algo-
rithm that is presented here requires between n2 + 0(n) and 112 + 0(n)
operations. The method is shown to be stable as long as the factor 1 is
well conditioned with respect to solving linear systems. These state-
ments are made precise in chapters TI and III.
4. Computational Results and Conclusions
Chapter V is concerned with presenting computational evidence in
support of the theoretical work described in chapters II, III, IV. The
computations were carried out at Argonne National Laboratory using an
IBM 370/195. All computations were done in double-precision arithmetic.
The updating algorithm has been tested for accuracy and timing
over a wide range of updating problems. We have included timings for
problems of various orders. The accuracy of solutions to linear
systems using the updating algorithm have been compared with solutions
obtained by computing and factoring A+azzt at each step. The results
are very encouraging. They indicate that the bounds- obtained in our
analysis are quite pessimistic and that the algorithm does not break
down even when the updating process is applied over many iterations.
The unconstrained optimization algorithm was applied to many of
the standard test problems which appear in the literature. Although
more work is needed to obtain an algorithm that can be recommended for
12
general use, the initial results show this algorithm to be competitive
with the algorithm of Gill and Murray [11]. In any case the underlying
idea is worthy of further research. It would be of great interest if
the ideas could be extended to a quasi-Newton method and to a con-
strained optimization algorithm.
Chapter II
Updating Factorizations of Symmetric Matrices
1. Introduction
Methods in numerical linear algebra are usually concerned with
the solution of a single linear problem. For example, a particular
method might be concerned with the solution of the linear system Ax - b
where A c Rnxn and x,b a R". Yet in practice we are often faced with
solving a sequence of linear problems which are closely related. For
instance, we may be interested in solving a sequence of nxn linear
systems
(1.1) Akxk - bk
A+1 - A + Uk
In many cases of interest Uk is of low rank. Often the rank of Uk is
one or two.
Direct methods for solving the main problems of numerical linear
algebra have come to rely heavily upon the use of matrix factorizations.
For full matrices the price (in terms of arithmetic operations) of such
factorizations is generally substantial. For instance, the relevant
factorization needed to solve (1.1) requires 0(n ) arithmetic operations
for each Ak. However, when Uk - Ak+1Ak has low rank, one might expect
that the factorization of A,+ 1 could be computed in an order of magni-
tude fewer operations using our knowledge of the factorization of A.
For example, in (1.1) we would aia for algorithms which require only
0(n2) arithmetic operations.
Nere we shall be concerned with factorisations used in solving
13
14
the problem (1.1) when the matrices Ak and Uk are symmetric, and where
each Uk is a rank one matrix. Then (1.1) has the form
(1.2) Akxk - bkkJ1,2,...
Ak+l - Ak + okzkz
where each zk c , k E R, Ak - A. This problem arises for instance
in quasi-Newton methods for optimization problems [7].
Thus we shall concern ourselves with obtaining the factorization
of
(1.3) A - A + ozzt
not by forming A explicitly, but by using the factorization of A. Such
a process is called updating a matrix factorization.
There are two important and very distinct cases:
(1) A is positive definite,
(ii) A is indefinite.
In case (i) A may be factored in a numerically stable way into
A" =LDLt,
where L i Rthxn is a unit lower triangular matrix, and 3c R xn is a
diagonal matrix with positive diagonal elements. To pivoting is re-
quired to obtain numerical stability in the positive' definite case.
However, in case (ii) such a factorization may not even exist. For
example consider the matrix
Sn 11 0*
A numerically stable method for obtaining a factorization of A in case
15
(1 ) is given in [5] by Bunch and Parlett, and is later revised in [4]
by. Bunch and Kaufman. By this method one obtains a permutation matrix
Q, a lower triangular matrix H, and a block diagonal matrix D such that
(1.4) QAQt MDMt.
The diagonal blocks of D are order one or two. Whenever Di+ 1 .i 0
then M+ 0. Also, MH = 1 for all i.
The case in (1.3) where both A and A are theoretically known to
be positive definite has been studied and updating algorithms are given
in [9,13,14]. The case where A and A are symmetric but possibly indefi-
nite has not been studied.
In the following sections we shall present and analyse an algo-
rithm for computing Q, M, D, when given the factorization (1.4), suchthat
(1.5) QA t -ME ,
where A is given by (1.3). The algorithm requires between n2 + 4n and
1,2 + 5 + arithmetic operations and at most 2n comparisons. Here
an arithmetic operation is considered to be a floating point multipli-
cation followed by an addition. Divisions are counted as multiplica-
tions. The operation count compares favorably with the alternative of
computing A + azzt and then factoring this matrix into . This
would requiree i2 + n multiplica.tons together with 2 additions to
form the new matrix. It would then require at most
jn3 +4 2 +j
operations to obtain the new decomposition. Therefore, a total of at
most
16
operations would be needed.
Thus it is advantageous to use the updating algorithm whenever
n > 10. However, .t should be emphasized that the upper bound on the
number of operations required by the updating algorithm is a worst case
bound. Computational results indicate that the worst case seldom
occurs. Therefore, we expect that in practice the crossover number
would be much smaller.
2. Description of the Algorithm
We shall begin by describing a basic algorithm with no pivoting.
The algorithms given in (9,141 for the positive definite case will be
presented as modifications to this basic algorithm. The modifications
were designed to insure numerical stability. The algorithm we present
for the indefinite case is also a modification of this basic algorithm.
However, it is necessarily more compliested since the pivoting mast be
updated.
Assume for the moment that no permutations were required to
obtain
A. UIt
with K (block) unit lower tr.taugular, and D block diagonal with one-by-
one or two-by-two diagonal blocks. TI we may write
Am danDt Iah,
where the D~ are the diagonal blocks of D and the N are the block
17
columns of M. Let -
A - A + ozz,
and let Hp - z. Denote
(~k) -jm ~ C)- I MDMtw = M
A ) pjk JJ jk
where pt" (pt t
Suppose that D e D + ap p is non-singular, and let
Dlbl op1.1 Then take M1 -M1+w(2)b1. Note that only the elements
below the identity part of M1 are altered;
x =+ ,
x .
where the x's and y's denote possibly non-zero quantities. We have that
(2.1) A " N1(D1 + ap pt)H + a lp w( 2)t + w(2)t1 4 )
+ A(2) + aw W(2)t
(M + v(2 )bi)D 0( + w(2)b )
+ a(--pc w(2)t + w2p )
D b (2 )t + w(2)b -
+ (a - b1 b)w(2)w(2)t11
+ A(2)
"re h Da t i + A(2) + a'w(2),(2)(t
Observe that the matrix A(2) + a'w(2)w(2)t hate fors
18
r0 0
and thus we may recursively apply this procedure to obtain
- m
A = IM~M_
j-1
as long as Dj=Dj + a p pt is non-singular for 1 < j < m. This assump-
tion on D is theoretically always satisfied in the positive definite
case. However, this cannot be guaranteed in the indefinite case. After
establishing some preliminary results concerning these computations we
shall discuss some of the numerical algorithms that have been proposed
for the positive definite case.
Lemma (2.1). Let D and D + appt be non-singular. Then the solution to
(i) (D + appt)b = ap
is given by b = OD lp, where 6 - a/(1 + aptD p). Moreover,
(ii) det(D + app ) - det D(1 + ap D p) and the
updated a' - a - btDb
is given by
(iii) a' - a1 + aptD p
Proof.:
(1) follows by substitution,
(ii) Sherman-Morrison formula (or direct computation),
(iii) follows by substitution.0
19
Thus if all the D (for 1 Ji j m)- are non-singular we have the formula
a for 1 <j <m ,1 + PjDj pj
and k
11 det D
ak kk f det D
hence det.AdeA
In the case that both A and A are positive definite, these formulas
point out the necessity of maintaining av with the same sign as a. We
note also that we may recursively compute t - as follows:
(2.2) t + l t + p D P ,
and we have the relation
t det D(2.3) j = -
t det D
N1 iWhen A is positive definite, dj- det D , d mdet D and
D - diag(d1,...,dn).
Now, often in practice one knows theoretically that the matrix A
should be positive definite when A is positive definite. In the case
that a is positive there is no difficulty since the recursion for the
t 's yields an increasing sequence, and d - d . Thus the d are
all positive and d> d . The following algorithm results for a > 0:
20
(2.4) t1 = a1, w -z, A MDMt
for i= 1 step 1untilndo
(1) p = w
(2)tl - i =p /d
(3) di = d (ti+l/ti)
(4) terminate if i = n
(5) bi = (pi/di)/ti+l
(6) w(i+l) - pii
(7) N H=H + biw(i+l)
Note that the number of arithmetic operations required is n2 + 0(n),
since only i operations are needed at steps 6 and 7.
Difficulties arise when a < 0 because round off error may cause
a t 1 to be positive, and hence di will be negative indicating that the
computed A is not positive definite.
Two remedies have been proposed. One of these [9] is to compute
the vector p such the Mp = z at the outset. It is noted that in the
application to quasi-Newton methods, the vector p is often available
anyway. If a < 0 then calculate the ti, for 2 < j < n+1 from the formu-
la (2.2). If one of the t should turn out to be positive then the t
are recalculated using
(2.5) tn+l = /cl ,
t - tj+1 - p2 /d , j=n,n-1,...,1
where a is the relative machine precision. These new values of t are
then used in place of the old ones in (2.4), steps 3 through 7. The
effect is to replace a by t1 which gives a problem that is close to the
21
original problem, and for which the computed A will be positive definite.
In [14]. another approach is taken which yields a similar algo-
rithm. The major differences being that tn+1 is set to c if some t is
positive and a backwards recurrence formula is used to compute H. Thus,
in place of (2.4) steps 6 and 7, we would have
(2.6) w(n+l) _ 0
for i - n step -1 until 1 do
(1) wi) -p
(2) Mi - M+ biw(l+1)
(3) w -w(i+l) + piMi.
However, there seems to be the need for additional storage in
(2.6). Note that the computation of w requires knowledge of Mi which
has presumably been overwritten at step (2) of (2.6).
In [9] an error analysis of this process has been given. That
analysis shows that
MDHt - A + zzt -" E ,
where the elements of E have first order terms in c which depend on the
ratio idin (2.4) step 7 is used. However, it is possible to show
that
(2.7) -M Hi(di/di) + biw~i)
and here the error terms depend on the ratio d which is less than 1
when a > 0. In both [9] and [14] one switches to (2.7) only if the
ratio 12 1/d1 becomes larger than some bound.
This leads us to the following algorithm which is a slight
22
modification of the composite t-method given in [9].
(2.8) (1) terminate if a - 0; put t1 - and w1 z;
(2) 'if o> 0'oto 6;
(3) if p is not available solve Mp = z for p;
(4) for i1,2,...,n do ti+1 ti + p2/d ;
(5) if any t1 > 0then
begin
tn+l - /aI;
for i=n,n-1,...,1 do t- = ti+1 - p /di;
end
(6) for il= 1 step_1until n do
S> 0 thenbe ini WI ; ti+ 1 ti + pi/di; end;
eg ti+l/t ; = - diei;
terminate if i = n
bi (pi/di)/ti+1;
IL 6e 4 then
Yi = t /ti+l'
-Yix+bw ;(i)
end
else
V(i+l) (i) - peH;
nMd + b(i+l)
"nnlnd .
23
The situation becomes completely different when the matrices A and A are
not assumed to be positive definite. In order to obtain a stable algo-
rithm for solving Ax = b pivoting must be used to factor A [3,5] and we
obtain
QAQt = MDMt
Moreover, the following example shows that Di in (2.1) may be singular
even though both A and A are non-singular.
0 1 0
Let A = 1 0 0 (= MDMt, where M=' I and D = A),
0 0
let a 2 , z = (1,-l,l) .
N 0 1 f 1 2 2
Then D = + '[1,-1] = 1is singular but1 0 -1 .2 2
A = A + azzt satisfies det A = - . Therefore, some pivoting strategy
must be employed to avoid the breakdown of the computation (2.1). The
main difficulty in updating the pivoting strategy is maintaining H in
triangular form.
We shall now describe the pivoting strategy given in [5] for the
Bunch-Parlett factorization in some detail. This strategy will be used
in a portion of the updating algorithm, so we include its description
for the sake of completeness.
Given a symmetric non-singular matrix A with elements aij the
factorization proceeds as follows:
Let 0 < < 1 be fixed.
Let v = max lail and let p - max lai| .1<1< i i0 j i j
24
If v > ap, let i be the smallest index such that taut| - v. Let Q be
the identity matrix with rows 1 and i interchanged. Then the matrix
Q1AQt has the element aij in the (1,1) position. The first step of the
factorization is to write
1 0 al 0 1 0 t
(ajl I 0 A'-al vt]V6l I
Thus
H1 Q1Ac4H -t - 81 )- AQ A(2)
where ,and A(2) A' - tw . If v < up, let i > jvl I
indices such that Iai j - . Let Q be the identity matrix with row i
interchanged with row 2 and row j interchanged with row 1. Then the
matrix Q1AQt has the element ai in the (2,1) position. In this case
the first step of the factorization is to write
Q1AQ1 -V As
I D I O t
VD-1I 0 A V~lt Vl I
Here V is the first two columns of Q1AQt below the (2,1) and (2,2)
positions, and D is a two-by-two matrix.. Also, dot -D uaua -
ai < (42-1)p2 < 0. Thus
-1 t-t - D 2))0
25
where H1 , :1 and A(2) At-VDlVt.
LD I
The factorization now proceeds by applying the same pivoting strategy to
the reduced matrix A2). The end result is that
ml1 -19k-1 ... - QAQ ... Qk - *ct 1 = D ,
where D is a block diagonal matrix with lxl or 2x2 diagonal blocks.
Hence,
A = Q1M 1 Q2M2 tsttkMkDMtQ ... tM2Q2M1Q1.
Since Qila Qifor i < i < k we may write
Q M Q2M2''Qk~ Qt12'''k
where Qt - QQ2''Qk,
and M = Qj +1Qj+2 ''QSYjQk k'Qj+2Q j+1
Then H has the same form as M and thus if we take
Mja MaM2 '''
then M is a block unit lower triangular matrix such that
QAQt aM t.
The number a is chosen to be (1+/7)/8. For fixed a, 0 < a < 1,
the strategy just described shall be called the diagonal pivoting
strategy S,. This factorization is almost as stable as Gaussian elimi-
nation with complete pivoting [3,5]. A modification of this strategy,
which is comparable to Gaussian elimination with partial pivoting, is
given in (41. The algorithm in [5] requires between-3 + 32 +
and 3 12In+n 2 + -3-ncomparisons, while the algorithm in [4] requires at
most n2-1 comparisons.
26
Now, in order to establish the theorem that we shall use to
construct the algorithm for updating this factorization shall need some
preliminary lemmas.
Lemma (2.2). Let AE R 5'" be symmetric with eigenvalues A 1< A2 '...' A1 2 n
and let A - A + azzt for some z e Rn, o e R. . If a >0' then A has eigen-
values Ai such that
while if a < 0 then the eigenvalues of A can be arranged so that
Al<A1 (** ... A .
Proof: [20] pp. 95-98. 0
Remark: In particular if A is non-singular then at most one of the Ai
is zero. -
With Lemma (2.2) and the, pivoting strategy just described we can
establish
Lemma (2.3). Let V = (,), where I E and V' E Rkx'. Suppose that
D = D E K is non-singular and that w ecR ,aE R. Define
(2.9) C E VDVt +wt.
Then there is an txt permutation matrix Q such that
(2.10) ( C(VVt + (vw)B(vw)t,
Wggrwhere
(1) 1 is a non-singular block diagonal matrix with lxl or 2x2
diagonal blocks,
(ii) V is block unit lower trapezoidal,
27
(iii) v - lJVEvRE ,
v
(iv) 1=iI, w c Rk ,and
(v) B E R2x2
Proof: Write
(2.11) C - (Vw) D : t0 0
I 0 D Us I 0
V' w' yet 2o) V' w')p u
where
w t=,w 1 E R, w' - w2-v 'w2
D - D + w Ow,
and a = ow .
Here u may be any positive. real number; if p is chosen small enough then
the diagonal pivoting, strategy Sa will give either
Q 0 t us Q0(2.12)r':( ff
(0 1 a pa 0 1)
OM D 0 1 M O
pbt 1 )0 92o')pbt1
or
0
I
0
0
0
1 ) 0
0
0
0
0
0
a
0
01
J2a'
- t
m
llbt
o o t
1 0 ,
01 J
where D is block diagonal with ixi or 2x2 diagonal'blocks, and M has the
corresponding block unit lower triangular structure. Since D is non-
singular, Lemma (2.2) implies that D has at most ore zero eigenvalue.
The diagonal pivoting strategy preserves the inertia of D. Therefore, D
is non-singular. Note that 8 = 0 in (2.13) if and only if D is singular
and in this case we cannot carry the decomposition further without per-
muting the last row and column of
(r ;
If (2.12) is obtained then
[Q o (Qt)0
0 I 0 I
Q 0 Qt 0 Q 0 D y Q Qt0)
O I IV'Q u ' 0 1 ust y2c 10 1 )V'Qt uW
I 0 K 0 D 0 M 0 t I 0
LV uw' pbt 1 ) 00 ') bt 0 ) V -w'K I 0)[D 0 M 0 t
VM +w'b w' 0 a' V + w'b w', hr
1 0 Q 0 QtI
V ) 0 I V' Qt V',Qt
28
29
Here we take
V = ~ , and w w'.
VM + w'bt
If (2.13) is obtained then
Q 0) Qt
00 I 0 I)
Q 0 Qt 0 Q 0 ID
" 0 I JV'Qt v' 0 1 us
I 0 01 0 0 0
V v-w'
ubt 0 1 0
I+ vt +w'bt
01
vi
0
W'
0
0
0
0
us Q t0
u 2 0
0 01-
0 0
001 (0
i9i j2 yb
0 0
VM
l 1'
Q 0 Q
, Qt , v 0 I)
0 0
1 0
0 1
t
IV
I+ vt+ W'btl
0 0
1 - '
01
vi
0of
v'
e0 Q 0- (I
where V v 1 l t - .1VQ
v _j, where cRk
and we take
.Vlr + va + 'b
This gives the desired result.
Observe that the scale factor a does not actually euter into the
here
0
i
a a -
a = w= '
30
computations and thus explicit scaling need not be implemented in a
code. Also, we note that for the intended application, we will have
< 3 in Lemma (2.3). WhenL < 3 we have
C (Vw) : (Vw)t0
with D of order at most 3. Then in the computations non-singular
matrices of order at most 4 are inserted between the factors on the
right and permutations are used to obtain
( Q 0 Q t( 0 D )( v -) t
0 IC 0 I -( ~ )0 1( vw
Thus, some fixed number of arithmetic operations are required to compute
D and B. Also, some fixed multiple of k arithmetic operations are
required to compute V.
Before we give the main theorem of this section we shall need to
establish one more lemma. The proof of the lemma is trivial but it is
included for the sake of clarity in some of the following computations.
Lmna (2.4). Let B [: :) x2,where.0 f0. ThenSo
S 1 -+Y A1 0 1
y 1 0 A -Y 1
with 1111 >I A|1
Pgoof: Let be the eigenvalues of B with lull | >1u 21. Since
R f 0, 3 has an eigenvector corresponding to p 1 of the fors (1. Thus
t: . 1 1
8 e y "pA
31
Therefore, 6+0y = and since 0
we have y - (U 1-0)/0.
Since B is symmetric it has an orthonormal system of eigen-
vectors. Thus
B-Y 1 0 AZ -Y 1)
where A - )-/(1+y2), 1,2.0
The following theorem will show how Lemma (2.3) can be used to
obtain an 0(n2) updating algorithm.
Theorem (2.1). Let A e R"n be non-singular with QAQt - MDMt. Suppose
that z c Rn, a c R are such that
A- A + zzt
is also non-singular. Then .A -t can be computed from the factor-
ization of A in 0(n2) arithmetic operations.
Proof: Let w = Qz. Then QAQt HDMkt + rww. We denote
I(k) IMD Mt, A(k) _ IHD t . First we may write
j=lj jj-kj
)QAQt : D 0)( w +A(2) C(l)+A(2)
V w2 0 c V w2
( ) , D . Dl. Then by Lemma (2.3) we may
V 2construct a permutation U1 such that
(i)(
v w v
or
0
1
v1
0 0 o
w' w'
or
I I t 0
1 1 D + wt' (0w,t
where dy0 0, D1 E R2x2 is non-singular, B E R2x2, and a' e R. Observe
U(2)also that [1 )A(2)( - A(2) If (iii) is achieved then the
0 "I 0 I
problem becomes
QoA4 - A() + A(2) + '
where Q1 (1 Q. Note that0 I
A .(2) + C' (0,w')
has the same form as the original problem but the dimension of the prob-
lea is decreased to n-1 or n-2.
In the following discussion we shall drop the primes and sub-
scripts from the expressions on the right of (2.14). Also, some of the
qualities appearing in (2.14) are redefined below.
If (i) holds in (2.14), then
32
(2.14)110 C(1) U1
L0 I 0 1(ii) (d)
v2 v2
33
wh l if ii1ho t h2)v w )v w)
while if (ii) holds then
0
Let I =M2.
0
Q1AQt ( + 1
Then we may write
(i) C(2)
Q1 AQ1 or
(ii)al)
0
0
w
0 o
B 31 + A(2)
- w
+ A(3)
+ C(2) + A(3)
In (i) we have
(2.15) c(2) V[a W 1k !)(: 2)(
v2 w2V 0 D2 v2 w2V
v w
and a similar expression in (ii). Here v - and w ( have
v2 w2been partitioned so that Vv1 and Vw1 are defined.
Now if (i) or (ii) occurred in (2.14) then
a s
B 8 3satisfies 181 < ajI|. Hence by Lemma (2.4)
BiY 1 J0 7Z 2
with .|Al I 1 12 1. Moreover,. A # 0 since A1 = 0 implies that B . 0.
Let 1
vi
v2
Using this express
(2.16) C
S1 0
with D
V I
Lemma (2.3) may be
w 1 0
w2 t2 w2 Y
ion in (2.15) gives
( v I w j 0 D2
v2 V 2 0
1 O|wO D 0(01Iw I0 v2 2
71
10 0 1 0 tnon-singular
0 D2 to I
applied to obtain
0 1 0 w o0 vl I w
A2 v2 V V2
1 01w0
0 I w
2 2
and v 2 = v2 -V1.
0 0 0 0t
1 0 B 1 0,
(:2 :)C2)(: ) ( ) ) + or
o
0 I V VQ' O(Oot)
-I
We take and . Case (ii) of (2.14) is similar. This
process may be continued until the full updated factorization has been
attained.
To see that only 0(n2) operations are needed, observe that a
small fixed number of arithmetic operations (bounded by b say) are re-
quired to obtain a new diagonal block. Manipulating the columns of the
triangular matrix M at step k requires some fixed multiple of (n - k)
arithmetic operations (bounded by a say). Thus, there are at most
34
It
Now,
35
a(n + (n-1) +...+ 1) + bn = an(n+1) + bn
arithmetic operations required. o
We remark now that the implementation does not actually rewrite
C(2) as in (2.16). Instead, (2.15) is written as
1 0 0 - a 0 s 1 0 0 i
(2.17) C(2)uj vi I wl 0 D2 0 )v Iw
22B1o1v2 V 2 0 O a v2 V 2
1 0 1 0 D b 1 0, 0 t
0 I10 .0 1 I 0-- - b a ---- l--
v 2 VI w2v 2 V 2
Db1 0 0 8 00 1 0 0 t
where b v I w 0 D2 0 v1I1 w .
b 0 0 1 -. O a 0 0 1
Multiplying the matrix factors and then equating matrix elements
will show that D in (2.17) is equal to the matrix D appearing in ex-
pression (2.11) of Lemma (2.3) if we had first obtained (2.16) and then
applied Lemma (2.3). After this form of C(2) has been obtained, the
factorization may proceed as described in Lemma (2.3).
We are ready now to give an Algol-like description of the imple-
mented algorithm. Some of the details have been left out for the sake
of simplicity. The most notable of these omissions is that when updat-
ing a diagonal block Dk we may obtain two lx1 blocks instead of a 2x2.
The explicit bookkeeping involved is not present in this somewhat
simplified description.
In the following description of the algorithm we shall make the
following conventions:
36
(1) The expression a : = b means b overwrites a.
(2) D will be a matrix of order at most 4. An expression of
the form D : - will mean we have increased the(0 a
size of D with elements defined as indicated. Similar
remarks will apply to the arrays V and B.- 0
(3) At step k, w will always have the form w = w1 ,where
w2
wvhas 1 or 2 components whenever Dk is lxl or 2x2
respectively. Matrices Q and Qk are permutation matrices.
Let QAQt I= M D ta, z be given. Let 0 < a < 1 be fixed. Thej-1jiii
following algorithm will compute MDM and Q such that
Q(A+cazzt)Qt = MjD M .j=1
(1) begin
v : =Qz; k : -1; j :-1;
tD1 +awlw ow
D := _ ; V : Ml; w:=w-Mw ;
(2) Ll : comment decompose D as described in Lemma (2.3);A 0wA11 _D
D = M
0 1 0 1 0 B
N A
D : =B; Dk
37
S: a order of kj;k :k+1;j :-j+ ;
D : - Dk + w w1
if (j - n and D is lx1) or (j = n - 1 and
D21|a> max(ID11 , 1D 22 1)) then to QUIT;
A D ow lD : - ; w : - w - w;V V:
Update Q with Qk;
L to LI;
end;
(4) if B is 2x2 then
begin
Q k 0 Qk 0-(MkV,w): 0 I(Vw)0 M
D : - D; t : - size(Dk); k : - k + 1; j : - j + t;
Sj > n then begin Dk:'- B ; o_to QUIT; end;
B 0 B2811 0 821
D : - 0 D k 0 ;
k21 0 B22
L -w11(V,.w) : - (V, K Jw) comment where (V
0 1 mn
S L w LiL;0D : "1 o D;
1 w1
to t~o.Ll;
end
QUIT:
end.
._
SL
,Mk) i
38
We refer the reader now to the brief flow diagram (Al) describ-
ing the pivoting strategy and to the program listing (A2) in the
appendix. The operation count that follows refers to that particular
implementation. The results of the operation count are given in Table 1.
By an L-step reduction we shall mean that t columns of H andthe corresponding diagonal blocks have been completely determined by the
algorithm we have just described. Operations at step-k which are
carried out on columns of M or the vector w contribute to the 0(n2) por-
tion of the operation count and will be referred to as operations of
type-A. Operations needed to update a diagonal block will be called
operations of type-B and they contribute only to the linear term in our
operation count. We shall consider an operation as a multiplication and
an addition; with this convention we are ignoring the important contri-
bution of interchanges to the 0(n2) term. The paths cited refer to the
flow diagram (Al) in the appendix.
Table 1 needs some explanation. The counts given under the
heading "path 1" refer to a successful one-step reduction without enter-
ing paths 2 or 3 (see Al). The operation counts given for paths 2 and 3
include those cases which begin with path 1 and end in paths 2 or 3.
Table 1Operations Required at Step k
type-A
type-B
comparisons
reduction
Path 1 Path 2 Path 3
4(n-(k+1))m _ 10(n - (k+2)) + 2 < m <- (n -(k ))+1 +211tn-( k+ +))2 ~
5 15 < m<19 42
1 4 6
1 2 3
m denotes number of operations.
39
The best possible situation occurs when Path 1 is taken at each step; we
then have that the total operations required are
n
S - = 2(n-j) + 5(n-l)j=1
= 2 n(n-l) + Sn
Sn2+ 4n
The worst possible case will now be considered.
Suppose
path 1 is taken for k = j,''.,k
path 2 is taken for k = 1 ,... ,k;2
path 3 is taken for k = ml,...,mk3
where n = k1 +. 2k2 + 3k3.
The total number of operations contributing to the n2 term is
then bounded by the sum S, where
S= 2(n-j1 + n-j2 +...+ n- jk1
+ 6(n - (Z1+1) + n - (L2 +1) +...+ n - (k l)) + k22
+ 11(n - (m1+2) + n - (2+2) +...+ n - (mk +2)) + 2k33
= 3(n -j 1 +n - j2+...+n- jk1)
+ 3 (in - t + n - (L 1+l)} +...+ {n - k2 + n - (c, +1))2 2
+ 3({n - ml + n - (ml+l) + n - (m1+2 )} +0.
+{n -m ,+n - (m +1)+n - (m +2)}) -2k 2 -7k 3
+ 2(n - (m1+2) +...+ n - (um +2))3
- (n - jl +...+ ' - 1k~ *1
40
Thus
k- n-2 +kkS n2 - 2 +2k 3n-2 3 m + ii-kn- 2k
=11 2 -1k3
Now,
k k
2k3n - 2 3 m < 2kn- 2 3 (1 +3(1-1)) - k3(2n-3k3) +k 3i=1 i11-
and
k n-k_12- ki I j -kn=- k +2k .
1 jun-k+l1
Thus
S < n2 - n + k3(2n-3k3 10) -Z k1 + k1 - 2k2
<2n 2-Z n + 3n-5) 2
11 2 29 25- n-6n+ ,
where we have maximized the expression k3(2n-3k3-10) over {k3 : k3 >0}.
The analysis is not valid unless n > 5.
Let us divide the counts for type-B operations- by the corres-
ponding reduction at step k. An upper bound for this number in the
worst case is 14. Thus 14n is an upper bound for the number of type-B
operations needed. Therefore, the worst case operation count is bounded
by
6n2 - 3 n+ 3 +14n
"- n2 +56n+ 3 25
The maximum number of comparisons needed is 2n.
41
3. A Pictorial Description of the Algorithm
In the last section we gave a formal description and proof of
correctness of an algorithm to update the factorization of a symmetric
matrix. The main difficulty in obtaining this algorithm was updating
the pivoting strategy while maintaining the triangular structure of M
and M.
The following diagram represents the algorithmi at step k.
I It I IA(k) C(k) A(k+l)
Figure 1Pivoting in the Updating Algorithm
In Fig. 1, A(k) represents that portion of the factorization of
A obtained up to step k. C'k) represents a working array that involves
information from the vector w and at most three columns of M. ik+s)
that portion of the factorization of A which has not yet been con-
sidered. From this diagram we see that the pivoting effects neither the
triangular structure of H that has already been computed nor the
triangular structure of that portion of M which has not yet been
42
considered.
One can also represent the operations on the elements in a dia-
gram. In the following, d's will represent elements of the diagonal
blocks Dk of D, m's will represent elements of the M.'s which occur
below the block diagonal of identity matrices in the matrix M, and w's
will represent elements of the vector w s Qz. Let a and A k) be as in
Section 2, and we assume that A(k) is in factored form, so only the
lower triangle and diagonal D need be stored. A "~" over an element
means that some operation has altered this element. If a 0 appears then
that element has been "zeroed out" and is not subject to further
alteration.
Only those elements that need to be stored are represented; the
elements known by definition are left blank; we store the diagonal
matrix D in place of the identity matrices in M. Row permutations are
denoted by ( orE- column permutations are denoted by L .The permutation matrices Q are not explicitly represented.
We shall illustrate the algorithm with a 5x5 example.
Case 1:
D1 is lxl,
(1) d 0
m w
A m A(2) + o w [ "
m w
m w
D is computed
(2)
and found to satisfy the
d
S m A (2)A#3
pivoting criteria,
0
+ /V+ w V
V
Case 2:
D is 2x2,
(1)
A-
D has been compu
(2) Comp
d 0
d d 0
m 3 +0 w [ ].
m A (.)w
M m w
ted and does not satisfy the criteria for a 2x2 pivot.
D1 0 Q 0ute Q1DQ - [ ,) (M1,V) -= [l
0 d 0 I
d 0
Cm d 0
- 3 3 +a w [ ].~ (2) ~m m A w
a m w3f
We are finished with D and are ready to apply the algorithm to the
diagonal element in the second position.
43
I.
0 I 0 1
44
Case 3:
D 1 .is 1x1 and does not satisfy the pivoting criteria with D2 also 1x1.
(1) d 0
m w
A- m m +a w (1.
m A w
a a w
(2) Compute (M1 , M2) m (M1 , M2)H, 40
N
N
a
U
U
U
I.
)D2
0
0
IV[ I
w
w
i.
(3) Apply Case 2.
Case 4:
D1 is 1x1 and
(1)
does not satisfy the pivoting criteria, and D2 is 2x2.
d0
w
d d +0 w
SA (3)wa a a V
Is
I
45
(2) Compute (M1 , M2) _ (M1 , H2)L, L( D1 )Lt
0 D2
Nd 0
d d 0
A a a a + oa 0 I.
Mm m A( 3 )
N At0
(3) Compute Q1, M such that Q1DQ- 1
0 D2
In (3) the diagonal pivoting strategy could have produced
several different block structures forD1 and D2 depending
on the matrix D. We only show the case is 2x2 and D2 is
1xl.
0.d a0m m 3 A +Q 0
0 I 0 I ;-Ow Ow m A(3)w
m m m w
(4) We are finished with D and are ready to apply the algo-
rit a to the diagonal element in the third position.
1 .
Chapter III
Error Analysis of the Updating Algorithm
1. Introduction
We have updated the symmetric indefinite factorization of
(1.1) A = A + azz
in order to solve the linear system
(1.2) Ax - b .
A method for solving (1.2) is considered to be stable if the computed
result xc satisfies
(1.3) (A+E)xc = b
where PEI is small compared to I|AI. (191 is the matrix norm induced by
a vector norm on Rn which we also denote by II-|.)
The following analysis is influenced by the error analysis of
the diagonal pivoting method given by Bunch [3]. The solution to (1.2)
is given in four steps: -.
(1.4) (i) . - HDt (update the decomposition),
(ii) Mc - b (find the new right-hand side c),
(iii) Dy = c (solve the 1x1 and 2x2 systems),
(iv) Mtx s y (obtain the final solution x).
We have presented an algorithm that is algebraically correct for
obtaining (i). There are standard methods for solving (ii), (iii), and
(iv). However, in finite precision arithmetic error is introduced at
each of the steps (i) - (iv).
46
47
Instead of obtaining the exact decomposition of A, we actuallyA A A A At t
obtain M = (M+AM), and D = (D+AD) such that MDM QAQ + S. Then whenA
equations (ii), (iii), and (iv) in (1.4) are solved, the errors SM1 ,A A
6D, GM2 are introduced at steps (ii), (iii), and (iv), respectively.A A
Thus, we actually compute M, D, c, y, x such that
(1.5) (i) Ilr = - Q(A+azzt)Q + S,
(ii) (H+M 1 )c = b,A A
(iii) (D+GD)y = c,
(iv) (M+6M2)x - y.
Now, M and D are the exact factors of A. Therefore, steps (i) -
(iv) give the exact solution to the system (A+F)x = b, where
(1.6) F= + + A t
+ )(D+8D) [M + (AH+aM 2) t
+ D(AM+6M2)t + S.
In this chapter if x = ale + a2 e2 +...+ akek, where a1 # 0, then
we write x = O(e) and say x is of order a as e-+0. If B is an nxn
matrix with elements bij, then we shall denote B - 0(c)B if
- bj i j(e), where j (e) = 0(e). In the following analysis we
shall obtain expressions of the form
(1.7) (1) (i+GM 1) = 0(e) + G(e),AM
(ii) (AD+GD) = 0(e)D + H(c),
(iii) (AMS2)t = 0(e)M + G(C).
Using (1.7) in (1.6) gives
48
(1.8) F = (0(e)H + G(e))[D + O(e)D][M + O(e)Mt + G(e)]t
+ (O(e)D)[M + 0(e)M + G(e) ]t
- + ND(0(e)M + G(e))t + MH(c)Mt + S
= 0(e)D + G(e)DMt
+ M(O(e)D)Mt
+ MD(0(e)Mt) + MD[G(e)] + MH(c)M'
+ 0(e2 )B + S.
The O(e2)B term in (1.8) is negligible when compared to the
dominant first order terms. The combined terms give
F = 0(e)MDMt + O(e2)B + S,
if G(e) a O(e)1, H(e) = 0(e)D, and S - 0(e)A. Then
(1.9) 11= -=0(e);
hence the method is stable.
However, we shall also see that the terms S, G, and E will in-
volve products of the entries of solutions to triangular systems
involving the original factor M. Thus if M is ill-conditioned,
(l|MII 1M11 is large compared to the number of significant digits avail-
able in our finite precision arithmetic) then the updating procedure
cannot guarantee that the constant in the 0(e) term in (1.9) is of
moderate size.
2. A Detailed Description of the Updating Algorithm
We shall now give a detailed floating point analysis of the
computations performed in our updating algorithm. There are two parts
to a step of the algorithm. An intermediate step of the algorithm
49
results in a sum of matrices of the form
(2.1) wk k (k-1) + Ck + A(k+t)
with
A(k-1) k -1 , A(k+L) ! M D M tj=1 jjjj=k+t jj
w(j)
where atis1 or 2. Let w (aKJfor1 < j < m, where w = M wWO)- j-1j
and [ j+1) [ - Mwi .2
Part 1 of a step consists in preparing the matrix Ck for part 2.
This involves possibly bringing the term Mk+D 6+e into the matrix Ck
and performing certain operations on the factors of Ck to obtain a
special form. Part 2 consists in permuting certain columns and elements
of the factors of Ck and obtaining the updated Mk and Dk'
We shall now describe an intermediate'step in detail.
Part 1.
The previous steps of the algorithm have resulted in
0 0 d 6 0 0 t
(a) Ck = "1 0 R Q 1 0
v(k) (k+l) v(k) w(k+l)J
with 161 aSI|, or in
o [alJw(k)t(b) Ck (k) J
50
We shall now drop the superscripts. If (a) holds then we replace Ck by
v I Ia0ll
I Ik+1 o 0
I Iv
Ckin[
0
0
I
H'
0
0
W2
w 2
I8
0
B
o 0 0
Dk+l 0 1
0 c1 v
0 - 0
D 0 1 2
0 Q v
v2
I I
Ik+1 0I I II IwJ
li
0
0
I
H'
0
0
w1
w2
where we have partitioned
wum[
v1
w 2 ) 9
We then compute
0
1
0
v 2-M'v2
0
0
I
M'
0
0
0
w 2 -M'wM' w2 -m'w1J
6v1 + Sy1
Dk+1 +vv +f(v +w+wv )+wwt
Bv +vw
s
viB + ol.
JNow we proceed
If (b)
to part 2.
holds then we replace Ck by.
0
1
v 2
v2
v2 )
(2.2)Ck a
0
0
M'
0
1
1
V-'
it
where
0
0
I
0
0
0
(D) [ Vla
8
+B1
Mlk+10
Ck
0 0
I wi
M' w2
Dk
0
Dk
0
0 lot
Q I w
0 00
I w ,Q
ml W2
where we have partitioned
wm[V,)
V2
0
Mt
We then compute
0 0 Dk+aww4 wl)
Ck = I 0 aw a
M' W2-M'w
and proceed to part 2.
Part 2.
Part 1 has resulted in a matrix of the
0
0
w
where D is a symmetric matrix of order at most
ing strategy Sa only to the matrix D producing
and a lxi or a 2x2 matrix 1) such that
1 0 0 0
C= 0 Q 0 Ck 0 Q
0 0 1 0 0
3. We then apply pivot-
a permutation matrix Q,
0
0
I
I t
51
(2.3)
0
I
H'
0 wlt
0
w2~ -M1
(2.4)
form
)t
,
0 0 D b 0
Ck " ' I 0 b t I
V w v
52
is given by one of the following forms:
(a)D (8) is lxi, b - (S), and 181 a
0 0 ao 00 o
(2.5) k 1 0 0 1 0
0
with a = v-O2/6. We then take Dk = (d), Mk 1 , and a is
replaced with . Return to part 1. vw(S/d)
(b) qt b(18 21:::),bt = (01,B2) I1iiI 18221, and
I2 aT22
|oi >_ a|l 21|. Then
(2.6) 11CLk 0 BL
0 B
where
2 /8- 21 11a22
s2
Lk =
0
1
8+2/811
yv + 2 O)
02 - (81821)/811
a- 028 J/
101I I101I II 1.1
+ v(B1//11 ) I v21
0
0
0
w
and we have partitioned V (v1 ,v2).
,
We then take
Dk c11 Mk
Ck1
0
1
a21 11
y + v2( 21/a11) + w(
0 0
0 0
1 0
v2V
(B)0 0
0 0
1 0
V 2 w
Return to part 1.
(c) D- (
(2.7)
6 u
621
621
622 )0
Ck I
V +wb -
and max(|o 16 , 16221) (11211.
0 .D
0 0
0 0
t -
V +wbtD-1
We then take
Dk -D, H, -
0
V + wbt -1
and replace a by a
au
(d) QDQt " 621a31
621 631
D2, al1 is
the pivot choice when
is applied to D, 6 >2 | > 11 for 1=2,3; and bt -( b2
53
, and
it
0
ort
olewi
I
- btD 1 b.
p
(2.8) Ck*
0 0 0 0,
1 0 0 0
21 /11 1 0 0
631/8 110 1 0
v1 v2 v3 w
a O 01 0 0 0 0
0 D2 b2j 1 0 0 0
0 b 821a/11 1 0 0
831 /11 0 1 0
v1 v2 v3 v
where ; = v + v2(821/811) + v3(831/811) + w(S /o
- 1 rd821D2 "D2 1 (21, 31
a31
bt - b - (811 )(21, 831)
Q=Q- s 2/a ,
and we have partitioned V - (v1 ,v2 ,v3)
We then take
k 1 1)k
0
1
821 /11
831 /11
v1
9
0 0 0
0 0 0
1 0 0
0 1 0
v1 v2 w-
D2 b 2
bc2 a J
0 0
0 0 0
1 0 0
0 1 0
li v2 'i
54
and t
55
We now bypass part 1 since Ck+l is already in the proper form needed to
apply part 2.
(e) QDQt =
the choice of a 2x2
bt - (b4, 2 ).
831
pivot
831
32 Sa applied to D resulted
832 33
D 1 ; Idet Dj > (l-c 2)(maxld |)2, and
in
(2.9) CkE
0
I
~td
S+vagt+ wbt
0
0
1
v 2
0 D1 0)
0 0 BJ
0
w
0
I
~wtdt
+vdt+ wbtv21
where
--1= (63] a )n32 1
= b("I-,1b 1 D1 ,
33 (31,832)D l( 31) 2 -b1 31
632 32
_2 -b -1 31 - bD02 11 32 1
and we have partitioned V
We then take
DkimDl,
a (V1, v2)*
0
I~etd
V1 + v2dt
and
It0
(0
1
v2
0
0
0
V
~rtdt
Optb
8
+ wi
D
r
1 9
56
0 0 0(B)
0 0 0 0
k+l 1 0 1 0
v 2 v2 V
Return to part 1.
3. Floating Point Analysis
Now that we have a detailed description bf the numerical opera-
tions performed, we are ready to examine the error introduced when these
operations are carried out in finite precision floating point arith-
metic. We shall work in base 0, t-digit floating point arithmetic. We
call e 201 the basic machine unit. Let
FL(0,t) - f9: 0 * Bk I Y ,0) 0 Y (B-1)
.U(=1
each 8 an integer,
1 <Y 1 (0-1), k any integer}.
We then have (10]
fL(01 *02) *_(e1*02)(1+e') ,
where Ic'I <c, whenever 01,e2 FL(B,t) are floating point numbers, and
* is one of the operations (+, -, ", /, and ft(01*02) is the nearest
number in FL(B,t) to the real number 81*02. We shall also write fL(B)
to denote the computed elements of the matrix (or vector) B.
Lemma (3.1). Consider the vector v(k) defined in part 1(a) of section 2.
Then the components vk) of v(k) satisfy
where
(3.1)
Proof. The vect
of the expression
or (2.9) is one
Since V in (2.4)
(3.2)
where
If v(k)
duced. However
ft(V k) - V(k) (+k)(C
IT k) (C) L3+k)c(.-)maxIv | .(k)k
for v(k) is the vector v2 appearing in one and only one
ins (2.6) or (2.9) at gcep k-I. The vector v2 in (2.6)
of the columns of the matrix V in the expression (2.4).
is given by (2.2) or by (2.3), we see that
o (V(k-1) - M'v(k-l)) or V wM',
0 (k-1)
I = Mk+i, and V1 ) (k-1)
M' 2
is defined as a column of M, then no error is intro-
if
(3.3) (k) -V(k-1) -
then let j be the largest index
by a column of M. Then
m'(k-i)
less than k for which v was defined
(34)0 0 k (t1(k)r(k) M ,
where we have partitioned v~ - i L)J so that Mvl4) makes sense. The
2
formula (3.4) can be derived from (3.3) using an inductive argument. We
recognize (3.4) as the process we would use to solve the linear system
57
58
0
(3.5) Mx - [V )
where v(k) is the result of the kth step of that process. Then it has
been shown [6] that
fL(v k)) - v(k) + 4 k)(E)
where
(3.6) I(k)(c)I|< (3+c)c(i-1) maxIv .)I
j<t<k
The bound (3.1) follows from (3.6), but is not as good as (3.6); however,
(3.6) cannot be obtained without prior knowledge of the index J. o
We shall comment now on the growth of the v in (3.4). Let us
consider equation (3.5) further. Since v is defined by that portion
of some column of M (say column i) which lies below the main diagonal,
we may write (3.5) as
MxW Me - eg
where ei is the basis vector defined by (ei) - { . Thus
M(x-e 1 ) ="-ei .
Therefore, the solution e1 - x is a column of M1. This shows that the
v in (3.4) are in fact composed of elements of M . We now observe
that undue growth in the v in (3.4) indicates severe ill-conditioning
of the matrix M with respect to solving linear equations.
Lemma (3.2). Consider the computed quantities fZ(wjk)). Then if
fL(v - )PIf(m))
S
59
we have the p satisfies the equation
(M-T)p = w
where the elements i of T satisfy
|1 | .<_(n+l)i~m |ec
Here, y is a constant of order unity and the mi are the elements of M.
Proof: One observes that the w1 are computed from the standard back
substitution algorithm. The result then follows from [20]. o
Let us drop the ft-notation and hereafter regard the quantities
WO as computed quantities. Then Lemma (3.2) shows that we may write
(1)
(3.7) MDMt + wwt= M(D+[. +[wi)...wm) ])Mt
w(1)
+ a(T : wt + w[w l .wm) t)
S(m)
w()
1)
wm)
Thus we shall now regard the vectors w(k) as exact quantities. The
error introduced from the computation of these quantities in finite
precision is expressed in the error matrix
S = aTpwt + wptTt + Tp Tt)
Lemma (3.3). Let vik) denote a component of v1k). (See Lemma (3.1),
equation (3.2).) Let Wk) denote a component of w(k) Then the floating
point computation of D results in
60
f(D) - D + E,
where E is a block diagonal matrix with the same structure as that of
D. Moreover, the 2x2 blocks of fZ(D) satisfy
k-(k) )(k)u 16|1 > max(I61 , |1,2where k1k )
21 22
and the elements eii of E satisfy
(3.8) e I|< Ce
where
0 < C < max (|6 |,a|k(v k) 21, 2|0kv(k) (k)I, |ak(wk))2I)49
(Here ak is "a" at the kth step, and 8 k is the "" appearing in part 1
at the kth step.)
Proof: From equations (2.2) and (2.3) we see that the updated diagonal
blocks Ik are obtained by decomposing matrices of the form
6 6ov1 + 8w1
D1 - v16 + w18 Dk+1 + 8v v + B(v w + w v )+ ww v18+aw ,
8 v1 + ola
with 161 < a| | if equation (2.2) was used, or
[Dk + oww owl
D2 "
if equation (2.3) was used. Here the v's, w's, 8, 8, a are the pre-
viously computed quantities at step k; we have left off the
superscripts.
61
Then
fL(DL) D + Eg, (Z is 1 or 2),
where a typical (ij) element of ft(D1) is of the form
E[6jj + 8v v (l+e )(1+E 2 )J(1+e 3 ) + 2Bviwj(1+e4 ) (1+e5)
}(1+e6 ) + OWw (1+e7)(l+c 8) (1+C9)
S ii (1+C3)(1+C 6) (1+C9) + 8vivj (l+c1 ) (1+c2) (l+e3)(1+e 6) (l+c9 )
+ 26v Wj (1+E4 ) (1+e 5 ) (1+e6 ) (1+e9) + ome (1+e7) (l+e8) (1+c9 )
where I |<.
Now, if re < 0.1 and II c< , then (1+p)r - 1 + rp', where
IP'I < 1.06e [19, ex. 4, p. 80], and if |P' |,..., rI < e then
(1+p1) ... (1+pr) - 1 + rp ,
where IpI < 1.06c. Thus we see that if the elements of DZ are denoted
by 4a) and the elements of Et are denoted by then() + ei - (1+3pl) + 8vivj(1+5a2)+
+ 2Bviwj(1+4p3) + uwej (1+3P4)
where p | < 1.06e for j - 1,2,3,4, and 181 < a |. Hence,
n e | < max(i | asv iv, 2|i epr|, i s t)15(1.06)e
Maximizing the quantities which appear in this expression gSives the
bound
(3.9) |e |< a(|o|1, |akk[v k) 2|, 2|0kv k) (k) k(()j 2|)15.9c.The case we have examined is clearly the worst case for the type
of analysis we have carried out. Thus we take (3.9) as our bound for
62
the elements of E , (L = 1,2).
A A
The next step in obtaining Dk is given by decomposing D1 or D2
according to one of the equations (2.5)-(2.9). Let us refer now to the
proof of Lemma (2.3) of Chapter II. Specifically we consider the decom-
position given in equation (2.12) of Chapter II. There it was shown
that a scale factor u may be implicitly introduced in the last row andw w
column of D1 or D2. This factor has no consequence on the final result.A A
However, when D or D2 is suitably scaled in this way then pivoting
strategy SG does not choose any of the elements in the last row or
column as pivot elements. We then obtain a computed factorization.
wwwt Q 0 A Q 0 tD = (D + E ) +F (i 1 or2).
0 1 0 1
The analysis given in [3] applied to this (at most) 4x4 case shows that
the elements f of F satisfyii
(3.10) f ()< max|6(t)|(34e).(31)ij ii
Now (3.10) together with (3.9) give the bound in (3.8).
We have given an analysis of all of the operations in part 1 and
of the formation of D. We now turn to an analysis of the final forma-
tion of the elements of H. We begin with
Lemma (3.4). Let a be the ijth element of M with i > j. Then
(3.11) fL( ) -= mij + ~igjij(E) + vij(E)
where
|nd(c) I<_c(3+E)
and
63
Ivi (c)I < maxIvk)I((3 + max[ , j-]) + O(c))ika1 a
Proof: We shall give a detailed analysis of the operations used in
forming M. These operations are described in part 2 a-e. We shall have
to examine each case separately. Cases a-e below refer to the opera-
tions performed in part 2 a-e, respectively. The errors resulting from
computing the quantities 0/6, b D have been accounted for in Lemma
(3.3). Therefore, we shall assume here that we have these quantities
exactly.
Casea: The vector Hk is computed by
0
ft(Hk) 1
fZ(v(k) + w(k+l)C/)
Now,
fL(v(k) + wk+l)CB/a) -
S(vk) + [W(k)(0/6)](1+e1))(l+e2
S[v k)+ w(k)/j)](1+E2) + k) (/8( +E E2
mik + +iki2 + k)1(/6) 1+ce2
- Omik + mik62 + (vik) + k)(g/ )(E +E E2)
- vik) (El+El2)
aik + mik c1+c2+ l2) (k)
64
Case b:Mk is computed by
0
~ 1
ft(Mk) _
21/611
fe(vl+ v2(621/611) + w(k+C
Let v have components vjj, j = 1,2.
Now,
fZ(mik) = fZ(v1 + v 2( 21/6 11) +Pik+1
_ [11 + v (a2111)(1+E )](1+E2
+ m(k+1) (01 /611)(1+c3)}(1+c4)
S[v 1 (1+E2) + v12(621/611)(1+E 1+c2+EcE2)
+ m k+1) B1/a ) (1+E3)] (1+E4)
-= +
+ 41eg[ 2+v2 21 011 1+ 2+ 12
+ e k+1) (B /6 )c 3
mik + mike4
+ (1+e4)[(vi + vi2 (621/611) + E+1)
+ v11(c2-E3) + Vi2(8 1/611)(E1+c2-E3+c1c2)J
amik + ~ik(c3+2Ec+C3c4) + v (c2-E3+E4(c2-E3U
+ v 12(a21 /6 U)(c1+e2-c3+c1c2)(1+c 4) .
65
Case c: is computed by
ft(Mk)
0
fL[(v ,v2) w(k+1)bt 1}
Let btDl - (51982) Then
fl(ik) (ft(vi + ik+1)i ), vi2 - i 2
Each of the components falls under the same analysis as case a. We
obtain
Lii md + ;ij(e+c2+c c2) - vi(e1+ee2)
and
Win ij+1 m ij+l
where M k
+ mij+l(Ec+ec+ce ) - vi2(ej+cEe)
- (ij ,mij+l) .
Case_ d: is computed by
f(Mk)
0
621/811
831/811
fL(v 1 + v2(821/511) + v3(631/611) + w(k+2) (B1/611))
So
ft(uik) - fL(vi1 + vi2(821/ 1l) + vi3(821/a 11) +t1(k+2)
" (v-i + v12(821/11)(1+Ec)J(1+e2)
+ vi3( 31/811)(1+E 3)}(1+c4) + m(k+2)(81/8 )(1+c5)] (1+c6)
0
* fVi 1 + Vi2(621f11) + v3(31/11) + ( k+2) (01/ai)
+ vi(2+E+e2E) + V 621611 (1+c24+e 2
+c4(E1+E2+E1 E2))
+ V 1 3(31/ 11) (3+E+ec+4) + (k+2)(B /6 )E51('+F6
mik + mik 6 + Vil[e 2+E4 +E 6+ 2E4+E 6(c2+E4+e2E4)J
+ V2 f[ 1 +2 + +E 6+E2 + E( 1 + 2 +ce2 )
+2 al 1 4 61 2 (1 1 2
+E6(c 1+E2f+4l12E4 (c1+c2+e1c2))]
+ 3v633+E1+c6+C 3 4 + 6 (c 2 ++c 3 c4 J
+ w (k+2) (0 E(5+e6+E56)
- t mik + UikC6 + ( 5+e6+c5c6) [Vi+ 2( +v 3()
+ w k+2)1
+ Vl c2+E4-E5+E 22 3-c5c640(c)2J
+ v [ E+ +E 4E + E2E 5E 6+E (E +E2+E 2)+O(E2)+ a [8-1 2 4 5 124 5 42
31 ) c3+c4 -c5+e 3c4 -. c5c6+O(c2) I
s computed by
0'
I
(631k 632 k
fC[(v ,v2) + v3(831'9 32)D,1 + w(k+2)(61.52)Dk
66
Case : Mk
ft(k )
67
where we have partitioned V - (v1,v2,v3), bt = (01'02'03). Thus if we
let (631,a32) -_(63l,632)3 l, 0'2 (.1,1 2)Dk1 be the quantities
computed in (2.9), we have
ft(Mik) - (ft(vii + vi3831 + W(k+2) 6), fL(v12 + vi3 32 + W k+2)02))
Then the components of M fall under the same- analysis as case b.
We have shown that if mi is computed from the formulas given in
part 2 a, b, c, d, and e using the computed quantities from part 1 and
from the formation of D, then
ft(~n ) -~ +~n y (e) + v (c).
Define v - max ~ 4k)|. Then taking absolute value, using the triangleis k
inequality, and recalling that ICkI < e gives
Iui(e)|< E(3+c)
and
ve(l+c) in case a,
v(3c(l + --a) + O(e2)) in case b,
lv j(c)1 < ve(l+c) in case c,
I 21 83v(e(3 +4i- + 3 -- ) + O(e)) in cased,
v(3c(l + max(|631, 32I) + 0(t2)) in case e.
By the properties of pivoting strategy S we have that
,' a1a d x(31*132-_
68
Thus,
Ivj(c)I < v(e(3 + max[ ,1 + o(c2 ))
in all cases. 0We now return to equations (1.5).. Regarding (1.5)(i) we have
shown that A5ij - MijIij(e) + vij(e), with bounds for yia(c) and vij(e)
given in Lemma (3.4). We also have that AD = E in Lemma (3.3) is block
diagonal with the same block structure as D. Using the analysis given
in [3, p. 667] we see that in (1.5)(i) and (iv) that
I62MijIi I1HijI 2 e[1 + 0(E)](n-2+1-j)IMijI
and in (l.5)(iii) we have
if Dii is a one-by-one block; otherwise, from [3]'
Cl(8D i (i4 lI ) lE1+0(0)|D Ir 1
I(6D)i,i+11 I(8D)i+1 ,i+iI i+l,i 1 1 J
Finally, we have that the error matrix S, defined in Lemma (3.2)
and discussed in the remarks following it, is bounded by
|IS I. {(2IIT|Ijjp|I,||w|, + DITII|plI|} aI
< {2n(n+l)y max Imi Imaxw Ik)jI wILc
+ e2[n(n+1)]2 2max~w k) 2}Q
where y is the constant appearing in Lenma (3.2). The matrix G(e) in
(1.7)(i) and (iii) is given by G - (v (c)). (Note that G is lower
triangular.)
We have the bound for F given by
69
(3.12) IFI|| < 2 max|vik)c( 3 + maxc ))IIMDrtLi,k
+ 2 max(|D |I, a| Sk v k))2|, 210kv k)w(k)
ijk
|ak(wk))21|)49EI|i1LIlM tI10
+ E[1 + O(6)]IIMDMtII + |Sli, + Q( 2 )
We have already mentioned that the iv(k)j and Iw(k)|, and |SII.
may grow with n for ill-conditioned matrices M.' However, the computa-
tional evidence indicates that the usual remarks concerning the solution
of triangular systems apply: in practice large growth does not occur in
these quantities.
In order to guarantee stability we must also show that the ak's
and 0k's are bounded. We shall do this by showing that the growth of
ok is bounded at each step of the algorithm. This is sufficient since
it can be demonstrated that the growth of sk+l is bounded as long as the
growth of ak+1 is bounded.
It will be necessary to impose an additional condition on the
acceptance of a lxl pivot. The number 6 in
( :)will be accepted as a lxl pivot if 161 1>_1| as before, or if
I Q6 > 002. This does not affect any of the preceding analysis.
We shall begin by establishing several preliminary lemmas.
70
Lemma (3.5). Let A be symmetric and suppose that A = MDMt. Let the
eigenvalues of A be A ! A <...< An, and let the eigenvalues of D be
1 < <2 <...<Un. Let k be the index such that A < 0 for 1 < j < k,
and A > 0 fork < j < n. Then A<cy for l < j < k, and X >_cu
for k < j < n, where / is the smallest singular value of M.
Proof: By the min -max theorem
tIA = min max
dim(S)=j xcS x x
x#0
= min max sDsdim(S)=j sS stM M-ts
s#0
. min max s ssdim(S)=j seS sts JstMlM-ts "
s#0
tSince 8est> c > 0, we have for j < k that
stM L M s
F t1A <c min max- - = -ca
dim(S)=i sES ss ss#0.
Similarly, for j > k we have A > cuAj. o
For the following discussion we shall also need to know the
smallest singular value of certain lower triangular matrices in order to
apply Lemma (3.5).
Lemma (3.6). If M - (Y J, then the smallest singular value of H is
/, where
c>.Y
71
1 0 0
If M = YO 1 0 , then the smallest singular value of M is /, where
Yi 0 1
c > 2 12
Proof: The smallest singular value of (is where
c - (Y2 + 2 - 4+42/2
1 0 0
The smallest singular value of [o 1 0 is /, where
Yi 0 1
C 2(YO+ T2 + 2 - (Y+y1 )2 + 4(+y))/2
If c - (a + 2 - a2 + 4a)/2 with a > 0, then
(a+22 2
C 2(a + 2 + )
2
a + 2 +a+4a
2
a + 2 + /a2+4a+4
2 12(a+2) a2
The lemma follows from this inequality. o
It will also be of interest in the following discussion to bound the
--norm of the inverse of a 2x2 block.
Lema C3.7). Suppose that D - [116 21 , with aca62 1 > max(161,1d221)
621 622Then
72
21
Proof_:
ID1 -11, L1DII, I
where |a I = 161622 6 >621 2 221'6 1 2A -2 1 2 -6
D L = 16211 + max(161 1 1,16221)
< (1+a)16211
u - (1+a) 1 - 1(1-a)Thus 1-162 J (1-a)I621 0
Two more lemmas are needed before we can establish the bounded-
ness of the ak0
Lemma (3.8). Let B e Rnxn be symmetric and nonsingular with eigenvalues
Al < 12 ''' An, where n > 2. Let z e Rn, and n c R. Let
B' = B + nzz.
Then n maxIBy| > A, where A = min |A .
Proof: Let B' have eigenvalues .1 < 2 '''.< n. From Lemma (2.2) of
Chapter II,
S> 0=> Ag y <An un
while
f < 0 -> p1 A1 un *A
If A1 > 0, then |ynrj> 1 1 -
If A1 < 0 and An > 0, then In1 - InI when n > 0 and I" 2I > IA11 when
n < 0.
73
If An < 0, then 1)u11 > 1-In
We conclude that
p = max I I >l1<Jn i
By the standard norm inequalities we obtain
n max|B, I >p
and the result is established. o
Lemma (3.9). Suppose that the k-th step of the updating algorithm has
resulted in
k)61 IIandtC
v w 8 a v w
with a 0, 161 < a|| and |o|6 < a2 .'Then
2 JO ldet -1l + a0 z A dz
(1+a)Idet (k) I(ll tjt fl -wl + 1 + )
D1 0
where M (H1,M2), D -
0 D2
QAQ - (M),M2 ^(M,M2)t0 D2
and D(k) is that portion of D obtained up to step k. Here a0 is the
starting "a", and a is the modified "a" after k steps of the algorithm.
Proof: At the k-th step we have
(k) (k)t - =(k)D(k)(k)t +
a (|0
M(k)l
I v
|4
2 IIC0 0
i o C63 (k) 02
0 (6-82/a)0 0
0
+48/A 0.8/a,Now, det(A + agzzt) - det A(1 + a ztAlz).
M(k)'|I
B= ( 01V
H2
D(k) 02
0 0
8 10V w
00)
2 C
wt + vt)
Also, if p -
00D2JI
1 0I(k)
I v
det A(1 + t -1SZ A z) .
0 det(B + app )
idet B(1 + ptB p)
2
" det w(k) 2( - )det D 2(1 + ap B p)
(3.13) Idet Al 1 + aOztAlz| - Idet U(k)|IIdet D211 - 10 2iSince 16a1 5 acs2, we have
+ ap Blp .
160-81 1 8 - |601 (1-)82
|6o-82| 5 |11 + 82 < (c+1)82
Also, one can show that
74
)t+ 2 2 2
t01V
lM2
Cthen
08/a
w+s/av, and
Thus
(3.14)
and
(3.15)
~(k)
M2*
i
|
|
|
.
75
(0
/ ,1 :-
2
- (i/Kw ,
A-1
where we use H2 to mean the inverse of the unit lower triangular matrix
that occupies the lower triangle of M2. Thus if " = wt 1tlMlw we
have
ptB 1 = 2 + .2 S
(a - Q}
Hence
(3.16) IPIt - 1 6 2 + I|a||o - 0
IIl(1-c) +
by (3.14). Using (3.15) in (3.13) gives
2 I 21Idet AIIl + a ztAlzI(3.17) IQ (1+a)8 2 2-/a| = (k) t -1 "
Idet Dk Idet D2111 + aptBlpl
Therefore, using (3.16) in (3.17) gives
(3.18) B2 ___ Idet Di |1 + aztAlzI-(1+G) Idet D )I(IoIII + 1 + 1 )
This is the desired result. 0
Observe that the quantities in (3.18) are independent of the updating
process except for |u|, and Idet D(k)|. Now we are ready to prove
76
Theorem (3.1). Suppose that the k-th step of the updating algorithm has
resulted in
C(k) 1 0 6 s 1 0t
V w) S a v w)
with 161 < a s1, and |61 as2. Then the next step.of the algorithm
will produce a a' of the form
a'u - b D bk
with |a'| bounded.
d S 1 1 d B -Y -Y
Proof: Let( ) 4 , and( : 2 withSas Y Y C L - 1
lul 1u |2| as in Lemma (2.4) of Chapter II. Let rn{ 0y1/(1+y2), j=1,2.
As we have already seen in Chapter II
equivalent to forming
the updating process is
1 0 w nl 0 0 1 0 wo t
(3.19) C(k+l)mvl I w l 0 D' 0 vl I w ,1 v2 V 2 0 0 n2- v2 V 2
where D' and I are, respectively, the next diagonal block of D and
V v
the corresponding column of M. In (3.19) w0 - -y, - v+yw, and
(wl) w-yv.
w2.
The next step is to form
1 0 w 1 0 Iw -t
C(k+l)m o0 Iiwl(2 2 IW t
wv2 2)[l 0 2 v2 2nd w2
A 2o where D ,and v2 v 2-Vv1.
vl I 0 D' vl I2
77
Then we form
1 010 t 1 0-t(k+)D|+ n2bb nbc(k+l s(0 "2 012_Coo:C = 0 I 0 _ 0 I o
2 V w 2 2b In2v 2 V I
where b - ,and w2 s2 -v2w0-Vw1 .
Finally, we pivot and obtain the updated diagonal block Dk+l'
Let D " D+n2bbt, and let = - maxDi |. Then IDkI > aC when Dk+1 is lxi.
If it is 2x2, . To avoid cumbersome notation we
shall let B- +Dk+l DrThe factoring of I n2J gives an updated ri - b2- 8b-
n2b n2
where b1 consists of components of b. It can be shown thatw
2 (l+y2 2 1 where w - . Comparing this with the updatinga i e ee wialgorithm described in Chapter II will show that
(3.20)
Now,
Hence, if a > 1
(3.21)
Let /R be the
a' = (1+y 2 2 0
In| <. n21 + n Ib B1 b1 I
. Ii2| + 3ri AbII IB1L
/2 then
3n2l b2In| < n2 + .1
smallest singular value of 1 ). From Lemma (3.6),v I
9 > 1/(1v1||2 + 2) .
By Lemma (3.8) we have that 3 >_A1 where A is the smallest eigenvalue of
D (in absolute value). However, Lemma (3.5) implies l>_6. min(|nr|, Ip),
78
and thus
(3.23) >1 min(|n |,1UI)
where U is the smallest eigenvalue of D' (in absolute value). Finally
we have that
(3.24) v = I||b1 | <I b||L < max ( I |, ||w || +I|I|I||v|IL) .
Combining inequalities (3.21)-(3.24) gives
v29n ~ lI2+2)
(3.25) |n| < n21 9+min(n1v,|)(1-a)
Since IIw and IIv L are bounded we have that V2 =0(y2) and
iv |N2 = 0(y2). Thus it is sufficient to bound the quantity
4
(3.26) n2mi(Iri1I,IpI)
Suppose that 18 l>o|. Since y = (a-U 2)/0 we have
lI (a+a) - sgn(+)(a-a)2 + 482 +1
1 + a + /(+a)2 + 4) + 1
Thus (3.26) is bounded since 1 < 1 and l is fixed. Therefore, we
shall assume that |0| < l<|. Now,
|21 - n21 2
However,
I A z. f l y1a 2 2
79
Let 4 * sgn(a+)/(am-)2+402. Then
2Y 112[(a-6)+*] 2[ (-a)-tt-26]
a 82?882a
a_+_ (a-a) 2 2 + 26 [ (a-6)+1]22a j 402 80-2a '
Thus 2
]2 iT+ 12
where T (-6) + Observe tha.
T< 1[+ +/J7] 2+4 J2
< Z1 + a + (1+u) 2+ 4.
By assumption a < 1 so that
2
a < T+ T2
We also have that
|421 - 2[(a) - $] 2 (Qa) -r+ +26]
(1 4822 I(a-6) + + |11.
Now,
1(0-8) + 4i| = tQ-|+ a-|61 2 4 482 >_ 2|a-a|.
Thus,
<2 2+ +16.2 Icv-l 1 a
80
Therefore,
1n2 I < (l+y2)-2(I S + 161+ IaI@+T2)
It follows that .
4
< -+ |8+||(T+22
To bournd -ni- we consider
In2 -IS - < 62 2
2 -2 2
But
A - (a+6) 2+ 2|+|(a-6)2+ 482+ (a-6) 2 +42
>_2a2 + 282 + 2(a2 62) + (-6) 2 + 482
- 5Q2 - 26 +62 + 48
>5Q2 + (4-2a) 2 >_ 52 .
Thus
But 2 [2
2+2
<-Q-L1++v l)+ 44482
81
It follows that
4 4--- _-- L' + a + (l+a) + 4
= n a ++16 n i
2 .4
<2 (1+a) 1+a+ (1+a)2+4
800
Using the same notation as in Lemma (3.9) and applying that result gives
a2 (1+a)|al|det -D(k)|1(|0||$| + 1 +)
Idet D | 1 + aztA
Observe that if the previous a's and 8's are bounded then Idet D(k)I is
bounded. Thus we have bounds
4 4
SKl and< K2
This gives
i < |q2 1(1 + max(K1,K2 ))
But
n2I < a|(1+y2)-2r + a + T + T2
Since a' - (1+y2 22 we have
all 1<01 71 + a + T + r2 (1 + max(Kl,K2) '
This concludes the proof. o
Theorem (3.1) provides a bound on the growth of a when pivoting is done.
The following theorem provides a bound in the remaining cases.
Theorem (3.2). If the algorithm updates a 2x2 block and accepts the
updated block as a 2x2 pivot or if a 14 pivot is accepted from the
82
updated block then
|a'|I ,.<or
.afl 1 +
If 6 is accepted from ( )as a lxi block then
Ia'| < |a| + max(la|,|S|)/a.
Proof:
Case 1. If 6 is accepted as a lx1 block as a result of
Sa
satisfying 161 > a|t.|, or 16a| > a0 2 then
'= a -I6S2
Hence lo'| < |a| +
< |a| + IaI/a -- |a|(1 + 1/a)
or 1a'| <I|d + 18|/c .
Case 2. Suppose that a 2x2 block is updated and accepted .as a 2x2
pivot. If the old block has elements 6 then the updated block has
elements of the form 6 - 6 + awjwj. The conditions that must be
satisfied are a6211 > max(16111,16 221), and a16211 > max(1, 11 1,91 221)'
Let' represent this 2x2 updated block, and let w - (w1 ,w2). Without
loss of generality we assume IwI, - 1w11. Then |lw|L < 21w1 |. In this
case we have
2 tw-1Q QnaawD w
83
so that
Ia'1 I< kI(1 + 2Ia 1w 1 2 1D 1.)
From Lemma (3.7) 1 1 .
Now,
IawlI - 16ii1 5 -I6 11+cw I < a1 21+ w2w1 |
Thus
IowlI ~I211 + IiI
5. I621I + 16211)
.(I621I + 1621 + aw2w1I + Iw 2w11)
26211 + a aw .
Hence (1-a)Iaw | < 2a16211, and we have
l '|l < |al 1 + 2 k-a I)
- |al1+ %)
Case 3. A 2x2 block is updated and it is found that
a1621 1 max(I1 111,1622 |). We use the same notation as for Case 2.
Without loss of generality we assume that 16I .|11 (otherwise 622
is brought to the 811 position). Then
o2,2
811
Now
lI8ll+ovi l . lkw I - 18,111
84
so that
1w < | + (a y.
Thus
1 aw1
1 +I111
But
. 2 2+w2w + law2wl)
" a (121 + |0w2w1 .)'
Thus
i| - aIaw2 w1 I
1 11+
a ! < 2 .I6iI -
A similar argument shows that
1221 + 16221< 1< 1+
S211 + k1w2w1 I1fl
NJ
If 1w1 > >Iw21 then (1-a)|aw1 < IawI - 1aww2 |, and inequality (3.27)
shows that
Ia1I 2d11 - -
However, if 1w21 > 1|w1then (1-)I|awI (1-)I|awI < Iaw$| - aI oww2 I,
(3.27)
aW2!
---- 1
Thus
(3.28)Iaw| - (2w1
~ < 2
(3.29)
611 1 a9a211
85
and inequality (3.28) gives inequality (3.29). "o
Theorem (3.1) shows that the growth of a can be sensitive to
near singularity in A. This can result in two ways. If a is much
larger than the eigenvalues of A then numerically A appears to be a rank
one matrix. Also, one of the updated eigenvalues can be shifted to zero.
This is reflected in the bounds obtained in Theorem (3.1) since one of
the bounds depends on - where P is an eigenvalue of D, and the other
boun4 depends upon 1/(l+aztA z). The quantity 1+az A z - 0 if and
only if A has a zero eigenvalue. We conclude that the use of the algo-
rithm should be restricted to cases where the matrices involved are well
conditioned. Finally, we do not expect this technique to generalize to
the LU decomposition of non-symmetric matrices since our results are
heavily dependent upon properties of symmetric matrices.
Chapter IV "
The Use of Directions of Negative Curvaturein a Modified Newton Iteration
1. Introduction
In this chapter we present an algorithm for obtaining a numeri-
cal approximation to the solution of the following problem:
(1.1) Let f:VP c Rn + R;
find x* E V such that
f(x*) f(x)
for all x in some neighborhood of x*.
For theoretical reasons we shall assume once and for all that f has two
continuous derivatives on V and that for any cx 0V , the level set
L(x0) - {x: f(x) < f(x 0 )} is a compact subset of V. Additional assump-
tions will be introduced as they are needed. The assumptions just
stated shall be referred to as assumptions (1.2).
Recall from Chapter I that we denote the gradient of f(x) by
g(x), and the Hessian by G(x). Given a sequence of vectors fXk( C V we
shall use the notationfk - f (xk) k - g(xk), and Gk - G(xk). We shall
sometimes omit the argument x and write f for f(x), and g for g(x),
etc., when there is no danger of confusion. Throughout this chapter we
use j* o to denote the Euclidian norm, and xzy to denote inuar products.
The algorithm we shall present may be classified as a descent
ethod.Usually a descent method determines a descent direction k at
the iterate xk (i.e. g sk < 0). Then a linear search is performed to
obtain ak > 0 such that f(xk+aksk) < k and we take xk+l * Xkaksk
86
87
Under some additional restrictions on the choice of, ak one can show that
lia sk/Ilsk|| - 0. The vector sk is usually related to gk in such a way
that this limit equalling zero implies that the iterates converge to a
point x* where g(x*) - 0.
In addition to obtaining such a point x* we would like to be
able to assert that G(x*) is positive definite for this would imply that
f(x*) < f(x) for all x in some neighborhood of x*. Of course, we shall
not be able to accomplish this goal, but through the use of directions
of negative curvature we shall be able to guarantee that G(x*) is posi-
tive semidefinite. For practical purposes this is very strong assertion.
For instance, if the Hessian were known to be nonsingular at all critical
points then the point x* would have to be a local minimum.
Recently the idea of using directions of negative curvature has
appeared in modified Newton algorithms [8,11,16). In particular we are
indebted to the paper of McCormick [16]. Indeed, theorem (3.1) is only
a slight modification of McCormick's result. However, this result led
us to consider a new line search strategy. The implementation of this
strategy which we present here is based in a fundamental way on the
factorization of symmetric matrices using the algorithm of Bunch and
Parlett [5] and this is discussed in section 4. In section 5 we give
termination criteria for the new univariate search strategy, and show
how this relates to previous strategies. Finally, in section 6 we give
a convergence result that includes various choices of descent directions
and we suggest a particular way to define a modified Newton iteration.
Since the algorithm is a descent method we shall begin with a
discussion of descent directions.
88
2. Descent Directions..
The following definitions will be useful throughout this
chapter.
Definition (2.1). Let f: R + R be twice differentiable in the open set
V.
(a) A point x in V is an indefinite point if G(x) has at least
one negative eigenvaLue.
(b) If x is an indefinite point then d is a direction of
negative curvature if dtG(x)d < 0.
(c) A pair of vectors (sd) is a descent air at the point x
if when x is not an indefinite point then gs < 0, gtd <0,
and dtGd = 0, while if x is an indefinite point then
gt < 0, g d < 0, and dtGd <0.
An example of a descent pair would be to take s = -g(x). Then
if G(x) is positive semidefinite take d = 0, and otherwise take
d = -sgn(gte)e where e is the unit eigenvector corresponding to the most
negative eigenvalue of G(x). We shall see that there are more attrac-
tive choices than this. However, regardless of the specific choice, a
descent pair fails to exist at x only if g(x) = 0, and G(x) is positive
semidefinite.
The search strategy we shall present departs from the usual
strategy discussed in the introduction. Instead of using only one
descent direction and searching in a line determined by that direction,
we shall consider searching along a curve of the form
(2.1) C: j1 - x++1 (a)s+4 2 (a)d: s a o)
89
with (s,d) a descent pair at x, and with $1(0)"- $2(0) - 0. We hope to
produce an a > 0 such that
(2.2) f (x ) < f (x)
If we let 0(a) - f(x ) we encounter a univariate minimization problem
where 0" is continuous as long as ',$' are continuous. The following
lemma gives sufficient conditions. under which (2.2) can be satisfied.
Lena (2.1). Let 0: R + R be twice continuously differentiable on the
open interval I which contains the origin, and assume that II [0,1).
Then there is an a > 0 such that
#(a) <0#(0) + '( 0)a + 0"(0)
for all ae [0,i] provided that either 0'(0) < 0, or 0'(0) " 0 and
*"(0) < 0.
Proof: The mean value theorem implies that for every a> 0 there exists
* c (0,a) such that
a2
;(a) - 0(0) + 0'(0)a + 0"(0) 2
1 ["6 - 9"0)a
Hence,
0(a) * 0(0) + u 0'(0)a + 0"(0) + r(),
where
r(a) - (1-u)[0'(0)a + 0"(0)
+ 1[(0~() - 0"(0)1a2
Since
1 r~a 0S *2 0,
there exists an a> 0 such that r(a) < 0 for all a c [,a]. o
90
This lemma not only tells us when (2.2) can be satisfied, but
also that the function f must decrease by a significant amount along the
curve xa. It also indicates that a larger decrease is likely when
*"(0) < 0. We, of course, want to use the simplest functions #1 and *2
which will guarantee that the hypothesis of Lemma (2.1) is satisfied.
Observe that if 0(a) - f(xa) with xa as in (2.1) then
(2.3) *'(0) - g(x)t(, (O)s + # (0)d)
(2.4) 0"(0) - g(x)t(,'(O)s + 4'(0)d)
+ (i(O)s + , (0)d)tG(x)(,j(0)s + "2(0)d)
Suppose that g s s g d " 0 at an indefinite point (this occurs for
instance at a saddle point). Then in order to insure 0"(0) < 0 without
imposing further conditions on a we must require 1(0) " 0, and9(0) > 0. Then (2.3) and (2.4). simplify to
(2.5) *'(O) - g(x)t(*I(O)d)
(2.6) *"(0) - g(x)t(,'j(O)s + 4'(O)d)
+ (#'(O)d)tG(x)(4 (0)d) .
When G(x) is positive definite then d " 0 must be satisfied in order for
(s,d) to be a descent pair. Thus 0'(0) - 0 and we east have +i(0) > 0
in order to insure 9"(O) < 0. Therefore, if "l(a) -" 10aiandj"0 j
*2(a) Iya then we must have 8 1 O with 82 > 0 and Y0 O1.01
with Yl > 0. The simplest functions of this type are, of course,
fl(a) " a2, +2(a) 0 a .
In this case
91
(2.7) '(0) - g(x) d
(2.8) O"(0) - 2 g(x)ts +.dtG(x)d
3. A Modification of the Armijo Steplength Procedure
In Section 2 vie introduced the notion of a descent pair. The
motivation for considering the use of a pair of vectors rather than the
simpler strategy of determining a single direction of descent will be
discussed now. We shall present here a modification of a theorem of
McCormick. In [16] McCormick gives a modification of the Armijo step-
length algorithm [2] which includes secovnJ derivative information in the
form of directions of negative curvature.
The steplength algorithm will be described now. Given
y,p E (0,1), let {xk: k'0,1,2,...{ be a sequence of points derived from
the given point x0 as follows:
Determine a descent pair (sk,dk) at xk and let ik be the smallest
non-negative integer i such that
(3.1) yki1k+Y2 sk Yidk
and
(3.2) f(yk,< k +iNj2[gsk + Gkdk] 1
Take Xk+1l* 7k,i. Lemma (2.1) .shows that the iterates are well defined,
and if a descent pair does not exist at xk then we accept xk as a solu-
tion to problem (1.1).
Theorem (3.1). Let f satisfy assumptions (1.2) and suppose that
IskI,|dkI are bounded independent of k. Then
92
(3.3) lim (-gsk 0k.
and
(3.4) lim (-d4Gkdk) * 0
k-Nr Pr_: The sequence {k} is a decreasing sequence which is bounded
below due to the continuity of f and the compactness of L(x0). Thus
liU (fk*k+1) " 0. There are two cases to consider.k-m
Case . Suppose the integers i} are bounded above by some m> 0.Then
fkfk- > -Y gksk + dGkdkj
Since
-gsk > 0 and -dkGkdk 0
the conclusion follows.
Case . The integers {ik} are not bounded above. Without loss of gen-
erality we assume that li k i - 4. By the definition of ik, if
S( 1kY ), then
(3.5) f lk+l f 4[gksk + dGkd4
However, due to our assumptions on f and L(x0 ), a Taylor series argument
and the fact that gkdk < 0 may be used to show that
(3.6) fk+l < k+ k kk + jmdkGkd] + r(xk'skdk'ak) ,
withr(xk* k"kkk
(3.7) liz -- 2 0k-o k
lnes, combining (3.5) and (3.6) gives
93
(3.8)-r(xk.sk.dk' k) t(i + dG ](3.) 2 -- u ksk +2 d Gkd .
The conclusion follows from (3.7) and (3.8). oThe result presented by McCormick did not specify a choice of
xk+l when xk was not an indefinite point, but did suggest the Newton
direction. In the case that xk was an indefinite point then
sk "sI k k'k with Pk a descent direction such that -gpk c1 5k 'Also, dk was required to be a unit vector such that dGkdk 5- c2 Gk where
kAGk is defined as the most negative eigenvalue of Gk. In the above
statements c1 ,c2 > 0. McCormick was able to conclude that if infinitely
many indefinite points fxk } were to occur in the sequence {xkI, then
any point of accumulation x of the sequence {xk } must satisfy g(x) 0,j
and G(x) is positive semidefinite with at least one zero eigenvalue. A
specific choice of sk and dk was not suggested.
Under the additional hypothesis that the number of critical
points in V is finite, and with a judicious choice of (sk,dk) one can
show that the iterates defined by (3.1) and (3.2) converge to a point x*
where g(z*) - 0, and G(x*) is positive semidefinite. However, Armijo
type steplength procedures do not take into account any information
about the shape of the function along the curve x. More sophisticated
strategies are available for determining the steplength %.
In the rest of this chapter we shall be concerned with the
choice of (skdk), and with a steplength procedure which specifies
criteria for terminating a univariate search along curves x of the form
(2.1). Finally, a convergence result will be given that indicates these
choices are quite reasonable.
94
4. Determining Directions of Negative Curvature
As we shall see, the results of Theorem (3.1) are useful only if
(4.1) (gksk +f0) > (gk + 0)
and
(4.2) (dkGkdk- .0) (Gk0)k
where AG is defined to be the most negative eigenvalue of Gk when g is
an indefinite point and zero otherwise. Intuitively, if (4.1) and (4.2)
hold then the iterates (xk} are converging to a critical point where the
Hessian is positive semidefinite. These statements will be made precise
in sections 5 and 6. Here we present various ways in which (4.2) can be
accomplished. Matrix factorizations will play an important role. The
factorizations we shall discuss in some detail are Gill and Murray's
modified Cholesky factorization [11], and the method of Bunch and
Parlett [5].
Gill and Murray present an algorithm which for any symetric
matrix A produces a unit lower triangular matrix L, a diagonal matrix D
with positive diagonal elements, and a diagonal matrix E with nonnegative
diagonal elements such that
A+E*LDL .
The elements of LDfand E are bounded relative to the maximum element of
A. This factorisation depends upon nonnegative parameters (8,0). The
parameter p is used to force a bound upon the elements of LD. The
parameter 6 In a sense determines the level of positive definiteness
that the matrix O+ is required to have. Given the parameter 6 > 0,
this factorisation proceeds much the same as the Cholesky factorisation
95
with the exception that when a diagonal element is 'found to be less than
or equal to 6, it is modified. This modification is expressed in the
diagonal matrix E.
It is possible to obtain a direction of negative curvature from
this factorization when 6 - 0. Assuming A has a negative eigenvalue,-
one computes an index L such that D -E < D -E for 1< j.1 n. Then
the solution d to the equation
Ltd ,
where eL is the unit vector whose t-th component is 1, can be shown to
be a direction of negative curvature.
With this factorization A can have a negative eigenvalue only if
8 is nonzero. However, when 6 > 0 it is possible for E to be nonzero
even though A is positive definite. Thus the direction d obtained above
cannot be guaranteed to be a direction of negative curvature unless
8 = 0. Unfortunately, when this factorization is used in a modified
Newton's method 6 > 0 must be specified to obtain a proof of convergence.
The factorization of Bunch and Parlett allows an alternative
that avoids this difficulty. We have already discussed this factoriza-
tion in chapters I and II, but we wish to emphasize here the properties
of this factorization relevant to this discussion.
Given any symetric matrix A the factorization will obtain a
permutation matrix Q, a block diagonal matrix D, and a unit lower tri-
angular matrix M such that
OAQt "lm t .
The matrices N and D satisfy
96
(4.3)
The following lemia will show how this factorization can be used
to obtain directions of negative curvature which satisfy (4.2).
Lemma (4.1). Let A - WBW where W E Rnx is nonsingular, and B E Rnxn
is symmetric. Assume that A has at least one negative eigenvalue. Let
{s : j-l,2,...,u} be unit eigonvectors for B corresponding to
eigenvalues
kLet z -" s
j-i
1 012.
where 1 < k < a and let
V y 3 0(4.7)
Then
AA - k2(K 2(W)3 2ii
The elements of M are bounded by a fixed positive
constant which is independent of the matrix A.
D is a block diagonal matrix with one-by-one or
two-by-two diagonal blocks.
D has the same number of positive, negative, and zero
eiganvalues as A (Sylvester's Inertia Theorem).
The number of 2x2 diagonal blocks plus the number of
negative diagonal elements which occur as lxl diagonal
blocks of D is equal to the number of negative eigen-
values of A. In the case that A is positive semi-
definite, D is a diagonal matrix with nonnegative
.diagonal elements.
(4.4)
(4.5)
(4.6)
97
where AA is the smallest eigenvalue of A, and K2(W)' = ||lWl111W1 is the
Euclidean condition number of W.
Proof: If x is a unit eigenvector for A corresponding to AA thentA
x &x =A and if u = Wtx then
A A=xtAx = uBu >A mu||2
Moreover, since Null < |wl|, and A3 < 0,
(4.8) %AA l ,l2
Now note that from (4.7)
k k k
y Ay =-l B z L= A <0.j= =l j=l
= t k
Since Il = 1W(111 kIW 11we havej-l jJ
k
(4.9) Z L 1Y k2 -1 2 - k2 -1 2
Together, inequalities (4.8) and (4.9) give the desired result. o
If Lemma (4.1) is to be useful, then Wy = z must be easy to
solve. Also, the eigensystem of B must be readily available, and the
factorization A = W3Vt should be relatively cheap to compute. These
requirements rule out a full eigensyste. decomposition of A and also the
factorization of Aasen (1] which gives B in tridiagonal form. However,
the Bunch-Parlett factorization certainly satisfies all these require-
ments with the additional feature that K2 (W) has a bound that is
independent of A.
Fletcher and Freeman [8] have suggested the use of this factori-
sation to obtain a direction of negative curvature. The direction they
98
suggest corresponds to taking k - m in Lemma (4.1).. However, Lemma
(4.1) suggests that the best direction to use is with k - 1 since this
reduces the magnitude of the constant k2 [K2(W)J2 and is slightly cheaper
to compute.
5. A Steplength Algorithm
Once a descent pair (sd) has been determined at a point x then
we are faced with the problem of determining a such that
f (x.) < f (x)
where zx - x+a2s+ad, 0 < a. One solution would be to determine a such
that.
(5.1) f(x) - min f(x)
but this is a very difficult computational problem. It is computation-
ally more desirable to replace the problem of satisfying (5.1) exactly
with the specification of criteria for terminating a univariate minimiza-
tion procedure that is designed to solve (5.1).
Such an approach is motivated by the success of previous algo-
rithms which have been used when a single descent direction is specified.
Given a descent direction s at a point x, one such algorithm is to ter-
minate the line search when an i has been found which satisfies
(5.2) g(xis)t o ng(x)ts,
and
(5.3) f(x+s) <cf(x) + aRp(x)is ,
where 0 < p p n < 1 are preassigned constants. If a sequence of points
{xk) aredterinedwhere x kuk skithx xk, s -sk, 0 ak
99
satisfying (5.2) and (5.3) for each k, then -
(8k+1-8k) tsk >- -(1-n)g sk
and hence
(5.4) igk+l gk|| > -(1-n)g sk/||sk *
It follows from (5.4) that
(5.5) k||sk ->$(-(l-n) sk/||sk)
where # is the reverse modulus of continuity of g [17, p. 482]. Since
fk+l < k, and f must be bouinded below on the compact set L(x0), we have
that lim (fk -k+) = 0 and thus (5.3) impliesk~.
(5.6) (ak| ) J) sk/|k I + 0 .
Since $(tk) + 0 implies tk + 0 it follows from (5.5) and (5.6) that
(5.7) 112 g sk/IIsk * 0 .k4.
Usually gk and sk are related so that (5.7) implies Igkl|| + 0 which in
turn implies IIs.JI +0. Thus it is concluded that IIX, l-Xk + 0 and
IIgJI + 0 as long as the ak are bounded. This is enough to insure that
lit.xk = **
with x* a critical point of f due to the following lemma given in [17].
Lemma (5.&). Let f: V c nR+ R be continuously differentiable on the
compact set 0cVP. Let
S = {x: x E V03 g(x) - 0) *
and assme that S is finite. If (xk} c V0 is a sequence such that
MRlmk+lxkl = 0 , lim gk|| = 0 ,k~ee k+
100
then lim xk = x*, where x* E S..-
Proof: See [17, p. 4761.
A full discussion of this type of strategy may be found in [17]. Par-
ticular algorithms of this type are given in [12,17]. The strategy has
a geometrical interpretation which is depicted in figure 2.
y = f(x) + ang(x)is
y = c + ang(x)ts
Figure 2
A Search Along x+os
Here U = n and a+ is the smallest positive root of the equation
g(x+as)ts = ng(x)ts. The local quadratic approximation to f(x+ap) is
(a) = f(x) + ag(x) s +a2atG(x)s
which is convex near a = C if G(x) is positive definite as shown in
figure 2. Condition (5.3) guarantees sufficient decrease of the func-
tion so that 'gk' + 0 which means that f(x+ap) lies below the top line
in figure 2. Condition (5.2) guarantees that the distance IxkI+1 - k
does not become arbitrarily small. The picture indicates that the only
possibility for a* < k to be small is that x is close to a local
minimum.
The termination criterion we shall give may. be viewed as an
extension of these ideas which are suitable for the situation when an
iterate xk is an indefinite point. We replace (5.2) and (5.3) with the
following rule. If (s,d) is a descent pair at x then we terminate the
search when a has been found which satisfies
(5.8)
and
(5.9)
g(x ) (2as + d) > ngts + 2a(gts + $dtGd)J
f(xm) -+f+ 2 ts + tGd
with 0 < U < n < 1 as before. Note that when d = 0 these conditionsreduce to those of (5.2) and (5.3). Again there is a geometrical inter-
pretation which is depicted in figure 3.
-" f + angtd
f(xa)
a* aI IJ 0m
y - f + n[ag d + a2(gts+dtcd)J
y = c + q,[agtd + a2 St tGd)J
Figure 3
A Search Along x+a2
-0
/0000
102
Here a* is the smallest positive root of the equation
g(xa) t(2as+d) - n[gtd + 2 (g s +d .Gd)J
The situation shown in figure 3 describes the shape of f(xa) along the
curve
C:{x: xa x + a2s+ad},
where x is an indefinite point (see figure 4).
Cc
Figure 4
The Curve x+a2s+ad
An additional requirement is placed on a steplength algorithm at an
indefinite point. Sufficient decrease of the function must be used to
force the negative eigenvalues of the Hessian to zero as well as to
force the gradient to zero. This is guaranteed by condition (5.9). In
addition to this we must not let -x+l -xkI become arbitrarily small.
This is accomplished by condition (5.8). The at pictured in figure 3 is
similar to its counterpart in figure 2. The picture suggests that the
only possibility for a* to become small is for the iterate xk to be
close to a local minimum. The inflection point which must occur along
the path C must either be crossed or become "flattened out" in the
iterative process.
We note with Fletcher and Freeman (8) that if a direction dk of
negative curvature alone is used (taking sk - 0) then the condition
103
I .ldkI - -ng dk is inappropriate for te-mination of the linear search
because gSdkemay be close to zero even far away from a minimum. They
found it necessary to give termination criteria based on an estimate of
the first derivative of f(x ) at the inflection point. The estimate
was obtained from the value of the derivative of a related quartic poly-
nomial at its corresponding inflection point.
The following lemma will show that conditions (5.8) and (5.9)
can be satisfied whenever a descent pair exists at a point x.
Leuma (5.2). Let 9: R + R be twice continuously differentiable in an
open interval I which contains the origin; and suppose that L(0) c I is
compact where L(0) - (a e I: 0(a) < 0(0)). Let v E [0,1) and n c (,l).
Then if 9'(0) < 0 and 0"(0) < 0 there is an a E (0,=) n I such that
(5.10) *'(a) >i n['(0) + 0"(0)] ,
and-2
(5.11) *(a) 9< (0) + (p['(0) + 9"(0) j]
Proof: The assumption that ('(0) < 0 and 9"(0) < 0 implies the existence
of 0 c Iwith 9(a) < 9(0) for 0 < a < 0. Let 8 = sup{B: 9(a) < 9(0)
with 0 < a < B}. Then B > 0, and the assumption on L(0) implies S EI
is finite. The continuity of 4 implies #(0) - 9(B). Thus
2(5.12) 0(B) 9(0) + u[0'(0)B + ("(0) L-]
Define h: I + R by
2h(a) " "(a) - 0(0) - n[0'(0)a + 9"(0) 2j].
Since pIj n we have h(0) > 0. Note also that h(0) " 0, h'(0) 1 0,
h"(0)< 0. This together with the continuity of h implies the existence
of 1 E (0,] such that h(01) - 0, and h(a) < 0 for all a e (0,B1 ). Nov
104
Rolle's theorem implies the existence of a e (0,0 1)'such that h'(;) = 0,
and (5.10) follows. Also, h(a) < 0 and U .1 imply (5.11).
If we take 9(a) - f(x ) then Lemma (5.2) implies that conditions (5.8)
and (5.9) can be satisfied. In the next section we will show how these
conditions may be used to prove the convergence of a modified Newton
method.
6. Convergence of the Hodified Newton Iteration
Nov we turn our attention to defining a modified Newton itera-
tion. We shall give a convergence result based on the use of descent
pairs and the stepiength algorithm discussed above. The proof proceeds
in two parts. The first result is somewhat independent of the defini-
tion of the iterates. The second part will use the particular way in
which the iterates are defined to establish convergence.
The general iteration from a point xk begins with determining a
descent pair (sk,dk) at xk. Let
(6.1) "k(a) f(xk + a 2k + adk)
Assume u c (0,1) and n c[i,l) are independent of k. Then ak > 0 is
determined such that
*(6.2) yk " Xk + aGk + akdk V,
2ak
(6.3) f(yk (xk) + p9 (0) j2
(6.4) *"(ak) > n [(0) + 4k(0)ak *
Take Xk+l 7= k'
One might note that due to (5.11) in the statement of Lemma (5.2)
105
we could require f(yk f(xk) + u cO(0)ak + 0'(0) instead of (6.3).
However, the additional term does not enhance the convergence result in
any way, while it does give a more stringent requirement to be satisfied
by the univariate search. The first step in the convergence result is
Theorem (6.1). Let f satisfy assumptions (1.2). Then the iteration
defined above satisfies
(6.5) lim - g sk = 0 ,k+
and
(6.6) lim 4Gkdk - 0
Proof: From (2.7) and (2.8) we have 6I(0) - g dk and 0k(0) = 2g4sk +
dkGkdk. Since (sk,dk) is a descent pair, $k(0) 1 0, and 0k(0) < 0.
Thus (6.3) implies that {x} cL(x0 ). By the continuity of f and
compactness of L(x0 ) we have lim(fk fk+l) = 0. Now
2
k -k+l > -U6"(0) > > 0, so that
(6.7) lim -ogksk -0,k-e-
and
(6.8) him -adtGkdk -0.
From condition (6.4) we obtain
O (ck) - # (0) - k#"(0) > -(1-n) ((0) + 9' (0)zkI *
and hence
(ak) - 4k(0) - c9Ak (0)->-(l-k )O(0)%*
An application of the mean value theorem now yields that for some
0k' (OGk)I
106
(6.9) *k(6k) - s.(0) > -(-) (0) .
The desired result now follows readily, for if either (6.5) or (6.6) do
not hold, then there is a subsequence {ki} and a o > 0 such that
(6.10) -0k (0) > a > 0 .i"
Hence (6.9) implies that {Gk } does not converge to zero. However, ifi
(a } does not converge to zero and (6.10) holds, then (6.7) and (6.8)
cannot be satisfied. This contradiction establishes the theorem. oThe {ak} of (6.2)-(6.4) are -o be determined by a univariate
minimization algorithm applied to Ok(a). Let 6 > 0 be fixed, and termi-
nate the search when 0 < a < 6 has been found such that (6.4) is
satisfied with ; in place of ak. If (6.3) is also satisfied we accept
-k= a. If either (6.4) cannot be satisfied (say within a fixed number
of steps) or if a does not satisfy (6.3) we take w to be the largestelement of the set {2I: 1-0,1,2,...} such that (6.3) is satisfied with
aw in place of ak and then accept ok = &nw. If infinitely many of the
ak's must be determined in this way, then Theorem (3.1) applies so that
(6.5) and (6.6) are still obtained. We shall call this process the
steplength rule SR(p,n,B).
Our next result will show that the iterates defined by this
steplength rule converge to a critical point of f where the Hessian is
positive semidefinite. It is here that specific properties of the
descent pairs (sk,dk) are crucial.
Theorem (6.2). Assume in addition to the hypothesis of Theorem (6.1)
that f has finitely many critical points in L(x0). Suppose that the
sequence {xk: k=0,1,2,...} has been obtained using the steplength rule
107
SR(v,i,B) where the descent pairs (skdk) satisfy
("sk|,' IIdkII: k=0,1,2,...} is bounded together with
(6.12) (gk + 0) -> (gk + 0 and sk-+ 0) ,
and
(6.13) (d(Gkdk G 0) (k + 0 and dk * 0) 9ask+'.. Then
lim3kX
with g(x*) - 0 and G(x*) positive semidefinite. Moreover, if infinitely
many of the xk are indefinite points, then G(x*) must have at least one
zero eigenvalue.
Proof: From Theorem (6.1) we see that lim g k - 0 and lip d Gkdk " 0.
By ( 6 .12) we have gk +0 and sk + 0. By (6.13) we haveAGk +00 and
dk + 0. Now,
Ixk41 - ij . 05k2II + BId jI
hence li Ixk, 1 - Ek 0.k4.e
Therefore, Lemma (5.1) applies and we obtain
limzXk ** *kim
with g(x*) - 0. Since AG + 0, we also have by the continuity of GGk
that G(x*) must be positive semidefinite. Moreover, if infinitely many
of the Xk are indefinite points then every neighborhood of x* contains
an indefinite point. Thus the continuity of G Implies that G(x*) has at
least one zero eigenvalue. o
Obviously, the proof of Theorem (6.1) rests on the steplength
108
rule, while the proof of Theorem (6.2) rests on the, particular choice of
the descent pairs. Many choices of sk are possible which satisfy (6.12).
Indeed, if A is any sequence of symmetric positive definite matrices
such that Ak |,IIA|II are bounded independently of k, then choosing sk
as the solution of
Aksk "-k
will satisfy (6.12).
In section 3 we gave several ways to choose the dk at an indefi-
nite point so that
(d4Gkdk Gk0) -> Ak0)
The additional requirements that dk must satisfy are obtained if we
replace dk with y(AGk)dk, where y is a function such that y(tk) + 0 u>
ktk4 +0, and where the sign is chosen to make g dk 0.
The iterates should also reduce naturally to Newton's iteration
as soon as a region is found where the Hessian is positive definite.
Indeed, the main motivation for this strategy is to obtain the iterates
using second derivative information which is based on the true quadratic
model at each xk. Of course, it is expected that in practice very few
indefinite points will be encountered during the iterative process. In
fact, Theorem (6.2) indicates that the strategy we have presented
actively seeks a region where the Hessian matrix is positive semi-
definite. If, for example, the Hessian G(x) is nonsingular whenever x
is a critical point of f then only finitely many of the iterates can be
indefinite points.
Finally, we shall suggest a way to obtain the descent pairs
109
(skdk) which satisfy all of the requirements of Theorem (6.2). In our
description we assume Gk - MkDk?4 is the Bunch-Parlett factorization of
the Hessian. Thus we have omitted explicit representation of the per-
mutations Qk which will be present in practice. Given f and x0 which
satisfy the hypothesis of Theorem (6.2), for k=0,1,2,,... define
(6.14) sk as the solution of
(kDkMk)sk =gk
where Dk - UkAkU is obtained from Dk by first
obtaining the eigensystem Dk = UkAkU of Dk and
then replacing the diagonal elements A(k) of Ak
with
max (A(k) 1 , en max ixik)i, c)
1<j<n 1<i<n
where c is the relative machine precision. In the decomposition of Dk
we have Uk k = I, and Ak diagonal. Note that only 0(n) arithmetic
operations are required to obtain Dk from Dk.
(6.15) dk is the solution to
Mtdk -= 'ID IhMk * Dk z
where AD is the most negative eigenvalue and ZkDk
the corresponding unit eigenvector of Dk. When Dk
does not have a negative eigenvalue we take dk = 0.
The compactness of L(x0 ) and the continuity of G imply that the
elements of Gk and the components of gk are uniformly bounded. Thus
(sk,dk) satisfy the requirements of a descent pair as well as (6.12)
and (6.13) due to the bound on the condition numbers K2Mk )
110
The above choice of (sk,dk) is somewhat ad 'hoc and we make no
mathematical statements concerning the desirability of this choice.
However, in the next chapter computational results will be reported
which show that this specification of (sk,dk) works reasonably well in
practice. We wish to emphasize that many other choices are possible.
We have not addressed the problem of providing an initial step a
to the univariate search. Many strategies for determining the initial
step are possible. However, we have not found a strategy with enough
theoretical basis to recommend it over something very simple such as
taking the initial step to be a - 1 each time. Note, however, that
whatever strategy is chosen must eventually take a = 1 in order to retain
the local quadratic rate of convergence enjoyed by Newton's method.
7. Conclusions
The algorithm we have just described has the following informal
description:
(7.1) Given c Pe
for k-0,1,2,...
(1) Determine a descent pair (sk,dk)
(2) Determine ak by SR(p,n,O)
LL) xk+l= k+ sk+akdk .
Step (1) involves evaluating and factoring the Hessian Gk. Step (2)
involves the use of a univariate search. that can satisfy SR(u,n,B).
The importance of this iteration is that it represents a natural
extension of previous theory to include second derivative information.
It avoids saddle points and possesses a strong theoretical convergence
111
property. Finally, the iteration, even in this preliminary stage of
development, performs well in practice.
Chapter V
Computational Results
1. Introduction
The purpose of this chapter is to present computational support
of' the theoretical results obtained in chapters II, III, and IV. The
updating algorithm was tested for timing and accuracy on a large number
of random updating problems. The optimization algorithm was tested on a
set of test problems which have been used extensively at Argonne
National Laboratory for such purposes [5]. In addition to this, the
algorithm was tested on some problems which demonstrate its behavior
when many indefinite points are encountered during an iteration.
2. Testing the Updating Algorithm
There are two important criteria for testing an updating algo-
rithm. The first criterion is that the updating algorithm actually
should represent a computational savings over the alternative of forming
the updated matrix and refactoring. The second criterion is that solu-
tions of linear equations using the updating method should be reasonably
close to solutions obtained by forming the updated matrix and
refactoring.
Timing the updating algorithm and comparing to the alternative
is a straightforward task. In order to address the question of accuracy
one crust decide what quantities should be measured and compared. For
each update it seems reasonable to compare
(2.1) IAx-bI/IbI
112
113
with
(2.2) IIAxu-bI/IIbII
for several right hand sides b. In (2.1) the vector xc is the solution
obtained by forming and refactoring the updated matrix. In (2.2) the
vector xu is the solution obtained by using the updating algorithm. The
quantity
(2.3) fxc u /Ixc
should also be computed.
The quantities in (2.1) and (2.2) measure the relative error in
the residual. This relative residual indicates how close the computed
solution is to satisfying the equation Ax = b relative to the size of
the right hand side b. The quantity (2.3) measures how much the answer
obtained by the updating method has deviated from the answer obtained by
computing and refactoring the updated matrix.
The process used to test these criteria can most easily be des-
cribed by means of an informal algorithm. Given a dimension n, we start
with A = In the nxn identity matrix. Then the following iteration is
carried out.
(2.4) A := I
for k-0,1,2,...m,
(1) z c Rn is chosen with random components in (-1,1);
(2) a R is a random number in (-100,100).
(3) A = A + asst
(3.1) Q Q = sUD by updating;
(3.2) Q = Mc c by forming A and factoring;
114
(4) for j-1,...,5
" (4.1) b c Rn is chosen with random components in
(-50,50);
(4.2) Solve Ax - b
(i) Using (3.1) to compute xu;
(ii) Using (3.2) to compute xc;
(4.3) Compute
J ~(i) |Axc -b||1/ I|b||;
(ii) ||Axu-bI||/|Ibil;
(iii) |xc xU||1/||b||;
The steps (3.1) and (3.2) of iteration (2.4) were timed. These
timings were averaged over the number m of.updates. Thus the time re-
quired by the updating algorithm can be compared to the time required by
the alternative of computing A and refactoring. The solution .to Ax = b
was computed for five different right hand sides after each update. This
was done to increase the chances of obtaining a large residual
IIAxu-bI /|bII. The quantities (2.1), (2.2), and (2.3) were averaged over
all iterations and right hand sides. The results are shown in tables 2
and 3.
Table 2 shows the above quantities for various values of the
dimension n. In Table 2 UAVE is the average value of i Axu-b Is/fIbl, CAVE
is the average value of NIAxc-bII/ Ibil, and AVERR is the .average value of
1 x4 u1 I tcjI. The quantity CTIME is the average time to compute and
refactor A and UTIME is the average time to update the factorization.
115
Table 2
Results for Increasing Order
The times shown here are in microseconds. The important thing
to note is the relationship of UTIME to CTIME as n increases. To see
that the numbers are in the correct proportion one should compare n2 to3
UTIHE and - to CTIME. Observe also that there is roughly only a one
digit loss of accuracy using the updating algorithm. For each of the
results in Table 2 we have taken m = 100 in (2.4).
Table 3 shows the results of a particular updating sequence
computed by the iteration (2.4). In this example n = 10. The updating
process was carried out for 1000 updates. The results show every fifth
update selected from the beginning,, middle, and end of these computa-
tions. In Table 3 the quantities are not averaged. UERR is
IAx-b|/IbI, CERR is IAxc-bII/tb, and XERR is Rxc", I/Ixc I for only
one right hand side. UTIME and CTIME are the timings for each indi-
vidual update in this case. For the entire sequence, the average
quantities were UAVE . 2 x 1013, CAVE = 3 x 10-15 , and AVERR =lx 10-13
n UAVE CAVE AVERR UTIME CTIME
5 6x10 1 4 4x10 15 4x1014 167 424
10 2 x 10-13 2 x 1014 3 x 10-13 320 1567
20 1 x 1013 3 x10 1 x1013 706 6459
30 3 x 10C13 7 x 10 14 2 x 10-13 1162 16606
40 8 x 10-13 2 x 1013 4 x 10 13 1819 32468
50 2 x 1012 3 x 10 13 4 x 10-13 2533 55016
Results of
Table 3
a Long Range of Updates
beginning
middle
end
These results indicate that the error analysis in Chapter III is
somewhat pessimistic. In particular, Table 3 shows that obtaining the
factorization by the updating method does not deteriorate much even over
a long range of updates. The timings show that the operation count
given in Chapter II was indeed a worst case analysis. They indicate
that the worst case rarely happens. This is demonstrated in Table 3
since for matrices of order 10 the operation count predicts that the
updating algorithm should require as much work as the alternative.
116
UERR CERR XERR UTIME CTIME
6 x 10 5 4 x 1015 8 x 1016 312 1563
7 x 10 3 x10 6 x10 313 1875
3 x 10-15 7 x 10-16 3 x 10-15 521 1979
1 x 10 5 x 10-15 2 x 10 1 5 - 417 2083
1 x 107 4 x 10-15 2 x 10- 417 1563
5x10 1 4 6x10 1 6 5x101 312 1667
2x10 1 3 1x10 15 7x10 1 4 313 1562
8 x 10 6 x 10 4 x 10 208 1563
7 x 10 1 x 10-15 5 x 1014 312 1980
1 x 101 4 x 1016 1 x 10-13 312 1667
7 x 10-1 3 3 x 10-15 2 x 10 208 1458
5 x 10 1 3 3 x 10 1 5 5 x 10 13 312 1667
5 x 10 3 1 x 10-15 2 x 1012 209 1458
2 x 1013 1 x 1015 5 x 10 209 1771
1 x1013 1 x 10-15 2 x 1013 312 1563
117
One disadvantage of the updating algorithm, is the length of
computer code necessary to describe the algorithm. The timing results
indicate that it would be a worthwhile project to see if the length of
code could be decreased; perhaps at the expense of increasing the
operation count slightly.
3. Testing the Modified Newton's Method
The unconstrained optimization algorithm described in Chapter IV
was tested on some standard minimization problems. The computer imple-
mentation is still under development. Therefore, the results presented
here are to be regarded as an indication that the method is promising.
There are a number of practical considerations that must be settled
before this algorithm can be recommended for general use.
One of the practical problems is the choosing of the descent
direction s at an indefinite point. We have described one way in
Chapter IV, but we feel that others should be tried. Also, it is not
clear what the scaling of the descent direction s should be relative to
the direction of negative curvature d.
Another problem is choosing the initial step for the linear
search procedure at an indefinite point. Enough information is avail-
able at an indefinite point to use a cubic polynomial to predict an
initial step. To do this, one interpolates f, f', f" at x where the
derivatives are taken along the curve x+a2s+ad. The resulting cubic
polynomial is then required to achieve a decrease A at its local minimum
a. The number A is the amount of decrease obtained on the last itera-
tion. This process uniquely defines a polynomial p. We then have
118
a = (3A-2f')/f" .
We also require that the initial step a0 satisfy .5 < a0 <1. Thus weA A A
take a0- a if .5 < a < 1. Otherwise we take the closest endpoint to a.
Obviously there is little theoretical justification for this choice of
a0, but it does an adequate job when safeguarded as mentioned.
Finally, there is always the task of choosing parameters. For
instance we must specify p, ri, and 0 for the steplength rule SR(p,n,8)
(see Chapter IV). In addition to this we must specify criteria for
accepting an iterate as an approximation to .a local minimum. This, of
course, requires the specification of other parameters.
In the following examples we have taken ' - 10 4, n = .9, and
8 - 106. An iterate xk is accepted as an approximation to a local mini-
mum when
(i) The Hessian is positive semidefinite,
(ii) I|fk k-11 (2+e) (l+lfk *
(iii) ak-1||sk-1|| < ( +e) (1+||xk *
(iv) gk 2/(l+|fkl2
Here c is the relative machine precision. The parameter r is specified
by the user but defaults to 10/ if. found to be smaller than c. For
these examples t is given the default value. These stopping criteria
are used in the Gill and Murray algorithm. We have adopted them in order
to obtain a good comparison of the two algorithms. These functions were
used as test problems:
Rosenbrock's Problem;
n a 2,
f - (1-x1)2 + 100(x2-22
standard start:
(3.2)
(3.3)
(3.4)
(3.1)
119
Powell's Function of Four Variables;
n = 4,
f = (x1+10x2)2 + 5(x3-x4)2 + (x 2-2x 3 )4 + 10(xl-x
standard start: (3.0, -1.0, 0.0, 1.0).
Brown's Function with Two Global Minima;
n = 2,
f - (x-x2 1)2+ x((-x 2) 2 + (x 2-0.5) 2 2
standard start: (0.1, 2.0).
Powell's Badly Scaled Function of Two Variables;
n = 2,
f = (104x1x2-l)2 + (el + eX2 - 1.0001)2
standard start: (0.0, 1.0).
Box's Function;
n = 3,
f . E(aehh _ x e n2 Gi - x (e"ei . e-106i) 2=1
where 6 i/10,
standard start: (0.0, 20.0, 20.0).
4,
(3.5)
(-1.2,1.0).
120
(3.6) Wood's Function; -
n - 4,
f - 100(x2 -x2) + (1-x1 )2 + 90(x4 -x3 )2 + (1-x3)2
+ 10.1((x2 x 1)2 + (4-1)2) + 19.8(x 2 -)(x 4 -),
standard start: (-3.0, -1.0, -3.0, -1.0).
(3.7) Penalty Function I;
n = 4,- n 2x2- 2
f-A j 1 (x 1 -l) 2 + B[ x - , where A =105,Bit 1
standard start: xi - i, (for 1 < i _ n).
(3.8) EXP6;
n = 6,23
f = x3e-x1Zi - 1 4 j-x2Zi + x6g-xSZ - Y) 2,i-i I
where yi - ezi - Se-lOzi + 3e4zi,
s- (0.1)1, (for 1 < i < 13),
standard start: (1.0, 2.0, 1.0, 1.0, 1.0, 1.0).
(3.9) Brown's Badly Scaled Problem;
n - 2,
f - (xi-106)2 + (x2-2x10-6)2 + (xlx2-2)2,
standard start: (1.0, 1.0).
(3.10) Beal's Function;
n - 2,3
f - (ci-x1 (1-x2 ))2i-l
where c 1 - 1.5, c2 - 2.25, c3 - 2.625,
standard start: (1.0, 1.0).
121
(3.11) Rosenbrock's Cliff Function; '
n = 2,
f = x 2 ( 1-x 2 ) + e20(x1-x2)100 (1J)
standard start: (0.0, -1.0).
(3.12) Cubic Function;
n = 3,7
f = Ef + 2,..
where f1 -=x1, f2 = f3 = 0.1x1(x2 1) 2, f4 2(x214,
f5 6 2 2 4f - =f 6 - 0.1x1 (x3-1) 2, f 7- (x3 l) 4,
standard start: (2.0, -3.0, 3.0).
(3.13) Gottfried's Function;
n - 2,
f - (x1 - 0.1136(x1+3x2)(l-x1))2
+ (x2 + 7.5(2x1 -x2)(l-x2))2
standard start: (0.5, 0.5).
(3.14) Four Cluster Function;
n 2,
f = (x1-x2 )(x 1-sin(x2))2
+ [(cos(x2)-x1)(x2-cos(x1))]2
standard start: (0, 0).
(3.15) Hyperbola-Circle Function;
a - 2,
f (xx2 1) 2 + (x+x 2 -4)2,
standard start: (0.0, 1.0).
122
Table 4 shows the results of these tests on problems (3.1) -
(3.15) with the starting point x taken to be the standard start. The
results of the Gill and Murray algorithm on the same problems are also
given in this table. For each problem the first entry is the result of
the algorithm presented in Chapter IV and the second entry is the result
of the Gill and Murray algorithm. The quantities represented are:
NITER = the number of Hessian evaluations,
NFEV = the number of function evaluations,
t28 a - NII
POSDEF = T if the Hessian was found to be positive
semidefinite at the solution, and F otherwise,
NEGCNT = the number of indefinite points encountered
during the iteration,
FLAG = 0 means normal termination.
1 means abnormal termination.
(We note that for either algorithm an abnormal
termination may .have been indicated even though
the approximation was close to the solution.)
123
Table 4
Results of Tests with Standard Starts
# NITER NFEV g g POSDEF NEGCNT FLAG
21 28 5x 10 19 T 0 03.1
23 29 6 x 10-27 0
29 30 5 x 10-2 6 T 0 03.2 ___
- 2525 5 x 10-21 -0
8 10 0.0 T 0 03.3_-
9 10 1 x 10 '.0
138 239 1 x 10 T 2 03.4
186 344 2 x 10 _ 1
14 18 2 x 10-4 T 2 0
.5 10 10 4 x 10761
38 48 2x10 T 1 0
3.6 39 50 1 x 10 1
34 43 2 x 10 T 0 0
36 44 1 xO 10 0
527 1000 .6 x 10-6 F 527 13.8-4
47 382 4 x 16-0
8 10 2 x 10 T 1 03.90
8 10 2 x 1010
9 11 9 x 102 0 T 2 03.10 7
9 71 0 20x10
27 28 3 x 102 T 0 03.11 -
28 29 1 x 1070 1
66 67 2 x 10 T . 0 03.12 - r0
332 3 33 1 x 10-0
8 16 9 x 10~b T 3 03.13
4 1354 4 x 10 1
11 12 1 x 107 T 0 03.14-
14 33 1 x 1072-_ 0
6 7 0.0 T 1 03.15 ..
7 8 9 x10 _ _ 0
124
The algorithm presented in Chapter IV requires the calculation
of the Hessian from an analytic expression in order for the underlying
theory to be valid. However, one may want to use the algorithm with a
finite difference approximation to the Hessian. In Table 5 the results
of using such an approximation on problems (3.1) - (3.15) are presented.
The headings in this table are as in Table 4. Again we use the standard
starts for x0 . It should be noted that except for Powell's Badly Scaled
Function (3.4), there is little difference between the behavior of the
algorithm with finite differences and with analytic derivatives.
Results from
Table 5
Using Finite Differences
I NITER NFEV g g POSDEF NEGCNT FLAG
3.1 21 28 1 x 10 18 T 0 0
3.2 29 30 7 x 10-26 T 0 0
3.3 8 10 3 x 10-28 T 0 0
3.4 553 1000 1 x 103 F 546 1
3.5 14 18 3 x 10~25 T 2 0
3.6 38 48 5 x 10 17 T 1 0
3.7 34 43 3 x 10 21 T 0 0
3.8 561 1000 8 x 10-6 F 561 1
3.9 11 23 2 x 10019 T 2 0
3.10 9 11 3 x 10 19 T 2 0
3.11 34 44 2 x 10716 T 16 0
3.12 66 67 2 x 10-52 T 0 0
3.13 8 16 3 x 10 17 T 3 0
3.14 11 12 2 x 10-20 T 1 0
3.15 6 7 6 x 10-30 T 1 0
The use of standard starting points on these test examples does
not fully reveal the performance of this algorithm. Some of the
125
standard starts are in regions such that little or, no negative curvature
is encountered during the iteration. In order to demonstrate how the
algorithm performs when many indefinite points are encountered, we
include results of the algorithm on problems (3.5), (3.8), (3.9), and
(3.13) with random starting points. These results are presented in
tables 6, 7, 8 and 9. In each table the results from ten random starting
points are given. For each point there are two entries. The first is
from the algorithm presented in Chapter IV and the second is the result
from Gill and Murray's algorithm on the same problem.
Table 6
Box's Function
tI NITER NFEV g g POSDEF NEGCNT FLAG
25 36 1 x 10- T 21 0
1 24 140 4x 10-$1
16 17 2x 10- 3 1 T 3 0
2 36 97 1x 101
14 15 3 x 10-3 1 T 3 03 --27__
27 70 5 x 10 1
20 26 1 x 10-2 0 T 2 0
4 20 20 6 x 10-330
26_"42 0.0 T 22 0
37 118 5-x 10-25 1
22 41 1 x 1032 T 17 0
26 64 1 x 10-22 120 33 1 x 10 22 T 18 0
19 26 1 x 10 27 0
18 22 4 x 10 28 T 3 0
9 9 2 x 10 21 0
16 20 5 x 10-25 T 1 0
14 11 x 10 2 0
12 16 8 x 10 32 T 7 010 -2533 95 2 x 1 1
126
Table 7
EXP6
NITER NFEV gtg POSDEF NEGCNT FLAG
485 730 2 x 1021 T 41 0
2 81 overflow 1
6 6 6 x 107 F 6 12 -6___
- 135 198 4 x106 -1
25 25 2 x 108 F 25 13-28
79 147 2 x 10-20
202 276 5 x 10-2e T 200 0
108 353 2 x 10 1
8 71 6 x 102 F 8 15
98 418 3 1
6 40 431 4 x 10 20 T 38 0
128 253 4 x 1076 1
335 543 2 x-10'1 9 T 59 07--4
12 227 9 x 10 41
8 98 138 5 x 10 19 T 32 0
88 474 4 1
865 1000 2 x 10 4 F 865 19 --
112 332 2 x 10 1
810 1000 5 x 10-5 F 810 110 .108 358 1 x 1
127
Table 8
Gottfried's Function
* NITER NFEV g g POSDEF NEGCNT FLAG
26 29 2 X 107 T 3 01
27 29 2 x10_0
18 26 3 x 10 1 8 T 3 0
19 22 5 x 10
12 17 8x10 T 3 0
14 15 1 x 10-2 0
21 29 1 x 10-20 T 6 0
1 1001 2 x 101
19 27 5 x 10-20 T 8 0
51 1001 2 x 105
13 15 1 x '10T _ _ T 3 06 -
1 1001 9 x 101 2 1
25 27 1 x 10 1 T 2 0
25 26 4 x 10- 2 2 -0
21 25 8 x 10 2 1 T 4 08 1
1 1001 2 x 10 5 1
15 16 1x1027 T 2 0
1 1001 4 x 10161
14 16 1x10-26 T 3 010 -1001 3 x 014
1 1001 3 x 10 '1
128
Table 9
Brown's Badly Scaled Problem
I NITER NFEV gtg POSDEF NEGCNT FLAG
22 43 2 x 10 4 T 14 010
24 105 2 x 10_4 0
21 43 2x104 T 14 0
2 21- 89 8 x 10191
21 40 2 x107 T 13 03_
21 91 4 x 10 1
19 45 2x10 T 8 04_
19 81 4 x 101 1
21 40 2 x 10~ T 14 0
21 90 4 x 10 1
19 42 2 x10 T 12 06
21 95 2 x 10- 9 1
21 41 2 x 10 1 9 T 14 07 -8
22 99 1 x 10 8 1
22 41 2x10 4 3 T 15 08
21 101 2 x 10-4 0
23 44 2 x 1071 T 16 09
23 111 1 x 10 1
10 19 *36 2 x 10 T 10 010121 97 4 x10 -4 1
129
Tests using random starts were made with the other functions as
well. However, with the exception of problems (3.5), (3.8), (3.9),
(3.10), and (3.13), the results shown in Table 4 were consistent with
results using random starts. On Beal's Function (3.10), the two algo-
rithms behaved quite differently depending upon the starting point. The
problem exhibited a lot of negative curvature. However, one algorithm
would do much better than the other on one starting point, but the oppo-
site situation would occur on another starting. point.
We have compared these results with the results obtained by the
algorithm of Gill and Murray [11], and have found them to be competitive.
This is encouraging since the Gill and Murray algorithm has undergone a
thorough development and is one of the best codes available.
The results shown here indicate that the method presented in
Chapter IV is promising. Further development is needed in the practical
problem areas discussed at the beginning of this section. However, the
evidence so far indicates that a fully developed algorithm has the
potential of being a reliable and efficient method for unconstrained
optimization.
130
References -
[1] Aasen, J. 0., "On the Reduction of a Symmetric Matrix to Tri-diagonal Form," BIT, 11, (1971), 233-242.
[2] Armijo, L., "Minimization of Functions Having Lipschitz ContinuousFirst Partial Derivatives," Pac. J. Math. 16, (1966), 1-3.
[3] Bunch, J. R., "Analysis of the Diagonal Pivoting Method," Siam J.Numer. Anal., 8, (1971), 656-680.
[4] Bunch, J. R. and Kaufman, L., "Some Stable Methods for Calculat-ing Inertia and Solving Symmetric Linear Systems," Math. of Comp.,31, (1977), 162-179.
[5] Bunch, J. R. and Parlett, B. N., "Direct Methods for SolvingSymmetric Indefinite Systems of Linear Equations," Siam J. Numer.Anal., 8, (1971), 639-655.
[6] Dahlquist, :. and Bj5rk, A., Numerical Methods, Prentice-Hall Inc.,New Jersey, 1974.
[71 Dennis, J. E. and More, J., "Quasi Newton Methods, Motivation andTheory," SIAM Review, 19, (1977), 46"89.
[8] Fletcher, R. and Freeman, T. L., "A Modified Newton Method forMinimisation," U. of Dundee Rept. No. 7, (1975).
[9] Fletcher, R. and Powell, M. J. D., "Modifications of LDLt Factor-izations," Math. of Comp., 28, (1974), 1067-1087.
[10] Forsythe, G. E. and Moler, B. C., Computer Solutions of LinearAlgebraic Systems, Prentice-Hall, 1967.
[11] Gill, P. E. and Murray, W., "Newton Type Methods for Unconstrainedand Linearly Constrained Optimization," Math. Prog., 7, (1974),311-350.
[12] Gill, P. E. and Murray, W., "Safeguarded Steplength Algorithms forOptimization Using Descent Methods," NPL Rept. NAC37, (1974).
[13] Gill, P. E., Golub, G. H., Murray, W., Saunders, M. A., "Methodsfor Modifying Matrix Factorizations," Math. of Comp., 28, (1974),505-535.
[14] Gill, P. E., Murray, W., Saunders, M. A., "Methods for Computingand Modifying the LDV Factors of a Matrix," Math. of Comp., 29,(1975), 1051-1077.
131
[15] Hillstrom, K. E., "A Simulation Test Approach to the Evaluationand Comparison of Unconstrained Nonlinear Optimization Algo-rithms," Rep. ANL-76-20, Argonne National Laboratory, Argonne, IL,(1976).
[16] McCormick, G., "Second Order Convergence Using a Modified ArmijoStep Size Rule for Function Minimization," The George WashingtonUniversity Serial T-328, (1976).
(17] Ortega, J. M. and Rheinbolt, W. C., Iterative Solution of Non-linear Equations in Several Variables, Academic Press, New York,(1970).
[18] Powell, M.J. D., "A New Algorithm for Unconstrained Optimization,"Nonlinear Programming, J. B. Rosen, 0. L. Mangasarian andK. Ritter, eds., Academic Press, New York, (1970).
[19] Stewart, G. W., Introduction to Matrix Computations, AcademicPress, New York, 1973.
[20] Wilkinson, J. H., The Algebraic Eigenvalue Problem, ClarendonPress, Oxford, 1965.
132
Appendix Al .
PATH 1STR
ENTER lxl
K = N STOP
Update Ifthe Kth T D > a aw(k) F next block 2x2 GO TOcolumn. 11 1 sPATH 3
K +K + 1 is
If Join the next
nextbloc 2x2lxl block to
istboc x form a 2x2
PATH 2
Update
2x2 block ENTER 2x2
Update column TaD >mx(ID DF facor n
K and K + 1 1 D |11 ,|2 )acrK: + + obtaining twoK~~~x +K+2x blocks
If 2x2- Update thenext block
Kth column.
is K+ K+ 1
lxl
GO TO PATH 1 GO TO PATH 2 GO TO ENTER 1x1
133
PATH 3
Jointhe next2x2 block
to form a 3x3
If
F m. D > mxD | T
Do a. Do a
1x1 pivot 2x2, pivot
and and
L factor factor
Update Update
the k-th- the K-th and
column.. K+1-s t columns
K+K+1 K+-K+2
GO TO ENTER 2x2 [GO TO ENTER lx l
Appendix A2
SUSPO)UTIy -SY:1u0(ANLD,N,SlGPAZCHANGE,0,W)OOUHtI.E PRF CTSIrIr SIGMAINTEGER "2,.;L').fllf!LE PRECISI'1 A(%LO.':I,Z (tJICHANGE(N) ,W(N)INTEGER (PJ3
C*********6**S*$A**$****r*r*******************.*****************CC THIS SURRCUTItX COMPUES THE UPDATED SYMMETRIC FACTRIZATIONC CF AN N X N SYI'4ETRIC PATR IX A 'FOLLOWEC BY A PAK CfE UPDATEC OF THE FORM A + SIGMA*ZZ' . IT IS ASSUMED THATCC OAO' " MC M'CC WITH D BLOCK 2IAGONAL CONSISTING OF 1 X 1 AND 2 X 2 DIAGONALC BLOCKS, AND M OCCUPYING THF LOWF; TRIANGLE CF THE PHYSICALC AR.RAY A. THE LOCK STRUCTURE OF 0 IS INDICATED EY THEC ARRAY CHANGE.C CHANGEIKI 1 I IF ENTRY K IS A 1 X 1 BLOCKC " 2 IF ENTRY K IS THE START OF A 2 X 2 HLCCKC a THE DETEIrINAt:T OF THE 2 X 2 BLOCK WHICHC STARTS AT ENTY K-1.C THE ARRAYS A.0.PIVUT ARE OVERWRITTEN WITH THE UPCATEDC FACTCRIZATIONCC MOM' " Q A + SIGPA*ZZ')C'CC THE UPPER TRIAN(LE CF THE PHYSICAL ARRAY A IS NOT ALTFREDC IN ANY MAN.P. THUS A COPY OF THE ORIGINAL MATRIX A NAY BEC STORED IN T'F UPPER TRIANGLE CF A IF A HAS CIMENSIONSC N X N+1. THE VECTOR Z IS NOT ALTEPEO.CCC *ss ***e*******
CC A IS A RFCTANGULAR.ARRAY WHOSE LEADING DIMENSION ISC NLD. THIS ARRAY IS ASSUMED TG CONTAIN THE SYMMETRICC MATRIX A IN FAC1OREC FCRM AS DESCRI ED AOVE. THEC LOWER TRIANGLE CF A CONTAINS THE MATKIX M. THE BLOCKC DIAGONAL MATRIX 0 IS STORED IN THE CORRESPONCINGC BLOCK OIAGONAL LfCATICNS OF TFE ARRAY A. THIS ISC POSSIbLE SINCE IF D(I,JI (I .NE. Ji IS I.CNZEFOC THEN M(I,JI IS ZERO. THEREFORE, THESE LOCATIONSC AS WELL AS THE DIAGONAL ENTRIES OF A MAY BF USED TOC TO STORE D.CC NLD THE LEADING DIMENSION CF THE ARRAY A.CC N THE DIMENSION OF THE MATRIX A.CC SIGMA THE SCALAR DESCRIBED ABOVE.CC 2 THE N DIMENSIONAL VECTCR IN THE UPDATING FORMULA.CC CHANGE THE N DI9FISIONAL ARPAY WHICH INDICATES THE BLOCKC STRUCTURE OF THE BLOCK DIAGONAL MATRIX 0. THEC CONTENTS OF THE ARRAY CHANGE ARE DESCRIBED ABOVE.CC 0 AN N DIMENSIONAL INTEGFR APRAY THAT INDICATES THEC PIVOTIN", NECESSARY TO OBTAIN THE FACTORIZATION.CC W AN N DIMENSIONAL LINEAR WORK ARRAY.C
CC***O*** **m*e****6****s*6666*6.s* s+*6s6sU*6* 666*666*666 6*6***66*6
DOUBLE PRECISION MAKEUMDOUBLE PRECISION ALFA,O11,O21.,31,D22,032,033, B1. B2, B3, UI,UO,
1 T,T1,T2,TCTL.L2INTEGER Q1,02,03,1,10 eII .K I , ,eKM1,KP1,KP2,KP3,J
cALF.A'Il.0DO * OSOPT Il7.ODOI )/8.00o
134
100.100.1100.2100.3100.4100.5100.,100.7100.8100.9101.101.1101.2101.3101.4101.5101.6101.7101.8101.9102.102.1102.2102.3102.4102.5102.6102.7102.8102.9103.103.1103.2103.3103.4103.5103.6103.7103.8103.9104.104.1104.2104.3104.4104.5104.6104.7104.8104.9105.105.1105.2105.3105.4105.5105.6105.7105.8105.9106.106.1106.2106.3106.4106.5106.6106 7106.8106.9
135
PO 100 Jul.N 107.WIJ121(4IJ11 107.1
100 '4TI'4UE 117.2Kul 107.3.
101 CONTINUF 107.4C 107.5C THE PROCESSING AcGI1.S HERE 107.1C 107.7
KDluK*1 107.6KP2=K+2 107.4IF1(K.LT.N).ANC.(CHAiGEIKP1I.LT.OI GO TO 115 108.
C 100.1C THE NEXT BLOCK IS A I X 1 P.LCK 10.2C 108.3
TzhIKI 100.481=SIGMA*T 108.5011aA(KK 1+31 *T 108.6IF(KP1 .GT. NJ GO TC 202 108.700 102 J=KP1,N 108.6
W(J)uW(J)-A(J,K)*T 108.9102 CONTItU= 109.202 CONTINUE 109.1103 (ONTINUE 109.2
C 109.3C ENTEP I X1 10.4C 109.5
IF(K.LT.N) GO TO 104 109.EA(K.K)=011 109.7
C 109.0C THE DECOMPCSITICN IS CCPPLTE.IF K=N 109.9C 110.
RETURN 110.1C 110.2
104 U1uDAPS(O11I 110.3UOu=ABS(81) 110.4IF((U1.LT.(ALCA*UOII .AND. (U1*DA1S(SIGMAI .LE. ALFA*UO*UOI 110.5
1 GO TO 106 . 110.6SIGMA=SIGMA-1*sb1/O1 1 110.791 81/C11 110.6A(KK1wD11 110.9
C 111.C UPDATE THE K-TH COLUMN OF M. 111.1C 111.2
IFIKP1 .GT. NI GO TO 205 111.300 105 J=KPI,N 111.4
A(JK3.A(J.K)+81*4(J) 111.5105 CONTINUE 111.6205 CONTINUE 111.7
K'KP1 111.6GO TO 101 111.9
C 112.106 II(CHANGE(KP2I.LE.03.AND.(KP2.LE.N)I GO TO 108 112.1
C 112.2C A 2 X 2 BLOCK IS FORMED BY CCMBINING THE NEXT 1 X 1 BLOCK 112.3
C WITH BLOCK K. 112.4
82=W(KP11 112.6Ts82 112.7C21=92*81 112.882=SIGMA*82 112.9022uA (KP I,KP1 1+02*T 113.L1'A (KPI .K 113.1022=022+L1*021 113.282=82+L 1*81 113.3021=021+L1'O11 113.4C22u022+L1*D21 113.5
113.6
C INCLUDE INFORMA TICN FPOM THE (K+ 1-ST COLUMN OF N. 113.7C..113.8IF KP2 .GT. NI GO TO 207 113.9
1Q7207
108
INCL9J9 INFORMATICN FROM THE (K+l)-ST ANDOF M.
(K+2)-NC CCLUMNS
19 (KD3.GT.':3 GO TO 20900 109 J KP3,N
(JI-WIJ)-(A(J,KP1)*T1+AIJ,KP2)*T2)A(Jg)nA(JK)-(A(J,KPi )*LI+A(J,KP2)*L2)
109 CnflTIIUE209 CON TINUE
Il1 MAXNUM(C11,022.033,111)U')IIAXNUMC21.031,032,10)IF (UI.LT.IALFA*UOI GO TO 112
A 1 IX 1 PIVOT WILL bE USED
I
CC
110210
C1l1@2w2Q3! 3
CALL pV 1X1011,D21,C31,022.032,033,81,2,83,CHANGEQ1,02,03,11,K,N)
K1=K-1+01SI GI4Am SIGMA-011*81*81
UPCATE THE K-TM CCLUMN OF M
tF(KP3.GT.N) GO TO 21000 110 J'KP3,N
TuAIJ.K)A(JK)-A( JKI)A C JK1)uTA(JK)=A(JK)+O21*AIJ,KP13+031*A(JKP2)+B1*W(J)
CONTINUECONTINUEKM1'K-1
De 107 J=KP?,NWIJ)aWIJI -.(JKPI)*TAIJKI=At J,K)-A(JKP1) 'L1rrNjTI:NUE
CAT I NUSrr TO 117
CONTI:SUE
IF THIS P^.RT IO! OF THE CGOE IS REAChEC WF ARE IN THC CASE OF A1 X 1 SINGULAR V.JCK FOLL CWEV BY A 2 X 2 BLOCK. THIS 2 X 2CLOCK IS JOINED TO THE 1 X 1 LOCK TO FORM A 3 X 3 MATPIX 0.
Ti:4(KP1)T2aW(KP2)R2=SIGMA *T1S3=SIGMA*T2022=A(lP1,K01)+u?*TL032=A(KP2,KP1 )+3*T1033=A(KP2,KP2I+33*TZO21=T1*81031=T2*1 *"L1=A(KP1,K)L2.AIKP2,K)T=L2*011133=033+L2*(2.00*D31+T)031=031+TC32=032+L1*031+L2*D21T=L1*011022=022+L 1*12 .0oro*o21T)C'- =021+T82=32L 1 *R1B3=A3+L2*B1KP3=K+ 3
136
114.114.1114.2114.3114.4114.5114.114.7114..8114.9115.115.1115.2115.3115.4115.5115.6115.7115.8115.4116.116.1116.2116.3116.4116.:116.6116.7116.8116.9117.117.1117.2117.3117.4117.5117.6117.7117.8117.9118.118.1118.2118.3118.4118.5118.6118.118.8118.9119.119.1119.2119.3119.*4119.5119.6119.7119.8119.9120.120.1120.*2120.3120.4120.5120.6120.7
ccc
137
C 120.eC INTERCHANGE TH COcKESPrONIG RfWS OF N. 120.9C 121.
IF (K'11 .LT. 1) GO TC 211 121.1CO 111 J=1KM1 121.2
T=A(K,J) 121.3A(K.J)=AlK1,J) 121.4
* A(K1,J)aT 121.5111 CONTINUE 121.6211 CONTI NUF 121.7
S3( K) 121.18Q(K)=O(K1) 121.90(KI)=I 122.
C 122.1A(K,K)=Dl 122.2AlKP1,K)=D21 122.3A(KP2,K)=031 122.4C1t'D?2 122.5C22=D33 122.6C21=032 122.7@1=82 122.882z83 122.9K=KPI 123.KP1=KP2 123.1KP2uKP2+1 123.2GO TO 117 123.3
112 CONTINUE 123.4C 123.5C A 2 X 2 PIVOT WILL BE USED 123.6C 123.7
01=1 123.802=2 123.9Q3=3 124.CALL PIV2X2(I11021,031.C22,032,033,B1,.8Z,3,CHANGE,S1G4A,C1, 124.1
1 Q2,03,I0,K,N) 124.2K1=K-1+Q1 124.3K2=K-1+02 124.4:l0(K) 124.5
O(K)=C(K1) 124.6CIKl)=I 124.7=13(KPl 3124.8
0(KPI)=0(K2) 124.90(K2)=I 125.
C 125.1C UPDATE THE K-TH AND (K+l-ST COLUMNS OF M. 125.2C 125.3
IF (KP3.GT.N) GO TO 213 125.400 113 JaKP1,N 125.5
T=AIJ,K) 125.EA(J,K)uA(J,KI) 125.7A(J,K1)=T 125.8T=A(J,KP1) 125.9A(J,KP1I)=AIJK2) 126.AIJ,K2)=T 126.1A(JK)=A(JK)+031*AtJ,KP2) e i1W ( J) 126.2AIJ,KPl)=A(J,KPI)+632*A(JKP2)+32*WIJ) 126.3
113 CONTINUr 126.4213 CONTINUE 126.5
C 126.6C INTERCHANGE THE CORAESPr-NfING ROWS OF M. 126.7C 126.8
KM1=K-1 126.900 114 Ju1,KM1 127.
T=A(K,J) 127.1A(KJ)=A(K1,J) 127.2A1K1,J)=T 127.3T AIKPI,J) 127.4A(KPI,J)*A(K2,J) 127.5A(K2,JI.T 127.6
114 CANTINUF . 127.7
138
A(KK)=O11 127.8A(K'1,I)=(.21 127.9A(KP1.K'l =3x22 128.A(K02,K) =03i 128.1A(KP2,KP1)=3Z 128.2011=033 128.3
- 8183 .126.4K=KP2 128.5KP1aK+1 128.6KP2aK+2 128.7GO T7 103 128.8
115 CONTINUE 128.9C 129.C THE DIAGONAL 3LOCK BEGINNING AT ENTRY K IS 2 X 2 . THE UPOATEC 129.1C C'AGGNAL RL(JCK U IS RkQUCIREr TO SATISFY 129.2C 129.3C ABSI021)*ALFA .G1. MAX(ABS(C11) , A8S(D22)). 129.4C 129.5C IF THIS IS POT SATISFIED THE BLOCK IS SPLIT INTO TC 1 X 1 BLOCKS. 129.6C 129.7C 129.8
T1=W(K) 129.9T2aW(KP1) 130.B1aSIGM^AAW(K) 130.182=S IGMfS*v( KP1) 130.2011=A(K,KI+i1*h(K) 130.3C21=A(KPIK)+2*h(K) 130.4022=A(KP1,K,1,a+32*W(KP1) 130.5IF (K01.GE.9I) GL TO 117 130.6CO 116 JaKP2,N 130.7
W(J)=(J)-EA(J,K)*T1 + A(J,KP1)*T2) 130.8116 CCNTINUE 130.9
C 131.117 CCNTINUE 131.1
C 131.2C ENTER 2 X 2 131.3C THE 2 X 2 BLOCK WILL BE PROCESSED 131.4C 131.5
T1 = 0.000 131.EU1uMAXNIU(011,022,TlI 1) 131.7IF (U1.GE.(ALFA*DABS(D21))I GO TO 119 131.8
C "131.9C A 2 X 2 PIVOT WILL BE USED 132.C 132.1
CET=011*022-021*021. 132.2CHANGE(K)v2 132.3CHANGE I'* 1) OET 132.4A(KK)uC11 132.5A(KP1,K)=121 132.6A(KP1,KP1)=u22 132.7IF (KP1.E. 41 RETURN 132.8T1u(D22m01-021*21 /OT 132.9T2aI-D?1*31+O11*52)/DOET 133.IF (KP2 .GT. N) GO TO 218 133.1
C 133.2C UPDATE THE K-TI LAND (K+1I-ST COLUMNS CF M. 133.3C 133.4
00 118 J=KP2,N 133.5T (J) 133.6A(J,KlIA(J,K)+TI*T 133.7A(JKP1)'A(JKP1)+T2*T 133.8
116 CONTI?4UE 133.9218 CONTINUE 134.
SIGMAS IGM4A-( T1I81+T2*82) 134.1K=KP2 .134.2GO TO 101 134.3
119 CONTINUE 134.4
139
C 134.bC A 1 x I PIVOT wilt SF USFO 134.6C 134.7
IF (fL.NE.2 10 TU L22 134.8C 134.9C INTEPCITANGE THE a4 S ANO COLUMNS CF M IF NCFSSAPY. 135.C 135.1
Ter 11' 135.2011=022 135.3^22=T 135.4T=il1 135.5L1=2 135.682aT 135.7I=O(K 135.8Q(K)Q(KPI) 135.9(KP1)=I 136.IF (KP2.GT.s41 GO I n220 136.100 120 J=KP2,A 136.2TuA(JKR 136.3A(JK)A(J.KP1) 13.4AIJKPI).T 136.5
120 CflNTINUA 136.6220 CONTINUE 136.7
KNM=K-1 136.8IF (KM1 .LT. 1) GE' TO 221 136.00 121 J=1,V41 137.
T=A(K,.1 137.1AiK,JuwA(KPl.J) 137.2A(KP1,J)=T 137.3
121 CONTINUE 137.4221 CONTiNUF 137.5
C 137.6122 CONT tIE .137.7
C 137.8C PROCESS THE TtJ 1 X 1 BLOCKS 137.9C 138.
CHANGE(Kl 1 138.1CHANGE (Kl1 138.2022=u22-(0213211 /Cl 3l138.3C21=021/011 138.4O2=02-81*021 138.5b1=O/011 138.6IF IKP2.GT.N1 GC TO 223 138.7
C 138.8C UPDATE THE K-TH CCLUtMh OF M. 138.9C 139.
:0 123 JuKP2"' 139.1AIJKIA(J,KI+C21'A(JKP1i+R1*W(J1 139.2
123 CONTINU= 139.3223 CONTINUE 139.4
A KKie011 139.5AIKP1.K1=f921 139.6
SIGMACSIGMA-81*C11*B1 139.7V1 1"022 139.881s82 139.9K.KP1 140.KPl KP2 140.1KP2uK+2 140.2O 10 103 140.3
ENO 140.4
SUORCUTI'4r PIVIXL(O11,CZL,t31,O22,032,C33,01,4s?,(3,CHA14GF.,
1 -l1,c2,3,11x,41DOUBLE PFC I SICI I 11 ,L?1,'3 I ,r22,r32.')33,o1, 12, 13IVTEGFQ 01,O2.':3,11.K,r-rOUI LE porCSIr.? CANCI (')
Ce**ee*s**.*** t" u,*a s* **;tt'i A **.****s* B,4***(ss**estste S*******CCC THIS SURFPUTIVF PE^F(MS A 1 x I PvI YT. GIVEN 4 3 X 3C SYMMETPIC PAT!AX O'IC!J) WItCI SATISFIES THE 1 x I PIVOTC CPITDeIA WITH f(h1,111 AS THC PIVPT ELF:4E T. Tit 3 X 3 MATRIXrC IS PERMUTED Tf sP IAG L(11,111 TC THE (1,1) PrSITINt4 ANCC THEN THE FIRST SEEP (F THE FACTOPIZATION IS OCfh IN PLACE.CCCs~ ****$* ***********s***«*** ****«,. *** ****C
DOUBLE PRECISE. 4 TINTCE KPIKP2KP 1K+1KP2w K+2GO TO 110,20,301,I11
THE AX ELEMENT IS 522
20 TuCllC11.022C22*r
TuC32C32"031C3 1.T
T"8282.81SluTCl.80281
GO TO 10
THE MAX ELEMENT IS E033
30 TaCIICl1'0330338T
T.C21021u0320327
1g8161.83S3=T
01.303.1
THE MAX ELEflENT IS 011
10 022.022-(021*0211/011032"C32-1031*0211/011033.033- 1t031 03/011stel/il
83983-S1*031021aC21/'11031&C31/011CHANGIK ul.RETURN940
140
CCC
C
C
CCC
C
C
c
ccc
140.5140.140.7140.es140.9141.141.1141.2141.3141.4141.5141.6141.7141.6141.n142.142.1142.2142.3142.4142.5142.C
142.7142.8142.9143.143.1143.2143.3143.4143.5143.6143.7143.8143.9144.144.1144.2144.3144.4144.5144.6144.7144.8144.9145.145.1145.2145.3145.4145.5145.6145.7145.8145.9146.146.1146.2146.3146.4146.5146.6146.7146.8146.9147.147.1147.2147.314.*414.7.5
141
SUBICUTINE PIV?%22D11,C21,C31,C,22f32.033,.1.8',B3,CHANX,~SIGMA, 147.61 1,02,03,ICKNI 147.7CfUBL' PRECISI'.1 11,t.21g022,031,D32,033,t1,2,B3,SICwA 147.6
.1INTFGER Ol.02,73,10,V.R 147.5SOUBL PR ECISIr MAE (K)-148.
C* ****#****## #i"4*ilII ****-.**.+.****.* "***#i##"#i*#~#!****** 148. 1
C 148.2C 148.3C THIS SUBROUTINE PEPFO"MS A 2 X 2 PIVCT 0,1 TI 3 X 3 MTPIX 18.4C t'uv (CIJ). THE 'AXIMUM OFF-IAGONAL ELEMFNT IS b."OUCH1 TG THE 148.5C 12,11 POSITION. ITS ORIGINAL LCCATICN IS INDICATED EY iNC 148.6C VARIABLE 10: 148.7C 148.8C 10=1 021 IS THE MAX ELEMENT 148.9C 10=2 031 IS THE MAX ELEMENT 149.C 10=3 032 IS THE MAX ELEMT .149.1C 149.2C THE FIRST STEP OF THE FACTOFIZATI^N CF THE MATRIX (DIJ) IS 149.3C CARRIED OUT IN PLACE USING THE 2X2 PIVOT. 149.4C 149.5C**********.s.*********.* *****.***t****** * ***********t**** 149.6C 149.7
INTEGER KP1,KP2 149.8COULEP FRECISION S.T,CET 149.9
C 150.KP 1.K+1 150.1KP2=K+2 150.2GO TO (10.20.301.10 150.3
20 CONTINUE 150.4C 150.5C P31 IS THE MAX ELEMLNT 150.6C 150.7
T=C22 150.8C22'033 150.9033.T 151.
C 151.1T=021 151.2C21=031 151.3031T .151.4
C 151.5T='2 151.6E2u83 151.7L3=T 151.8
C 151.902-3 152.03=2 152.1
CO TC 10 152.230 CONTINUE 152.3
C 152.4C 032 IS THE MAX ELEMENT 152.5C 152.E
Ts011 152.7C11.022 152.8022.033 152.9033=T 153.
C 153.1Tir21 153.2C21'032 153.3C32"031 153.4031ST 153.5
C 153.6T.SI 153.71lug 153.882.83 153.933.? 154.
C. 154.101.2 154.2C2.3 154.303.1 154.4
10 CONTINUE . 154.5
142
C .154.6C ^21 IS THE VAX FLEMENT 4.7C THF 2 X 2 PTVrT IS )U HfPF I 4.8C 154.9
lC<zCll*?.22-C21*121 155.18(0220C31-^21!232)/OET 155.1Ss l-C21*D31+C1 1 *.321/1 -u155.7
" 43sU3-1 T*Ml+S*4?) 155.3D32 I33-1 T*t'31+S*0321 155.4031=T 155.5032=5 155.6T (O22"P1-92l*F2 1/DET 155.7Ss (-C21*B 1+0 1P?)I/CET 155.8SIGMAsSIG9IA-(T*vl +S*B21 155.9B8=T 156.R22S 156.1CHAyi;PIK!s2 156.2CHANGE(KPIIaCET 156.3EHANGEIKP2) : 1 156.4RETURN 156.5ENC 156.6
COURLF PRECISE. FUNCTION MAXNUM(AB,C,I) 156.7DOU00 PRECIs:GN A.B.C 156.8INTEGER I 156.9
C#*****s***sm *******a.3***s****##************S ******s*.***t* 57.C 157.1C THIS FUNCTION FINDS THE MAXIMUM OF THE ABSOLUTE VALUES OP A BC 157.2C ANC I JCICATFS ;HICH OF THF. VALUES IS SELECT:: EY SETTING 157.3C I s 1.2,3 RESPECTIVELY. 157.4C 157.C**********************.*** ***** *****************t********S*** 157.6
POU3L= PRECISION S.T 157.7li 157.8TECASS(A) 157.9S=CABS(S) 158.IF (S.LE.TI GO 10 10 58.1
TsS 156.21.2 158.3
10 CCNTINUS 158.4S=CABS(C) 158.5IF (S.LE.T) GO TO 20 158.6
T=S 158.71.3 158.8
20 CONTINUE 158.9NAXNUMUT 159.
PETURN 159.1END 159.2
SUERCUTINE S0LVC(A,?,LC,N,CHAAGE,4,X,IFAILIINTIGfR NNLD,IFAILDOUBLE .PR-CISIIal A t t. .!,XIF),CHANGE(: IINT=GCP Q7h()
CC THIS ' I3(ROUT IN= CCMPUTPS THE SOLUTICN TO AX B.BCC THE M TPIX A IS ASSUrEC TO BE IN THE FACTOPEO F)RMCC QAG' M04"CC WHERE N IS L'LJCK UNIT LWF.R TRIANGULAR AND 0 IS BLOCKC DIAGONAL WITH 1X1 A' 22 OIAGNAL bLOCKS. IT I SC ASSUME THAT 4, 0 PE CUTPUT FR9 THE POUT INE SYMuPCC AND THAT THAT THESE APPAYS AVE STfPE! IN THE LCWEFC T.IAN'LE OF A. ON InPUT THE ARRAY X CONTAINS THEC RIGHT IAND S IDE B AIvf CN OUTPUT X CONrA.INS THEC SOLUTION VCTOR. IFAIL = IF THE SYSTEM ISC SINGULAR (NO S^LUTICr. I+: THIS CASE) CTHEPWISE IFAILC IS RETURNED WITh THE VALUE 0 tA SOLUTION WAS OEtTAINECI'.CC
CCC A THE ARPAY A IS .=CTANGULAR WITH LEADING CIMFNSICN LO.C THE SECOND DIMENSION rUST fE GREATER THAN OP E;UAL TC A.NC THE ARRAY A IS ASSUIEO TO HAVE THE FACTRIZATICN OF THEC MATRIX A AS DESCU T ED IN THE SU5FGUTtINE SYMUPO.CC NLC THE LEADING DIPENSICN OF THE ARRAY A.CC N THE DIMENSION: CF THE MATRIX A.CC CHANGE AN N OI9FNSIONAL VECTCR WHICH CONTAINS A DESCF IDTICNC OF THE SLOCK STRUCTI'PF EOF , AND THE CETFO MitAa TC OF EACH 2X2 DIAGONAL CF C. SEE THE QCCJMENTA ICt: FPC THE SUROUTIE SYMUPO FOR A CRE COMPLETF CFS(FIPTIONC OF THE CONTENTS OF CHANGE.CC 0 AN N DIMENSIONAL INTEGER ARRAY WHICH CONTAINS 'HEC PIVOTING USLO Ti) OBTAIN THE FACTCIZATION OF A.CC X AN N 0IMENSIOCAL VECTCR. THE CONTENTS OF X AREC DESCRIBED ABOVE.CC IFAIL AN INTEGEP VARIABLE THAT INDICATES WHEA A IS SINGULAR.C THE CONTENTS OF IFAIL ARE DESCP1VEO ACVE.CCCe******.e*.**.*** .***..****.**.**.*********#******.*****.C
DOUBLE PRECISION T,SINTEGFP I,J,K,IP1, IP2
00UBLE PRECISION W(53)IFAIL 000 10 J " 1N
WtJI a XI0(JI)10 CONTINUE
CC BACKSOLVE THE LOWER TIANGULAR SYSTEM AND INVERT THE CIAGONALC BLCCKS.C
1I 1'20 IF LIt.GE. NI GO TO 60
IP1 " I + 1IF (CHAGF(IP1) .GT. 01 GO TO 40IF LCHANGF(IP11 .Ea. ocO1 GO TO 1000
143
159.3159.4159.5159.E159.7159.8159.9190.160.1160.21(0.3160.4160.5160.6160.7160.8160.9161.161.1161.2161.3161.4161.5161.6161.7161.8161.9162.162.1162.2162.3162.4142.5162.6162.7162.8162.91F3.163.1163.2163.3163.4163.5163.1163.7163.8163.9164.164.1164.2164.3164.4164.5164.6164.7164.8164.9165.165.1165.2165.3165.4165.5165.6165.7165.8165.9196..166.1
144
C 1 IC.7C WE HAVE A 2 X2 PIVOT AT STEP I 1(6.3C Ltf..4
IP2 a + 2 Ite..S Wil . 1 (..6
T "W(IPI) 146.7IF IP2 .GT. N:) GC TO 130 166 .07 30 J " !P2,N 166.9
WJ) WIJI - IS*AiJI) + T*A(JIP1)) 167.30 CONTIPIUE 1(7.1130 CONTINUE 167.2
WI!) (AtIP1.IP11*S - A(iPt,l)*T)/CHAtIGE(IPll 167.?W(IPI) a i-A(IP1,I)*S + AiI,I)*Tl/CHAN(IE(IP1' 167.41 iP2 167.5GO TO 20 167.6
C 167.740 CONTINUE 167.8
C 167.9C WE HAVE A 1 X 1 PIVOT AT STEP I 166.C 166.1
T " hill 166.2nf 50 J IP1,N 1(1.3
W(J) W(J) - A(Jl)*T 16.450 CONTINUE 168.5
IF (A(l,!) .c:. 0.000) GO TO 1000 168.6Wil ) WCII/AtI.I) 168.71 IP1 166.EGO TC 20 16.9
C 16q.60 1 " N 169.1
C 169.2C INVEST THE LAST DIAGONAL OLCCK ANC INITIALIZE 169.3C FOR THE FORWARD SOLUTIC' 169.4C 169.5
IF ICHANG(t) .GT. 0.0th) GC TO 65 ' 169.6IF (CHANGFiI) .FQ. OCO) GO TC 1000 169.7
C 169.8C THE LAST BLOCK IS 2 X 2 169.9C IT HAS ALREACY 8FEEJ INVERTED 170.C 170.1
IPI .1 -. 1 170.21 s 1 - 2 170.3GO TO 70 170.4
C 170.565 CONTINUE 170.6
C 170.7C THE LAST BLOCK IS I X 1 170.8C 170.9
IF IA(N,N) .EO. 0.000) GO TO 101)0 171.hIN) WIN)/AIN,N) 171.1IP1' * 1 171.21 I- 1 171.3
C 171.470 CONTINUE 171.5
C 171.6C FORWARD SOLVE THE REMAINING UPPER TRIANGULAR SYSTEM 111.7C 171.8
IF 11.LE. 01 GO TO 1001 171.9IF ICHANGEII) .GT. 0.0CO1 GO TO 90 172.
C 172.1C 2X 2 PIVOT 172.2C 172.3
IP2 IP1 172.4IP1 1 172.5I 1-1 177.600 so J * IP2.N 172.7
W IPI" * WIPI) - AIJ,IP1l*W(JI 172.8hiI " Wil) - AIJ ll*W(J) 172.9
S0 CONTINUE ,173.
IPI * I
GOT CT 70CC x PIVOTC
90 00 100 J s IPlNWit) a ii) - A1JI)*W(J)
100 COP TINUEIP1 = II = I-"1GO TO 70
CC1000 C nnTINUECC VHF MATRIX IS SINGULARC
I ?IL = 1
C1001 C
CCC
THI:. 'HE NORA4L i.ETUR . . A SOLUTION WAS FOUN'
00 110 J .11X(Q(J aWJ)
110 CONTINUERETURNEND
145
173.1173.2173.3173.4173.5173.7173.7173.9173.c174.174.1174.2174.3174.4174.5174.7174.7174. G174.'q175.175.1175.2175.3175.4175.5175.6175.7175.E175.'176.
