Computational Physics 301: Exercise 1 - Matrices and Linear

Algebra

Thomas Wigg

February 14, 2014

University of Bristol, Bristol, UK

This exercise involves using programs written in the Clanguage to calculate the inverse of a square matrix in threedi�erent ways: analytically, using LU decomposition, andsingular value (SV) decomposition.

1 Matrix Inversion for Linear Alge-

bra

1.1 Analytical Calculation of Inverse Ma-

trix

A set of simultaneous equations can be solved using matricesby writing them in corresponding matrix equations. In thesimple case of a system with two unknowns, the equations

a11x1 + a12x2 = c1 (1)

a21x1 + a22x2 = c2 (2)

can be written in matrix form as(a11 a12a21 a22

)(x1x2

)=

(c1c2

)(3)

This format for two equations can be expanded to incor-porate a set of simulatious equations of any size. This canbe generalised as

Ax = c, (4)

where A is the matrix of coe�cients, x is the vector ofunknown variables and c is the vector of constants.The solution to the set of simultaneous equations can be

found by multiplying both sides of equation 4 by the inverseof A to give

A−1Ax = A−1c, (5)

which then simpli�es to

x = A−1c (6)

The inverse of A is given by

A−1 =adj.A

|A|, (7)

where adj.A is the adjoint of A, which is the transpose ofthe matrix of cofactors of A, and |A| is the determinant ofA.The determinant of a matrix of any order can be calcu-

lated by Laplace expansion[1] according to

|A| = ai1Ci1 + ai2Ci2 + · · ·+ ainCin =

n∑j=1

aijCij , (8)

where Cij is the cofactor of aij . The Laplace expansion inequation 8 is performed along the ith row. It can equallybe performed along the jth column. The cofactor, Cij , iscalculated by

Cij = (−1)i+j |Aij|, (9)

where |Aij|is the determinant of the matrix A with rowi and column j removed. The transpose of the matrix ofcofactors is simply found by

CTij = Cji. (10)

That is, changing the row and column coordinate of thecofactor value.

1.2 Task 1 - Inverting a Matrix using An-

alytical Methods

The �rst task we are asked to complete is to write a pro-gram to calculate the inverse of a square matrix of chosenorder using the analytical techniques describe above. Dueto the recursive nature of this method of calculating deter-minants using Laplace's equation, it was necessary to writea function which is able to call upon itself. The determi-nant function used performs Laplace expansion along the

1

�rst row for simplicity, and continues to call upon itself un-til the input matrix is 1 x 1 (single value), at which point anif() command returns that single input value. The programcontains an if() function that returns a statement that theinverse of the entered matrix is not mathematically possibleshould the determinant be 0.A very similar process is then used to progress through

each of the elements of the original matrix and calculate thematrix of cofactors, which calls upon the determinant func-tion to calculate the determinant of the reduced matricesfor each of the elements. Finally, the transpose of the co-factor matrix is computed and the inverse matrix calculatedand returned. The program was initially in a state whereit would record the user-entered matrix elements from thecommand window. This is a useful feature were I to bereleasing the program as a standalone matrix inverter, butobviously as the order increases, entering all the matrix el-ements for, say, a 10 x 10 matrix becomes impractical. In-stead, I adapted the program to compute the inverse of amatrix written in the routine itself. Because of this, thereare some ine�cient variable assignments but as the programworks e�ectively, I did not see a need to clean it up.To remove the need for me to write each matrix by

hand, I wrote a routine to generate a matrix of pseudo-random numbers of user-entered order using the rand() andRAND_MAX functions. The program writes the comma-separated matrix to a �le which can then be copied into theanalytical inversion program. This routine can be seen inappendix A.Figure 1 shows a comparison of the execution times for

computing the inverses of matrices of increasing order us-ing analytical methods. For each order, the pseudo-randommatrix was inverted �ve times and the mean execution timecalculated. There was very little �uctuation in the execu-tion time, but that which occurred can be attributed toexternal processes on the system using a fraction of theavailable processing power.Figure 1 shows that the execution time to compute the in-

verse of matrices of order up to eleven remains very small,but past this the execution time rises exponentially, withthe inversion of a 13 x 13 matrix taking nearly an hour anda half. This result implies the calculation of the inverse of ahigher order matrix will take many hours and, as such, I didnot attempt any matrices of order greater than 13. Whilstthe execution times will vary greatly for di�erent systems(due to technical speci�cations), the same exponential in-crease in calculation time will inevitably be seen for someorder due to the nature of the increase of the number ofcalculations required to compute the inverse of a matrix ofincreased order.In fact, the exponential �t used is only a good approx-

imation to the true relationship and the total number of

Figure 1: Graph showing the execution time (in seconds)against the matrix order for a program which computes theinverse of an entered matrix analytically. The �t is of expo-nential form such that y = 0.34204+4.74445×10−12e2.6637.

calculations required to compute the inversion (and hencethe execution time), scales according to

N = (n− 1)n! + n!− 1, (11)

where N is the total number of operations required and n isthe order of the entered matrix. This relationship is foundfrom the fact that for the computation of the determinantthere will be n! summand, where each summand requires(n− 1) multiplications and �nally to combine this into theinverted matrix requires n! − 1 summands[1]. As is obvi-ous from this relationship (and is correlated by the resultsseen in �gure 1), a small increase in the order of the ma-trix results in a large increase in the number of requiredcomputations.To test the accuracy of this analytical method, the re-

sults were compared to hand calculated inverse matricesof order 3 (for higher order than this, matrix inversionbecomes a tiresome exercise in bookkeeping) which wereveri�ed against the values returned by an online matrixcalculator[2]. The precision of a double precision �oatingpoint number is seventeen signi�cant �gures[3], so this wouldlogically imply that this is the maximum precision to whichthe matrix elements can be stored. This was the case as,when the routine was modi�ed to print out the elementsto greater than seventeen signi�cant �gures the programsimply returned zeros for the additional places. Consider-ing the accuracy of the returned inverse elements, it wasfound that they were accurate up to the number of signi�-cant �gures of the true answer. As an example, one of thetrue (hand-calculated) inverse matrix elements of a matrix

2

produced by the pseudo-random generator was 0.845723and the program returned the value 0.84572324197412796,which rounds to the true value. I am unsure as to whythe routine generates these additional values rather thanreturning zeros, but I presume it must be due to the recur-sive nature of the caluclations and rounding errors beingcarried forwards.

2 Algorithms for Linear Algebra

2.1 LU Decomposition

LU decomposition describes breaking down the matrix Ain equation 4 into the product of two matrices such that

A = LU, (12)

where the matrix L is lower triangular (only has non-zeroelements on the main diagonal and below) and U is uppertriangular (as with L but only has non-zero elements on andabove the main diagonal)[4]. For a 3 x 3 matrix in elementform, equation 11 would be written as

a11 a12 a13a21 a22 a23a31 a32 a33

=

l11 0 0l21 l22 0l31 l32 l33

u11 u12 u130 u22 u230 0 u33

.

(13)

Equation 4 can now be written as

Ax = LUx = c . (14)

By making the substitution

Ux = y (15)

such that

Ly = c (16)

solving equation 13 becomes trivial. By considering equa-tion 14 as an example, it can be seen from looking at the�rst row of the L matrix that this will simply reduce to theexpression y1 = c1

l11. This can then be substituted into the

calculation for y2 and so on. This can be generalised to givean expression for the value of any unknown variable as

yk =ck −

∑k−1j=1 akjyj

akk, k = 2, 3, · · · , n . (17)

2.2 Singular Value Decomposition

Singular value decomposition is again a means of simplify-ing the method of solving equation 4. This is achieved bydecomposing A into the product of three matrices such that

A = UDVT, (18)

where the matrix D is diagonal (only non-zero elementsalong main diagonal) and the matrices U and D areothogonal[4]. This equation is easier to solve as the in-verse of an orthogonal matrix is simply its transpose andthe inverse of a diagonal matrix is also simply found by

d11 0 · · · 00 d22 · · · 0...

.... . .

...0 0 · · · dnn

−1

=

1

d110 · · · 0

0 1d22

· · · 0...

.... . .

...0 0 · · · 1

dnn

.

(19)From this, the inverse of A is given by

A−1 = VD−1UT. (20)

2.3 Task 2 - Comparing the Performance

of Analytical, LUD and SVD Methods

for Inverting Matrices

2.3.1 Comparison of Speed of Execution of Ana-lytical, LUD and SVDMethods for InvertingMatrices

The second task required programs to be written that com-pute the inverse of an entered matrix using LUD and SVDroutines from the GNU Scienti�c Library (GSL)[5]. GSLmatrix functions only recognise vectors and matrices in thegsl_vector and gsl_matrix forms respectively. Because ofthis, any matrix arrays or vectors to be manipulated by theGSL functions must �rst be stored in the aforementionedforms.LU Decomposition - The matrix is �rst decomposed

using the gsl_linalg_LU_decomp function and then simplyinverted using the gsl_linalg_LU_invert function.SV Decomposition - The matrix is �rst decom-

posed into the matrices U, D and V using thegsl_linalg_SV_decomp function. As the GSL functionsonly allow the multiplication of two matrices at a time, it isthen necessary to use a sequence of two gsl_blas_dgemmfunctions to compute A−1 = VD−1UT.Figure 2 shows a comparison of the execution times for

the LUD and SVD methods as well as including the pro-jection of the analytical method from �gure 1. As with the

3

analytical method, each matrix was produced using the ma-trix generator and its inverse computed �ve times with themean execution time shown on the graph.

Figure 2: Graph comparing the execution times (in seconds)of a program calculating the inverse of a matrix using an-alytical methods (cyan points) and programs utilising theGSL functions to compute the matrix using LU decompo-sition (blue points) and singular value decomposition (redpoints). The cyan line is the exponential projection of theanalytical method seen in �gure 1. The blue line is a �t ofy = 0.19043+0.000170032x1.978564 projecting the executiontime of the matrix inversion by LUD methods. The red lineis a �t of y = 0.16435 + 0.000129852x2.00012 projecting theexecution time of the matrix inversion using SVD.

It is evident from �gure 2 that the LUD and SVD meth-ods of computing the matrix inverse are much faster thanthe analytical method. The analytical method became im-practically slow at an order of 13, whereas the LUD andSVD methods were still able to calculate the inverse of a350 x 350 matrix in under 20 seconds. It is also clear thatwhilst at lower orders the LUD and SVDmethods return thematrix inverse in roughly the same time, the LUD methodbecomes more e�cient at higher orders. I would've testedthis further, for matrices of much higher order, but the rou-tine crashed as soon as a matrix order of 361 was entered. Ican only assume this is because the GSL functionsa are lim-ited in the dimensions of the matrix they can manipulate.It is also worth noting that the execution time for both theLUD and SVD methods scales approximately to the squareof the matrix order.

k Inv. Acc. (signi�cant �gures)

1× 10−1 161× 10−2 131× 10−3 121× 10−4 61× 10−5 71× 10−6 61× 10−7 81× 10−8 61× 10−9 71× 10−10 51× 10−11 41× 10−12 31× 10−13 21× 10−14 0

Table 1: Accuracy of inverted matrix elements (in signif-icant �gures) of the SVD method of matrix inversion forvarious values of k compared to the values returned by anonline matrix inverse calculator.

2.3.2 Accuracy of LUD and SVD Routines WhenEntered Matrix is Close to Singular

The �nal part of this task asked us to compare the returnedinverse matrix using the LUD and SVD methods with thetrue inverse, when the entered matrix is close to singular,and compare the accuracies of both methods. For the pur-poses of this sub-task, I will be using the close-to-singularmatrix

A =

1 1 11 2 −12 3 k

, (21)

where k is a small number. For the LUD method of inver-sion, the routine always returns each element of the inverseto seventeen signi�cant �gures (the precision of a doubleprecision �oating point number). The routine is always ac-curate to the number of signi�cant �gures of the true value,and should the true value be of fewer signi�cant �guresthan seventeen, the routine will generate the remaining sig-ni�cant �gures such that if rounded to the number of sig-ni�cant �gures of the true value, the returned value is thesame as the true value; why this occurs is curious. For avalue of k = 1× 10−16 the routine crashes and returns thatthe determinant of the input matrix is equal to zero (andhence the input matrix is singular), presumably because theroutine is rounding the value of k to zero.

The SVDmethod of computing the inverse of the close-to-singular matrices behaved somewhat di�erently to its LUDcounterpart. Whereas the LUD routine computed the in-

4

verse matrix elements, to the same number of signi�cant�gures as the true value, for all values of k up to the pointat which is crashes, the accuracy of the SVD routine becamegradually worse as the value of k was reduced in factors often. The accuracy of the inverse matrix elements returnedby the SVD routine with their corresponding values of kcan be seen in table 1. For a value of k = 1 × 10−14, theSVD routine returned a completely inaccurate inverted ma-trix. For both the LUD and SVD routines, the results werecompared to an online matrix inverse calculator[2] and theroutines were adjusted to print the values of the inversematrix elements to the highest possible precision.

3 Physics Problem: Football Sta-

dium Camera

�A remote overhead camera at a football stadium is sus-pended by three cables attached to the roof. Each cableis fed from a motorised drum so the camera can be movedaround by changing the lengths of the cables. The camerahas a mass of 50 kg and the attachment points are all 70 mfrom the centre of the pitch forming an equilateral trianglein a horizontal plane. Using an appropriate matrix algo-rithm, calculate and plot the tension in one of the cables asa function of the camera's position as it is moved in a hori-zontal plane a constant depth (10 m) below the attachmentpoints. What is the maximum tension and where does itoccur? [You may ignore the mass of the cables].� - Exercise1 handout

Figures 3, 4 and 5 show the physical setup of the aboveproblem. In �gure 3, the length of the sides of the equalat-eral triangle was simply calculated using the cosine ruleand consequently the height of the triangle was found usingPythagoras' theorem. Using �gures 4 and 5 it is possible toconstruct a set of simultaneous equations relating the com-ponents of the tension in each wire. These equations are asfollows:

T1sinθ1 + T2sinθ2 + T3sinθ3 = 50g (22)

−T1cosθ1cosφ1 + T2cosθ2cosφ2 + T3cosθ3sinφ3 = 0 (23)

−T1cosθ1sinφ1 + T2cosθ2sinφ2 + T3cosθ3cosφ3 = 0 (24)

From these it is possible to write a corresponding matrixequation such that

sinθ1 sinθ2 sinθ3−cosθ1cosφ1 cosθ2cosφ2 cosθ3sinφ3−cosθ1sinφ1 cosθ2sinφ2 cosθ3cosφ3

T1T2T3

=

50g00

.

(25)

Figure 3: Diagram showing the stadium from above withthe camera centred and attached to the three anchor pointson the stadium roof by negligible mass cables.

It would be possible to solve this expression as it stands,but for simplicity of coding and understanding I expressedthe trigonometric expressions in equation 25 in terms of thex and y coordinates of the camera, where the x and y axesare de�ned in �gure 3.

Figure 4: Diagram showing the camera at an arbitrary posi-tion above the pitch. The relative angles and cable tensionsare also noted.

5

Figure 5: Diagram showing one of the cables and the camerain the vertical plane. The tension in the cable and theweight of the camera are also noted.

This results in the matrix expression

10a

10b

10c

−xa

70√3−xb

35√3−xc

−ya −y

c105−y

c

T1T2T3

=

50g00

, (26)

where a =√x2 + y2 + 100,

b =√(70√3− x)2 + y2 + 100 and

c =√(35√3− x)2 + (105− y)2 + 100 (substituted for

ease of use).

Unfortunately due to time constraints I have been un-able to implement the routines written earlier to solve thisphysics problem. Should I have had time, I would've usedthe LU decomposition method to solve equation 26, simplybecause the code for this routine is marginally simpler. Bypremultiplying both sides of equation 26 with the inverseof the coordinate equation would allow the tensions in eachwire to be calculated. I would then have asked the rou-tine to compute the tension in one of the wires for manypossible points over the pitch, varying the coordinates inincrements of, say, ten centimetres. This information couldthen be represented in a topographic map of the stadium.I am disappointed that my time-management is such thatI will not be able to complete this problem as I feel I wasprogressing well with the exercise.

References

[1] Linear Algebra: Modern Introduction, D. Poole, ThirdEdition, Cengage Learning, 4, 277-280, (2011).

[2] Online Matrix Calculator (with inverion function-ality), http://www.bluebit.gr/matrix-calculator/, (ac-cessed 14/02/2014).

[3] Programming C# 4.0, I. Gri�ths, M. Adams, J. Liberty,Sixth Edition, O'Reilly Media Inc., 2, 32, (2010).

[4] Numerical Recipes: The Art of Scienti�c Computing,W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P.Flannery, Third Edition, Cambridge University Press,2, 49-75, (2007).

[5] GNU Scienti�c Library (GSL),http://www.gnu.org/software/gsl/, 1.16, (accessed14/02/2014).

6