Post on 16-Jul-2020
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Systems of linear equations
Paul Schrimpf
UBCEconomics 526
September 10, 2013
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
More cardinality
• Let A and B be sets, the Cartesion product of A and Bis
A × B := {(a,b) : a ∈ A,b ∈ B}
QuestionIf A and B are countable is A × B countable?
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
More cardinality
TheoremIf A and B are countable, then so is A × B.
Proof.• A and B are countable, so we can write A = {a1,a2, ...}
and B = {b1,b2, ...}.• Consider
(a1,b1) (a1,b2) (a1,b3) · · ·(a2,b1) (a2,b2) (a2,b3) · · ·(a3,b1) (a3,b2) (a3,b3) · · ·
... . . .
• Count along the diagonals
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
More cardinality
QuestionIf A1,A2,A3, ... are each countable is ∪∞
n=1An countable?
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
More cardinality
TheoremIf {An} are countable, then so is ∪∞
n=1An.
Proof.• An is countable, so we can write An = {a1n,a2n, ...}• Consider
a11 a12 a13 · · ·a21 a22 a23 · · ·a31 a32 a33 · · ·
... . . .
• Count along the diagonals
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
..1 Introduction and definitions
..2 Gaussian elimination
..3 Existence of solutions
..4 Uniqueness of solutions
..5 Set of solutions
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Systems of linear equations
• Example:
5x1 − 7x2 =9−8x1 + x2 =0.
• In general:
a11x1 + a12x2 + ...+ a1nxn =b1
a21x1 + a22x2 + ...+ a2nxn =b2......
am1x1 + am2x2 + ...+ amnxn =bm,
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Coefficient matrix
• Coefficient matrix Aa11 · · · a1n...
...am1 · · · amn
x1
...xn
=
b1...
bm
Ax =b
• Augmented coefficient matrix A
A =
a11 · · · a1n b1...
......
am1 · · · amn bm
=(Ab
)
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Example: Markov model ofemployment
• Let st =
{1 if employed at time t0 if unemployed at time t
• Random process described by P(st |st−1, st−2, ...)
• Markovian: P(st |st−1, st−2, ...) = P(st |st−1)• Probability of being employed tomorrow only depends
on whether you’re employed today and not the moredistant past
• Stationary distribution: q stationary if whenP(st = 1) = q(1) and P(st = 0) = q(0) today, then alsohave P(st+1 = 1) = q(1) and P(st+1 = 0) = q(0)tomorrow
q(s) =∑
s0∈{0,1}P(s|s0)q(s0)
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Example: Markov model ofemployment
• Stationary distribution satisfies:
q(1) =P(1|1)q(1) + P(1|0)q(0)q(0) =P(0|1)q(1) + P(0|0)q(0)
1 =q(1) + q(0)
system of linear equations for unknowns q(1) and q(2)• Coefficient matrix and augmented coefficient matrix = ?
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Questions to be answered:
Given a system of linear equations:..1 Does any solution exist?..2 How many solutions?..3 How can a solution be computed?
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Equation/row operations
• Familiar with solving equations by:• Substitution• Elimination
• Use equation (row) operations that preserve set ofsolutions
..1 Multiply an equation by a non-zero constant,
..2 Add a multiple of one equation to another, and
..3 Interchange two equations.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Row echelon form
• Row echelon form: each row begins with more zerosthan the row above it or the row is all zeros
• Examples: (a11 a120 a22
)(
a11 a12 a130 0 a23
)• Once a system is in row echelon form it is easy to solve
by back substitution
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Gaussian elimination
• Systematic way to transform system of equations to rowechelon form
..1 Identify the first column to contain any non-zeroelements, call this column c∗.
..2 Interchange rows so that a nonzero entry appears at thetop of column c∗.
..3 Add a multiple of the first row to each of the rows belowso that the entries in column c∗ below the first row arezero.
..4 Repeat 1-2 on the submatrix consisting of the lowerright part of the original matrix below the first row and tothe right of column c∗. Stop if this submatrix has nocolumns or has no rows.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Gaussian elimination: example
x1 + x2 + 3x3 = 02x1 + 3x2 + 7x3 = 93x1 + 5x2 + 13x3 = 1
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Transform the following system into row echelon form:
x + 2y − z =24y + z =5
−2x − 4y + 2z =1.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Existence of solutions
• rank of A is the number of nonzero rows in its rowechelon form
• Is rank well-defined?
LemmaThe rank of a matrix A is always less than or equal to thenumber of columns of A and less than or equal to thenumber of rows of A.
LemmaLet A be a coefficient matrix and A be an augmentedcoefficient matrix. Then rankA ≤ rankA.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Existence of solutions
Theorem (Existence of solutions)A system of linear equations with coefficient matrix A andaugmented coefficient matrix A has a solution (perhapsmore than one) if and only if rankA = rankA.
Proof.See notes.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Consider the system:
4y + z =5x + 2y − z =2−8y − 2z =− 10.
• What if b3 = −10?
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Multiple solutions means infinitesolutions
LemmaSuppose x1 and x2 are two distinct solutions to the systemof equations Ax = b. Then the system of equations has(uncountably) infinitely many solutions.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Solution existence for any b
Theorem (Solution existence)A system of linear equations with coefficient matrix A willhave a solution for any choice of b1, ..., bm if and only ifrankA is equal to the number of rows of A.
CorollaryFor any system of equations with more equations thanvariables, (i.e. an overdetermined system) there exists achoice of b such that no solutions exist.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Uniqueness
Theorem (Solution uniqueness)Any system of equations with coefficient matrix A has atmost one solution for any b1, ..., bm if and only if rankAequals the number of columns of A.
CorollaryIf rankA is less than the number of columns of A then eitherno solutions exists or multiple solutions exists.
CorollaryA is nonsingular (always has a unique solution) if and only ifA has an equal number of columns and rows (A is square)and has rank equal to its number of columns (or rows).
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Summary
rankA = rankA rankA < rankArankA = columns(A) rankA < n
0 X1 X∞ X
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Sets of solutions
−2x + y = 1 x − y − z = 0−2x + y + 3z = 1 −2x − y + z = 2
-10
-5
0
5
10
15
-6 -4 -2 0 2 4 6
y
x
-2
0
2
4
6
8
10
12
x
-1
0
1
2
3
4
5
6
y
0
1
2
3
4
5
6
7
z
-10
-5
0
5
10
-10
-5
0
5
10-20
-10
0
10
20
30
z
x
y
z
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
DefinitionThe set S ⊆ Rn is called a linear subspace if it is closedunder (i) scalar multiplication and (ii) addition in other words,if
(i) for every (x1, ..., xn) ∈ S and a ∈ R, we have(ax1,ax2, ..., axn) ∈ S, and
(ii) for every (x1, ..., xn) ∈ S and (y1, ..., yn) ∈ S, we have(x1 + y1, ..., xn + yn) ∈ S
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
DefinitionA set of vectors in Rn, {xj = (x j
1, ..., xjn)}J
j=1, is linearlyindependent if the only solution to
J∑j=1
cjxj = 0
is c1 = c2 = ... = cJ = 0.
DefinitionThe dimension of a linear subspace S ⊆ Rn is the cardinalityof the largest set of linearly independent elements in S.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Theorem (Rouche-Capelli)A system of linear equations with n variables has a solutionif and only if the rank of its coefficient matrix, A, is equal tothe rank of its augmented matrix, A. If a solution exists andrankA is equal to its number of columns, the solution isunique. If a solution exists and rankA is less than its numberof columns, there are infinite solutions. In this case the setof solutions is of the form1
{s + x∗ ∈ Rn : s ∈ S and Ax∗ = b}
where S is the linear subspace of dimension n − rankAdefined by S = {s ∈ Rn : As = 0} and x∗ is any singlesolution to Ax = b.
1A set of this form is called an affine subspace. It is a linear subspacethat has been shifted so that it no longer necessarily contains the origin.
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Example: Markov model ofemployment
• Employment st ∈ {u,e}.• Markovian: P(st |st−1, st−2, ...) = P(st |st−1)
• Stationary distribution: (πe, πu) such that if st = e withprobability πe and u with probability πu, then st+1 hassame distribution, i.e.
P(e|e)πe + P(e|u)πu =πe
P(u|e)πe + P(u|u)πu =πu
Systems oflinear
equations
Paul Schrimpf
Cardinalityreview
Introductionand definitions
Gaussianelimination
Existence ofsolutions
Uniqueness ofsolutions
Set ofsolutions
..........
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
.....
.....
......
.....
......
.....
.....
.
Example: Markov model ofemployment
• Stationary distribution is a solution to:
(pee − 1)πe + peuπu =0pueπe + (puu − 1)πe =0
πe + πu =1.
• Questions:..1 Does any solution exist?..2 How many solutions exist?..3 How can a solution be computed?