Linalg Tutorial - Stanford University

Quick Tour of Linear Algebra and Graph Theory

Quick Tour of Linear Algebra and GraphTheory

CS224w: Social and Information Network AnalysisFall 2012 Yu ”Wayne” Wu

Based on Borja Pelato’s version in Fall 2011


Basic Linear Algebra

Matrices and Vectors

Matrix: A rectangular array of numbers, e.g., A ∈ Rm×n:

A =

a11 a12 . . . a1na21 a22 . . . a2n

......

...am1 am2 . . . amn

Vector: A matrix consisting of only one column (default) orone row, e.g., x ∈ Rn

x =

x1x2...

xn



Matrix Multiplication

If A ∈ Rm×n, B ∈ Rn×p, C = AB, then C ∈ Rm×p:

Cij =n∑

k=1

AikBkj

Special cases: Matrix-vector product, inner product of twovectors. e.g., with x , y ∈ Rn:

xT y =n∑

i=1

xiyi ∈ R



Matrix Multiplication



Properties of Matrix Multiplication

Associative: (AB)C = A(BC)

Distributive: A(B + C) = AB + ACNon-commutative: AB 6= BA

Proof of associativity:Let L = (AB)C and R = A(BC), then we can showLij =

∑pk = 1

∑nl = 1 (ail ◦ blk ) ◦ ckj =∑n

l = 1∑p

k = 1 ail ◦(blk ◦ ckj

)= Rij .



Operators and properties

Transpose: A ∈ Rm×n, then AT ∈ Rn×m: (AT )ij = Aji

Properties:(AT )T = A(AB)T = BT AT

(A + B)T = AT + BT



Identity Matrix

Identity matrix: I = In ∈ Rn×n:

Iij =

{1 i=j,0 otherwise.

∀A ∈ Rm×n: AIn = ImA = A



Diagonal Matrix

Diagonal matrix: D = diag(d1,d2, . . . ,dn):

Dij =

{di j=i,0 otherwise.



Other Special Mtrices

Symmetric matrices: A ∈ Rn×n is symmetric if A = AT .Orthogonal matrices: U ∈ Rn×n is orthogonal ifUUT = I = UT U



Linear Independence and Rank

A set of vectors {x1, . . . , xn} is linearly independent if@{α1, . . . , αn}:

∑ni=1 αixi = 0

Rank: A ∈ Rm×n, then rank(A) is the maximum number oflinearly independent columns (or equivalently, rows)Properties:

rank(A) ≤ min{m,n}rank(A) = rank(AT )rank(AB) ≤ min{rank(A), rank(B)}rank(A + B) ≤ rank(A) + rank(B)



Rank from row-echelon forms



Matrix Inversion

If A ∈ Rn×n, rank(A) = n, then the inverse of A, denotedA−1 is the matrix that: AA−1 = A−1A = IProperties:

(A−1)−1 = A(AB)−1 = B−1A−1

(A−1)T = (AT )−1

The inverse of an orthogonal matrix is its transpose



Eigenvalues and Eigenvectors

A ∈ Rn×n, λ ∈ C is an eigenvalue of A with thecorresponding eigenvector x ∈ Cn (x 6= 0) if:

Ax = λx

eigenvalues: the n possibly complex roots of thepolynomial equation det(A− λI) = 0, and denoted asλ1, . . . , λn



Eigenvalues and Eigenvectors Properties

Usually eigenvectors are normalized to unit length.If A is symmetric, then all the eigenvalues are real and theeigenvectors are orthogonal to each other.tr(A) =

∑ni=1 λi

det(A) =∏n

i=1 λi

rank(A) = |{1 ≤ i ≤ n|λi 6= 0}|



Matrix Eigendecomposition

A ∈ Rn×n, λ1, . . . , λn the eigenvalues, and x1, . . . , xn theeigenvectors. P = [x1|x2| . . . |xn], D = diag(λ1, . . . , λn), then:



Matrix Eigendecomposition

Therefore, A = PDP−1.In addition:

A2 = (PDP−1)(PDP−1) = PD(P−1P)DP−1 = PD2P−1

By induction, An = PDnP−1.A special case of Singular Value Decomposition



Convex Optimization

A set of points S is convex if, for any x , y ∈ S and for any0 ≤ θ ≤ 1,

θx + (1− θ)y ∈ S

A function f : S → R is convex if its domain S is a convexset and

f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)

for all x , y ∈ S, 0 ≤ θ ≤ 1.



Logistic Regression



Gradient Descent



Submodularity

A function f : S → R is submodular if for any subset A ⊆ B,

f (A ∪ {x})− f (A) ≥ f (B ∪ {x})− f (B)

Submodular functions allow approximate discreteoptimization.



Greedy Set Cover

If the optimal solution contains m sets, greedy algorithm finds aset cover with at most m loge n sets.



Proofs

Induction:1 Show result on base case, associated with n = k0

2 Assume result true for n ≤ i . Prove result for n = i + 13 Conclude result true for all n ≥ k0

Example:For all natural number n, 1 + 2 + 3 + ...+ n = n∗(n+1)

2Base case: when n = 1, 1 = 1.Assume statement holds for n = k , then1 + 2 + 3 + ...+ k = k∗(k+1)

2 .We see 1+2+3+ ...+(k +1) = k∗(k+1)

2 +(k +1) = (k+1)(k+2)2 .



Proofs

Contradiction (reductio ad absurdum):1 Assume result is false2 Follow implications in a deductive manner, until a

contradiction is reached3 Conclude initial assumption was wrong, hence result true

Example:Let’s try to prove there is no greatest even integer.First suppose there is one, name N, and for any even integer n,we have N ≥ n.Now let define M as N + 2. Then we see M is even, but alsogreater than N. Thus by contradiction we prove the statement.



Graph theory

Definitions: vertex/node, edge/link, loop/cycle, degree,path, neighbor, tree, clique,. . .Random graph (Erdos-Renyi): Each possible edge ispresent with some probability p(Strongly) connected component: subset of nodes that canall reach each otherDiameter: longest minimum distance between two nodesBridge: edge connecting two otherwise disjoint connectedcomponents



Basic algorithms

BFS: explore by “layers”DFS: go as far as possible, then backtrackGreedy: maximize goal at each stepBinary search: on ordered set, discard half of the elementsat each step



BFS



DFS



Complexity

Number of operations as a function of the problemparameters.Examples

1 Find shortest path between two nodes:DFS: very bad idea, could end up with the whole graph as asingle pathBFS from origin: good ideaBFS from origin and destination: even better!

2 Given a node, find its connected componentLoop over nodes: bad idea, needs N path searchesBFS or DFS: good idea

Linalg Tutorial - Stanford University

Documents

Transcript of Linalg Tutorial - Stanford University