Numerical Algorithms Matrix multiplication Numerical solution of Linear System of Equations.
Fast Matrix Multiplication Algorithms
Transcript of Fast Matrix Multiplication Algorithms
Fast Matrix Multiplication
Algorithms
Why should we care?
Complexity of matrix multiplication = Complexity of βalmost allβ matrix problems
Solving linear systems
Evaluating determinants
LU factorization
Many more
P. BΓΌrgisser, M. Clausen, M. A.
Shokrollahi
Algebraic complexity theory.
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
Strassenβs algorithm
π = ππ + πβ=π1+π2
π‘ = ππ + ππ=π3+π4
π = ππ + ππ=π5+π4 βπ2 +π6
π’ = ππ + πβ=π5+π1 β π3+π7
Strassenβs algorithm
1. π1 = π΄1π΅1 = π(π β β) = ππ β πβ
2. π2 = π΄2π΅2 = (π + π)β = πβ + πβ
3. π3 = π΄3π΅3 = (π + π)π = ππ + ππ
4. π4 = π΄4π΅4 = π(π β π) = ππ β ππ
5. π5 = π΄5π΅5 = π + π π + β = ππ + πβ + ππ + πβ
6. π6 = π΄6π΅6 = (π β π)(π + β) = ππ + πβ β ππ β πβ
7. π7 = π΄7π΅7 = (π β π)(π + π) = ππ + ππ β ππ β ππ
π = ππ + πβ=π1+π2
π‘ = ππ + ππ=π3+π4
π = ππ + ππ=π5+π4 βπ2 +π6
π’ = ππ + πβ=π5+π1 β π3+π7
Strassenβs algorithm
1. π1 = π΄1π΅1 = π(π β β) = ππ β πβ
2. π2 = π΄2π΅2 = (π + π)β = πβ + πβ
3. π3 = π΄3π΅3 = (π + π)π = ππ + ππ
4. π4 = π΄4π΅4 = π(π β π) = ππ β ππ
5. π5 = π΄5π΅5 = π + π π + β = ππ + πβ + ππ + πβ
6. π6 = π΄6π΅6 = (π β π)(π + β) = ππ + πβ β ππ β πβ
7. π7 = π΄7π΅7 = (π β π)(π + π) = ππ + ππ β ππ β ππ
π = ππ + πβ=π1+π2
π‘ = ππ + ππ=π3+π4
π = ππ + ππ=π5+π4 βπ2 +π6
π’ = ππ + πβ=π5+π1 β π3+π7
T n
= 7Tn
2+ Ξ(n2)
Time
complexity:
π(π2.808)
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
1978: Pan, π < 2.796 (V. Y. Pan. Strassenβs algorithm is not optimal)
Bilinear Algorithms
Given two matrices π΄, π΅ππ =
π,π π’ππππ΄ π, π π,π π£ππππ΅ π, π , πlinear combinations.
π΄π΅ π, π =
π
π€πππππ
The minimum number of products r that a bilinear algorithm can use to
compute the product of two π Γ π matrices is called the rank of π Γ π matrix
multiplication π ( π, π, π )
The product of two kπ Γ ππ matrices can be viewed as the product of two π Γk matrices the entries of which are π Γ π matrices
We can create a recursive algorithm ALG for multiplication of π Γ k.
View the ππ Γ ππ as π Γ k matrices with entries ππβ1 Γ ππβ1 matrices
The recursive approach using an upper bound of π on π ( π, π, π ) gives a
bound π < logπ π, (the number of additions that one has to do in each step is
no more than 3ππ2
As long as π < π3 we get a non trivial bound for π
Strassen: π = 2, r = 7
Pan: π = 70, π = 143640
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
1978: Pan, π < 2.796 (V. Y. Pan. Strassenβs algorithm is not optimal)
1979: Bini (border rank), π < 2.78 (D. Bini, M. Capovani, F. Romani, and G.
Lotti. π(π2.7799) complexity for n n approximate matrix multiplication)
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
1978: Pan, π < 2.796 (V. Y. Pan. Strassenβs algorithm is not optimal)
1979: Bini (border rank), π < 2.78 (D. Bini, M. Capovani, F. Romani, and G.
Lotti. π(π2.7799) complexity for n n approximate matrix multiplication)
1981: Shonhage Ο-theorem(asymptotic sum inequality), π < 2.548
in the same paper, π < 2.522 (A. Schonhage. Partial and total matrix
multiplication)
Approximate Bilinear Algorithms (ABA)
In bilinear algorithms the coefficients π’πππ, π£πππ, π€πππ were constants.
In ABA these coefficients are linear combinations of the integer powers of a
indeterminate π.
The entries of π΄π΅ are then only approximately computed: π΄π΅ π, π = ππ€πππππ + π(π)
π(π): linear combination of positive powers of π.
When π β 0, then the product is almost exactly.
The minimum number of products π for an ABA to compute the product of two π Γ π matrices, is called border rank of a matrix multiplication π ( π, π, π )
Bini showed that when dealing with the asymptotic complexity of matrix
multiplication, approximate algorithms suffice obtaining bounds for π
If π ( π, π, π ) β€ π, then π β€ logπ π
Bini used 10 entry products to multiply a 2 Γ 3matrix with a 3 Γ 3matrix π = 12,π = 1000
Shonhage Ο-theorem: Suppose we have an upper bound of π on the border
rank of computing π independent instances of matrix multiplication with dimensions ππ Γππ by ππ Γππ for π = 1, β¦ , π. Then π < 3π, where π(ππππππ)
π= π
In particular he showed that one can approximately compute the product of a
3 Γ 1 by 1 Γ 3 vector and the product of a 1 Γ 4 by 4 Γ 1 vector together using
only 10 products, whereas any exact bilinear algorithm needs at least 13
products.
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
1978: Pan, π < 2.796 (V. Y. Pan. Strassenβs algorithm is not optimal)
1979: Bini (border rank), π < 2.78 (D. Bini, M. Capovani, F. Romani, and G.
Lotti. π(π2.7799) complexity for n n approximate matrix multiplication)
1981: Shonhage Ο-theorem(asymptotic sum inequality), π < 2.548
in the same paper, π < 2.522 (A. SchΒ¨onhage. Partial and total matrix
multiplication)
1982: Romani, π < 2.517 (F. Romani. Some properties of disjoint sums of
tensors related to matrix multiplication.)
A brief historyβ¦
Until the late 1960βs: naΓ―ve algorithm, π(π3)
1969: Strassenβs algorithm, π(π2,808) (V. Strassen. Gaussian elimination is not
optimal)
1978: Pan, π < 2.796 (V. Y. Pan. Strassenβs algorithm is not optimal)
1979: Bini (border rank), π < 2.78 (D. Bini, M. Capovani, F. Romani, and G.
Lotti. π(π2.7799) complexity for n n approximate matrix multiplication)
1981: Shonhage Ο-theorem(asymptotic sum inequality), π < 2.548
in the same paper, π < 2.522 (A. SchΒ¨onhage. Partial and total matrix
multiplication)
1982: Romani, π < 2.517 (F. Romani. Some properties of disjoint sums of
tensors related to matrix multiplication.)
1982: Coppersmith & Winograd, π < 2.496 (D. Coppersmith and S.
Winograd. On the asymptotic complexity of matrix multiplication)
Continueβ¦
1986: Strassen, laser method, π < 2.479 (entirely new approach on the
matrix multiplication problem) (V. Strassen. The asymptotic spectrum of
tensors and the exponent of matrix multiplication)
Continueβ¦
1986: Strassen, laser method, π < 2.479 (entirely new approach on the
matrix multiplication problem) (V. Strassen. The asymptotic spectrum of
tensors and the exponent of matrix multiplication)
1989: Coppersmith & Winograd, combine Strassenβs laser method with a
novel from analysis based on large sets avoiding arithmetic progression,
π < 2.376 (D. Coppersmith and S. Winograd. Matrix multiplication via
arithmetic progressions.)
Continueβ¦
1986: Strassen, laser method, π < 2.479 (entirely new approach on the
matrix multiplication problem) (V. Strassen. The asymptotic spectrum of
tensors and the exponent of matrix multiplication)
1989: Coppersmith & Winograd, combine Strassenβs laser method with a
novel from analysis based on large sets avoiding arithmetic progression,
π < 2.376 (D. Coppersmith and S. Winograd. Matrix multiplication via
arithmetic progressions.)
2003: Cohn & Umans: group theoretic framework for designing and
analyzing matrix multiplication algorithms
Continueβ¦
1986: Strassen, laser method, π < 2.479 (entirely new approach on the
matrix multiplication problem) (V. Strassen. The asymptotic spectrum of
tensors and the exponent of matrix multiplication)
1989: Coppersmith & Winograd, combine Strassenβs laser method with a
novel from analysis based on large sets avoiding arithmetic progression,
π < 2.376 (D. Coppersmith and S. Winograd. Matrix multiplication via
arithmetic progressions.)
2003: Cohn & Umans: group theoretic framework for designing and
analyzing matrix multiplication algorithms
2005: Cohn, Umans, Kleinberg, Szegedy, π(π2.41) (H. Cohn, R. Kleinberg, B.
Szegedy, and C. Umans. Group-theoretic algorithms for matrix
multiplication)
Conjectures that can lead to: π = 2.
Continueβ¦
1986: Strassen, laser method, π < 2.479 (entirely new approach on the
matrix multiplication problem) (V. Strassen. The asymptotic spectrum of
tensors and the exponent of matrix multiplication)
1989: Coppersmith & Winograd, combine Strassenβs laser method with a
novel from analysis based on large sets avoiding arithmetic progression,
π < 2.376 (D. Coppersmith and S. Winograd. Matrix multiplication via
arithmetic progressions.)
2003: Cohn & Umans: group theoretic framework for designing and
analyzing matrix multiplication algorithms
2005: Cohn, Umans, Kleinberg, Szegedy, π(π2.41) (H. Cohn, R. Kleinberg, B.
Szegedy, and C. Umans. Group-theoretic algorithms for matrix
multiplication)
Conjectures that can lead to: π = 2.
2014 Williams: π < 2.373 (Multiplying matrices in π π2.373 time)
Continueβ¦
For some π they provide a way to
multiply π Γ π matrices using π βͺ π3
multiplications and apply the technique
recursively to show that π < logππ
Basic group theory definitions
Group
A group is a non empty set πΊ with a binary operation β defined on πΊ such that the
following conditions hold:
1. For all π, π, π β πΊ, we have π β π β π = (π β π) β π
2. There exists an element 1β πΊ such that 1 β π = π and π β 1 = π for all π β πΊ
3. For all π β πΊ there exists an element πβ1 β πΊ such that π β πβ1 = 1 andπβ1 β π = 1
Order of a group
The order πΊ of a group is its cardinality, i.e. the number of elements in its set.
Cyclic Group
A group is said to be cyclic if it is generated by a single element.
(We say that π generates πΊ if πΊ = π if every element of πΊ can be written as a finite
product of elements from π and their inverses. Note that the order of an element πof a group is the order of the subgroup π it generates)
Abelian Group
The group πΊ is said to be abelian if π β π = π β π far all π, π β πΊ.
Group Algebra
The group algebra πΉ πΊ of πΊ is defined to be the πΉ-vector space with basis the
elements of πΊ endowed with the multiplication extending that on πΊ. Thus:
1. An element of πΉ πΊ is a sum πβπΊ πππ, ππβπΉ
2. Two elements πβπΊ πππ, πβπΊ ππβ²π of πΉ πΊ are equal if and only if ππ = ππ
β² for all π
3. πβπΊ πππ πβπΊ ππβ²π = πβπΊ ππ
β²β²π , ππβ²β² = π1π2 = πππππ
β²
Homomorphism
A homomorphism for a group πΊ to a group πΊβ² is a map πΆ: πΊ β πΊβ² such that πΆ ππ =πΆ(π)πΆ(π) for all π, π β πΊ. An isomorphism is a bijective homomorphism.
Multiplying polynomials via FFT
Standard method requires time complexity of π(π2)
We think of the coefficient vectors of the polynomials as elements of the
group algebra β πΊ of a finite group πΊ
If the group is large (order at least 2π), convolution of two vectors in the
group algebra corresponds to the polynomial product.
Multiplying polynomials via FFT
Standard method requires time complexity of π(π2)
We think of the coefficient vectors of the polynomials as elements of the
group algebra β πΊ of a finite group πΊ
If the group is large (order at least 2π), convolution of two vectors in the
group algebra corresponds to the polynomial product.
Discrete convolution
Suppose we have two complex vectors in πΈπ:π = π§0 π§1 β― π§πβ1
π π = π¦0 π¦ β― π¦πβ1π
The discrete convolution of these two vectors is another vector, which we
denote π β π, defined componentwise by (π β π)π= π=0πβ1 π§πβππ¦π , π = 0,1,2, β¦
Multiplying polynomials via FFT
Standard method requires time complexity of π(π2)
We think of the coefficient vectors of the polynomials as elements of the
group algebra β πΊ of a finite group πΊ
If the group is large (order at least 2π), convolution of two vectors in the
group algebra corresponds to the polynomial product.
Convolution in the group algebra can be computed quickly using the FFT.
Time complexity of FFT and inverse FFT: π(πππππ)
Discrete Fourier Transform for
polynomials
Discrete Fourier Transform
Embed polynomials as elements of the group algebra β[πΊ]:
Let πΊ = π§ be a cyclic group of order π β₯ 2π. Define
π΄ = π=0πβ1πππ§
π π΅ = π=0πβ1πππ§
π
Discrete Fourier Transform is an invertible linear transformation
π·: β[πΊ] β β πΊ , such that
π· π΄ = ( π=0πβ1πππ₯0
π , π=0πβ1πππ₯1
π , β¦ , π=0πβ1πππ₯πβ1
π , ), π₯π = π2ππ
ππ
Then π΄ π΅ = π·β1(π·( π΄)π·( π΅))
Embedding matrices A, B into elements π΄, π΅of the group algebra β[πΊ]
Cohn & Umans
Matrix multiplication can be embedded into the group algebra of a finite
group πΊ (πΊ must be non-abelian)
Let πΉ be a field and π, π and π be subsets of πΊ.
π΄ = (ππ ,π‘)π βπ,π‘βπ and B = (ππ‘,π’)π‘βπ,π’βπ
are π Γ π and π Γ π , indexed by elements of π, π and π, π respectively.
Then embed π΄, π΅ as elements π΄, π΅ β πΉ πΊ :
π΄ = π βπ,π‘βπ ππ ,π‘π β1π‘ and π΅ = π‘βπ,π’βπ ππ‘,π’π‘
β1π’
Using the FFT
As with the polynomial method, the Fourier transform provides an efficient way
to compute the convolution product.
For a non-abelian group a fundamental theorem of Weddeburn says that the
group algebra is isomorphic, via a Fourier transform, to an algebra of block
diagonal matrices having block dimensions π1β¦ππ, with ππ2 = πΊ .
Convolution in β πΊ is thus transformed into block diagonal matrix
multiplication.
Embed matrices A,
B into elements π΄, π΅ of the group
algebra β[πΊ]
Multiplication of π΄and π΅ in the
group algebra is carried out in the Fourier domain
after performing the Discrete
Fourier Transform (DFT) of π΄ and π΅
FFT
The product π΄ π΅ is found
by performing the inverse
DFT
Inverse FFT
Entries of the matrix
AB can be read off from
the group algebra product π΄ π΅
Triple Product Property
The approach works only if the group πΊ admits an embedding of matrix
multiplication into its group algebra.
The coefficients of the convolution product correspond to the entries of the
product matrix.
Such an embedding is possible whenever the group πΊ has three subgroups,
π»1, π»2, π»3 with the property that whenever β1π π»1, β2π π»2 and β3π π»3 with
β1β2β3 = 1, then β1 = β2 = β3 = 1
(The condition can be generalized to subsets of πΊ rather than subgroups)
Beating the sum of cubes
In order for π to be less than 3, the group must satisfy more conditions.
In particular, it must be the case that:
π»1 π»2 π»3 > ππ3,
ππ: the block dimensions of the block matrices
Group Theory Definitions
Permutation Groups
Let π be a set and let ππ¦π π be the set of bijections π: π β π
ππ¦π π is a group, called the group of symmetries of π. For example, the permutation group on n letters ππ is defined to be the group of symmetries of the set 1,β¦ , π β it has order π!.
Groups Acting on Sets
Let π be a set and let πΊ be a group. A left action of πΊ on π is a mapping π, π₯ βΌππ₯: πΊ Γ π β π such that
a) 1π₯ = π₯, for all π₯ β π
b) (π1π2)π₯ = π1(π2π₯), all π1, π2 β πΊ, π₯ β π
A set together with a (left) action of πΊ is called a (left) πΊ-set. An action is trivial if ππ₯ =π₯ for all π β πΊ
Direct product
When πΊ and π» are groups, we can construct a new group πΊ Γ π», called the
(direct) product of πΊ and π». As a set, it is the Cartesian product of πΊ and π», and
multiplication is defined by: π, β πβ², ββ² = (ππβ², βββ²)
Normal subgroups
A subgroup π of πΊ is normal, denoted π β² πΊ, if πππβ1 = π for all π β πΊ
Semidirect product
A group πΊ πs a semidirect product of its subgroups π and π if π is normal and the
homomorphism πΊ β πΊ π induces an isomorphism π β πΊ π.
We write πΊ = π β π.
Wreath Product
The wreath product of two groups π΄ and π΅ is constructed in the following way.
Let π΄π΅ be the set of all functions defined on π΅ with values in π΄.
With respect to the componentwise multiplication, this set is a group which is the
complete direct product of π΅ copies of π΄.
The semidirect product π of π΅ and π΄π΅ is called the Cartesian wreath product of
π΄ and π΅, and is denoted by π΄ππ π΅.
If instead of π΄π΅ one takes the smaller group π΄(π΅) consisting of all functions with
finite support, that is, functions taking only non-identity values on a finite set of
points, then one obtains a subgroup ofπ called the wreath product of π΄ and π΅and is denoted by π΄ π€π π΅.
Beating the sum of cubes, finallyβ¦
The elusive group πΊ that managed to βbeat the sum of cubesβ turned out to
be a wreath product of:
Abelian group of order 173
Symmetric group of order 2
Symmetric group of order 2
Wreath Product
Abelian group of order 173
π π2.91
Beating the sum of cubes, finallyβ¦
The elusive group πΊ that managed to βbeat
the sum of cubesβ turned out to be a wreath
product of:
Abelian group of order 173
Symmetric group of order 2
Elementary fact of group representation theory
The index of the largest abelian subgroup of a
group is an upper bound on the size of the
maximum block of the block diagonal matrix
representation given by Wedderburnβs
theorem.
For a non-abelian group a
fundamental theorem of
Weddeburn says that the
group algebra is isomorphic,
via a Fourier transform, to an
algebra of block diagonal
matrices having block
dimensions π1β¦ππ, with ππ2 = πΊ .
Improving the bounds for π
Szegedy realized that some of the combinatory structures of the 1987
Coppersmith - Winograd paper could be used to select the three subsets in
the wreath product groups in amore sophisticated way.
The researchers managed to achieve exponential bound: π < 2.48
The researchers distilled their insights into two conjectures, one that has an
algebraic flavor and one that has a combinatorial.
A 6Γ6 strong USP, along with 2 of its 18
pieces