Sparse Matrices and Their Data Structures (PSC §4.2)

Sparse matrix data structures

Sparse Matrices and Their Data Structures(PSC §4.2)

1 / 19


Basic sparse technique: adding two vectors

I Problem: add a sparse vector y of length n to a sparse vectorx of length n, overwriting x, i.e.,

x := x + y.

I x is a sparse vector means that xi = 0 for most i .

I The number of nonzeros of x is cx and that of y is cy .

2 / 19


Example: storage as compressed vector

I Vectors x, y have length n = 8.

I Their number of nonzeros is cx = 3 and cy = 4.

I A compressed vector data structure for x and y is:x [j ].a = 2 5 1

x [j ].i = 5 3 7

y [j ].a = 1 4 1 4

y [j ].i = 6 3 5 2

I Here, the jth nonzero in the array of x hasnumerical value xi = x [j ].a and index i = x [j ].i .

I How to compute x + y?

3 / 19


Addition is easy for dense storage

I The dense vector data structure for x, y, and x + y is:0 0 0 5 0 2 0 1

0 0 4 4 0 1 1 0

0 0 4 9 0 3 1 1

I A compressed vector data structure for z = x + y is:z [j ].a = 3 9 1 1 4

z [j ].i = 5 3 7 6 2

I Conclusion: use an auxiliary dense vector!

4 / 19


Location array

The array loc (initialised to −1) stores the location j = loc[i ]where a nonzero vector component yi is stored in the compressedarray.

y [j ].a = 1 4 1 4

y [j ].i = 6 3 5 2

j = 0 1 2 3

yi = 0 0 4 4 0 1 1 0

loc[i ] = −1 −1 3 1 −1 2 0 −1

i = 0 1 2 3 4 5 6 7

5 / 19


Algorithm for sparse vector addition: pass 1

input: x : sparse vector with cx nonzeros, x = x0,y : sparse vector with cy nonzeros,loc : dense vector of length n,loc[i ] = −1, for 0 ≤ i < n.

output: x = x0 + y,loc[i ] = −1, for 0 ≤ i < n.

{ Register location of nonzeros of y}for j := 0 to cy − 1 do

loc[y [j ].i ] := j ;

6 / 19


Algorithm for sparse vector addition: passes 2, 3

{ Add matching nonzeros of x and y into x}for j := 0 to cx − 1 do

i := x [j ].i ;if loc[i ] 6= −1 then

x [j ].a := x [j ].a + y [loc[i ]].a;loc[i ] := −1;

{ Append remaining nonzeros of y to x }for j := 0 to cy − 1 do

i := y [j ].i ;if loc[i ] 6= −1 then

x [cx ].i := i ;x [cx ].a := y [j ].a;cx := cx + 1;loc[i ] := −1;

7 / 19


Analysis of sparse vector addition

I The total number of operations is O(cx + cy ), since there arecx + 2cy loop iterations, each with a small constant numberof operations.

I The number of flops equals the number of nonzeros in theintersection of the sparsity patterns of x and y. 0 flops canhappen!

I Initialisation of array loc costs n operations, which willdominate the total cost if only one vector addition has to beperformed.

I loc can be reused in subsequent vector additions, becauseeach modified element loc[i ] is reset to −1.

I If we add two n × n matrices row by row, we can amortise theO(n) initialisation cost over n vector additions.

8 / 19


Accidental zero

0 2000 4000 6000 8000 10000 12000 14000 16000

0

2000

4000

6000

8000

10000

12000

14000

16000

nz = 99147

Spy plot of the original matrix

17, 758× 17, 758 matrix memplus with 126, 150 entries, including27,003 accidental zeros.

I An accidental zero is a matrix element that is numerically zerobut still occurs as a nonzero pair (i , 0) in the data structure.

I Accidental zeros are created when a nonzero yi = −xi isadded to a nonzero xi and the resulting zero is retained.

I Testing all operations in a sparse matrix algorithm for zeroresults is more expensive than computing with a fewadditional nonzeros.

I Therefore, accidental zeros are usually kept.9 / 19


No abuse of numerics for symbolic purposes!

I Instead of using the symbolic location array, initialised at −1,we could have used an auxiliary array storing numerical values,initialised at 0.0.

I We could then add y into the numerical array, update xaccordingly, and reset the array.

I Unfortunately, this would make the resulting sparsity patternof x + y dependent on the numerical values of x and y: anaccidental zero in y would never lead to a new entry in thedata structure of x + y.

I This dependence may prevent reuse of the sparsity pattern incase the same program is executed repeatedly for a matrixwith different numerical values but the same sparsity pattern.

I Reuse often speeds up subsequent program runs.

10 / 19


Sparse matrix data structure: coordinate scheme

I In the coordinate scheme or triple scheme, every nonzeroelement aij is represented by a triple (i , j , aij), where i is therow index, j the column index, and aij the numerical value.

I The triples are stored in arbitrary order in an array.

I This data structure is easiest to understand and is often usedfor input/output.

I It is suitable for input to a parallel computer, since allinformation about a nonzero is contained in its triple. Thetriples can be sent directly and independently to theresponsible processors.

I Row-wise or column-wise operations on this data structurerequire a lot of searching.

11 / 19


Compressed Row Storage

I In the Compressed Row Storage (CRS) data structure, eachmatrix row i is stored as a compressed sparse vector consistingof pairs (j , aij) representing nonzeros.

I In the data structure, a[k] denotes the numerical value of thekth nonzero, and j [k] its column index.

I Rows are stored consecutively, in order of increasing i .

I start[i ] is the address of the first nonzero of row i .

I The number of nonzeros of row i is start[i + 1]− start[i ],where by convention start[n] = nz(A).

12 / 19


Example of CRS

A =

0 3 0 0 14 1 0 0 00 5 9 2 06 0 0 5 30 0 5 8 9

, n = 5, nz(A) = 13.

The CRS data structure for A is:a[k] = 3 1 4 1 5 9 2 6 5 3 5 8 9

j [k] = 1 4 0 1 1 2 3 0 3 4 2 3 4

k = 0 1 2 3 4 5 6 7 8 9 10 11 12

start[i ] = 0 2 4 7 10 13

i = 0 1 2 3 4 5

13 / 19


Sparse matrix–vector multiplication using CRS

input: A: sparse n × n matrix,v : dense vector of length n.

output: u : dense vector of length n, u = Av.

for i := 0 to n − 1 dou[i ] := 0;for k := start[i ] to start[i + 1]− 1 do

u[i ] := u[i ] + a[k] · v [j [k]];

14 / 19


Incremental Compressed Row Storage

I Incremental Compressed Row Storage (ICRS) is a variant ofCRS proposed by Joris Koster in 2002.

I In ICRS, the location (i , j) of a nonzero aij is encoded as a 1Dindex i · n + j .

I Instead of the 1D index itself, the difference with the 1D indexof the previous nonzero is stored, as an increment in the arrayinc. This technique is sometimes called delta indexing.

I The nonzeros within a row are ordered by increasing j , so thatthe 1D indices form a monotonically increasing sequence andthe increments are positive.

I An extra dummy element (n, 0) is added at the end.

15 / 19


Example of ICRS

A =

0 3 0 0 14 1 0 0 00 5 9 2 06 0 0 5 30 0 5 8 9

, n = 5, nz(A) = 13.

The ICRS data structure for A is:a[k] = 3 1 4 1 5 9 2 . . . 0

j [k] = 1 4 0 1 1 2 3 . . . 0

i [k] · n + j [k] = 1 4 5 6 11 12 13 . . . 25

inc[k] = 1 3 1 1 5 1 1 . . . 1

k = 0 1 2 3 4 5 6 . . . 13

16 / 19


Sparse matrix–vector multiplication using ICRS

input: A: sparse n × n matrix,v : dense vector of length n.

output: u : dense vector of length n, u = Av.

k := 0; j := inc[0];for i := 0 to n − 1 do

u[i ] := 0;while j < n do

u[i ] := u[i ] + a[k] · v [j ];k := k + 1;j := j + inc[k];

j := j − n;

Slightly faster: increments translate well into pointer arithmeticof programming language C; no indirect addressing v [j [k]].

17 / 19


A few other data structures

I Compressed column storage (CCS), similar to CRS.

I Gustavson’s data structure: both CRS and CCS, but storingnumerical values only once. Offers row-wise and column-wiseaccess to the sparse matrix.

I The two-dimensional doubly linked list: each nonzero isrepresented by i , j , aij , and links to a next and a previousnonzero in the same row and column. Offers maximumflexibility: row-wise and column-wise access are easy andelements can be inserted and deleted in O(1) operations.

I Matrix-free storage: sometimes it may be too costly to storethe matrix explicitly. Instead, each matrix element isrecomputed when needed. Enables solution of huge problems.

18 / 19


Summary

I Sparse matrix algorithms are more complicated than theirdense equivalents, as we saw for sparse vector addition.

I Sparse matrix computations have a larger integer overheadassociated with each floating-point operation.

I Still, using sparsity can save large amounts of CPU time andalso memory space.

I We learned an efficient way of adding two sparse vectors usinga dense initialised auxiliary array. You will be surprised to seehow often you can use this trick.

I Compressed row storage (CRS) and its variants are usefuldata structures for sparse matrices.

I CRS stores the nonzeros of each row together, but does notsort the nonzeros within a row. Sorting is a mixed blessing:it may help, but it also takes time.

19 / 19

Sparse Matrices and Their Data Structures (PSC §4.2)

Documents

Transcript of Sparse Matrices and Their Data Structures (PSC §4.2)