Parallelizing Gauss-Seidel Solver/Pre-conditioner

Aim: To parallelize a Gauss-Seidel Solver, which can be used as a pre-conditioner for the finite element code poicyc.cpp

Parallelizing a GS solver involves two steps:

1. Re-arrange the stiffness matrix to permit a parallel solution

2. Solve the re-arranged stiffness matrix on multiple CPUs

After parallelizing:

A. Interface the above with poicyc.cpp to solve large finite element problems on multiple CPUs

B. The Gauss-Seidel solver can be extended to SOR solver with the inclusion of factor omega and interpolating with previous vector

Part 1: Re-ordering the Stiffness Matrix

Why: The stiffness matrix needs to be re-ordered such that its main diagonal comprises of diagonal sub-matrices. This is achieved by re-numbering the nodes using a ‘coloring scheme’. The new re-numbered mesh will have the structure on the right.

Stiffness Matrix Before Re-ordering

Stiffness Matrix After Two-color Re-ordering

Node Coloring and Re-ordering

General Procedure:

• For each node in the mesh connectivity, search for all its neighbours. This includes all nodes in all the elements that this node is connected to.

• Assign a color (i.e. a numerical label) to each node one by one, with the condition that a node cannot have the same color as any of its neighbouring nodes.

Structure of the Code:

• For each node, check neighbours, and store all those nodes

• For each node, assign color ‘0’. Then assign the smallest numerical label that its neighbours does not have.

• Collect all nodes of a color, and number them in numerically increasing order. Do the same for all nodes of all colors.

• Re-order the element connectivity based on the new numbering scheme. Create new vectors for ‘p’ (coordinate data) and ‘t’ (connectivity).

Images of the Mesh and Stiffness Matrices after Re-ordering

FE mesh of the cube(uses 10-node tetrahedrons)

Nodal coloring of a 2-D 6-node Triangular mesh

0 50 100 150 200

0

50

100

150

200

nz = 3638

Stiffness Matrix After Re-ordering (for the cubic mesh in previous slide)

FE mesh of the cube(uses 10-node tetrahedrons)

Note multiple Diagonal ‘sub-matrices’ along main Diagonal-one for each color 13 Colors required above

0 50 100 150 200

0

50

100

150

200

nz = 3638

Stiffness Matrix After Re-ordering for a 25,000/Half-million node mesh

Solving the New Stiffness Matrix by Gauss-Seidel

Gauss-Seidel Solution for: Ax=B

1. Start with a trial vector x 1 (=B) Compute: B_new = B - U x 1

2. Solve: [D + L] = B_new3. Update x & repeat Steps 1 through 3

With re-ordered matrix, we have a parallel solution:

1. All rows in step 1 can be solve simultaneously (parallel)2. Forward substitution can be parallelized thus:

1. Solve D_1 = x _UPPER HALF (simple division: x(i)= b(i)/a(i) )2. Compute y = B_new –L x _UPPER HALF

3. Solve x_2 = D_2 /y3. For multiple colors, do step 1, then repeat steps 2 and 3 for each color

U

L

D_1

D_2

Parallelizing the GS code (on two processes)

Step 1Cyclically, submit each row to a process to Compute RHS – U*x

Gather all rows, assembleB-new and broadcast toAll procs.

About half the rows areComputed by process 1

Send ComputedRows to Proc. 0

Step 2: Compute solutionTo diagonal elements of Each sub-matrix (on proc. 0)

Step 3: Cyclically submitEach row to a process andCompute: y = B-Lx x= y/d

Gather all rows, assembleB-new and broadcast toAll procs.

About half the rows areComputed by process 1

Send Computedx to Proc. 0

Iterate amongThese steps

B/Cast B-new

For allcolors

B/Cast x

Run-Times on various processors

25,000 node model

500,000 node model

For 300 iterations

1 p 2 p 4 p

27.2 s 50.7 s 9.2 s

9m 48 s 19m 33s

Node coloring:

2-3 s

17 s