Data Objects
• Object creation
• Object assembly
• Setting options
• Viewing
• User-defined customizations
• Vectors (Vec)– focus: field data arising in nonlinear PDEs
• Matrices (Mat)– focus: linear operators arising in nonlinear PDEs (i.e., Jacobians)
tutorial outline: data objects
tutorial outline: data objects
beginnerbeginner
beginnerbeginner
intermediateintermediate
advancedadvanced
intermediateintermediate
Vectors
• Fundamental objects for storing field solutions, right-hand sides, etc.
• VecCreateMPI(...,Vec *)– MPI_Comm - processors that share the
vector– number of elements local to this processor– total number of elements
• Each process locally owns a subvector of contiguously numbered global indices
data objects: vectors
data objects: vectorsbeginnerbeginner
proc 3
proc 2
proc 0
proc 4
proc 1
Vectors: Example#include "vec.h"int main(int argc,char **argv){ Vec x,y,w; /* vectors */ Vec *z; /* array of vectors */ double norm,v,v1,v2;int n = 20,ierr; Scalar one = 1.0,two = 2.0,three = 3.0,dots[3],dot;
PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); ierr = OptionsGetInt(PETSC_NULL,"n",&n,PETSC_NULL); CHKERRA(ierr); ierr =VecCreate(PETSC_COMM_WORLD,PETSC_DECIDE,n,&x); CHKERRA(ierr); ierr = VecSetFromOptions(x);CHKERRA(ierr); ierr = VecDuplicate(x,&y);CHKERRA(ierr); ierr = VecDuplicate(x,&w);CHKERRA(ierr); ierr = VecSet(&one,x);CHKERRA(ierr); ierr = VecSet(&two,y);CHKERRA(ierr); ierr = VecSet(&one,z[0]);CHKERRA(ierr); ierr = VecSet(&two,z[1]);CHKERRA(ierr); ierr = VecSet(&three,z[2]);CHKERRA(ierr); ierr = VecDot(x,x,&dot);CHKERRA(ierr); ierr = VecMDot(3,x,z,dots);CHKERRA(ierr);
Vectors: Example ierr = PetscPrintf(PETSC_COMM_WORLD,"Vector length %d\n",(int)dot); CHKERRA(ierr); ierr = VecScale(&two,x);CHKERRA(ierr); ierr = VecNorm(x,NORM_2,&norm);CHKERRA(ierr); v = norm-2.0*sqrt((double)n); if (v > -1.e-10 && v < 1.e-10) v = 0.0; ierr = PetscPrintf(PETSC_COMM_WORLD,"VecScale %g\n",v); CHKERRA(ierr); ierr = VecDestroy(x);CHKERRA(ierr); ierr = VecDestroy(y);CHKERRA(ierr); ierr = VecDestroy(w);CHKERRA(ierr); ierr = VecDestroyVecs(z,3);CHKERRA(ierr); PetscFinalize(); return 0;}
Vector Assembly
• VecSetValues(Vec,…)– number of entries to insert/add
– indices of entries
– values to add
– mode: [INSERT_VALUES,ADD_VALUES]
• VecAssemblyBegin(Vec)• VecAssemblyEnd(Vec)
data objects: vectors
data objects: vectors
beginnerbeginner
Parallel Matrix and Vector Assembly
• Processors may generate any entries in vectors and matrices
• Entries need not be generated on the processor on which they ultimately will be stored
• PETSc automatically moves data during the assembly process if necessary
data objects: vectors and matrices
data objects: vectors and matrices
beginnerbeginner
Selected Vector Operations
Function Name Operation
VecAXPY(Scalar *a, Vec x, Vec y) y = y + a*xVecAYPX(Scalar *a, Vec x, Vec y) y = x + a*yVecWAXPY(Scalar *a, Vec x, Vec y, Vec w) w = a*x + yVecScale(Scalar *a, Vec x) x = a*xVecCopy(Vec x, Vec y) y = xVecPointwiseMult(Vec x, Vec y, Vec w) w_i = x_i *y_iVecMax(Vec x, int *idx, double *r) r = max x_iVecShift(Scalar *s, Vec x) x_i = s+x_iVecAbs(Vec x) x_i = |x_i |VecNorm(Vec x, NormType type , double *r) r = ||x||
Sparse Matrices
• Fundamental objects for storing linear operators (e.g., Jacobians)
• MatCreateMPIAIJ(…,Mat *)– MPI_Comm - processors that share the matrix
– number of local rows and columns
– number of global rows and columns
– optional storage pre-allocation information
data objects: matrices
data objects: matrices
beginnerbeginner
Parallel Matrix Distribution
MatGetOwnershipRange(Mat A, int *rstart, int *rend)– rstart: first locally owned row of global matrix
– rend -1: last locally owned row of global matrix
Each process locally owns a submatrix of contiguously numbered global rows.
proc 0
} proc 3: locally owned rowsproc 3proc 2proc 1
proc 4
data objects: matrices
data objects: matrices
beginnerbeginner
Matrix Assembly
• MatSetValues(Mat,…)– number of rows to insert/add
– indices of rows and columns
– number of columns to insert/add
– values to add
– mode: [INSERT_VALUES,ADD_VALUES]
• MatAssemblyBegin(Mat)• MatAssemblyEnd(Mat)
data objects: matrices
data objects: matrices
beginnerbeginner
Viewer Concepts
• Information about PETSc objects– runtime choices for solvers, nonzero info for matrices, etc.
• Data for later use in restarts or external tools– vector fields, matrix contents – various formats (ASCII, binary)
• Visualization– simple x-window graphics
• vector fields• matrix sparsity structure
beginnerbeginner viewersviewers
Viewing Vector Fields• VecView(Vec x,Viewer v);• Default viewers
– ASCII (sequential):VIEWER_STDOUT_SELF
– ASCII (parallel):VIEWER_STDOUT_WORLD
– X-windows:VIEWER_DRAW_WORLD
• Default ASCII formats– VIEWER_FORMAT_ASCII_DEFAULT– VIEWER_FORMAT_ASCII_MATLAB– VIEWER_FORMAT_ASCII_COMMON– VIEWER_FORMAT_ASCII_INFO– etc.
viewersviewersbeginnerbeginner
Solution components, using runtime option -snes_vecmonitor
velocity: u velocity: v
temperature: Tvorticity:
Viewing Matrix Data
• MatView(Mat A, Viewer v);
• Runtime options available after matrix assembly– -mat_view_info
• info about matrix assembly
– -mat_view_draw • sparsity structure
– -mat_view • data in ASCII
– etc.
viewersviewersbeginnerbeginner
Linear Solvers
Goal: Support the solution of linear systems,
Ax=b,
particularly for sparse, parallel problems arising
within PDE-based models
User provides:– Code to evaluate A, b
solvers:linear
solvers:linearbeginnerbeginner
Linear Solver Object
SLES Object
Coeff.Matrix
Precond.Matrix
KSP Object
PC object
- Method: CG, GMRES etc.- Convergence tolerance- ...
- Pctype: ILU, LU etc.
Basic Linear Solver Code (C/C++)
SLES sles; /* linear solver context */Mat A; /* matrix */Vec x, b; /* solution, RHS vectors */int n, its; /* problem dimension, number of iterations */
MatCreate(MPI_COMM_WORLD,n,n,&A); /* assemble matrix */VecCreate(MPI_COMM_WORLD,n,&x); VecDuplicate(x,&b); /* assemble RHS vector */
SLESCreate(MPI_COMM_WORLD,&sles); SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN);SLESSetFromOptions(sles);SLESSolve(sles,b,x,&its);
solvers:linear
solvers:linearbeginnerbeginner
Basic Linear Solver Code (Fortran)SLES sles Mat AVec x, binteger n, its, ierr
call MatCreate(MPI_COMM_WORLD,n,n,A,ierr) call VecCreate(MPI_COMM_WORLD,n,x,ierr)call VecDuplicate(x,b,ierr)
call SLESCreate(MPI_COMM_WORLD,sles,ierr)call SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN,ierr)call SLESSetFromOptions(sles,ierr)call SLESSolve(sles,b,x,its,ierr)
C then assemble matrix and right-hand-side vector
solvers:linear
solvers:linearbeginnerbeginner
Customization Options
• Procedural Interface– Provides a great deal of control on a usage-by-usage
basis inside a single code
– Gives full flexibility inside an application
• Command Line Interface– Applies same rule to all queries via a database
– Enables the user to have complete control at runtime, with no extra coding
solvers:linear
solvers:linear
intermediateintermediate
Setting Solver Options within Code
• SLESGetKSP(SLES sles,KSP *ksp)– KSPSetType(KSP ksp,KSPType type)– KSPSetTolerances(KSP ksp,double
rtol,double atol,double dtol, int maxits)– ...
• SLESGetPC(SLES sles,PC *pc)– PCSetType(PC pc,PCType)– PCASMSetOverlap(PC pc,int overlap)– ...
solvers:linear
solvers:linear
intermediateintermediate
• -ksp_type [cg,gmres,bcgs,tfqmr,…]
• -pc_type [lu,ilu,jacobi,sor,asm,…]
• -ksp_max_it <max_iters>• -ksp_gmres_restart <restart>• -pc_asm_overlap <overlap>• -pc_asm_type
[basic,restrict,interpolate,none]• etc ...
Setting Solver Options at Runtime
solvers:linear
solvers:linearbeginnerbeginner
1
intermediateintermediate
2
1
2
SLES: Review of Basic Usage
• SLESCreate( ) - Create SLES context• SLESSetOperators( ) - Set linear operators• SLESSetFromOptions( ) - Set runtime solver
options for [SLES, KSP,PC]
• SLESSolve( ) - Run linear solver• SLESView( ) - View solver options
actually used at runtime (alternative: -sles_view)
• SLESDestroy( ) - Destroy solver
beginnerbeginnersolvers:linear
solvers:linear
SLES: Review of Selected Preconditioner Options
Functionality Procedural Interface Runtime Option
Set preconditioner type PCSetType( ) -pc_type [lu,ilu,jacobi, sor,asm,…]
Set level of fill for ILU PCILULevels( ) -pc_ilu_levels <levels>Set SOR iterations PCSORSetIterations( ) -pc_sor_its <its>Set SOR parameter PCSORSetOmega( ) -pc_sor_omega <omega>Set additive Schwarz variant
PCASMSetType( ) -pc_asm_type [basic, restrict,interpolate,none]
Set subdomain solver options
PCGetSubSLES( ) -sub_pc_type <pctype> -sub_ksp_type <ksptype> -sub_ksp_rtol <rtol>
And many more options...solvers: linear: preconditioners
solvers: linear: preconditionersbeginnerbeginner
1
intermediateintermediate
2
1
2
SLES: Review of Selected Krylov Method Options
solvers: linear: Krylov methods
solvers: linear: Krylov methodsbeginnerbeginner
1
intermediateintermediate
2And many more options...
Functionality Procedural Interface Runtime Option
Set Krylov method KSPSetType( ) -ksp_type [cg,gmres,bcgs, tfqmr,cgs,…]
Set monitoring routine
KSPSetMonitor() -ksp_monitor, –ksp_xmonitor, -ksp_truemonitor, -ksp_xtruemonitor
Set convergence tolerances
KSPSetTolerances( ) -ksp_rtol <rt> -ksp_atol <at> -ksp_max_its <its>
Set GMRES restart parameter
KSPGMRESSetRestart( ) -ksp_gmres_restart <restart>
Set orthogonalization routine for GMRES
KSPGMRESSetOrthogon alization( )
-ksp_unmodifiedgramschmidt -ksp_irorthog
1
2
SLES: Example Programs
• ex1.c, ex1f.F - basic uniprocessor codes • ex2.c, ex2f.F - basic parallel codes • ex11.c - using complex numbers
• ex4.c - using different linear system and preconditioner matrices• ex9.c - repeatedly solving different linear systems
• ex15.c - setting a user-defined preconditioner
And many more examples ...
Location: www.mcs.anl.gov/petsc/src/sles/examples/tutorials/
solvers:linear
solvers:linear
1
2
beginnerbeginner
1
intermediateintermediate
2
advancedadvanced
3
3
Example: ex1.c (Sequential Tridiagonal Solver)
int main(int argc,char **args){ Vec x, b, u; /* approx solution, RHS, exact solution */ Mat A; /* linear system matrix */ SLES sles; /* linear solver context */ PC pc; /* preconditioner context */ KSP ksp; /* Krylov subspace method context */ double norm; /* norm of solution error */ int ierr, i, n = 10, col[3], its, flg, size; Scalar neg_one = -1.0, one = 1.0, value[3];
PetscInitialize(&argc,&args,(char *)0,help); MPI_Comm_size(PETSC_COMM_WORLD,&size); if (size != 1) SETERRA(1,0,"This is a uniprocessor example only!"); ierr = OptionsGetInt(PETSC_NULL,"-n",&n,&flg); CHKERRA(ierr); ierr = MatCreate(PETSC_COMM_WORLD,n,n,&A); CHKERRA(ierr); value[0] = -1.0; value[1] = 2.0; value[2] = -1.0; for (i=1; i<n-1; i++ ) { col[0] = i-1; col[1] = i; col[2] = i+1; ierr = MatSetValues(A,1,&i,3,col,value,INSERT_VALUES); CHKERRA(ierr); } i = n - 1; col[0] = n - 2; col[1] = n - 1; ierr = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES); CHKERRA(ierr); i = 0; col[0] = 0; col[1] = 1; value[0] = 2.0; value[1] = -1.0; ierr = MatSetValues(A,1,&i,2,col,value,INSERT_VALUES); CHKERRA(ierr); ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRA(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRA(ierr);
Example: ex1.c (cont’d)
ierr = VecCreate(PETSC_COMM_WORLD,PETSC_DECIDE,n,&x); ierr = VecSetFromOptions(x);CHKERRA(ierr); ierr = VecDuplicate(x,&b); CHKERRA(ierr); ierr = VecDuplicate(x,&u); CHKERRA(ierr); ierr = VecSet(&one,u); CHKERRA(ierr); ierr = MatMult(A,u,b); CHKERRA(ierr); ierr = SLESCreate(PETSC_COMM_WORLD,&sles); CHKERRA(ierr); ierr =SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN); ierr = SLESGetKSP(sles,&ksp); CHKERRA(ierr); ierr = SLESGetPC(sles,&pc); CHKERRA(ierr); ierr = PCSetType(pc,PCJACOBI); CHKERRA(ierr); ierr = KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT, PETSC_DEFAULT); CHKERRA(ierr); /* Set runtime options, e.g., -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol> */ ierr = SLESSetFromOptions(sles); CHKERRA(ierr); ierr = SLESSolve(sles,b,x,&its); CHKERRA(ierr);
Example: ex1.c (cont’d)
ierr = SLESSolve(sles,b,x,&its); CHKERRA(ierr); ierr = SLESView(sles,VIEWER_STDOUT_WORLD); CHKERRA(ierr); ierr = VecAXPY(&neg_one,u,x); CHKERRA(ierr); ierr = VecNorm(x,NORM_2,&norm); CHKERRA(ierr); if (norm > 1.e-12) PetscPrintf(PETSC_COMM_WORLD,"Norm= %g, Iters= %d\n",norm,its); else PetscPrintf(PETSC_COMM_WORLD,"Norm < 1.e-12, Iters= %d\n",its); ierr = VecDestroy(x); CHKERRA(ierr); ierr = VecDestroy(u); CHKERRA(ierr); ierr = VecDestroy(b); CHKERRA(ierr); ierr = MatDestroy(A);CHKERRA(ierr); ierr = SLESDestroy(sles); CHKERRA(ierr); PetscFinalize(); return 0;}
Example: ex2.c (Parallel Laplace Solver)
int main(int argc,char **args){ Vec x, b, u; /* approx solution, RHS, exact solution */ Mat A; /* linear system matrix */ SLES sles; /* linear solver context */ PetscRandom rctx; /* random number generator context */ double norm; /* norm of solution error */ int i, j, I, J, Istart, Iend, ierr, m = 8, n = 7, its, flg; Scalar v, one = 1.0, neg_one = -1.0; KSP ksp; PetscInitialize(&argc,&args,(char *)0,help); ierr = OptionsGetInt(PETSC_NULL,"-m",&m,&flg); CHKERRA(ierr); ierr = OptionsGetInt(PETSC_NULL,"-n",&n,&flg); CHKERRA(ierr); ierr = MatGetOwnershipRange(A,&Istart,&Iend); CHKERRA(ierr); for ( I=Istart; I<Iend; I++ ) { v = -1.0; i = I/n; j = I - i*n; if ( i>0 ) {J = I - n; MatSetValues(A,1,&I,1,&J,&v,INSERT_VALUES); } if ( i<m-1 ) {J = I + n; MatSetValues(A,1,&I,1,&J,&v,INSERT_VALUES); } if ( j>0 ) {J = I - 1; MatSetValues(A,1,&I,1,&J,&v,INSERT_VALUES); } if ( j<n-1 ) {J = I + 1; MatSetValues(A,1,&I,1,&J,&v,INSERT_VALUES); } v = 4.0; MatSetValues(A,1,&I,1,&I,&v,INSERT_VALUES); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRA(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRA(ierr);
Top Related