Basic Concepts of FEM Framework & API
description
Transcript of Basic Concepts of FEM Framework & API
1
Basic Concepts Basic Concepts of FEM of FEM Framework & APIFramework & API
Gunavardhan Gunavardhan KakulapatiKakulapati
(([email protected]@cs.uiuc.edu))
2
IntroductionIntroduction FEM Framework overviewFEM Framework overview Using the framework APIUsing the framework API InstallationInstallation Conversion ExampleConversion Example
3
MotivationMotivation
Finite Element Method Finite Element Method Used extensivelyUsed extensively Computationally intensiveComputationally intensive
Do it in parallel !!Do it in parallel !!
4
FEM ProgramsFEM Programs
FEM programs manipulate FEM programs manipulate elements and nodeselements and nodes
Element is a portion of problem Element is a portion of problem domain, surrounded by nodesdomain, surrounded by nodes
Element computation: based on Element computation: based on fieldsfields of surrounding nodes of surrounding nodes
Elements contribute to Elements contribute to fieldsfields of of surrounding nodessurrounding nodes
5
FEM MeshFEM Mesh
ElemenElementt
Surrounding Surrounding NodesNodes
E1E1 N1N1 N3N3 N4N4
E2E2 N1N1 N2N2 N4N4
E3E3 N2N2 N4N4 N5N5
6
FEM Program StructureFEM Program Structure
read mesh, connectivity, boundary conditionstime loop element loop- Element deformation applies forces to surrounding nodes node loop- Forces and boundary conditions change node positions end time loopwrite out mesh data for postprocessing
7
ParallelizationParallelization
Partition the FEM Mesh into Partition the FEM Mesh into multiple chunksmultiple chunks
Distribute elements, replicate Distribute elements, replicate shared nodesshared nodes
Shared nodes = Communication !!Shared nodes = Communication !! Partition so that communication is Partition so that communication is
minimizedminimized
8
PartitioningPartitioning
ElemenElementt
Surrounding Surrounding NodesNodes
E1E1 N1N1 N3N3 N4N4
E2E2 N1N1 N2N2 N3N3
ElemenElementt
Surrounding Surrounding NodesNodes
E1E1 N1N1 N2N2 N3N3
Shared NodesShared Nodes
AA BB
N2N2 N1N1
N4N4 N3N3
9
Parallel FEM ProgramParallel FEM Program
read/get chunk mesh data, connectivity, shared nodeschunk time loop element loop- Element deformation applies forces to surrounding nodes <update forces on shared nodes> node loop- Forces and boundary conditions change node positions end time loopwrite chunk mesh data for postprocessing
>>
10
FEM Framework: GoalsFEM Framework: Goals
Separate parallel implementation Separate parallel implementation from numerical algorithmsfrom numerical algorithms
Parallel version to closely resemble Parallel version to closely resemble the serial programthe serial program
Allow features like load balancing, Allow features like load balancing, visualization.visualization.
11
FEM Framework: FEM Framework: ResponsibilitiesResponsibilities
Charm++(Dynamic Load Balancing, Communication)
FEM Framework(Update of Nodal properties, Reductions over nodes or partitions)
FEM Application(Initialize, Registration of Nodal Attributes, Loops Over Elements, Finalize)
METIS I/O
Partitioner Combiner
12
FEM Framework ProgramFEM Framework Program
May contain user-written, library-called May contain user-written, library-called subroutines:subroutines: initinit driverdriver mesh_updatedmesh_updated (more on this later…) (more on this later…)
init init andand mesh_updatedmesh_updated are called on are called on processor 0processor 0
driverdriver is called on every chunk is called on every chunk
13
Structure of an FEM Structure of an FEM ApplicationApplication
init()
Update Update Update
driver driver driver
Shared Nodes Shared Nodes
14
initinit
subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine
15
driverdriver
subroutine driver get local mesh chunk time loop FEM computations update shared node fields more FEM computations end time loop end subroutine
16
Data outputData output
Parallel output at the end of driverParallel output at the end of driver Possible to visualize the data rather Possible to visualize the data rather
than printing it out (netfem).than printing it out (netfem). Framework calls like FEM_Mesh_updated. Framework calls like FEM_Mesh_updated.
(more on this later …)(more on this later …)
>>>>
17
Framework CallsFramework Calls
FEM_Set_*FEM_Set_* Called from initialization to set the serial meshCalled from initialization to set the serial mesh Framework partitions mesh into chunksFramework partitions mesh into chunks
FEM_Create_fieldFEM_Create_field Registers a node data field with the framework, Registers a node data field with the framework,
supports user data typessupports user data types FEM_Update_fieldFEM_Update_field
Updates node data field across all processorsUpdates node data field across all processors Handles all parallel communicationHandles all parallel communication
Other parallel calls (Reductions, etc.)Other parallel calls (Reductions, etc.)
18
Framework Calls: MeshFramework Calls: Mesh
FEM_Set_*FEM_Set_* From From initinit, these routines describe the , these routines describe the
mesh to the FEM Frameworkmesh to the FEM Framework Framework calls the partitionerFramework calls the partitioner
FEM_Get_*FEM_Get_* From each chunk, the local portion of From each chunk, the local portion of
the mesh is obtained by the the mesh is obtained by the driverdriver
19
FEM_*_elemFEM_*_elemFEM_Set_elem(int elType,int nEl,
int doublePerEl,int nodePerEl);
subroutine FEM_Set_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType,nEl,doublePerEl,nodePerEl
FEM_Get_elem(int elType,int* nEl, int* doublePerEl,int* nodePerEl);
subroutine FEM_Get_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType integer, intent(out) :: nEl,doublePerEl,nodePerEl
20
FEM_*_nodeFEM_*_node
subroutine FEM_Set_node(nNode,doublePerNode) integer, intent(in) :: nNode,doublePerNode
subroutine FEM_Get_node(nNode,doublePerNode) integer, intent(out) :: nNode,doublePerNode
21
Element ConnectivityElement Connectivitysubroutine FEM_Set_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Get_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Set_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nEl,nodePerEl) :: conn
subroutine FEM_Get_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nEl,nodePerEl) :: conn
22
Additional Data for Nodes Additional Data for Nodes and Elementsand Elements
subroutine FEM_Set_node_data_r(data) REAL*8, intent(in), dimension(doublePerNode,nNode) :: data
subroutine FEM_Get_node_data_r(data) REAL*8, intent(out), dimension(doublePerNode,nNode) :: data
subroutine FEM_Set_elem_data_r(data) REAL*8, intent(in), dimension(doublePerElem,nElem) :: data
subroutine FEM_Get_elem_data_r(data) REAL*8, intent(out), dimension(doublePerElem,nElem) :: data
23
Node FieldsNode Fields
Framework handles combining data for Framework handles combining data for shared nodes and keeps them in syncshared nodes and keeps them in sync
Framework does not understand Framework does not understand meaning of node fields, only their meaning of node fields, only their location and typeslocation and types
Framework needs to be informed of Framework needs to be informed of locations and types of fieldslocations and types of fields
Create_fieldCreate_field once, once, Update_fieldUpdate_field every every timesteptimestep
24
FEM_Create_fieldFEM_Create_field
To handle the updating of shared To handle the updating of shared node values.node values.
Tell the framework where the Tell the framework where the shared data items of each node shared data items of each node are.are.
Creates a “field” and pass the field Creates a “field” and pass the field ID for updating shared nodal ID for updating shared nodal values.values.
25
FEM_Create_simple_fieldFEM_Create_simple_field
function integer :: FEM_Create_simple_field( base_type, vec_len) integer, intent(in) :: base_type, vec_len
Base_type
•FEM_BYTE- INTEGER*1, or CHARACTER*1 •FEM_INT- INTEGER*4 •FEM_REAL- REAL*4 •FEM_DOUBLE- DOUBLE PRECISION, or REAL*8
26
Create_simple_field Create_simple_field ExampleExample
! 3D Force for each node! stored as 3*n real*8 array
REAL*8 ALLOCATABLE, DIMENSION(:) :: nodeForce INTEGER :: fid
... allocate nodeForce as 3*n_nodes...
fid = FEM_Create_simple_field(FEM_DOUBLE,3)
27
FEM_Create_fieldFEM_Create_field
function integer :: FEM_Create_Field(base_type, vec_len, offset, dist)
integer, intent(in) :: base_type, vec_len, offset, dist
28
Node FieldsNode Fields
29
Create_field ExampleCreate_field Example
! 3D force is contained as fXYZ variable! in a user-defined type node_type
TYPE(node_type), ALLOCATABLE, DIMENSION(:) :: nodes INTEGER :: fid
...allocate nodes array as n_nodes...
fid = FEM_Create_Field(FEM_DOUBLE,3, offsetof(nodes(1), nodes(1)%fXYZ), offsetof(nodes(1), nodes(2)) )
30
Update and Reduce FieldUpdate and Reduce Fieldsubroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes
subroutine FEM_Reduce_Field(fid,nodes,outVal,op) integer, intent(in) :: fid,op varies, intent(in) :: nodes varies, intent(out) :: outVal op is
•FEM_SUM •FEM_MIN •FEM_MAX
31
UtilityUtility
function integer :: FEM_Num_Partitions()
function integer :: FEM_My_Partition()
function double precision :: FEM_Timer()
subroutine FEM_Print_Partition()
subroutine FEM_Print(str) character*, intent(in) :: str
32
Advanced FEM callsAdvanced FEM calls
FEM_Update_meshFEM_Update_mesh Reassembles chunks of the meshReassembles chunks of the mesh
FEM_Add_nodeFEM_Add_node Adds a new node into the meshAdds a new node into the mesh
33
FEM_Update_meshFEM_Update_mesh Reassembles all the chunksReassembles all the chunks Can be called only from driver Can be called only from driver Must be called from all chunksMust be called from all chunks Useful scenariosUseful scenarios
Giving out data as the simulation runsGiving out data as the simulation runs Repartition the meshRepartition the mesh Serial output when simulation finishesSerial output when simulation finishes
34
FEM_Update_meshFEM_Update_mesh
C call:C call:
void FEM_Update_mesh(int callMeshUpdated, int doWhat)
Fortran call:Fortran call:subroutine FEM_Update_mesh(callMeshUpdated,doWhat)
integer, intent(in) :: callMeshUpdated, doWhat
35
FEM_Update_mesh FEM_Update_mesh parametersparameters
callMeshUpdatedcallMeshUpdated is non-zero => is non-zero => call call mesh_updated mesh_updated (callMeshUpdated)(callMeshUpdated)
doWhat:doWhat:doWhatdoWhat RepartitionRepartition FEM_Update_meshFEM_Update_mesh
00 NoNo Non-blocking Non-blocking mesh_updatedmesh_updated
11 YesYes Blocks for repartitioningBlocks for repartitioning
22 NoNo Blocking mesh_updatedBlocking mesh_updated
36
FEM_Update_mesh FEM_Update_mesh examplesexamples
FEM_Update_meshFEM_Update_mesh(k,0) (k,0) Call mesh_updated(k) on assembled Call mesh_updated(k) on assembled
mesh, while the driver continuesmesh, while the driver continues FEM_Update_meshFEM_Update_mesh(k,1) (k,1)
Repartition after mesh_updatedRepartition after mesh_updated FEM_Update_meshFEM_Update_mesh(k,2)(k,2)
Block driver routines till Block driver routines till mesh_updated(k)mesh_updated(k)
37
Adding NodesAdding Nodes
One can add new nodes, and One can add new nodes, and update connectivity in driverupdate connectivity in driver Use Use FEM_Set_*FEM_Set_* subroutines subroutines New nodes are considered privateNew nodes are considered private
Framework can repartition the Framework can repartition the meshmesh Optionally calls user’s Optionally calls user’s mesh_updatedmesh_updated
subroutinesubroutine
38
FEM_Add_nodeFEM_Add_node
C call:C call:
void FEM_Add_node(int localIdx, int nBetween, int * betweenNodes)
Fortran call:Fortran call:
subroutine FEM_Add_node(localIdx,nBetween,betweenNodes)
integer, intent(in) :: localIdx,nBetween integer, intent(in) :: betweenNodes(nBetween)
>>
39
Installing FEM Installing FEM FrameworkFramework
40
Where to Get It ?Where to Get It ?FEM Framework is included in Charm++ distribution, available under CVSCSH:setenv CVSROOT ":pserver:[email protected]:/cvsroot"Or BASH:export CVSROOT=":pserver:[email protected]:/cvsroot"
You should now be able to do a
> cvs login(no password needed, just type [Enter] at prompt)
and then
> cvs co -P charm
to get the entire Charm++ source.
41
How to Build It ?How to Build It ?
> cd charm
and do
> ./build FEM net-linux -O
This will make a net-linux directory, with bin, include, lib etc subdirectories.
Platforms: net-sol, mpi-origin, mpi-linux etc.
42
How to Write Programs ?How to Write Programs ?
Write from scratchWrite from scratch Concepts and API discussed earlierConcepts and API discussed earlier Read the FEM-Framework Manual Read the FEM-Framework Manual
http://charm.cs.uiuc.edu/manuals/femhttp://charm.cs.uiuc.edu/manuals/fem
Convert existing programConvert existing program To be covered in next sectionTo be covered in next section
43
How to Compile & Link ?How to Compile & Link ?
Use “Use “charmccharmc”: available under bin”: available under bin a multi-lingual compiler driver, understands a multi-lingual compiler driver, understands
f90f90 Knows where modules and libraries areKnows where modules and libraries are Portable across machines and compilersPortable across machines and compilers
LinkingLinking use use “-language femf” “-language femf” : for F90: for F90 Use Use “–language fem”“–language fem” : for C/C++ : for C/C++
See example MakefilesSee example Makefiles pgms/charm++/fem/…pgms/charm++/fem/…
44
How to Run ?How to Run ?
Just run it! (net- versions only)Just run it! (net- versions only) Serial, but nice for debugging/testingSerial, but nice for debugging/testing
Use CharmrunUse Charmrun A portable parallel job execution scriptA portable parallel job execution script Specify number of processors: Specify number of processors: +pN+pN Special “Special “nodelistnodelist” file for net-* versions” file for net-* versions Multiple chunks per processor: use Multiple chunks per processor: use +vpM+vpM
45
Charmrun ExampleCharmrun Example
>>>>
./charmrun pgm +p4 +vp8
Nodelist File: $(HOME)/.nodelist
group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.eduetc…
46
FEM Framework FEM Framework Conversion Conversion exampleexample
47
A Serial ProgramA Serial Program
Input
Processing
Output
48
A Parallel Framework A Parallel Framework ProgramProgram
InputParallelProcessing
Output ParallelInfrastructure
49
Real Names of PiecesReal Names of Pieces
InitDriver
Finalize orMesh_updated Charm++ FEM
Framework
50
Serial Example ProgramSerial Example Program
F90 exampleF90 example Reads input mesh in “Triangle” formatReads input mesh in “Triangle” format Does simple explicit mechanics Does simple explicit mechanics
computation (CST triangles)computation (CST triangles) Writes Tecplot outputWrites Tecplot output
Not a toy example (by Philippe Not a toy example (by Philippe Geubelle)Geubelle) Reads a parameter fileReads a parameter file Applies boundary conditionsApplies boundary conditions Uses real mechanicsUses real mechanics
51
FEM Framework VersionFEM Framework Version
Uses all the same code as serial Uses all the same code as serial versionversion
Main program becomes init subroutineMain program becomes init subroutine Split up init and driverSplit up init and driver
Copy entire mesh into framework Copy entire mesh into framework (FEM_Set)(FEM_Set)
Copy portion of mesh out (FEM_Get)Copy portion of mesh out (FEM_Get) Changes to time loop are minimalChanges to time loop are minimal Output poses an interesting problemOutput poses an interesting problem
52
Splitting the programSplitting the program
Declare variablesDeclare variables
call readTriGlobalscall readTriGlobals InitInit
Read mesh conn. DataRead mesh conn. Data
Dynamics loopDynamics loop
Calc forcesCalc forces
Calc accl,velCalc accl,vel
Apply bound. conditionsApply bound. conditions DriverDriver
Update displacementsUpdate displacements
Output current dataOutput current data
53
Init – Basic differencesInit – Basic differences
Serial versionSerial version
Declare variablesDeclare variables
. . .. . .
call readTriGlobalscall readTriGlobals
. . .. . .Read mesh conn. dataRead mesh conn. data
Parallel versionParallel version
Declare variablesDeclare variables
. . .. . .
call readTriGlobalscall readTriGlobals
. . .. . .Read mesh conn. dataRead mesh conn. data
. . .. . .Call the set methodsCall the set methods
54
DriverDriver
Serial versionSerial version
Dynamics loopDynamics loop
Calc forcesCalc forces
Calc accl,velCalc accl,vel
Apply bound. CondApply bound. Cond
Update dispUpdate disp
Output current Output current datadata
Parallel versionParallel version
Call get methodsCall get methods
Create fieldCreate field
Dynamics loopDynamics loop
Calc forcesCalc forces
Update fieldUpdate field
Calc accl,velCalc accl,vel
Apply bound. CondApply bound. Cond
Update dispUpdate disp
Visualize current Visualize current datadata
55
More information: More information: Read the manual and FEM framework Read the manual and FEM framework
details at details at http://charm.cs.uiuc.eduhttp://charm.cs.uiuc.edu