HPC: a Paradigm for the ANSYS Software

HPC: a Paradigm for HPC: a Paradigm for the ANSYS Softwarethe ANSYS Software

Michel RochetteMichel RochetteR&D DirectorR&D Director

© 2009 ANSYS, Inc. All rights reserved. 1 ANSYS, Inc. Proprietary© 2009 ANSYS, Inc. All rights reserved. 1 ANSYS, Inc. Proprietary

Outline

• ANSYS, Inc ,

• High Performance Computing

• Performance - Fluids

• Performance - StructurePerformance Structure

• Healthcare Challenges and HPC

• Cerebral Aneurysms

• Osteoporosis• Osteoporosis

• Conclusion

© 2009 ANSYS, Inc. All rights reserved. 2 ANSYS, Inc. Proprietary

World’s LargestSimulation CommunitySimulation Community

20 3Network of sales channel partners in 40+ countriesApprox. 1,700 employees / 40+ sales offices on 3 continents


20 major development centers on 3 continents

© 2008 ANSYS, Inc. All rights reserved. 2/08 after Q4 2007 ANSYS, Inc. Proprietary

2008 & 2009 Target % of revenue spending on R&D: 15%

World’s LargestSimulation CommunitySimulation Community

15 000 T t l C t> 15,000 Total Customers> 250,000 Commercial Seats> 220,000 University Seats> 200 Channel Partners> 150 I d t P t

20 3Network of sales channel partners in 40+ countriesApprox. 1,800 employees / 40+ sales offices on 3 continents

> 150 Industry Partners


20 major development centers on 3 continents

© 2008 ANSYS, Inc. All rights reserved. 2/08 after Q4 2007 ANSYS, Inc. Proprietary

2008 & 2009 Target % of revenue spending on R&D: 15%

In-Depth TechnologySpanning Multiple DomainsSpanning Multiple Domains

Technical Breadth

Tet/PrismHex/Hex Core

StructuredUnstructured

Multi-ZoneBody-Fitted Cartesian

Patch IndependentMore…

Meshing

Conduction

ThermalCompressible

FluidsTech Quasi static (Low Freq)

ElectromagneticsStructuralLarge Displacements

y

ConvectionRadiationPhase ChangeMass Transport

IncompressibleLaminar Flow

phnical D

ep

Full WaveJoule Heating

Finite StrainContactMultibody DynamicsShock & Vibration

TurbulenceMultiphase Flow

ppth

Eddy currentsCurrent flow

Implicit & Explicit

More…

pReacting FlowMore…

St d St t T i t H i & M d l

Circuit CouplingMore…More…


Steady-State, Transient, Harmonic & Modal

Linear & Nonlinear

High Performance Computing

High Performance ComputingComputingComputing


HPC Value Proposition

• High Performance Computing (HPC) -and ANSYS parallel performance -provides you with:– Faster turnaround timeFaster turnaround time

• For single simulations, or multiple simultaneous jobs

Ability to consider bigger more detailed– Ability to consider bigger, more-detailed models• Full assemblies, direct CAD-to-Mesh

– Ability to consider more complex physics• Time varying simulations, multiphysics

– Capacity to consider multiple design p y p goptions• Parametric modeling, DOE


Higher productivity, improved insight, and faster development of better products

Parallel Performance for Higher-Fidelity Simulation Higher Fidelity Simulation

• More complete models– From single blade passage to full 360– Computational workload increases by 90xp y– Turnaround time maintained at ~7 hours

• ANSYS CFX parallel 90 process

• More complex physics• More complex physics– From steady “RANS” model to unsteady “LES”– Computational workload increases by >100x– File I/O time also increases significantly– Turnaround time maintained at ~8 hours

• ANSYS FLUENT Parallel 256 processANSYS FLUENT Parallel 256 process


ANSYS 12.0 – HPC Leadership

• ANSYS 12 0 development included a major investment inANSYS 12.0 development included a major investment in HPC:

• Focus on processor and platform optimization– Optimized for the latest multicore chips– Support for clusters running Linux and Windows HPC Server 2008

• Outstanding parallel scaling improvementsOutsta d g pa a e sca g p o e e ts• Linear performance out to 1000’s of cores (Fluids)• Teraflop performance and scaling to 512 core (Mechanical)

S t f ll l I/O• Support for parallel I/O• Support for “billion” scale simulations

“ANSYS has consistently focused software design on the full range of computing platforms ― from desktop to supercomputer ― with their technology yielding great performance on the latest high-performance computing (HPC) solutions. This ensures that our mutual customers can tackle ever more complex and high-fidelity simulations while still achieving


p g y gthe turnaround time required for product development decision making.”― Richard Dracott, General Manager, HPC Division, Intel, U.S.A.

Performance - FluidsPerformance - Fluids


ANSYS CFD 12.0Quad-Core Processor ScalingQuad-Core Processor Scaling

• ANSYS CFD 12 0ANSYS FLUENT and ANSYS CFX

Dual Socket Quad Core Performance• ANSYS CFD 12.0 incorporates extensive tuning for multicore 8 00

8,00

7,16 7 18 7,58 7 55

Dual-Socket Quad-Core PerformanceIntel Xeon 5500 (“Nehalem”)

gperformance– Compiler optimizations,

optimized core mapping and 5,00

6,00

7,00

8,00 , 7,18

6,23

7,55

6,796,47

Spee

dup

optimized core mapping and binding, enhanced cache re-use

• Result: ~80-95% of ideal 2,00

3,00

4,00

,

Cor

e So

lver

S

speedup!

8 Way Parallel Value 12

48

0,00

1,00

8-Way Parallel ValueRun 3-5M cell CFD simulationsReduce execution time by 6-7x

Answers in 2 hours instead of 12

1


Answers in 2 hours, instead of 12

ANSYS CFD 12.0Multi-Processor ScalingMulti-Processor Scaling

CFX Solver scalabilityANSYS CFX 12.0 Scaling PerformanceAMD O t 2218

80 00

100.00

120.00AMD Opteron 2218

Truck_Poly_14M Benchmark

40.00

60.00

80.00

Spee

dup

Ideal

Transonic Airfoil 10M Nodes

0.00

20.00

0 16 32 48 64 80 96

b f

Infiniband

GigabitEth t

• ANSYS FLUENT 12.0 delivers dramatic scaling P ll l S li V l

number of coresNumber of Cores

improvement in this release• ANSYS CFX 12.0 scaling also outstanding

Parallel Scaling Value14M cell simulation, 512-way

Assume 500 iterationsAns ers in 1 ho r instead of 150


Answers in 1 hour, instead of 150

ANSYS CFD 12.0Massive Simulation SupportMassive Simulation Support

• ANSYS FLUENT 12.0 includes architecture updates to support today’s largest simulations

• Demonstrated ability to solve model sizes yof one billion cells and above.

• Sample timing: 3.5 hours for 500 iterations on 768 cores; 1.3 hours for

Solver Speedup

2048

ANSYS FLUENT 12.0 Scale-OutSGI Altix ICE 8200EX – 2048 Cores

;data write.

• Demonstrated linear scaling at 1024 cores(111M cell benchmark)

991

1581

1024

1536

er S

peed

up Truck_111M Benchmark

( )• 78% of ideal at 2048 cores

128265

521

0

512

Solv

e

Speedup

Ideal

Mega-Modeling ValueConsider highest fidelity models.Develop detailed understanding.


00 512 1024 1536 2048

CoresNumber of Cores

p gGet answers back 1000x faster.

ANSYS CFD 12.0Parallel and Serial - I/OParallel – and Serial - I/O

• Parallel I/O provides unique scaling ANSYS FLUENT 12.0breakthrough at large scale – Addresses the I/O bottleneck– New file pdat

Serial I/O Parallel I/O

ANSYS FLUENT 12.0

New file .pdat– Uses MPI-IO and parallel file system– All nodes write to a single file– Files are interoperable

• Traditional “serial” I/O also improved• ~2x speedup in read/write ANSYS CFX

– Parallelized data compression• ~3-9x speedup for data write ANSYS FLUENT

Parallel I/O ValueParallel I/O Value10M cell simulation, 192-way

1000 iterations, 100 data writesI/O reduced to 20% from 70% of time


I/O reduced to 20% from 70% of timeAnswers in 1 hour, instead of 2

Performance - StructuresPerformance - Structures


ANSYS Mechanical 12.0Quad-Core Processor ScalingQuad-Core Processor Scaling

• ANSYS Mechanical 12 0 ANSYS Mechanical 12.0• ANSYS Mechanical 12.0 delivers improved scaling on latest quad-core processors from AMD and Intel 8,83

Dual-Socket Quad-Core PerformanceIntel Xeon 5500 Processor ("Nehalem")

from AMD and Intel• Fully loaded cores

• Typically 5 - 6x speedup on 6,00

7,00

8,00

9,008,00

5,745,46

8,83

5,53 5,83

peed

upyp y p pdual-socket quad-core systems

• 3 hour simulation now completed in 30 minutes! 2 00

3,00

4,00

5,00

,

4,55

Cor

e So

lver

Sp

completed in 30 minutes!

12

480,00

1,00

2,00

8-Way Parallel Value1

Data from Cray CX-1 Personal Supercomputer

3-5M DOF mechanical simulationsReduce execution time by 5-6x

Answers in minutes, instead of hours


Data from Cray CX 1 Personal Supercomputer Running Microsoft Windows HPC Server 2008On your desktop, or on a cluster

ANSYS Mechanical 12.0Multi-processor ScalingMulti-processor Scaling

ANSYS Mechanical 12.010M DOF Distributed ANSYS PCG Solver

ANSYS Mechanical 12.03M DOF Distributed ANSYS Sparse Solver

50607080

eedu

p

10M DOF Distributed ANSYS PCG SolverIntel Xeon 5500 Processor Series ("Nehalem")

3000

4000

5000

AMD Opteron 2360 ("Barcelona")QLogic TrueScale Infiniband

Tim

e (s

ec)

1020304050

Cor

e So

lver

Spe

0

1000

2000

3000

Cor

e So

lver

T

01 2 4 8 32 64 128

C

Number of Cores

16 32 64 128 256Number of Cores

• ANSYS Mechanical 12.0 delivers enhanced S S ec a ca 0 de e s e a cedscaling due to:

• improved domain decomposition• Improved load balancing

Parallel Scaling Value3M DOF benchmark

~12 hours (one processor)1 h (16 )

p g• distributed matrix generation

• Teraflop performance demonstrated on 512 cores

~1 hour (16 cores)~15 minutes (128 cores)

Value of 12 hours saved: $1200


Healthcare Challenges Healthcare Challenges


The Healthcare Worldwide ChallengeChallenge

• Healthcare expenses in North Americap– In 2007, healthcare spending in the U.S. reached $2.3 trillion. It is

projected to reach $4.2 trillion by 2016.I 2005 th U S t 16% f it d ti d t (GDP)– In 2005, the U.S. spent 16% of its gross domestic product (GDP) on healthcare.

• Worldwide healthcare concernsWorldwide healthcare concerns– Healthcare spending accounted for:

• 10.9% of GDP in Switzerland• 10.7% in Germany• 9.7% in Canada• 9.5% in France (Organization for Economic Cooperation and Development)% ( g p p )

• Increasing or maintaining the quality and comfort of healthcare in the future will be a major challenge.


Engineering Challenges

• Cost-effectiveness– Prevention and early detection– Treatment efficacy– Personalized medicine– New treatment approval

Fl d h t t f i th

• Comfort and social trends– Portable medicine

Flow and heat transfer in the human eye

– Less invasive surgery – Minimizing impact on active life

• Innovation– Drug efficacy


g y– VPH (virtual physiological human) Contours of oxygen concentration plotted on

vertical planes inside incubator

Courtesy Silesian University of Technology.

Maturity of Engineering Simulation TechnologySimulation Technology

• Existing medical imaging techniques to provide accurate g g g q ppatient specific morphology (0.1 mm)

• The newest techniques (ultrasound elastography, MRI) will id di l t d l iti ( bl tprovide displacements and velocities (comparable to our

simulation results)• Robustness and speed for fluid flow and structural analysisRobustness and speed for fluid flow and structural analysis• Major progress in CAD import and meshing

– Automatic process flow from medical imaging to clinician p g ginterpretation of simulation results

• Access to living tissues material properties • Valuable clinical demonstrations using ANSYS software suite


ObjectivesObjectives

• To use existing simulation software on patient g pspecific organs, bones, tissues, arteries …in order to “predict” a specific biomedical behavior p p

• This simulation, based on the combination of bio-imaging and simulation technologies will help theimaging and simulation technologies, will help the clinicians for diagnosis or surgery planning


ApplicationsApplications• Osteoporosis: risk of fracture, cement injection

• Cerebral Aneurysms: risk of rupture stent• Cerebral Aneurysms: risk of rupture, stent

• Coronary, Carotid: Modulography, rupture of atherome plaque, morphology, diagnostics, surgery planningp gy, g , g y p g

• Per-Op surgery software for Abdominal Aortic Aneurysms (from 2D angiography to 3D image of the deformed artery)

• Ophthalmology: Cornea and crystalline lens stiffness, Patient specific presbyopia laser surgery

• Early Cancer Detection using inverse problem between medical imaging (MRI or US) and simulation

F 2D X i t f ll 3D d l• From 2D X-ray views to a full 3D model

• From 4D images to organ motion and morphing (cardiac simulation)


• Aesthetic Surgery (soft tissues deformation)

• Dental implant, …

HPC challenges for Medical SimulationSimulation

• Patient specific simulation for diagnosis and surgeryp g g y– Collaborative Development with Labs and Clinicians to develop the vertical

application – Integration of medical images and engineering simulation for patient specificIntegration of medical images and engineering simulation for patient specific

clinical applications– Multiscale simulation: protein, cell, tissue, organ, body, population

• Coupling between lower and upper scales• Coupling between lower and upper scales

– Accurate engineering simulation on patient data to be computed in a reasonable time

• Some applications require quasi real time (per op surgery)• Some applications require quasi real time (per-op surgery)

– Database of simulation results taking into account the human variability• Huge design of experiments

N t ti f di l i i i i l ti• Next generation of medical imaging using simulation technologies– 4D images (US, MRI) will be filtered using model motion and morphing


g ( , ) g p g– Incomplete Medical Simulation models could be completed using imaging

data (Boundary Conditions, materials)

HPC Focus on 2 medical applications

HPC Focus on 2 medical applicationsapplicationsapplications

Cerebral AneurysmCerebral Aneurysm


Cerebrovascular Diseases

• Cerebrovascular disease: StrokeCerebrovascular disease: Stroke– 15% hemorrhagic (aneurysms)– 85% ischemic (stenosis )85% ischemic (stenosis …)

• Aneurysm prevalence is between 2% and 4%.In US 12 millions people have aneurysm– In US 12 millions people have aneurysm

• The incidence (risk of rupture) is 10/100,000 per yearThis risk co ld be pre ented sing endo asc lar• This risk could be prevented using endovascular surgery (stent)

This surgery is expensive and risky (2 to 5%)– This surgery is expensive and risky (2 to 5%)


Status Today

Initiation

GrowthR t

10/100K/yearRupture


2-4%2,000 - 4,000 / 100KSource: Humphrey JD, Cardiovascular

solid mechanics 2001

Risk Assessment: State of the Art

• Today a cerebral aneurysm is considered as a virtualToday a cerebral aneurysm is considered as a virtual time bomb in your head

• Hemodynamics (blood flow) governs the pathology• Hemodynamics (blood flow) governs the pathology from the initiation to the growth and rupture

Aneurysm risk assessment only based on morphology• Aneurysm risk assessment only based on morphology

• The decision for surgery is based on simple criteria b i d h li i l i f iabout aneurysm size and other clinical information– E.g. Cerebral Aneurysms beyond 5mm are treated;

• Even if an existing cerebral aneurysm has a statistical risk around 1% it is a wise decision to treat it through


endovascular surgery

@neurIST - www.aneurist.org

• @neurIST is a major multidisciplinary European initiative within the SixthEuropean initiative within the Sixth Framework Programme*

• The project brings togetherThe project brings together– Neurosurgeons– Neuroradiologists– Epidemiologists– Engineers– Biologists– Computer scientists from 32 European institutions

Members of the consortiumMembers of the consortium

• The aim of the project– To develop a usable interface for personalised

risk assessment and treatment of patients with pcerebral aneurysm and subarachnoid haemorrhage

• 4 years project Jan 2006 until Dec 2009


4 years project, Jan 2006 until Dec 2009

Cerebral Aneurysm

• Medical imaging software can extract Vessel surface extraction

g gpatient-specific aneurysm geometry and blood flow

• ANSYS suite of software creates a patient specific virtual model of thepatient-specific virtual model of the aneurysm region

• Risk of rupture evaluatedWall shear stress in aneurysm

Patient medical image (3DRA)

– Wall shear stress in aneurysm during the cardiac cycle

• Proactive treatments can be prescribed

Reduction to region of interest

• HPC requirements for unstented and stented aneurysms


Computational meshHaemodynamic results

@neurIST Haemodynamic Toolchain

OsteoporosisVPHOP

OsteoporosisVPHOPVPHOPVPHOP


Osteoporosis is a killer

• OP fractures kill as many women as breast ycancer.

• 30 to 50% of all women and 15 to 30% of all• 30 to 50% of all women and 15 to 30% of all men will face an osteoporotic fracture in their lifetimelifetime.

• 4,000,000 fractures every year cost Europe €30,000,000,000.

• Forcast to double by 2050.Forcast to double by 2050.• 250,000 elders will dies of related

li ti ithi 12 th ll th ill© 2009 ANSYS, Inc. All rights reserved. 32 ANSYS, Inc. Proprietary

complications within 12 month; all others will remain impaired.

Not enough technology

• The technology in current clinicalThe technology in current clinical practice is clearly insufficient.

• The accuracy in predicting• The accuracy in predicting fractures is as low as 60%.E if th d t• Even if we see the drugs are not working we wait for the fracture,

d l th i ll fi itand only then surgically fix it.


Better chances to deliver

• Bone physiology is as complex as any other organ

• But the biomechanics of bone fracture is in itself a purely ymechanical event.

• This is one of the few domainsThis is one of the few domains where organ-level models already achieve predictivealready achieve predictive accuracies of over 90%


Predictive, Personalised

• P2 medicine: the VPHOP project will make possible Predictive and Personalised (P2) medicine for osteoporosis– Predictive: multiscale models representing the

skeletal mechanobiology from the whole body down to the molecular constituents, simulate the skeletalthe molecular constituents, simulate the skeletal loading in various conditions and predict if the bones will fracture in each of them

– Personalised: The multiscale model is personalised using specific patient information. The more available information, the more personalised the modelinformation, the more personalised the model becomes


The Hypermodel: primary systemic relationshipsprimary systemic relationships

Tissue-level

Boundary Conditions

ConstitutiveEquation

Organ-level model

model

BoR

emod

Body-level model

onedelling

FailureCriterion

Cell-levelmodel

Constituent-levelmodel

Criterion


Organ level

Resorption ReversalHYPERMODELHYPERMODEL

Resorption Reversal

Resting Formation


HPC Challenges for Clinical PracticePractice

• Multiscale Simulation• Multiscale Simulation– Computation of the risk of femoral or vertebral fracture– Simulation taking into account the microstructure and cell level g

remodelling– Several hundreds millions of cells on a huge cluster (5000 cores

in CINECA cluster)in CINECA cluster)

• Interventional treatment planning– Predicting the most clinical location within each bone– Predicting the most clinical location within each bone – Predicting the changes in risk due to interventional augmentation– HPC requirements for the compatibility of clinical practice


Conclusion

• HPC is strategic for Computer Aided Engineering:g p g g– To consider more detailed models (size x100)– To consider more complex physics p p y– To consider parametric modeling and design of

experiments• Extreme scalability demonstrated on thousands of

cores for CFD• Extreme scalability for structural is the current step• These HPC innovations will open new applications in

personalized medicine• A new era for health technologies


HPC: a Paradigm for the ANSYS Software

Documents

Transcript of HPC: a Paradigm for the ANSYS Software