Data Formats ( HPC Visualization with ParaView Workshop) Shuaib Arshad April 23, 2014.
VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing...
Transcript of VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing...
![Page 1: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/1.jpg)
VIS AND ANALYSIS R&D
FOR LARGE-SCALE DATA
Salishan 2013 – LA-UR-13-21885
Jon Woodring
et al.: Jim Ahrens, John Patchett, Chris
Sewell, Li-Ta Lo, Chris Mitchell, Pat Fasel,
Joanne Wendelberger, Kary Myers, Curt
Canada, Rick Knight, Hilary Abhold
![Page 2: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/2.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Science!
Theory
Simulation Experiment
After ~1940
![Page 3: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/3.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Modern Data Driven Science Research
LANL Data Science at Scale Team
Theory
Simulation Experiment
![Page 4: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/4.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
My Vision for HPC Simulation Science
“Error Bars on Everything!”
Experimental
Design
Simulation
Data
Collection &
Retrieval
Hypothesis
Quantified
Analysis
input space and
analysis plan
selected experiments
selected outputs
data sampling
quantified results to test
hypotheses and make
decisions
UQ+HPC+DISC+DA
V
automatic
error bars
on results
![Page 5: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/5.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Vision for HPC Scientific Workflow
R, ParaView,
VTK, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions (human)
results
scientific workflow over time
![Page 6: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/6.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Some Current Research to Get There
1) Mesh sharing for no-copy (shallow) run-time translation
Exascale
Jon Woodring, Tim Tautges (ANL), Tom Peterka (ANL), Venkat Vishwanath (ANL), Berk Geveci (Kitware)
2) Data selection and 3) Quantified analysis for managing simulation data
Data intensive
Jon Woodring, Kary Myers, Joanne Wendelberger, Jim Ahrens, Chris Brislawn, Sue Mniszewski
4) Co-design of burst buffers for analysis use cases
Exascale and data intensive
Jon Woodring, Chris Mitchell, Aaron Torres, Mat Maltrud, Rick Knight
![Page 7: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/7.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Exascale and Data Intensive R&D
Ian Gorton, Paul Greenfield, Alex Szalay, Roy Williams, "Data-Intensive
Computing in the 21st Century," Computer, pp. 30-32, April, 2008.
7
1
2 & 3 4
![Page 8: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/8.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
1) Mesh sharing for no-copy (shallow)
run-time translation
R,
ParaView/VT
K, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions
results
scientific workflow over time
![Page 9: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/9.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Exascale Memory Constraints
Share via On Demand Translation
Memory constrained per process – going down per core
Total memory going up, but not as fast as cores
DOE codes share data (memory coupled):
Multi-physics, in situ, IO libraries, etc.
Usually duplicate data from one code to the other, wastes memory – “deep copy” of data
Share pointers (references) to the data structures?
Fragile and prone to error
Do on-demand, fine-grained data translation
i.e., do translation in small-chunk “get datum” methods
Translation shim code (sometimes called a “thunk”)
Does it scale? How much memory do we save?
![Page 10: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/10.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Mesh-based Data (MOAB and VTK)
Fine-grained, on demand conversion
Many DOE codes use a mesh-based data model
(for finite element methods, vis and analysis, etc.),
with different interfaces and implementations
Unstructured Mesh Data
to Convert at Run-time:
Point list (coordinates)
Cell list (cell types, point
connectivities)
Attribute data (point and
cell data)
Field data (data set
information: time step,
block number, etc.)
![Page 11: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/11.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Most of the work is in id translation
during a “get point” or “get cell”
VTK and MOAB
address points and
cells in different ways
VTK dense, 0-indexed,
two namespaces
(points and cells)
MOAB dense and
sparse, one namespace
(all entities)
Done with a map and some math:
dest = M.lower_bound(src) + src
When we “get cell” from VTK, it
gets converted into a “get cell” for
MOAB at run-time
![Page 12: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/12.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Copy Memory Performance (Moonlight)
Deep Copy costs x9 as much memory
![Page 13: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/13.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Render Time
worst: DL980 x1.05, ML x1.15 slower
![Page 14: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/14.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Promising Results: Time overhead is not
too bad, memory savings is good
Probably suitable for one shot operations: in situ,
IO; maybe for suitable for compute heavy: clip,
contour, render testing seems to indicate so
Could be optimized – no optimization currently
Need to try this with row-oriented (array of structs)
meshes and IO libraries
Eternal struggle of code reuse vs. peak
performance priorities
Makes it easy to share data, trading compute speed
![Page 15: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/15.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
2) Intelligent Data selection for
managing simulation data
R,
ParaView/VT
K, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions
results
scientific workflow over time
![Page 16: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/16.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Data Glut for HPC and Scientist
We need data reduction
I/O is the bottleneck for both the large-scale (HPC)
simulation and analysis tool (I/O bound)
Most of post-processed or batch visualizations and analyses
time is spent in I/O just loading the data
Simulations are already throwing away most of it
Cognitive bandwidth bottleneck (human bound)
The data sizes have already exceeded the human cognitive
capacity to be able to look at all of the raw data
In situ is one way to tackle it, but at a price, losing the
“discovery” from human-in-the-loop
![Page 17: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/17.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Interactive Analysis vs. In Situ Analysis
How do we keep the “discovery”?
raw data
interactivity
raw data products* discovery
refine (if missed something, rerun the sim)
*some in situ output allows interaction: explorable images, etc.
discovery
“dynamic”
“static”*
sparse
in situ
![Page 18: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/18.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Adaptive In Situ: Feature selection
Intersecting Interactive & In Situ
discovery
events flow control (lightweight)
(heavyweight)
detectors
re-run the sim (less often than normal in situ)
triggers
dynamic computational
raw data
products data
![Page 19: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/19.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Typical Uniform Output
No knowledge is applied to selection
![Page 20: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/20.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Statistically Driven Data Selection
Optimize Bandwidth (Human and HPC)
![Page 21: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/21.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Selected Frames on Density in XRage
Uses histogram metrics to compare
![Page 22: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/22.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Slide 22
The choice of α governs the triggering mechanism of the f-test.
14 distinct regions selected with α = 5 × 10-6:
Selection of Time Steps from Simulation
Using f-test for linear time prediction
![Page 23: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/23.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Demonstration with LCROSS satellite
impact simulation in Xrage with f-test
![Page 24: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/24.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
3) Quantified Visualization and
Analysis for Managing Data Reduction
R,
ParaView/VT
K, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions
results
scientific workflow over time
![Page 25: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/25.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Simple Example of Too Many Data
We can reduce it, but what is lost?
The data on the right aren’t
even “large” – there are “only”
2563 (16 million points) on a
megapixel display
Even for visualization, we often
need to reduce the data, in
addition to reducing data for
storage bottlenecks
(compression, sampling, in situ,
etc.)
But, what do you lose?
![Page 26: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/26.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Always measure the differences
between reduced data and original
resolution data before storing a
reduction (the data aren’t “lost” yet)
Comparing provides for a bounding
metric on the stored data – a
provenance of the transformation
Record the differences at all stages of
analysis and reduction to quantify the
differences in the scientific workflow
Quantify (Provenance Error Log) the
Differences in the Data (Always!)
![Page 27: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/27.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Provenance Error Log Comparing a
0.19% Sample to Full Resolution Data
![Page 28: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/28.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Provenance of Comparing Halo
Analysis on Different Data Reductions
![Page 29: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/29.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
8bpp
+/-1.49012e-08
0.5 bpp
+/-8.58903e-05
Point-wise Differences, Max error, and
Isocontour Error Provenance
![Page 30: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/30.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Maximum Error vs. Bit Rate after
Compression Error Provenance
![Page 31: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/31.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
4) Co-design of burst buffers for
visualization and analysis use cases
R,
ParaView/VT
K, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions
results
scientific workflow over time
![Page 32: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/32.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Burst Buffers Are Cost Effective Fast IO
at Scale – Needed for Fault Tolerance
Mean time to failure shortens (more frequent) as supercomputers scale
up. We use checkpoint restart to deal with failure, but it requires fast I/O,
which can be expensive. Burst buffers are a cost effective solution for
fast bandwidth…
Can burst buffers be used for analysis?
![Page 33: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/33.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Hypothesis: Analysis Capability with
Burst Buffers Optimizes Time to Results
Different analyses are suitable for different stages in
the pipeline – IO characteristics vs. compute
Some are suited for post, some are suited for in situ,
while others are suited for “in transit” on burst buffers
![Page 34: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/34.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Supercomputing ‘11 and ‘12 demos
show viability of “In Transit” analysis
HiGrad Firetec (‘11) and POP (‘12) ran live
Faster turn around time to results due to pipeline parallelism of the burst buffers
![Page 35: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/35.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Hardware and Software Co-design
with the sim and analysis use cases
![Page 36: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/36.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Using POP (Parallel Ocean Program)
as the use case – Different things to try
Heavy I/O vs. heavy compute – MOC vs. Tracers
Different software stacks to run on the burst buffer
Manage the burst buffer: I/O on buffers, analysis scheduling with simulation, data product creation
Combine industry data intensive knowledge with HPC technologies and DOE analysis software
MPI with DISC? DISC with ParaView/VisIt/EnSight?
How much faster can we turn around results for the scientist?
Bring to bear all of the previous research I have discussed to create a prototype of the “vision”
![Page 37: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/37.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Vision for HPC Scientific Workflow
R, ParaView,
VTK, VisIt
Matlab, etc.
simulation
and in situ
data intensive
archival and
analysis
experimental
design
sims to run
simulation
archival &
analysis
provenance &
data
UQ &
visualization
hypothesis testing & decisions (human)
results
scientific workflow over time
![Page 38: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/38.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
Acknowledgements
ASC CSSE
ASCR Core and SciDAC
LANL LDRD
![Page 39: VIS AND ANALYSIS R&D FOR LARGE-SCALE DATA · 3) Quantified Visualization and Analysis for Managing Data Reduction R, ParaView/VT K, VisIt Matlab, etc. simulation and in situ data](https://reader034.fdocuments.us/reader034/viewer/2022050109/5f46b0be23e85f71e60f361d/html5/thumbnails/39.jpg)
Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA
A Vision for Data Driven Science
Experimental
Design
Simulation
Data
Collection &
Retrieval
Hypothesis
Quantified
Analysis
input space and
analysis plan
selected experiments
selected outputs
data sampling
quantified results to test
hypotheses and make
decisions
UQ+HPC+DISC+DA
V
automatic
error bars
on results