Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the...
Transcript of Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the...
![Page 2: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/2.jpg)
2
Copy files for the hands-on
• You can download the material for most of the hands on from the web site https://tools.bsc.es/tools-hands-on.
• No binaries are provided, but you can follow the Extrae part with your own code.
> ls -l tools-material … clustering/ … dimemas/ … extrae/ … traces/
![Page 3: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/3.jpg)
3
Using Extrae in 3 steps
1. Adapt your job submission scripts
2. Configure what to trace (optional) • XML configuration file • Example configurations at $EXTRAE_HOME/share/example
3. Run it!
• For further reference check the Extrae User Guide:
• https://tools.bsc.es/sites/default/files/documentation/html/extrae/index.html
• Also distributed with Extrae at $EXTRAE_HOME/share/doc
![Page 4: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/4.jpg)
4
Step 1: Adapt the job script to load Extrae (LD_PRELOAD)
#!/bin/bash # @ initialdir = . # @ output = lulesh2_27p.out # @ error = lulesh2_27p.err # @ total_tasks = 27 # @ cpus_per_task = 1 # @ wall_clock_limit = 00:10:00 mpirun -np 27 ./lulesh2.0 -i 10 -s 65
job_27p.sh
> vi tools-material/extrae/job_27p.sh
Request resources
Run the program
![Page 5: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/5.jpg)
5
Step 1: Adapt the job script to load Extrae (LD_PRELOAD)
#!/bin/bash # @ initialdir = . # @ output = lulesh2_27p.out # @ error = lulesh2_27p.err # @ total_tasks = 27 # @ cpus_per_task = 1 # @ wall_clock_limit = 00:10:00 module load extrae TRACE_NAME=lulesh2_27p.prv mpirun -np 27 ./trace.sh ./lulesh2.0 –i 10 -s 65
job_27p.sh
> vi tools-material/extrae/job_27p.sh
Load Extrae
Activate Extrae in the execution
![Page 6: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/6.jpg)
#!/bin/bash # @ initialdir = . # @ output = lulesh2_27p.out # @ error = lulesh2_27p.err # @ total_tasks = 27 # @ cpus_per_task = 1 # @ wall_clock_limit = 00:10:00 module load extrae TRACE_NAME=lulesh2_27p.prv mpirun -np 27 ./trace.sh ./lulesh2.0 –i 10 -s 65
6
Step 1: Adapt the job script to load Extrae (LD_PRELOAD)
> vi tools-material/extrae/trace.sh
#!/bin/bash # Configure Extrae export EXTRAE_CONFIG_FILE=./extrae.xml # Load the tracing library (choose C/Fortran) export LD_PRELOAD=${EXTRAE_HOME}/lib/libmpitrace.so #export LD_PRELOAD=${EXTRAE_HOME}/lib/libmpitracef.so # Run the program $*
Select your type of application
Select “what to trace”
trace.sh
![Page 7: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/7.jpg)
7
Step 1: Which tracing library? • Choose depending on the application type
Library Serial MPI OpenMP pthread CUDA
libseqtrace
libmpitrace[f]1
libomptrace
libpttrace
libcudatrace
libompitrace[f] 1
libptmpitrace[f] 1
libcudampitrace[f] 1
1 include suffix “f” in Fortran codes
![Page 8: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/8.jpg)
8
Step 3: Run it!
• Submit your job
> cd tools-material/extrae > qsub job_27p.sh
![Page 9: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/9.jpg)
9
All done! Check your resulting trace • Once finished you will have the trace (3 files):
• To proceed with the example traces already generated here:
• Now let’s look into it !
> ls –l tools-material/extrae ... lulesh2_27p.pcf lulesh2_27p.prv lulesh2_27p.row
> ls tools-material/traces
![Page 10: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/10.jpg)
10
Install Paraver • Download from https://tools.bsc.es/downloads
Pick your version
wxparaver-4.7.2-win.zip
wxparaver-4.7.2-mac.zip
wxparaver-4.7.2-Linux_x86_64.tar.gz (64-bits)
wxparaver-4.7.2-Linux_i686.tar.gz (32-bits)
![Page 11: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/11.jpg)
11
Install Paraver (II) • Download tutorials:
• Documentation • Tutorial guidelines
Download links
![Page 12: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/12.jpg)
• Command-line
12
Uncompress, rename & move
> tar xf wxparaver-4.7.2-linux-x86_64.tar.gz
> mv wxparaver-4.6.2-linux-x86_64 paraver
> tar xf paraver-tutorials-20150526.tar.gz
> mv paraver-tutorials-20150526 paraver/tutorials
![Page 13: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/13.jpg)
13
Check that everything works • Start Paraver
• Check that tutorials are available
> paraver/bin/wxparaver
Click on Help Tutorials
![Page 14: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/14.jpg)
14
First steps of analysis
• Load the trace with Paraver
• Follow Tutorial #3
• Introduction to Paraver and Dimemas methodology
Click on Help Tutorials
Click on File Load Trace Browse to “lulesh2_27p.prv”
![Page 15: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/15.jpg)
• Click on “mpi_stats.cfg” • Check the Average for the column labeled “Outside MPI”
15
Measure the parallel efficiency
Load balance
Parallel efficiency
Comm efficiency
![Page 16: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/16.jpg)
• Click on “2dh_usefulduration.cfg” (2nd link) Shows time computing
16
Computation load & time distribution
Perfect work distribution (straight line)
Time imbalance (zig-zag)
Sockets with 5 processes
Sockets with 4 processes
![Page 17: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/17.jpg)
Work imbalance
(zig-zag)
Perfect work distribution (straight line)
• Click on “2dh_useful_instructions.cfg” (3rd link) Shows amount of work
17
Computation load & time distribution
Perfect work distribution (straight line)
![Page 18: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/18.jpg)
1. Click on “Open Filtered
Control Window”
Select this area (by drag-and-dropping)
• Go from the table to the timeline
Where does this happen?
![Page 19: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/19.jpg)
Zoom into 1 of the iterations (by drag-and-dropping)
Where does this happen?
Right click Fit Semantic Scale
Fit both
![Page 20: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/20.jpg)
20 TimeIncrement CommSend CommMonoQ
Where does this happen?
Slow Fast • & at the same time? Imbalance
• Reference to the source code: Hints Callers Caller function
![Page 21: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/21.jpg)
21
Save CFG’s (2 methods)
1. Right click on timeline
• From the contextual menu
![Page 22: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/22.jpg)
22
Save CFG’s (2 methods)
2. Main Paraver window
2. Select 3. Save
• From Paraver main window
![Page 23: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/23.jpg)
23
CFG’s distribution • Paraver comes with many more included CFG’s
![Page 24: Extrae & Paraver Hands-On - BSC-CNS · 7 Step 1: Which tracing library? •Choose depending on the application type Library Serial MPI OpenMP pthread CUDA libseqtrace libmpitrace[f]1](https://reader030.fdocuments.us/reader030/viewer/2022040401/5e7b807676edb92d7a0c513d/html5/thumbnails/24.jpg)
24
Hints: a good place to start! • Paraver suggests CFG’s based on the information present in the trace