Automated Instrumentation and Monitoring System (AIMS)
description
Transcript of Automated Instrumentation and Monitoring System (AIMS)
![Page 1: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/1.jpg)
Automated Instrumentation and Monitoring System (AIMS)
URL hhttp://science.nas.nasa.gov/Software/AIMS
Version 3.7
Languages ANSI C, Fortran 77
Platforms IBM SP with IBM MPI or MPICHSun, SGI, and HP workstations with MPICHSGI Power Challenge with SGI MPI
![Page 2: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/2.jpg)
AIMS Components
• Source code instrumentors– xinstrument– batch_inst
• Monitoring library
• Analysis tools– View Kernel (VK)– tally statistics generator
![Page 3: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/3.jpg)
xinstrument
• GUI allows user to select specific source code constructs to be instrumented
• Default is to instrument all communication routines
• Other possibilities
– All subroutines
– All I/O
– Enable by Type
– Point and click on particular constructs in Construct Tree diagrams
![Page 4: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/4.jpg)
xinstrument (cont.)
• Regards source code as nested collection of constructs
– conditionals
– loops
– subroutines
– communication calls
• Instrumented construct is replaced or surrounded by calls to AIMS monitor routines
• Execution of instrumented construct generates time-stamped event
![Page 5: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/5.jpg)
xinstrument display
![Page 6: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/6.jpg)
Files Menu Selections
Selection Action
Load Modules Brings up dialog box for file selection.Specify language/platform beforeloading.File containing main program must beloaded.
Instrument AllModules
Instruments all files loaded
Exit
![Page 7: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/7.jpg)
Options Menu Selections
Selection ActionAll Subroutines Instruments entry to
and exit from allsubroutines
All CommunicationConstructs
Instruments all sendsand receives
All I/O Instruments all I/Oactivities
Enable by Type Brings up dialog boxfor specifying whichconstructs toinstrument
![Page 8: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/8.jpg)
Enable by Type dialog box
![Page 9: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/9.jpg)
Advanced Options Menu Selections
Selection ActionInstrument Selected Modules Instruments constructs selected in
the Construct-Tree PaneEnable Selected Modules by Type An Enable by Type dialog box is
brought up and applied to filesselected in the Construct-Tree Pane
Remove Selected Modules Don’t instrument constructs selectedin the Construct-Tree Pane
Profile Load or Save profile
Monitor file Load or Save AIMS.monrc file
Set Preprocessor Options Brings up dialog box forpreprocessor options for the parser,such as –D and -I
![Page 10: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/10.jpg)
Files created by xinstrument
• The following files are created by xinstrument and placed in ./inst or designated output directory
– Instrumented source code
– AIMS.monrc - parameters from Settings pane plus other options
– Application database
• default name is APPL_DB
• Stores information about static structure of application
• Used by analysis tools to relate traced events to Instrumented constructs
• Incorporated at beginning of trace file
– Profile
• Table of flags, one for each construct in The application database
• Can be edited to change instrumentation
![Page 11: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/11.jpg)
batch_inst• Command-line instrumentor
batch_inst [-options] [filenames]
where options include:-output <pathname> directory for instrumented code-origin <pathname> root of source tree-enable <type> <type> = (all | procs | io |
default | reset)-pp_options <switches> preprocessor options-platform (f77-mpi | c-mpi) platform-verbose
![Page 12: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/12.jpg)
Compiling and Running Instrumented Code
• Copy necessary files to inst directory– Makefile– Header files– Source files not instrumented– Input files
![Page 13: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/13.jpg)
Example directory structure
![Page 14: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/14.jpg)
Compiling and Running (cont.)
• Link with AIMS MPI monitor libraryMON_LIB = $(AIMS_DIR)/lib
app: $(F77) -o app_program $(APP_OBJS) $(MON_LIB)/mpilib.a
• Set $AIMS_TMP_DIR environment variable to location to store local trace files
•Make sure AIMS scripts and executables are in your path•Per-process trace files are automatically collected and merged at end of execution
![Page 15: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/15.jpg)
Visualizing Trace Files with VK
• View Kernel (VK) animates a trace file• VCR-like controls for tracefile playback• Can set breakpoints by time or in specific source
code constructs• Source code click-back capability• Timeline display• Spokes view animates messages passed between
tasks
![Page 16: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/16.jpg)
Timeline Display
• Toggles between three different views
• OverVIEW– Horizontal bars represent tasks
– Colors represent different instrumented subroutines
– White space indicates task is blocked waiting to complete send or receive
– XX pattern indicates time spent writing AIMS trace files to disk
– Lines between bars represent messages transmitted between taSks
– Use keystroke and click combinations to get more information
![Page 17: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/17.jpg)
AIMS OverVIEW with source code clickback
![Page 18: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/18.jpg)
Click-back keystroke combinations
Informationdesired
OverVIEWobject
Mouse button Key
Routine name/causefor idle
subroutine bar middle
Construct tree ofroutine
subroutine bar middle control
Statistics aboutmessage
message line left
Construct tree ofsend task
message line left control
Source of send task message line left shift
Source of receivetask
message line right shift
![Page 19: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/19.jpg)
Timeline Display (cont.)
• I/OverVIEW
– Colored bars indicate read, write, seek, etc.
– Height of bar represents size of message
• MsgVIEW
– Idle time shown with two colors, one for send idling and one for receive idling
– Time spent computing shown as white space
– Message lines
– Height of bar represents message size
![Page 20: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/20.jpg)
AIMS MsgVIEW
![Page 21: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/21.jpg)
AIMS I/OverVIEW
![Page 22: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/22.jpg)
•blue - idle receive
•green - busy
•yellow - idle send
•hatched - flushing
Spokes View
![Page 23: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/23.jpg)
Controlling Scale and Speed of Playback
• No scrolling or zooming capabilities
• Set jump factor between 0 and 1 to speed up animation
• Set pause times or breakpoints to slow down animation
• Set scale to view larger or smaller time interval (default is 100 milliseconds)
![Page 24: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/24.jpg)
tally
• Processes trace file to produce resource utilization statistics on node-by-node and routine-by-routine basis
• tally output can be input to statistical drawing packages such as Excel or WingZ
![Page 25: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/25.jpg)
tally [options] [sorted tracefile]
where options include:
-help Prints usage message
-proc[=Name] Print information for procedure(s)
-node[=Node] Print information for node(s)
-ncpu Print information aboutnormalized cpu usage
-msg Print information about messagesizes per node
-all Print all information(proc+node+ncpu+msg)
![Page 26: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/26.jpg)
tally output - tally.summary
• Information for each procedure/function:– busy time: time spent performing useful work
– global blocking: time spent in global blocking operation
– send blocking: time spent in send operation
– receive blocking: time spent in receive operation
– life time: exclusive time
– percentage communication: percentage of total execution time spent in communication
– communication index: time spent in routine with respect to total time of program, as well as percentage of time spent in communication in this routine
![Page 27: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/27.jpg)
tally.summary (cont.)
• Information for each node (and routine):– busy time– global blocking– send blocking– recv blocking– percentage communication
![Page 28: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/28.jpg)
tally output - ncpu.summary
• NCPU for a given subroutine and a given k is the amount of CPU time used by that subroutine when k processors are busy, divided by k.
• Routine Concurrency - amount of time spent by each subroutine when k copies were executing simultaneously (indicates degree to which each routine was parallelized)
![Page 29: Automated Instrumentation and Monitoring System (AIMS)](https://reader034.fdocuments.us/reader034/viewer/2022051402/56815a9a550346895dc8181e/html5/thumbnails/29.jpg)
Graphs created by Excel 4.0 from tally output