Tidal Datums Text and Figures Source: NOAA/NOS CO-OPS Tidal Datums related publications.
Holistic Measurement Driven System Assessment …•Real-time collection and processing for 28,000...
1
Holistic Measurement Driven System Assessment (HMDSA) Exascale-ready operations, improved performance, and balanced system design via comprehensive system monitoring and analysis National Center for Supercomputing Applications and University of Illinois: Bill Kramer (PI) Greg Bauer, Brett Bode, Mike Showerman, Jeremy Enos, Aaron Saxton, Saurabh Jha, Zbigniew Kalbarczyk, Ravi Iyer Sandia National Laboratories: James Brandt, Ann Gentile Institutional Collaborators: Boston University, New Mexico State University, University of Central Florida • Data-driven and Machine Learning (ML) based mechanisms for data collection, analysis, and automated runtime feedback and control • Scalable APIs that enable flexible runtime interactivity (e.g., streaming analytics) • Platform independent open-source solutions • Enable highly efficient HPC system usage and inform future system improvements to produce more science • Transform real-time data into actionable intelligence at runtime • Lightweight, low overhead mechanisms enable high fidelity (e.g. sub-second) synchronized, whole system numeric and event data capture with negligible impact on application runtime • Discovery and analysis of performance and resilience related phenomena via integrated system logs and numeric data • High resolution extraction and classification of phenomena with respect to locality, severity, and temporal extent • Low latency feedback of analysis results to system software, applications, and system managers Is my performance variation due to system conditions or code changes? How can I know if the system is having problems? What are the architectural requirements given the site’s workload? How can a system provide more effective and efficient services? User System Managers & Users Architects & Acquisitions Teams Architects, System Managers & Support Staff Capability: Topology-aware scheduling (TAS) based on congestion measurements Different Cray Aries with competing application, different Aries, same Aries (self-contention) Insight: Identifying issues with network routing and congestion mitigation procedures Percentage of congestion time recovered by performing dynamic re-mapping. Capability: Diagnose file-system issues using periodic latency measurements of file system operations (e.g. write) Transient network issue Server degraded Significant latency ⇤ Capability: Diagnose network issues using extracted congestion regions (e.g. Cray Gemini interconnect) Capability: Combined reasoning based on changes in application characteristics (e.g.,MPI message size) and system characteristics (e.g., network congestion) to diagnose cause(s) of performance variation and predict application runtime. Capability: Dynamic task re-mapping based on congestion measurements Insight: Assessing dynamic range of memory needs for facilities’ workflows Answering Key Operational Questions Capability: Diagnose code-related issues (e.g. impact of MPI message size on runtime) 0 20 40 60 256 1,024 4,096 16,384 65,536 262,144 Runtime Message Size (B) Architecture supports big data analysis, provenance, and curation. Currently in production on Blue Waters: • Real-time collection and processing for 28,000 nodes and all sub-systems * Process 20 B datums per day * Active storage of 11 trillion datums • 8,100 active unique log and data event types * 50,000 different event types seen to date * Up to 1,470M log and data events/day (30M avg) Capability: Inform design decisions for future systems Time Key Strengths Exascale-ready Features 10% 55% 35% Worst % input queue stall across directions on Cray Gemini interconnect X±,Y ±,Z± Duration of congestion 10 1 10 2 10 3 10 4 Minutes Impact of TAS on system performance 2014-08-01 2015-02-01 2015-05-01 2015-11-01 Utilization Injection (B/s) Figures of merit on Cray Aries interconnect system Total peak memory (TB)1400 0 1000 600 200 400 800 1200 Node count 0 5000 10000 15000 20000 Log10 wall hours T = t Monitoring Clustering Inference State Estimation (Diagnosis) State Prediction Zombie process Memory leak; app slow tile link down fs traffic issue unroll factor, k=3 healthy Node memory leak .. .. 1 healthy N 2 State 1/Time between errors Server load Numerical features (from performance counters) Log clustering (e.g., modeling error propagation) 15,872 XE node NAMD job NIC input (GB/s) per project 0 6000 Time 0 500 1000 1500 2000 2500 0 10 20 30 40 Time (Seconds) #Run Prediction Real T = t + δ t Textual features (from raw syslogs) Feature Extraction and Training Visualization Numerical clustering (e.g., Identify performance anomalies) .. Pattern .. 2018/10/23 10:12 .. 2018/10/23 10:08 host svc pid * [num] LNET Received down event for * 1098 .. 23 956 .*LustreError:.*@@@ bulk PUT failed.* Count Last seen Ptn ID 357 Significance and Impact 0.0 0x0 0.053 CoMD 0.0 0.0 nid000[21-22] 10396 2018-09-21 00:21:25 0.61 2018-09-21 00:19:16 2.75 0.0 nid000[53,59] 2018-09-21 00:19:19 2018-09-21 00:21:28 0.042 0.06 CoMD 0.0 0x0 10397 0.59 0.1 0.0 0.0 1.55 2018-09-21 00:22:02 0.29 2018-09-21 00:19:26 nid000[60-61] 0.21 0x0 10398 0.041 lulesh Comm Perf 0.0 App Perf 0.59 PAPI Perf 0.0 Anomalies 0x1 Mem Score 0.042 NW Congest 1.57 0.0 HPages End Time 2018-09-21 00:30:40 Start Time nid000[53,59] 2018-09-21 00:28:40 CoMD App ID Node ID Job ID 10399 0% 10% 20% 30% 40% 50% Hops Bandwidth Stalls Individual data points Aggregated data points Online congestion analysis Job profile System profile Multivariate analyses of job and system Visualization Data Availability (seconds) Historical data analysis ML-based feedback and control Expert-driven Improvements Data characterization 10 0 0.001 1.0 1000 10 3 10 6 10 9 10 12 Data Size (B) Scalable System Design and Architecture Actionable Expert and ML-Driven Analysis https://hmdsa.github.com/hmdsa • ML/AI • Log and Event Analysis Actionable Intelligence Data Collection Data Collection Feedback and Control Analysis Admin Web MOTD, Log Browsing, Data Mini, Web Portal API Dashboards Analysis Research System Visualization Email, Text and Event Notifications • ML/AI • Signal Analysis • Log and Event Analysis • Event Prediction and Correlation HIGH PERFORMANCE/PROJECT DRIVEN ANALYSIS AND PREDICTION TOOLS PROCESSOR INTERCONNECT STORAGE Event Logs Blade and Cabinet Controller Sensor Data LDMS Job Application Resource and Performance Data Notification System Machine in the Loop Analysis Feedback Human in the Loop Analysis Feedback Power System Facility Systems Environmental Systems BUILDING AND FACILITIES Parser Syslog-NG Database and Active Data Repository Web Server Long Term Data Repository Z+ Y+ X+ 006 005 004 003 007 106 105 104 103 107 016 015 014 013 017 116 115 114 113 117 Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Transcript of Holistic Measurement Driven System Assessment …•Real-time collection and processing for 28,000...
Holistic Measurement Driven System Assessment (HMDSA)Exascale-ready operations, improved performance, and balanced system design via comprehensive system monitoring and analysisNational Center for Supercomputing Applications and University of Illinois: Bill Kramer (PI) Greg Bauer, Brett Bode, Mike Showerman, Jeremy Enos, Aaron Saxton, Saurabh Jha, Zbigniew Kalbarczyk, Ravi IyerSandia National Laboratories: James Brandt, Ann GentileInstitutional Collaborators: Boston University, New Mexico State University, University of Central Florida
• Data-driven and Machine Learning (ML) based mechanisms for data collection, analysis, and automated runtime feedback and control
• Scalable APIs that enable flexible runtime interactivity (e.g., streaming analytics)• Platform independent open-source solutions
• Enable highly efficient HPC system usage and inform future system improvements to produce more science
• Transform real-time data into actionable intelligence at runtime
• Lightweight, low overhead mechanisms enable high fidelity (e.g. sub-second) synchronized, whole system numeric and event data capture with negligible impact on application runtime
• Discovery and analysis of performance and resilience related phenomena via integrated system logs and numeric data
• High resolution extraction and classification of phenomena with respect to locality, severity, and temporal extent
• Low latency feedback of analysis results to system software, applications, and system managers
Is my performance variation due to system conditions or code changes?
How can I know if the system is having problems?
What are the architectural requirements given the site’s workload?
How can a system provide more effective and efficient services?
User System Managers & Users Architects & Acquisitions Teams Architects, System Managers & Support Staff
Capability: Topology-aware scheduling (TAS) based on congestion measurements
Different Cray Aries with competing application, different Aries, same Aries (self-contention)
Insight: Identifying issues with network routing and congestion mitigation procedures
Percentage of congestion time recovered by performing dynamic re-mapping.
Capability: Diagnose file-system issues using periodic latency measurements of file system operations (e.g. write)
Transient network issue
Server degraded
Significant latency⇤<latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="6ZJdJx/dJXOivdTVjtldzAgAoiI=">AAAB33icbZDNSgMxFIXv1L9aq1a3boJFEBdlxo0uBTcuK9paaIeSSe+0oZnMkNwRytBHcONCEd/KnW9j+rPQ1gOBj3MScu+JMiUt+f63V9rY3NreKe9W9qr7B4e1o2rbprkR2BKpSk0n4haV1NgiSQo7mUGeRAqfovHtLH96RmNlqh9pkmGY8KGWsRScnPVQXEz7tbrf8Odi6xAsoQ5LNfu1r94gFXmCmoTi1nYDP6Ow4IakUDit9HKLGRdjPsSuQ80TtGExH3XKzpwzYHFq3NHE5u7vFwVPrJ0kkbuZcBrZ1Wxm/pd1c4qvw0LqLCfUYvFRnCtGKZvtzQbSoCA1ccCFkW5WJkbccEGunYorIVhdeR3al43A8b0PZTiBUziHAK7gBu6gCS0QMIQXeIN3T3mv3seirpK37O0Y/sj7/AEWd4xl</latexit><latexit sha1_base64="kyuN1vIYBJRaKnyGTrLbNR9pxJU=">AAAB33icdZDNSgMxFIXv+Ftr1erWTbAI4qJkxtLWneDGZUX7A+1QMmmmDc1khiQjlKGP4MaFIr6VO9/GTFtBRQ8EPs5JyL0nSATXBuMPZ219Y3Nru7BT3C3t7R+UD0sdHaeKsjaNRax6AdFMcMnahhvBeoliJAoE6wbT6zzvPjCleSzvzSxhfkTGkoecEmOtu+x8PixXcPWyWfdqdYSrGDdcz83Ba9Quasi1Tq4KrNQalt8Ho5imEZOGCqJ138WJ8TOiDKeCzYuDVLOE0CkZs75FSSKm/Wwx6hydWmeEwljZIw1auN9fZCTSehYF9mZEzET/znLzr6yfmrDpZ1wmqWGSLj8KU4FMjPK90YgrRo2YWSBUcTsrohOiCDW2naIt4WtT9D90vKpr+RZDAY7hBM7AhQZcwQ20oA0UxvAIz/DiCOfJeV3WteasejuCH3LePgFzsIyn</latexit><latexit sha1_base64="FMQgioaDhwa8qjy71m2JlueoG60=">AAAB6nicdVDLSgNBEOyNrxhfUY9eBoMgHsLsGpJ4C3rxGNE8IFnC7GQ2GTL7YGZWCEs+wYsHRbz6Rd78G2eTCCpa0FBUddPd5cWCK43xh5VbWV1b38hvFra2d3b3ivsHbRUlkrIWjUQkux5RTPCQtTTXgnVjyUjgCdbxJleZ37lnUvEovNPTmLkBGYXc55RoI92mZ7NBsYTLF/WqU6kiXMa4Zjt2Rpxa5byCbKNkKMESzUHxvT+MaBKwUFNBlOrZONZuSqTmVLBZoZ8oFhM6ISPWMzQkAVNuOj91hk6MMkR+JE2FGs3V7xMpCZSaBp7pDIgeq99eJv7l9RLt192Uh3GiWUgXi/xEIB2h7G805JJRLaaGECq5uRXRMZGEapNOwYTw9Sn6n7Sdsm34DS41Lpdx5OEIjuEUbKhBA66hCS2gMIIHeIJnS1iP1ov1umjNWcuZQ/gB6+0TkbqN+A==</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit><latexit sha1_base64="+Rzta32q7ZzBpV3C+tL0S74zD9I=">AAAB6nicdVBNS8NAEJ3Ur1q/qh69LBZBPJSkhrbeil48VrQf0Iay2W7apZtN2N0IJfQnePGgiFd/kTf/jZu2goo+GHi8N8PMPD/mTGnb/rByK6tr6xv5zcLW9s7uXnH/oK2iRBLaIhGPZNfHinImaEszzWk3lhSHPqcdf3KV+Z17KhWLxJ2extQL8UiwgBGsjXSbns0GxZJdvqhXK24V2WXbrjkVJyOVmnvuIscoGUqwRHNQfO8PI5KEVGjCsVI9x461l2KpGeF0VugnisaYTPCI9gwVOKTKS+enztCJUYYoiKQpodFc/T6R4lCpaeibzhDrsfrtZeJfXi/RQd1LmYgTTQVZLAoSjnSEsr/RkElKNJ8agolk5lZExlhiok06BRPC16fof9KulB3Db9xS43IZRx6O4BhOwYEaNOAamtACAiN4gCd4trj1aL1Yr4vWnLWcOYQfsN4+AZL6jfw=</latexit>
Capability: Diagnose network issues using extracted congestion regions (e.g. Cray Gemini interconnect)
Capability: Combined reasoning based on changes in application characteristics (e.g.,MPI message size) and system characteristics (e.g., network congestion) to diagnose cause(s) of performance variation and predict application runtime.
Capability: Dynamic task re-mapping based on congestion measurementsInsight: Assessing dynamic range of
memory needs for facilities’ workflows
Answering Key Operational Questions
Capability: Diagnose code-related issues (e.g. impact of MPI message size on runtime)
0204060
256 1,024 4,096 16,384 65,536 262,144
Run
time
Message Size (B)
Architecture supports big data analysis,
provenance, and curation. Currently in production
on Blue Waters:
• Real-time collection and processing for 28,000 nodes and all sub-systems
* Process 20 B datums per day * Active storage of 11 trillion datums • 8,100 active unique log and data event types
* 50,000 different event types seen to date * Up to 1,470M log and data events/day (30M avg)
Capability: Inform design decisions for future systems
Tim
e
Key Strengths
Exascale-ready Features
10% 55%35%Worst % input queue stall across directions on Cray Gemini interconnect
X±, Y±, Z±<latexit sha1_base64="crSquu8g0DmovnGAUHFxadB7iWU=">AAAB+XicbVDLSgMxFL3js9bXqEs3wSK4kDIjgl0W3LisYB/aDiWTZtrQJDMkmUIZ+iduXCji1j9x59+YTmehrQdyOZxzL/fmhAln2njet7O2vrG5tV3aKe/u7R8cukfHLR2nitAmiXmsOiHWlDNJm4YZTjuJoliEnLbD8e3cb0+o0iyWD2aa0EDgoWQRI9hYqe+6nV4iLtFjXp9s7bsVr+rlQKvEL0gFCjT67ldvEJNUUGkIx1p3fS8xQYaVYYTTWbmXappgMsZD2rVUYkF1kOWXz9C5VQYoipV90qBc/T2RYaH1VIS2U2Az0sveXPzP66YmqgUZk0lqqCSLRVHKkYnRPAY0YIoSw6eWYKKYvRWREVaYGBtW2YbgL395lbSuqr7l99eVeq2IowSncAYX4MMN1OEOGtAEAhN4hld4czLnxXl3Phata04xcwJ/4Hz+AAGJko8=</latexit><latexit sha1_base64="crSquu8g0DmovnGAUHFxadB7iWU=">AAAB+XicbVDLSgMxFL3js9bXqEs3wSK4kDIjgl0W3LisYB/aDiWTZtrQJDMkmUIZ+iduXCji1j9x59+YTmehrQdyOZxzL/fmhAln2njet7O2vrG5tV3aKe/u7R8cukfHLR2nitAmiXmsOiHWlDNJm4YZTjuJoliEnLbD8e3cb0+o0iyWD2aa0EDgoWQRI9hYqe+6nV4iLtFjXp9s7bsVr+rlQKvEL0gFCjT67ldvEJNUUGkIx1p3fS8xQYaVYYTTWbmXappgMsZD2rVUYkF1kOWXz9C5VQYoipV90qBc/T2RYaH1VIS2U2Az0sveXPzP66YmqgUZk0lqqCSLRVHKkYnRPAY0YIoSw6eWYKKYvRWREVaYGBtW2YbgL395lbSuqr7l99eVeq2IowSncAYX4MMN1OEOGtAEAhN4hld4czLnxXl3Phata04xcwJ/4Hz+AAGJko8=</latexit><latexit sha1_base64="crSquu8g0DmovnGAUHFxadB7iWU=">AAAB+XicbVDLSgMxFL3js9bXqEs3wSK4kDIjgl0W3LisYB/aDiWTZtrQJDMkmUIZ+iduXCji1j9x59+YTmehrQdyOZxzL/fmhAln2njet7O2vrG5tV3aKe/u7R8cukfHLR2nitAmiXmsOiHWlDNJm4YZTjuJoliEnLbD8e3cb0+o0iyWD2aa0EDgoWQRI9hYqe+6nV4iLtFjXp9s7bsVr+rlQKvEL0gFCjT67ldvEJNUUGkIx1p3fS8xQYaVYYTTWbmXappgMsZD2rVUYkF1kOWXz9C5VQYoipV90qBc/T2RYaH1VIS2U2Az0sveXPzP66YmqgUZk0lqqCSLRVHKkYnRPAY0YIoSw6eWYKKYvRWREVaYGBtW2YbgL395lbSuqr7l99eVeq2IowSncAYX4MMN1OEOGtAEAhN4hld4czLnxXl3Phata04xcwJ/4Hz+AAGJko8=</latexit><latexit sha1_base64="crSquu8g0DmovnGAUHFxadB7iWU=">AAAB+XicbVDLSgMxFL3js9bXqEs3wSK4kDIjgl0W3LisYB/aDiWTZtrQJDMkmUIZ+iduXCji1j9x59+YTmehrQdyOZxzL/fmhAln2njet7O2vrG5tV3aKe/u7R8cukfHLR2nitAmiXmsOiHWlDNJm4YZTjuJoliEnLbD8e3cb0+o0iyWD2aa0EDgoWQRI9hYqe+6nV4iLtFjXp9s7bsVr+rlQKvEL0gFCjT67ldvEJNUUGkIx1p3fS8xQYaVYYTTWbmXappgMsZD2rVUYkF1kOWXz9C5VQYoipV90qBc/T2RYaH1VIS2U2Az0sveXPzP66YmqgUZk0lqqCSLRVHKkYnRPAY0YIoSw6eWYKKYvRWREVaYGBtW2YbgL395lbSuqr7l99eVeq2IowSncAYX4MMN1OEOGtAEAhN4hld4czLnxXl3Phata04xcwJ/4Hz+AAGJko8=</latexit>
Duration of congestion
101<latexit sha1_base64="YrVdlPNfALdGzzL4857zU8mu1zM=">AAAB63icbZDLSgMxFIZP6q3WW9Wlm2ARXJUZEeyy4MZlBXuBdiyZNNOGJpkhyQhl6Cu4caGIW1/InW9jpp2Ftv4Q+PjPOeScP0wEN9bzvlFpY3Nre6e8W9nbPzg8qh6fdEycasraNBax7oXEMMEVa1tuBeslmhEZCtYNp7d5vfvEtOGxerCzhAWSjBWPOCU2t3zv0R9Wa17dWwivg19ADQq1htWvwSimqWTKUkGM6fteYoOMaMupYPPKIDUsIXRKxqzvUBHJTJAtdp3jC+eMcBRr95TFC/f3REakMTMZuk5J7MSs1nLzv1o/tVEjyLhKUssUXX4UpQLbGOeH4xHXjFoxc0Co5m5XTCdEE2pdPBUXgr968jp0ruq+4/vrWrNRxFGGMziHS/DhBppwBy1oA4UJPMMrvCGJXtA7+li2llAxcwp/hD5/AAk6jYg=</latexit><latexit sha1_base64="YrVdlPNfALdGzzL4857zU8mu1zM=">AAAB63icbZDLSgMxFIZP6q3WW9Wlm2ARXJUZEeyy4MZlBXuBdiyZNNOGJpkhyQhl6Cu4caGIW1/InW9jpp2Ftv4Q+PjPOeScP0wEN9bzvlFpY3Nre6e8W9nbPzg8qh6fdEycasraNBax7oXEMMEVa1tuBeslmhEZCtYNp7d5vfvEtOGxerCzhAWSjBWPOCU2t3zv0R9Wa17dWwivg19ADQq1htWvwSimqWTKUkGM6fteYoOMaMupYPPKIDUsIXRKxqzvUBHJTJAtdp3jC+eMcBRr95TFC/f3REakMTMZuk5J7MSs1nLzv1o/tVEjyLhKUssUXX4UpQLbGOeH4xHXjFoxc0Co5m5XTCdEE2pdPBUXgr968jp0ruq+4/vrWrNRxFGGMziHS/DhBppwBy1oA4UJPMMrvCGJXtA7+li2llAxcwp/hD5/AAk6jYg=</latexit><latexit sha1_base64="YrVdlPNfALdGzzL4857zU8mu1zM=">AAAB63icbZDLSgMxFIZP6q3WW9Wlm2ARXJUZEeyy4MZlBXuBdiyZNNOGJpkhyQhl6Cu4caGIW1/InW9jpp2Ftv4Q+PjPOeScP0wEN9bzvlFpY3Nre6e8W9nbPzg8qh6fdEycasraNBax7oXEMMEVa1tuBeslmhEZCtYNp7d5vfvEtOGxerCzhAWSjBWPOCU2t3zv0R9Wa17dWwivg19ADQq1htWvwSimqWTKUkGM6fteYoOMaMupYPPKIDUsIXRKxqzvUBHJTJAtdp3jC+eMcBRr95TFC/f3REakMTMZuk5J7MSs1nLzv1o/tVEjyLhKUssUXX4UpQLbGOeH4xHXjFoxc0Co5m5XTCdEE2pdPBUXgr968jp0ruq+4/vrWrNRxFGGMziHS/DhBppwBy1oA4UJPMMrvCGJXtA7+li2llAxcwp/hD5/AAk6jYg=</latexit><latexit sha1_base64="YrVdlPNfALdGzzL4857zU8mu1zM=">AAAB63icbZDLSgMxFIZP6q3WW9Wlm2ARXJUZEeyy4MZlBXuBdiyZNNOGJpkhyQhl6Cu4caGIW1/InW9jpp2Ftv4Q+PjPOeScP0wEN9bzvlFpY3Nre6e8W9nbPzg8qh6fdEycasraNBax7oXEMMEVa1tuBeslmhEZCtYNp7d5vfvEtOGxerCzhAWSjBWPOCU2t3zv0R9Wa17dWwivg19ADQq1htWvwSimqWTKUkGM6fteYoOMaMupYPPKIDUsIXRKxqzvUBHJTJAtdp3jC+eMcBRr95TFC/f3REakMTMZuk5J7MSs1nLzv1o/tVEjyLhKUssUXX4UpQLbGOeH4xHXjFoxc0Co5m5XTCdEE2pdPBUXgr968jp0ruq+4/vrWrNRxFGGMziHS/DhBppwBy1oA4UJPMMrvCGJXtA7+li2llAxcwp/hD5/AAk6jYg=</latexit>
102<latexit sha1_base64="PZxAGS0Kb78BfW5FxNWqqFbrefk=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2WmCHZZcOOygr1AO5ZMmmlDcxmSjFCGvoIbF4q49YXc+TZm2llo6w+Bj/+cQ875o4QzY33/29vY3Nre2S3tlfcPDo+OKyenHaNSTWibKK50L8KGciZp2zLLaS/RFIuI0240vc3r3SeqDVPywc4SGgo8lixmBNvcCvzH+rBS9Wv+QmgdggKqUKg1rHwNRoqkgkpLODamH/iJDTOsLSOczsuD1NAEkyke075DiQU1YbbYdY4unTNCsdLuSYsW7u+JDAtjZiJynQLbiVmt5eZ/tX5q40aYMZmklkqy/ChOObIK5YejEdOUWD5zgIlmbldEJlhjYl08ZRdCsHryOnTqtcDx/XW12SjiKME5XMAVBHADTbiDFrSBwASe4RXePOG9eO/ex7J1wytmzuCPvM8fCr6NiQ==</latexit><latexit sha1_base64="PZxAGS0Kb78BfW5FxNWqqFbrefk=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2WmCHZZcOOygr1AO5ZMmmlDcxmSjFCGvoIbF4q49YXc+TZm2llo6w+Bj/+cQ875o4QzY33/29vY3Nre2S3tlfcPDo+OKyenHaNSTWibKK50L8KGciZp2zLLaS/RFIuI0240vc3r3SeqDVPywc4SGgo8lixmBNvcCvzH+rBS9Wv+QmgdggKqUKg1rHwNRoqkgkpLODamH/iJDTOsLSOczsuD1NAEkyke075DiQU1YbbYdY4unTNCsdLuSYsW7u+JDAtjZiJynQLbiVmt5eZ/tX5q40aYMZmklkqy/ChOObIK5YejEdOUWD5zgIlmbldEJlhjYl08ZRdCsHryOnTqtcDx/XW12SjiKME5XMAVBHADTbiDFrSBwASe4RXePOG9eO/ex7J1wytmzuCPvM8fCr6NiQ==</latexit><latexit sha1_base64="PZxAGS0Kb78BfW5FxNWqqFbrefk=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2WmCHZZcOOygr1AO5ZMmmlDcxmSjFCGvoIbF4q49YXc+TZm2llo6w+Bj/+cQ875o4QzY33/29vY3Nre2S3tlfcPDo+OKyenHaNSTWibKK50L8KGciZp2zLLaS/RFIuI0240vc3r3SeqDVPywc4SGgo8lixmBNvcCvzH+rBS9Wv+QmgdggKqUKg1rHwNRoqkgkpLODamH/iJDTOsLSOczsuD1NAEkyke075DiQU1YbbYdY4unTNCsdLuSYsW7u+JDAtjZiJynQLbiVmt5eZ/tX5q40aYMZmklkqy/ChOObIK5YejEdOUWD5zgIlmbldEJlhjYl08ZRdCsHryOnTqtcDx/XW12SjiKME5XMAVBHADTbiDFrSBwASe4RXePOG9eO/ex7J1wytmzuCPvM8fCr6NiQ==</latexit><latexit sha1_base64="PZxAGS0Kb78BfW5FxNWqqFbrefk=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2WmCHZZcOOygr1AO5ZMmmlDcxmSjFCGvoIbF4q49YXc+TZm2llo6w+Bj/+cQ875o4QzY33/29vY3Nre2S3tlfcPDo+OKyenHaNSTWibKK50L8KGciZp2zLLaS/RFIuI0240vc3r3SeqDVPywc4SGgo8lixmBNvcCvzH+rBS9Wv+QmgdggKqUKg1rHwNRoqkgkpLODamH/iJDTOsLSOczsuD1NAEkyke075DiQU1YbbYdY4unTNCsdLuSYsW7u+JDAtjZiJynQLbiVmt5eZ/tX5q40aYMZmklkqy/ChOObIK5YejEdOUWD5zgIlmbldEJlhjYl08ZRdCsHryOnTqtcDx/XW12SjiKME5XMAVBHADTbiDFrSBwASe4RXePOG9eO/ex7J1wytmzuCPvM8fCr6NiQ==</latexit>
103<latexit sha1_base64="GMo+Q+KfYZbr7EJTRGDvhwksukA=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2VGBbssuHFZwV6gHUsmzbShuQxJRihDX8GNC0Xc+kLufBsz7Sy09YfAx3/OIef8UcKZsb7/7a2tb2xubZd2yrt7+weHlaPjtlGpJrRFFFe6G2FDOZO0ZZnltJtoikXEaSea3Ob1zhPVhin5YKcJDQUeSRYzgm1uBf7j1aBS9Wv+XGgVggKqUKg5qHz1h4qkgkpLODamF/iJDTOsLSOczsr91NAEkwke0Z5DiQU1YTbfdYbOnTNEsdLuSYvm7u+JDAtjpiJynQLbsVmu5eZ/tV5q43qYMZmklkqy+ChOObIK5YejIdOUWD51gIlmbldExlhjYl08ZRdCsHzyKrQva4Hj++tqo17EUYJTOIMLCOAGGnAHTWgBgTE8wyu8ecJ78d69j0XrmlfMnMAfeZ8/DEKNig==</latexit><latexit sha1_base64="GMo+Q+KfYZbr7EJTRGDvhwksukA=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2VGBbssuHFZwV6gHUsmzbShuQxJRihDX8GNC0Xc+kLufBsz7Sy09YfAx3/OIef8UcKZsb7/7a2tb2xubZd2yrt7+weHlaPjtlGpJrRFFFe6G2FDOZO0ZZnltJtoikXEaSea3Ob1zhPVhin5YKcJDQUeSRYzgm1uBf7j1aBS9Wv+XGgVggKqUKg5qHz1h4qkgkpLODamF/iJDTOsLSOczsr91NAEkwke0Z5DiQU1YTbfdYbOnTNEsdLuSYvm7u+JDAtjpiJynQLbsVmu5eZ/tV5q43qYMZmklkqy+ChOObIK5YejIdOUWD51gIlmbldExlhjYl08ZRdCsHzyKrQva4Hj++tqo17EUYJTOIMLCOAGGnAHTWgBgTE8wyu8ecJ78d69j0XrmlfMnMAfeZ8/DEKNig==</latexit><latexit sha1_base64="GMo+Q+KfYZbr7EJTRGDvhwksukA=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2VGBbssuHFZwV6gHUsmzbShuQxJRihDX8GNC0Xc+kLufBsz7Sy09YfAx3/OIef8UcKZsb7/7a2tb2xubZd2yrt7+weHlaPjtlGpJrRFFFe6G2FDOZO0ZZnltJtoikXEaSea3Ob1zhPVhin5YKcJDQUeSRYzgm1uBf7j1aBS9Wv+XGgVggKqUKg5qHz1h4qkgkpLODamF/iJDTOsLSOczsr91NAEkwke0Z5DiQU1YTbfdYbOnTNEsdLuSYvm7u+JDAtjpiJynQLbsVmu5eZ/tV5q43qYMZmklkqy+ChOObIK5YejIdOUWD51gIlmbldExlhjYl08ZRdCsHzyKrQva4Hj++tqo17EUYJTOIMLCOAGGnAHTWgBgTE8wyu8ecJ78d69j0XrmlfMnMAfeZ8/DEKNig==</latexit><latexit sha1_base64="GMo+Q+KfYZbr7EJTRGDvhwksukA=">AAAB63icbZDLSgMxFIbPeK31VnXpJlgEV2VGBbssuHFZwV6gHUsmzbShuQxJRihDX8GNC0Xc+kLufBsz7Sy09YfAx3/OIef8UcKZsb7/7a2tb2xubZd2yrt7+weHlaPjtlGpJrRFFFe6G2FDOZO0ZZnltJtoikXEaSea3Ob1zhPVhin5YKcJDQUeSRYzgm1uBf7j1aBS9Wv+XGgVggKqUKg5qHz1h4qkgkpLODamF/iJDTOsLSOczsr91NAEkwke0Z5DiQU1YTbfdYbOnTNEsdLuSYvm7u+JDAtjpiJynQLbsVmu5eZ/tV5q43qYMZmklkqy+ChOObIK5YejIdOUWD51gIlmbldExlhjYl08ZRdCsHzyKrQva4Hj++tqo17EUYJTOIMLCOAGGnAHTWgBgTE8wyu8ecJ78d69j0XrmlfMnMAfeZ8/DEKNig==</latexit>
104<latexit sha1_base64="DHVFfGxZOb3ak9mLS6hpGea9ycs=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCozIthlwY3LCvYC7VgyaaYNzWVIMkIZ+gpuXCji1hdy59uYaWehrT8EPv5zDjnnjxLOjPX9b6+0sbm1vVPereztHxweVY9POkalmtA2UVzpXoQN5UzStmWW016iKRYRp91oepvXu09UG6bkg50lNBR4LFnMCLa5FfiP18Nqza/7C6F1CAqoQaHWsPo1GCmSCiot4diYfuAnNsywtoxwOq8MUkMTTKZ4TPsOJRbUhNli1zm6cM4IxUq7Jy1auL8nMiyMmYnIdQpsJ2a1lpv/1fqpjRthxmSSWirJ8qM45cgqlB+ORkxTYvnMASaauV0RmWCNiXXxVFwIwerJ69C5qgeO769rzUYRRxnO4BwuIYAbaMIdtKANBCbwDK/w5gnvxXv3PpatJa+YOYU/8j5/AA3GjYs=</latexit><latexit sha1_base64="DHVFfGxZOb3ak9mLS6hpGea9ycs=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCozIthlwY3LCvYC7VgyaaYNzWVIMkIZ+gpuXCji1hdy59uYaWehrT8EPv5zDjnnjxLOjPX9b6+0sbm1vVPereztHxweVY9POkalmtA2UVzpXoQN5UzStmWW016iKRYRp91oepvXu09UG6bkg50lNBR4LFnMCLa5FfiP18Nqza/7C6F1CAqoQaHWsPo1GCmSCiot4diYfuAnNsywtoxwOq8MUkMTTKZ4TPsOJRbUhNli1zm6cM4IxUq7Jy1auL8nMiyMmYnIdQpsJ2a1lpv/1fqpjRthxmSSWirJ8qM45cgqlB+ORkxTYvnMASaauV0RmWCNiXXxVFwIwerJ69C5qgeO769rzUYRRxnO4BwuIYAbaMIdtKANBCbwDK/w5gnvxXv3PpatJa+YOYU/8j5/AA3GjYs=</latexit><latexit sha1_base64="DHVFfGxZOb3ak9mLS6hpGea9ycs=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCozIthlwY3LCvYC7VgyaaYNzWVIMkIZ+gpuXCji1hdy59uYaWehrT8EPv5zDjnnjxLOjPX9b6+0sbm1vVPereztHxweVY9POkalmtA2UVzpXoQN5UzStmWW016iKRYRp91oepvXu09UG6bkg50lNBR4LFnMCLa5FfiP18Nqza/7C6F1CAqoQaHWsPo1GCmSCiot4diYfuAnNsywtoxwOq8MUkMTTKZ4TPsOJRbUhNli1zm6cM4IxUq7Jy1auL8nMiyMmYnIdQpsJ2a1lpv/1fqpjRthxmSSWirJ8qM45cgqlB+ORkxTYvnMASaauV0RmWCNiXXxVFwIwerJ69C5qgeO769rzUYRRxnO4BwuIYAbaMIdtKANBCbwDK/w5gnvxXv3PpatJa+YOYU/8j5/AA3GjYs=</latexit><latexit sha1_base64="DHVFfGxZOb3ak9mLS6hpGea9ycs=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCozIthlwY3LCvYC7VgyaaYNzWVIMkIZ+gpuXCji1hdy59uYaWehrT8EPv5zDjnnjxLOjPX9b6+0sbm1vVPereztHxweVY9POkalmtA2UVzpXoQN5UzStmWW016iKRYRp91oepvXu09UG6bkg50lNBR4LFnMCLa5FfiP18Nqza/7C6F1CAqoQaHWsPo1GCmSCiot4diYfuAnNsywtoxwOq8MUkMTTKZ4TPsOJRbUhNli1zm6cM4IxUq7Jy1auL8nMiyMmYnIdQpsJ2a1lpv/1fqpjRthxmSSWirJ8qM45cgqlB+ORkxTYvnMASaauV0RmWCNiXXxVFwIwerJ69C5qgeO769rzUYRRxnO4BwuIYAbaMIdtKANBCbwDK/w5gnvxXv3PpatJa+YOYU/8j5/AA3GjYs=</latexit>
Min
utes
Impact of TAS on system performance2014-08-01 2015-02-01 2015-05-01 2015-11-01
Utiliz
atio
n
Inje
ctio
n (B
/s)
Figures of merit on Cray Aries interconnect system
Tota
l pea
k m
emor
y (T
B)
1400
0
1000
600
200
400
800
1200
Node count0 5000 10000 15000 20000
Log1
0 w
all h
ours
T=
t<latexit sha1_base64="yG/FEJE0/45oWvz+KXJnH97KrH4=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEexFKHjxWKGphTaUzXbTLt1swu5EKKG/wYsHRbz6g7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0sbm1vZOebeyt39weFQ9PumYJNOM+yyRie6G1HApFPdRoOTdVHMah5I/hpO7ef3xiWsjEtXGacqDmI6UiASjaC2/TW4JDqo1t+4uRNbBK6AGhVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9ywqGnMT5ItlZ+TCOkMSJdo+hWTh/p7IaWzMNA5tZ0xxbFZrc/O/Wi/DqBHkQqUZcsWWH0WZJJiQ+eVkKDRnKKcWKNPC7krYmGrK0OZTsSF4qyevQ+eq7ll+uK41G0UcZTiDc7gED26gCffQAh8YCHiGV3hzlPPivDsfy9aSU8ycwh85nz+v8Y3n</latexit><latexit sha1_base64="yG/FEJE0/45oWvz+KXJnH97KrH4=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEexFKHjxWKGphTaUzXbTLt1swu5EKKG/wYsHRbz6g7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0sbm1vZOebeyt39weFQ9PumYJNOM+yyRie6G1HApFPdRoOTdVHMah5I/hpO7ef3xiWsjEtXGacqDmI6UiASjaC2/TW4JDqo1t+4uRNbBK6AGhVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9ywqGnMT5ItlZ+TCOkMSJdo+hWTh/p7IaWzMNA5tZ0xxbFZrc/O/Wi/DqBHkQqUZcsWWH0WZJJiQ+eVkKDRnKKcWKNPC7krYmGrK0OZTsSF4qyevQ+eq7ll+uK41G0UcZTiDc7gED26gCffQAh8YCHiGV3hzlPPivDsfy9aSU8ycwh85nz+v8Y3n</latexit><latexit sha1_base64="yG/FEJE0/45oWvz+KXJnH97KrH4=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEexFKHjxWKGphTaUzXbTLt1swu5EKKG/wYsHRbz6g7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0sbm1vZOebeyt39weFQ9PumYJNOM+yyRie6G1HApFPdRoOTdVHMah5I/hpO7ef3xiWsjEtXGacqDmI6UiASjaC2/TW4JDqo1t+4uRNbBK6AGhVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9ywqGnMT5ItlZ+TCOkMSJdo+hWTh/p7IaWzMNA5tZ0xxbFZrc/O/Wi/DqBHkQqUZcsWWH0WZJJiQ+eVkKDRnKKcWKNPC7krYmGrK0OZTsSF4qyevQ+eq7ll+uK41G0UcZTiDc7gED26gCffQAh8YCHiGV3hzlPPivDsfy9aSU8ycwh85nz+v8Y3n</latexit><latexit sha1_base64="yG/FEJE0/45oWvz+KXJnH97KrH4=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEexFKHjxWKGphTaUzXbTLt1swu5EKKG/wYsHRbz6g7z5b9y2OWjrCwsP78ywM2+YSmHQdb+d0sbm1vZOebeyt39weFQ9PumYJNOM+yyRie6G1HApFPdRoOTdVHMah5I/hpO7ef3xiWsjEtXGacqDmI6UiASjaC2/TW4JDqo1t+4uRNbBK6AGhVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9ywqGnMT5ItlZ+TCOkMSJdo+hWTh/p7IaWzMNA5tZ0xxbFZrc/O/Wi/DqBHkQqUZcsWWH0WZJJiQ+eVkKDRnKKcWKNPC7krYmGrK0OZTsSF4qyevQ+eq7ll+uK41G0UcZTiDc7gED26gCffQAh8YCHiGV3hzlPPivDsfy9aSU8ycwh85nz+v8Y3n</latexit>
Mon
itorin
g
Clu
ster
ing
Infe
renc
e Stat
e Es
timat
ion
(Dia
gnos
is)
Stat
e Pr
edic
tion
Zom
bie
proc
ess
Memory leak; app slow
tile
link
dow
n fs
traffi
c iss
ue
unroll factor, k=3
healthy
Node
memory leak.. ..
1
healthyN
2
State
1/Time between errors
Serv
er lo
ad
Numerical features (from performance counters)
Log clustering (e.g., modeling error propagation)
15,872 XE node NAMD job
NIC
inpu
t (G
B/s)
per
pro
ject
060
00
Time0
500
10001500
20002500
0 10 20 30 40
Tim
e (S
econ
ds)
#Run
Prediction Real
T=
t+
� t<latexit sha1_base64="XIrC3cerEYYuWYddHnotb/m+27g=">AAAB+XicbZDLSsNAFIYnXmu9RV26GSyCIJREBLsRCm5cVugN2hAmk9N26OTCzEmhhL6JGxeKuPVN3Pk2TtsstPWHgY//nMM58wepFBod59va2Nza3tkt7ZX3Dw6Pju2T07ZOMsWhxROZqG7ANEgRQwsFSuimClgUSOgE44d5vTMBpUUSN3GaghexYSwGgjM0lm/bTXpPkV7TfggSmY++XXGqzkJ0HdwCKqRQw7e/+mHCswhi5JJp3XOdFL2cKRRcwqzczzSkjI/ZEHoGYxaB9vLF5TN6aZyQDhJlXox04f6eyFmk9TQKTGfEcKRXa3Pzv1ovw0HNy0WcZggxXy4aZJJiQucx0FAo4CinBhhXwtxK+YgpxtGEVTYhuKtfXof2TdU1/HRbqdeKOErknFyQK+KSO1Inj6RBWoSTCXkmr+TNyq0X6936WLZuWMXMGfkj6/MHYBuSKg==</latexit><latexit sha1_base64="XIrC3cerEYYuWYddHnotb/m+27g=">AAAB+XicbZDLSsNAFIYnXmu9RV26GSyCIJREBLsRCm5cVugN2hAmk9N26OTCzEmhhL6JGxeKuPVN3Pk2TtsstPWHgY//nMM58wepFBod59va2Nza3tkt7ZX3Dw6Pju2T07ZOMsWhxROZqG7ANEgRQwsFSuimClgUSOgE44d5vTMBpUUSN3GaghexYSwGgjM0lm/bTXpPkV7TfggSmY++XXGqzkJ0HdwCKqRQw7e/+mHCswhi5JJp3XOdFL2cKRRcwqzczzSkjI/ZEHoGYxaB9vLF5TN6aZyQDhJlXox04f6eyFmk9TQKTGfEcKRXa3Pzv1ovw0HNy0WcZggxXy4aZJJiQucx0FAo4CinBhhXwtxK+YgpxtGEVTYhuKtfXof2TdU1/HRbqdeKOErknFyQK+KSO1Inj6RBWoSTCXkmr+TNyq0X6936WLZuWMXMGfkj6/MHYBuSKg==</latexit><latexit sha1_base64="XIrC3cerEYYuWYddHnotb/m+27g=">AAAB+XicbZDLSsNAFIYnXmu9RV26GSyCIJREBLsRCm5cVugN2hAmk9N26OTCzEmhhL6JGxeKuPVN3Pk2TtsstPWHgY//nMM58wepFBod59va2Nza3tkt7ZX3Dw6Pju2T07ZOMsWhxROZqG7ANEgRQwsFSuimClgUSOgE44d5vTMBpUUSN3GaghexYSwGgjM0lm/bTXpPkV7TfggSmY++XXGqzkJ0HdwCKqRQw7e/+mHCswhi5JJp3XOdFL2cKRRcwqzczzSkjI/ZEHoGYxaB9vLF5TN6aZyQDhJlXox04f6eyFmk9TQKTGfEcKRXa3Pzv1ovw0HNy0WcZggxXy4aZJJiQucx0FAo4CinBhhXwtxK+YgpxtGEVTYhuKtfXof2TdU1/HRbqdeKOErknFyQK+KSO1Inj6RBWoSTCXkmr+TNyq0X6936WLZuWMXMGfkj6/MHYBuSKg==</latexit><latexit sha1_base64="XIrC3cerEYYuWYddHnotb/m+27g=">AAAB+XicbZDLSsNAFIYnXmu9RV26GSyCIJREBLsRCm5cVugN2hAmk9N26OTCzEmhhL6JGxeKuPVN3Pk2TtsstPWHgY//nMM58wepFBod59va2Nza3tkt7ZX3Dw6Pju2T07ZOMsWhxROZqG7ANEgRQwsFSuimClgUSOgE44d5vTMBpUUSN3GaghexYSwGgjM0lm/bTXpPkV7TfggSmY++XXGqzkJ0HdwCKqRQw7e/+mHCswhi5JJp3XOdFL2cKRRcwqzczzSkjI/ZEHoGYxaB9vLF5TN6aZyQDhJlXox04f6eyFmk9TQKTGfEcKRXa3Pzv1ovw0HNy0WcZggxXy4aZJJiQucx0FAo4CinBhhXwtxK+YgpxtGEVTYhuKtfXof2TdU1/HRbqdeKOErknFyQK+KSO1Inj6RBWoSTCXkmr+TNyq0X6936WLZuWMXMGfkj6/MHYBuSKg==</latexit>
Textual features (from raw syslogs)Fe
atur
e Ex
trac
tion
and
Trai
ning
Visu
aliz
atio
n
Numerical clustering(e.g., Identify performance anomalies)
..
Pattern
..
2018/10/23 10:12
..
2018/10/23 10:08
host svc pid * [num] LNET Received down event for *1098
..
23956 .*LustreError:.*@@@ bulk PUT failed.*
Count Last seenPtn ID
357
Significance and Impact
0.00x00.053CoMD 0.0 0.0nid000[21-22]10396 2018-09-21 00:21:25 0.612018-09-21
00:19:16 2.75
0.0nid000[53,59] 2018-09-21 00:19:19
2018-09-21 00:21:28 0.0420.06CoMD 0.00x010397 0.590.1
0.0 0.01.552018-09-21 00:22:02 0.292018-09-21
00:19:26nid000[60-61] 0.21 0x010398 0.041lulesh
Comm Perf
0.0
App Perf
0.59
PAPI Perf
0.0
Anomalies
0x1
Mem Score
0.042
NW Congest
1.570.0
HPagesEnd Time
2018-09-21 00:30:40
Start Time
nid000[53,59] 2018-09-21 00:28:40CoMD
App ID Node IDJob ID
10399
0% 10% 20% 30% 40% 50%
HopsBandwidth
Stalls
Individual data points
Aggregated data points
Online congestion analysis
Job profile
System profile
Multivariate analyses of job and system
Visualization
Dat
a Av
aila
bilit
y (s
econ
ds)
Historical data analysis
ML-based feedback
and control
Expe
rt-dr
iven
Im
prov
emen
ts Data characterization
100<latexit sha1_base64="9mf83UWna4yoN55fyt1OTHldI/8=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2i6DHghePFewHtGvJpmkbm02WZFYoS/+DFw+KePX/ePPfmLZ70OoLgYd3ZsjMGyVSWPT9L6+wtr6xuVXcLu3s7u0flA+PWlanhvEm01KbTkQtl0LxJgqUvJMYTuNI8nY0uZ7X24/cWKHVHU4THsZ0pMRQMIrOagX+febP+uWKX/UXIn8hyKECuRr98mdvoFkac4VMUmu7gZ9gmFGDgkk+K/VSyxPKJnTEuw4VjbkNs8W2M3LmnAEZauOeQrJwf05kNLZ2GkeuM6Y4tqu1uflfrZvi8CrMhEpS5IotPxqmkqAm89PJQBjOUE4dUGaE25WwMTWUoQuo5EIIVk/+C61aNXB8e1Gp1/I4inACp3AOAVxCHW6gAU1g8ABP8AKvnvaevTfvfdla8PKZY/gl7+MbylGOjQ==</latexit><latexit sha1_base64="9mf83UWna4yoN55fyt1OTHldI/8=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2i6DHghePFewHtGvJpmkbm02WZFYoS/+DFw+KePX/ePPfmLZ70OoLgYd3ZsjMGyVSWPT9L6+wtr6xuVXcLu3s7u0flA+PWlanhvEm01KbTkQtl0LxJgqUvJMYTuNI8nY0uZ7X24/cWKHVHU4THsZ0pMRQMIrOagX+febP+uWKX/UXIn8hyKECuRr98mdvoFkac4VMUmu7gZ9gmFGDgkk+K/VSyxPKJnTEuw4VjbkNs8W2M3LmnAEZauOeQrJwf05kNLZ2GkeuM6Y4tqu1uflfrZvi8CrMhEpS5IotPxqmkqAm89PJQBjOUE4dUGaE25WwMTWUoQuo5EIIVk/+C61aNXB8e1Gp1/I4inACp3AOAVxCHW6gAU1g8ABP8AKvnvaevTfvfdla8PKZY/gl7+MbylGOjQ==</latexit><latexit sha1_base64="9mf83UWna4yoN55fyt1OTHldI/8=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2i6DHghePFewHtGvJpmkbm02WZFYoS/+DFw+KePX/ePPfmLZ70OoLgYd3ZsjMGyVSWPT9L6+wtr6xuVXcLu3s7u0flA+PWlanhvEm01KbTkQtl0LxJgqUvJMYTuNI8nY0uZ7X24/cWKHVHU4THsZ0pMRQMIrOagX+febP+uWKX/UXIn8hyKECuRr98mdvoFkac4VMUmu7gZ9gmFGDgkk+K/VSyxPKJnTEuw4VjbkNs8W2M3LmnAEZauOeQrJwf05kNLZ2GkeuM6Y4tqu1uflfrZvi8CrMhEpS5IotPxqmkqAm89PJQBjOUE4dUGaE25WwMTWUoQuo5EIIVk/+C61aNXB8e1Gp1/I4inACp3AOAVxCHW6gAU1g8ABP8AKvnvaevTfvfdla8PKZY/gl7+MbylGOjQ==</latexit><latexit sha1_base64="9mf83UWna4yoN55fyt1OTHldI/8=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2i6DHghePFewHtGvJpmkbm02WZFYoS/+DFw+KePX/ePPfmLZ70OoLgYd3ZsjMGyVSWPT9L6+wtr6xuVXcLu3s7u0flA+PWlanhvEm01KbTkQtl0LxJgqUvJMYTuNI8nY0uZ7X24/cWKHVHU4THsZ0pMRQMIrOagX+febP+uWKX/UXIn8hyKECuRr98mdvoFkac4VMUmu7gZ9gmFGDgkk+K/VSyxPKJnTEuw4VjbkNs8W2M3LmnAEZauOeQrJwf05kNLZ2GkeuM6Y4tqu1uflfrZvi8CrMhEpS5IotPxqmkqAm89PJQBjOUE4dUGaE25WwMTWUoQuo5EIIVk/+C61aNXB8e1Gp1/I4inACp3AOAVxCHW6gAU1g8ABP8AKvnvaevTfvfdla8PKZY/gl7+MbylGOjQ==</latexit>
0.001<latexit sha1_base64="xv3Kk8IL3yKvc2JrFmnkpVnp3Bk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KkkR9Fjw4rGCaQttKJvtpl262YTdiVBCf4MXD4p49Qd589+4aXPQ1gfLPt6bYWZemEph0HW/nY3Nre2d3cpedf/g8Oi4dnLaMUmmGfdZIhPdC6nhUijuo0DJe6nmNA4l74bTu8LvPnFtRKIecZbyIKZjJSLBKFrJdxuu6w1r9eIvQNaJV5I6lGgPa1+DUcKymCtkkhrT99wUg5xqFEzyeXWQGZ5SNqVj3rdU0ZibIF8sOyeXVhmRKNH2KSQL9XdHTmNjZnFoK2OKE7PqFeJ/Xj/D6DbIhUoz5IotB0WZJJiQ4nIyEpozlDNLKNPC7krYhGrK0OZTtSF4qyevk06z4Vn+cF1vNcs4KnAOF3AFHtxAC+6hDT4wEPAMr/DmKOfFeXc+lqUbTtlzBn/gfP4AK1uNiw==</latexit><latexit sha1_base64="xv3Kk8IL3yKvc2JrFmnkpVnp3Bk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KkkR9Fjw4rGCaQttKJvtpl262YTdiVBCf4MXD4p49Qd589+4aXPQ1gfLPt6bYWZemEph0HW/nY3Nre2d3cpedf/g8Oi4dnLaMUmmGfdZIhPdC6nhUijuo0DJe6nmNA4l74bTu8LvPnFtRKIecZbyIKZjJSLBKFrJdxuu6w1r9eIvQNaJV5I6lGgPa1+DUcKymCtkkhrT99wUg5xqFEzyeXWQGZ5SNqVj3rdU0ZibIF8sOyeXVhmRKNH2KSQL9XdHTmNjZnFoK2OKE7PqFeJ/Xj/D6DbIhUoz5IotB0WZJJiQ4nIyEpozlDNLKNPC7krYhGrK0OZTtSF4qyevk06z4Vn+cF1vNcs4KnAOF3AFHtxAC+6hDT4wEPAMr/DmKOfFeXc+lqUbTtlzBn/gfP4AK1uNiw==</latexit><latexit sha1_base64="xv3Kk8IL3yKvc2JrFmnkpVnp3Bk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KkkR9Fjw4rGCaQttKJvtpl262YTdiVBCf4MXD4p49Qd589+4aXPQ1gfLPt6bYWZemEph0HW/nY3Nre2d3cpedf/g8Oi4dnLaMUmmGfdZIhPdC6nhUijuo0DJe6nmNA4l74bTu8LvPnFtRKIecZbyIKZjJSLBKFrJdxuu6w1r9eIvQNaJV5I6lGgPa1+DUcKymCtkkhrT99wUg5xqFEzyeXWQGZ5SNqVj3rdU0ZibIF8sOyeXVhmRKNH2KSQL9XdHTmNjZnFoK2OKE7PqFeJ/Xj/D6DbIhUoz5IotB0WZJJiQ4nIyEpozlDNLKNPC7krYhGrK0OZTtSF4qyevk06z4Vn+cF1vNcs4KnAOF3AFHtxAC+6hDT4wEPAMr/DmKOfFeXc+lqUbTtlzBn/gfP4AK1uNiw==</latexit><latexit sha1_base64="xv3Kk8IL3yKvc2JrFmnkpVnp3Bk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KkkR9Fjw4rGCaQttKJvtpl262YTdiVBCf4MXD4p49Qd589+4aXPQ1gfLPt6bYWZemEph0HW/nY3Nre2d3cpedf/g8Oi4dnLaMUmmGfdZIhPdC6nhUijuo0DJe6nmNA4l74bTu8LvPnFtRKIecZbyIKZjJSLBKFrJdxuu6w1r9eIvQNaJV5I6lGgPa1+DUcKymCtkkhrT99wUg5xqFEzyeXWQGZ5SNqVj3rdU0ZibIF8sOyeXVhmRKNH2KSQL9XdHTmNjZnFoK2OKE7PqFeJ/Xj/D6DbIhUoz5IotB0WZJJiQ4nIyEpozlDNLKNPC7krYhGrK0OZTtSF4qyevk06z4Vn+cF1vNcs4KnAOF3AFHtxAC+6hDT4wEPAMr/DmKOfFeXc+lqUbTtlzBn/gfP4AK1uNiw==</latexit>
1.0<latexit sha1_base64="811cKSCjx8chQSC2kQHS1iS9azI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix4MVjRdMW2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ7fzeueJayMS9YjTlAcxHSkRCUbRWg9e3R1Ua27dXYisg1dADQq1BtWv/jBhWcwVMkmN6XluikFONQom+azSzwxPKZvQEe9ZVDTmJsgXq87IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQuZ3k6HQnKGcWqBMC7srYWOqKUObTsWG4K2evA7tRt2zfH9VazaKOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+yPn8AU6wjRc=</latexit><latexit sha1_base64="811cKSCjx8chQSC2kQHS1iS9azI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix4MVjRdMW2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ7fzeueJayMS9YjTlAcxHSkRCUbRWg9e3R1Ua27dXYisg1dADQq1BtWv/jBhWcwVMkmN6XluikFONQom+azSzwxPKZvQEe9ZVDTmJsgXq87IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQuZ3k6HQnKGcWqBMC7srYWOqKUObTsWG4K2evA7tRt2zfH9VazaKOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+yPn8AU6wjRc=</latexit><latexit sha1_base64="811cKSCjx8chQSC2kQHS1iS9azI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix4MVjRdMW2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ7fzeueJayMS9YjTlAcxHSkRCUbRWg9e3R1Ua27dXYisg1dADQq1BtWv/jBhWcwVMkmN6XluikFONQom+azSzwxPKZvQEe9ZVDTmJsgXq87IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQuZ3k6HQnKGcWqBMC7srYWOqKUObTsWG4K2evA7tRt2zfH9VazaKOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+yPn8AU6wjRc=</latexit><latexit sha1_base64="811cKSCjx8chQSC2kQHS1iS9azI=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix4MVjRdMW2lA22027dLMJuxOhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvGEqhUHX/XZKG5tb2zvl3cre/sHhUfX4pG2STDPus0QmuhtSw6VQ3EeBkndTzWkcSt4JJ7fzeueJayMS9YjTlAcxHSkRCUbRWg9e3R1Ua27dXYisg1dADQq1BtWv/jBhWcwVMkmN6XluikFONQom+azSzwxPKZvQEe9ZVDTmJsgXq87IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQuZ3k6HQnKGcWqBMC7srYWOqKUObTsWG4K2evA7tRt2zfH9VazaKOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+yPn8AU6wjRc=</latexit>
1000<latexit sha1_base64="G52InPjjCdo0joqGvYYaB9+mWeo=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCqZItRlwY3LCvYC7VAyaaYNTTJDkhHK0Fdw40IRt76QO9/GTDsLbf0h8PGfc8g5f5gIbizG315pa3tnd6+8Xzk4PDo+qZ6edU2caso6NBax7ofEMMEV61huBesnmhEZCtYLZ3d5vffEtOGxerTzhAWSTBSPOCU2t3yM8ahaw3W8FNoEv4AaFGqPql/DcUxTyZSlghgz8HFig4xoy6lgi8owNSwhdEYmbOBQEclMkC13XaAr54xRFGv3lEVL9/dERqQxcxm6Tkns1KzXcvO/2iC10W2QcZWklim6+ihKBbIxyg9HY64ZtWLugFDN3a6ITokm1Lp4Ki4Ef/3kTeg26r7jh5taq1HEUYYLuIRr8KEJLbiHNnSAwhSe4RXePOm9eO/ex6q15BUz5/BH3ucPv/ONUw==</latexit><latexit sha1_base64="G52InPjjCdo0joqGvYYaB9+mWeo=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCqZItRlwY3LCvYC7VAyaaYNTTJDkhHK0Fdw40IRt76QO9/GTDsLbf0h8PGfc8g5f5gIbizG315pa3tnd6+8Xzk4PDo+qZ6edU2caso6NBax7ofEMMEV61huBesnmhEZCtYLZ3d5vffEtOGxerTzhAWSTBSPOCU2t3yM8ahaw3W8FNoEv4AaFGqPql/DcUxTyZSlghgz8HFig4xoy6lgi8owNSwhdEYmbOBQEclMkC13XaAr54xRFGv3lEVL9/dERqQxcxm6Tkns1KzXcvO/2iC10W2QcZWklim6+ihKBbIxyg9HY64ZtWLugFDN3a6ITokm1Lp4Ki4Ef/3kTeg26r7jh5taq1HEUYYLuIRr8KEJLbiHNnSAwhSe4RXePOm9eO/ex6q15BUz5/BH3ucPv/ONUw==</latexit><latexit sha1_base64="G52InPjjCdo0joqGvYYaB9+mWeo=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCqZItRlwY3LCvYC7VAyaaYNTTJDkhHK0Fdw40IRt76QO9/GTDsLbf0h8PGfc8g5f5gIbizG315pa3tnd6+8Xzk4PDo+qZ6edU2caso6NBax7ofEMMEV61huBesnmhEZCtYLZ3d5vffEtOGxerTzhAWSTBSPOCU2t3yM8ahaw3W8FNoEv4AaFGqPql/DcUxTyZSlghgz8HFig4xoy6lgi8owNSwhdEYmbOBQEclMkC13XaAr54xRFGv3lEVL9/dERqQxcxm6Tkns1KzXcvO/2iC10W2QcZWklim6+ihKBbIxyg9HY64ZtWLugFDN3a6ITokm1Lp4Ki4Ef/3kTeg26r7jh5taq1HEUYYLuIRr8KEJLbiHNnSAwhSe4RXePOm9eO/ex6q15BUz5/BH3ucPv/ONUw==</latexit><latexit sha1_base64="G52InPjjCdo0joqGvYYaB9+mWeo=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuCqZItRlwY3LCvYC7VAyaaYNTTJDkhHK0Fdw40IRt76QO9/GTDsLbf0h8PGfc8g5f5gIbizG315pa3tnd6+8Xzk4PDo+qZ6edU2caso6NBax7ofEMMEV61huBesnmhEZCtYLZ3d5vffEtOGxerTzhAWSTBSPOCU2t3yM8ahaw3W8FNoEv4AaFGqPql/DcUxTyZSlghgz8HFig4xoy6lgi8owNSwhdEYmbOBQEclMkC13XaAr54xRFGv3lEVL9/dERqQxcxm6Tkns1KzXcvO/2iC10W2QcZWklim6+ihKBbIxyg9HY64ZtWLugFDN3a6ITokm1Lp4Ki4Ef/3kTeg26r7jh5taq1HEUYYLuIRr8KEJLbiHNnSAwhSe4RXePOm9eO/ex6q15BUz5/BH3ucPv/ONUw==</latexit>
103<latexit sha1_base64="MXHYMkXcl7hzFe5bJrn2G/U+57Q=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZKuiy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrKLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1lpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/AM7gjpA=</latexit><latexit sha1_base64="MXHYMkXcl7hzFe5bJrn2G/U+57Q=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZKuiy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrKLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1lpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/AM7gjpA=</latexit><latexit sha1_base64="MXHYMkXcl7hzFe5bJrn2G/U+57Q=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZKuiy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrKLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1lpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/AM7gjpA=</latexit><latexit sha1_base64="MXHYMkXcl7hzFe5bJrn2G/U+57Q=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZKuiy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrKLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1lpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/AM7gjpA=</latexit>
106<latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="wXbxPfgkp2vES0YFIKuya0dyvkg=">AAAB4nicbZDNSgMxFIXv1L9aq1a3boJFcFVmXKhLwY3LCnZaaMeSSe+0sZlkSDJCGfoOblwo4kO5821MfxbaeiDwcU5C7j1xJrixvv/tlTY2t7Z3yruVver+wWHtqBoalWuGLaaE0p2YGhRcYstyK7CTaaRpLLAdj29nefsZteFKPthJhlFKh5InnFHrrDDwH4vLab9W9xv+XGQdgiXUYalmv/bVGyiWpygtE9SYbuBnNiqotpwJnFZ6ucGMsjEdYtehpCmaqJhPOyVnzhmQRGl3pCVz9/eLgqbGTNLY3UypHZnVbGb+l3Vzm1xHBZdZblGyxUdJLohVZLY6GXCNzIqJA8o0d7MSNqKaMusKqrgSgtWV1yG8aASO730owwmcwjkEcAU3cAdNaAGDJ3iBN3j3lPfqfSzqKnnL3o7hj7zPH7JSjUs=</latexit><latexit sha1_base64="wXbxPfgkp2vES0YFIKuya0dyvkg=">AAAB4nicbZDNSgMxFIXv1L9aq1a3boJFcFVmXKhLwY3LCnZaaMeSSe+0sZlkSDJCGfoOblwo4kO5821MfxbaeiDwcU5C7j1xJrixvv/tlTY2t7Z3yruVver+wWHtqBoalWuGLaaE0p2YGhRcYstyK7CTaaRpLLAdj29nefsZteFKPthJhlFKh5InnFHrrDDwH4vLab9W9xv+XGQdgiXUYalmv/bVGyiWpygtE9SYbuBnNiqotpwJnFZ6ucGMsjEdYtehpCmaqJhPOyVnzhmQRGl3pCVz9/eLgqbGTNLY3UypHZnVbGb+l3Vzm1xHBZdZblGyxUdJLohVZLY6GXCNzIqJA8o0d7MSNqKaMusKqrgSgtWV1yG8aASO730owwmcwjkEcAU3cAdNaAGDJ3iBN3j3lPfqfSzqKnnL3o7hj7zPH7JSjUs=</latexit><latexit sha1_base64="PXz9oLg5y6XWkcACjucFco/rjCA=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2e1CPBS8eK9gPaNeSTdM2NpssyaxQlv4HLx4U8er/8ea/MW33oNUXAg/vzJCZN0qksOj7X15hbX1jc6u4XdrZ3ds/KB8etaxODeNNpqU2nYhaLoXiTRQoeScxnMaR5O1ocj2vtx+5sUKrO5wmPIzpSImhYBSd1Qr8++xi1i9X/Kq/EPkLQQ4VyNXolz97A83SmCtkklrbDfwEw4waFEzyWamXWp5QNqEj3nWoaMxtmC22nZEz5wzIUBv3FJKF+3Mio7G10zhynTHFsV2tzc3/at0Uh1dhJlSSIlds+dEwlQQ1mZ9OBsJwhnLqgDIj3K6EjamhDF1AJRdCsHryX2jVqoHjW79Sr+VxFOEETuEcAriEOtxAA5rA4AGe4AVePe09e2/e+7K14OUzx/BL3sc30i+Ojw==</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit><latexit sha1_base64="1o3fCNG9ptKMYqyByEsA4K/6ABc=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIuqy4MZlBXuBdiyZNNPGZpIhyQhl6Du4caGIW9/HnW9j2s5CW38IfPznHHLOHyaCG+t536iwtr6xuVXcLu3s7u0flA+PWkalmrImVULpTkgME1yypuVWsE6iGYlDwdrh+GZWbz8xbbiS93aSsCAmQ8kjTol1Vsv3HrLLab9c8areXHgV/BwqkKvRL3/1BoqmMZOWCmJM1/cSG2REW04Fm5Z6qWEJoWMyZF2HksTMBNl82yk+c84AR0q7Jy2eu78nMhIbM4lD1xkTOzLLtZn5X62b2ug6yLhMUsskXXwUpQJbhWen4wHXjFoxcUCo5m5XTEdEE2pdQCUXgr988iq0alXf8d1FpV7L4yjCCZzCOfhwBXW4hQY0gcIjPMMrvCGFXtA7+li0FlA+cwx/hD5/ANNvjpM=</latexit>
109<latexit sha1_base64="/Eov8KxIcRiFNUm+MxJxHteUijA=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIqi7ghuXFewF2rFk0kwbm0mGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5gIbqznfaPC2vrG5lZxu7Szu7d/UD48ahmVasqaVAmlOyExTHDJmpZbwTqJZiQOBWuH45tZvf3EtOFK3ttJwoKYDCWPOCXWWS3fe8iup/1yxat6c+FV8HOoQK5Gv/zVGyiaxkxaKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdShIzE2Tzbaf4zDkDHCntnrR47v6eyEhszCQOXWdM7Mgs12bmf7VuaqOrIOMySS2TdPFRlApsFZ6djgdcM2rFxAGhmrtdMR0RTah1AZVcCP7yyavQqlV9x3cXlXotj6MIJ3AK5+DDJdThFhrQBAqP8Ayv8IYUekHv6GPRWkD5zDH8Efr8Adf+jpY=</latexit><latexit sha1_base64="/Eov8KxIcRiFNUm+MxJxHteUijA=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIqi7ghuXFewF2rFk0kwbm0mGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5gIbqznfaPC2vrG5lZxu7Szu7d/UD48ahmVasqaVAmlOyExTHDJmpZbwTqJZiQOBWuH45tZvf3EtOFK3ttJwoKYDCWPOCXWWS3fe8iup/1yxat6c+FV8HOoQK5Gv/zVGyiaxkxaKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdShIzE2Tzbaf4zDkDHCntnrR47v6eyEhszCQOXWdM7Mgs12bmf7VuaqOrIOMySS2TdPFRlApsFZ6djgdcM2rFxAGhmrtdMR0RTah1AZVcCP7yyavQqlV9x3cXlXotj6MIJ3AK5+DDJdThFhrQBAqP8Ayv8IYUekHv6GPRWkD5zDH8Efr8Adf+jpY=</latexit><latexit sha1_base64="/Eov8KxIcRiFNUm+MxJxHteUijA=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIqi7ghuXFewF2rFk0kwbm0mGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5gIbqznfaPC2vrG5lZxu7Szu7d/UD48ahmVasqaVAmlOyExTHDJmpZbwTqJZiQOBWuH45tZvf3EtOFK3ttJwoKYDCWPOCXWWS3fe8iup/1yxat6c+FV8HOoQK5Gv/zVGyiaxkxaKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdShIzE2Tzbaf4zDkDHCntnrR47v6eyEhszCQOXWdM7Mgs12bmf7VuaqOrIOMySS2TdPFRlApsFZ6djgdcM2rFxAGhmrtdMR0RTah1AZVcCP7yyavQqlV9x3cXlXotj6MIJ3AK5+DDJdThFhrQBAqP8Ayv8IYUekHv6GPRWkD5zDH8Efr8Adf+jpY=</latexit><latexit sha1_base64="/Eov8KxIcRiFNUm+MxJxHteUijA=">AAAB7XicbZDLSgMxFIZP6q3WW9Wlm2ARXJWZIqi7ghuXFewF2rFk0kwbm0mGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5gIbqznfaPC2vrG5lZxu7Szu7d/UD48ahmVasqaVAmlOyExTHDJmpZbwTqJZiQOBWuH45tZvf3EtOFK3ttJwoKYDCWPOCXWWS3fe8iup/1yxat6c+FV8HOoQK5Gv/zVGyiaxkxaKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdShIzE2Tzbaf4zDkDHCntnrR47v6eyEhszCQOXWdM7Mgs12bmf7VuaqOrIOMySS2TdPFRlApsFZ6djgdcM2rFxAGhmrtdMR0RTah1AZVcCP7yyavQqlV9x3cXlXotj6MIJ3AK5+DDJdThFhrQBAqP8Ayv8IYUekHv6GPRWkD5zDH8Efr8Adf+jpY=</latexit>
1012<latexit sha1_base64="Aq1Hwb6oeUcWWb6ioDDq0HBNdm0=">AAAB7nicbZBNS8NAEIYn9avWr6hHL4tF8FSSIuix4MVjBfsBbSyb7aZdutmE3YlQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3jCVwqDnfTuljc2t7Z3ybmVv/+DwyD0+aZsk04y3WCIT3Q2p4VIo3kKBkndTzWkcSt4JJ7fzeueJayMS9YDTlAcxHSkRCUbRWh3fe8z9+mzgVr2atxBZB7+AKhRqDtyv/jBhWcwVMkmN6fleikFONQom+azSzwxPKZvQEe9ZVDTmJsgX687IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQua3k6HQnKGcWqBMC7srYWOqKUObUMWG4K+evA7tes23fH9VbdSLOMpwBudwCT5cQwPuoAktYDCBZ3iFNyd1Xpx352PZWnKKmVP4I+fzBz44jso=</latexit><latexit sha1_base64="Aq1Hwb6oeUcWWb6ioDDq0HBNdm0=">AAAB7nicbZBNS8NAEIYn9avWr6hHL4tF8FSSIuix4MVjBfsBbSyb7aZdutmE3YlQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3jCVwqDnfTuljc2t7Z3ybmVv/+DwyD0+aZsk04y3WCIT3Q2p4VIo3kKBkndTzWkcSt4JJ7fzeueJayMS9YDTlAcxHSkRCUbRWh3fe8z9+mzgVr2atxBZB7+AKhRqDtyv/jBhWcwVMkmN6fleikFONQom+azSzwxPKZvQEe9ZVDTmJsgX687IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQua3k6HQnKGcWqBMC7srYWOqKUObUMWG4K+evA7tes23fH9VbdSLOMpwBudwCT5cQwPuoAktYDCBZ3iFNyd1Xpx352PZWnKKmVP4I+fzBz44jso=</latexit><latexit sha1_base64="Aq1Hwb6oeUcWWb6ioDDq0HBNdm0=">AAAB7nicbZBNS8NAEIYn9avWr6hHL4tF8FSSIuix4MVjBfsBbSyb7aZdutmE3YlQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3jCVwqDnfTuljc2t7Z3ybmVv/+DwyD0+aZsk04y3WCIT3Q2p4VIo3kKBkndTzWkcSt4JJ7fzeueJayMS9YDTlAcxHSkRCUbRWh3fe8z9+mzgVr2atxBZB7+AKhRqDtyv/jBhWcwVMkmN6fleikFONQom+azSzwxPKZvQEe9ZVDTmJsgX687IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQua3k6HQnKGcWqBMC7srYWOqKUObUMWG4K+evA7tes23fH9VbdSLOMpwBudwCT5cQwPuoAktYDCBZ3iFNyd1Xpx352PZWnKKmVP4I+fzBz44jso=</latexit><latexit sha1_base64="Aq1Hwb6oeUcWWb6ioDDq0HBNdm0=">AAAB7nicbZBNS8NAEIYn9avWr6hHL4tF8FSSIuix4MVjBfsBbSyb7aZdutmE3YlQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3jCVwqDnfTuljc2t7Z3ybmVv/+DwyD0+aZsk04y3WCIT3Q2p4VIo3kKBkndTzWkcSt4JJ7fzeueJayMS9YDTlAcxHSkRCUbRWh3fe8z9+mzgVr2atxBZB7+AKhRqDtyv/jBhWcwVMkmN6fleikFONQom+azSzwxPKZvQEe9ZVDTmJsgX687IhXWGJEq0fQrJwv09kdPYmGkc2s6Y4tis1ubmf7VehtFNkAuVZsgVW34UZZJgQua3k6HQnKGcWqBMC7srYWOqKUObUMWG4K+evA7tes23fH9VbdSLOMpwBudwCT5cQwPuoAktYDCBZ3iFNyd1Xpx352PZWnKKmVP4I+fzBz44jso=</latexit>
Data Size (B)
Scalable System Design and Architecture Actionable Expert and ML-Driven Analysis
https://hmdsa.github.com/hmdsa
• ML/AI• Log and Event Analysis
Actionable IntelligenceData Collection
DataCollection
Feedback and Control
Analysis
Admin WebMOTD, Log Browsing,
Data Mini, WebPortal API
Dashboards
Analysis Research
System Visualization
Email, Text andEvent Notifications• ML/AI
• Signal Analysis• Log and Event Analysis• Event Prediction and Correlation
HIGH PERFORMANCE/PROJECT DRIVENANALYSIS AND PREDICTION TOOLS
PROCESSORINTERCONNECT STORAGE
EventLogs
Blade andCabinet
ControllerSensor Data
LDMS
Job ApplicationResource and Performance Data
NotificationSystem
Machine in the Loop Analysis FeedbackHuman in the LoopAnalysis Feedback
PowerSystem
FacilitySystems
EnvironmentalSystems
BUILDING AND FACILITIES
Parser
Syslog-NGDatabase and Active Data Repository
WebServer
Long Term Data Repository
Z+
Y+
X+006005004003 007
106105104103 107
016015014013 017
116115114113 117
Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Gentile, Ann
Gentile, Ann
Gentile, Ann
https://hmdsa.github.io/hmdsa
