ShakeAlert CISN Testing Center (CTC) Development
description
Transcript of ShakeAlert CISN Testing Center (CTC) Development
![Page 1: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/1.jpg)
SCEC: An NSF + USGS Research Center
ShakeAlert CISN Testing Center (CTC) Development
Philip MaechlingInformation Technology ArchitectSouthern California Earthquake Center (SCEC)14 October 2010
![Page 2: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/2.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) system to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
2
CTC Progress in 2010
![Page 3: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/3.jpg)
EEW Testing Center Provides On-going Performance Evaluation
• Performance summaries available through login (www.scec.org/eew)• Evaluation results for 2010 include 144 M4+ earthquakes in CA Testing
Region• Cumulative raw summaries (2008-present) posted at
(scec.usc.edu/scecpedia/Earthquake_Early_Warning)
![Page 4: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/4.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) system to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
4
CTC Progress in 2010
![Page 5: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/5.jpg)
Earthquake Catalog
Retrieve Data
FilterCatalog
Filtered Earthquake
Catalog
CISN Testing Center (CTC) Forecast Evaluation Processing System
CISN EEW Testing Center (CTC) and Web Site
ANSS Earthquake
Catalog
CISN Decision Modules
CISN User Modules
Load Reports
ShakeAlert Earthquake ParameterForecast
ShakeAlert Ground Motion
Forecast
Observed ANSS EQ Parameter and
Ground Motion Data
ShakeAlert Forecast
Information
Evaluation testscomparing
Forecasts andObservations
ShakeMap RSS Feed Ground Motion
Observations
![Page 6: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/6.jpg)
![Page 7: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/7.jpg)
![Page 8: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/8.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
8
CTC Progress in 2010
![Page 9: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/9.jpg)
![Page 10: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/10.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-present).
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
10
CTC Progress in 2010
![Page 11: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/11.jpg)
11
Current ShakeAlert CTC retrieves ShakeMap RSS data and plots observations for all Mag 3.0+ earthquakes in California Testing Region as shown (left).
CTC Progress in 2010
![Page 12: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/12.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-Present)
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
12
CTC Progress in 2010
![Page 13: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/13.jpg)
Earthquake Catalog
Retrieve Data
FilterCatalog
Filtered Earthquake
Catalog
CISN Testing Center (CTC) Forecast Evaluation Processing System
CISN EEW Testing Center (CTC) and Web Site
ANSS Earthquake
Catalog
CISN Decision Modules
CISN User Modules
Load Reports
ShakeAlert Earthquake ParameterForecast
ShakeAlert Ground Motion
Forecast
Observed ANSS EQ Parameter and
Ground Motion Data
ShakeAlert Forecast
Information
Evaluation testscomparing
Forecasts andObservations
ShakeMap RSS Feed Ground Motion
Observations
![Page 14: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/14.jpg)
1. Operating algorithm evaluation system with California-based performance reports and raw data available (2008-Present)
2. Changed our automated software testing infrastructure from web-based (Joomla) to server-based (CSEP) system.
3. Added ShakeMap RSS reader into CSEP for use as a source of authorized observational data that will be used to evaluate earthquake parameter and ground motion forecasts.
4. Implemented a prototype EEW forecast evaluation test as plotting of PGV used in ShakeMaps for each event.
5. Began nightly automated retrieval of observational data from ShakeMap RSS and create observation-based ground motion maps.
6. Started implementation of ground motion forecast evaluation defined in 2008 CISN Testing Document.
14
CTC Progress in 2010
![Page 15: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/15.jpg)
15
CTC Progress in 2010
Initial CTC Evaluation Test is defined in 2008 CISN EEW Testing Document (as updated July 2010). Previous Algorithm Testing Center did not implement this summary. Access to ShakeMap RSS ground motion observations makes automated implementation practical.
![Page 16: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/16.jpg)
1. Prioritization of forecast evaluation tests to be implemented2. SCEC science planning of EEW forecast evaluation experiments3. Use of EEW in time-dependent PSHA information4. Consider Extending ShakeMap format as CAP-based forecast
exchange format.– Send forecasts information (and time of report) to produce:– ShakeMap Intensity Maps– ShakeMap Uncertainties Maps
5. Consider ShakeAlert interfaces to support comparative EEW performance tests. Provide access to information for each trigger:
– Stations Used In Trigger– Stations Available when declaring Trigger– Software Version declaring Trigger
16
Scientific and Technical Coordination Issues
![Page 17: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/17.jpg)
End
17
![Page 18: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/18.jpg)
Proposed CTC Evaluation Tests
18
![Page 19: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/19.jpg)
Rigorous CISN EEW testing will involve the following definitions:– Define a forecast– Define testing area– Define input data used in forecasts– Define reference observation data– Define measures of success for forecasts
19
Design of an Experiment
![Page 20: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/20.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
20
Proposed Performance Measures
![Page 21: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/21.jpg)
Summary 1.1: Magnitude X-Y Diagram
Measure of Goodness: Data points fall on diagonal line
Relevant: T2,T3,T4Drawbacks: Timeliness
element not representedWhich in series of magnitude
estimates should be used in plot.
21
Experiment Design
![Page 22: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/22.jpg)
Summary 1.2: Initial magnitude error by magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T2,T3,T4Drawbacks: Timeliness
element not represented
22
Experiment Design
![Page 23: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/23.jpg)
Summary 1.3: Magnitude accuracy by update
Measure of Goodness: Data points fall on horizontal line
Relevant: T3,T4Drawbacks: Timeliness
element not represented
23
Experiment Design
![Page 24: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/24.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
24
Proposed Performance Measures
![Page 25: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/25.jpg)
25
Experiment Design
Summary 2.1: Cumulative Location Errors
Measure of Goodness: Data points fall on vertical zero line
Relevant: T3, T4Drawbacks: Does not
consider magnitude accuracy or timeliness
![Page 26: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/26.jpg)
Summary 2.2: Magnitude and Location error by time after origin
Measure of Goodness: Data points fall on horizontal zero line
Relevant: T3, T4Drawbacks: Event-specific
not cumulative
26
Experiment Design
![Page 27: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/27.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
27
Proposed Performance Measures
![Page 28: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/28.jpg)
28
Experiment DesignSummary 3.1 : Intensity Map
ComparisonsMeasure of Goodness: Forecast
map matches observed mapRelevant: T4Drawbacks: Not a quantitative
results
![Page 29: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/29.jpg)
Summary 3.2: Intensity X-Y Diagram
Measure of Goodness: Data points fall on diagonal line
Relevant: T1,T2,T4Drawbacks: Timeliness
element not representedWhich in series of intensity
estimate should be used in plots T3.
29
Experiment Design
![Page 30: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/30.jpg)
Summary 3.3: Intensity Ratio by Magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T1,T2,T4Drawbacks: Timeliness element
not representedWhich intensity estimate in
series should be used in plot.
30
Experiment Design
![Page 31: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/31.jpg)
Summary 3.3: Predicted to Observed Intensity Ratio by Distance and Magnitude
Measure of Goodness: Data points fall on horizontal line
Relevant: T1,T2,T4Drawbacks: Timeliness element
not representedWhich intensity estimate in
series should be used in plot.
31
![Page 32: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/32.jpg)
Summary 3.3: Evaluate Conversion from PGV to Intensity
Group has proposed to evaluate algorithms by comparing intensities and they provide a formula for conversion to Intensity.
32
![Page 33: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/33.jpg)
Summary 3.4: Evaluate Conversion from PGV to Intensity
Group has proposed to evaluate algorithms by comparing intensities and they provide a formula for conversion to Intensity.
33
![Page 34: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/34.jpg)
34
Experiment DesignSummary 3.5: Statistical Error
Distribution for Magnitude and Intensity
Measure of Goodness: No missed events or false alarms in testing area
Relevant: T4Drawbacks:
![Page 35: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/35.jpg)
35
Experiment DesignSummary 3.6: Mean-time to
first location or intensity estimate (small blue plot)
Measure of Goodness: Peak of measures at zero
Relevant: T1,T2,T3,T4Drawbacks: Cumulative and
does not involve accuracy of estimates
![Page 36: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/36.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
36
Proposed Performance Measures
![Page 37: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/37.jpg)
37
Experiment DesignNo examples for System Performance Summary defined as
Summary 4.1: Ratio of reporting versus non-reporting stations:
![Page 38: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/38.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
38
Proposed Performance Measures
![Page 39: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/39.jpg)
39
Experiment DesignSummary 5.1: Missed event and
False Alarm MapMeasure of Goodness: No
missed events or false alarms in testing area
Relevant: T3, T4Drawbacks: Must develop
definitions for missed events and false alarms, Does not reflect timeliness
![Page 40: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/40.jpg)
40
Experiment DesignSummary 5.2: Missed event and
False Alarm MapMeasure of Goodness: No
missed events or false alarms in testing area
Relevant: T3, T4Drawbacks: Must develop
definitions for missed events and false alarms, Does not reflect timeliness
![Page 41: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/41.jpg)
Summary Reports for each M ≥ M-min: Key documents is 3 March 2008 document which specifies six types of tests.
– Summary 1: Magnitude– Summary 2: Location– Summary 3: Ground Motion– Summary 4: System Performance– Summary 5: False Triggers– Summary 6: Missed Triggers
41
Proposed Performance Measures
![Page 42: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/42.jpg)
42
Experiment DesignSummary 6.1: Missed
Event mapMeasure of Goodness: No
missed events in testing region
Relevant: T3, T4Drawbacks: Must define
missed event. Does not indicate timeliness
![Page 43: ShakeAlert CISN Testing Center (CTC) Development](https://reader030.fdocuments.us/reader030/viewer/2022033104/56816145550346895dd0bf70/html5/thumbnails/43.jpg)
End
43