The Sochi 2014 Winter Olympics - · PDF fileThe Sochi 2014 Winter Olympics Forecast...
23
XXII Olympic Winter Games Pertti Nurmi Matias Brockmann & Sigbritt Näsman Acknowledgements : WMO FROST-2014 Expert Team & FMI verification system development team The Sochi 2014 Winter Olympics Forecast Verification Framework Some Early Results ( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )
Transcript of The Sochi 2014 Winter Olympics - · PDF fileThe Sochi 2014 Winter Olympics Forecast...
XXII Olympic Winter Games
Pertti NurmiMatias Brockmann & Sigbritt Näsman
Acknowledgements : WMO FROST-2014 Expert Team & FMI verification system development team
The Sochi 2014 Winter OlympicsForecast Verification Framework
Some Early Results( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )
XXII Olympic Winter Games
Laura ski stadium ~ 1400 m
RusSki Gorki
Rosa Khutor
Sanki
Olympics Mountain Cluster sports venues
Aibga ridge :Alpine skiing top ~2300 m
4
XXII Olympic Winter Games
Model Analysis / FC hours Contributor FMI Verification Notes
Deterministic ForecastsCOSMO-RU - 7 km 00, 06, 12, 18 UTC / + 78 hr HMC,Russia ***COSMO-RU - 2 km 00, 06, 12, 18 UTC / + 42 hr ” ***COSMO-RU - 1 km - ” to be considered as RDPGEM - 2.5 km 23 UTC / + 27 hr Env. Canada ***GEM - 1 km 21 UTC / + 25 hr ” ***GEM - 0.25 km 00 UTC / + 24 hr ” *** Processing difficultiesHARMONIE-Sochi - 1 km 00, 06, 12, 18 UTC / + 36 hr FMI, Finland ***NMMB - 1 km 00, 12 UTC / + 24 hr NOAA,USA ***KMA - KMA, S-Korea to be considered as RDPARPA (Sochi-mini) 00, 12 UTC / + 72 hr ARPA SIMC, Italy ***INCA - ZAMG, AustriaJoint Hourly / + 48 hr “consensus” ***
Ensemble ForecastsAladin-LAEF-EPS - 11 km - ZAMG,AustriaGLAMEPS - 11 km 06, 18 UTC / + 54 hr HIRLAM, Norway ***GLAMEPS - updated T Hourly / + 48 hr ” ***HarmonEPS - 2.5 km 06, 18 UTC / + 30 hr ” *** to be considered as RDPCOSMO-RU-EPS - 2 km - HMC,Russia to be considered as RDPCOSMO-S14-EPS - 7 km - ARPA SIMC, ItalyNMMB-EPS - 7 km - NOAA, USA
NowcastingMeteoExpert - single station 10 minutes / + 4 hr IRAM,Russia ***Joint Hourly / + 48 hr “consensus” ***INCA ZAMG,AustriaINTW Env.Canada to be considered as RDPCARDS Env.Canada to be considered as RDPABOM Env.Canada to be considered as RDPRW Model ( Harmonie-driven ) FMI, Finland Verification within external CoMoSeF project
FROST-2014 : Models - Contributors
6
XXII Olympic Winter Games
7
Verification Issues … #1 All FROST-2014 contributors verify their own fc products General practice!
… BUT can be biased NOT sufficient / adequate
Centralized (HMC, Russia) AND ”External” verification by FMI… also to “validate verification results”
Two complementary approaches (like Vancouver Olympics SNOW-V10)
a) High temporal resolution Point Verification User needs : FDP* Users require thresholds for their decisions Sports activities and events take place at (or along) points
b) Research-oriented verification, Detailed model comparisons : RDP*
* FDP = Forecast Demonstration Project / RDP = Research Demonstration Project / WMO terminology
XXII Olympic Winter Games
8
Verification Issues … #2 Hi-fidelity site-specific (point) observations highly important
Enhanced observation network : 31 selected stations used in verification
On-line production, storage + dissemination from all forecast contributors
Common data base structure : ftp site
”Freeze” the FDP system at certain stage
“ No changes allowed in the last 3-4 months before the Olympics ”
Standardized XML formats for fc & obs dissemination + storage !
Comprehensive post-event verification possible only after all data archived and QC’d
Case studies Hard to generalize results fall under impact assessment and/or RDP
Only some did
Formal verification period : 15 January – 15 March = 2 monthsOlympics : 7-23 February + Paralympics : 7-16 March
XXII Olympic Winter Games
9
Verification Issues … #3 Pointwise verification of all forecasts, tuned at all forecast = decision points
a) Assumption : Reference baseline quality from a coarse, large-scale modelb) Various hi-res models from contributors with various time/space resolutionsc) “Consensus” products “Joint model”d) Inter-comparison between, and added value of (a) to (c) Impacts End-Products by the Olympics forecaster team not disseminated for verification
Definition of each sports site/event relevant obs stations + fc variables and thresholds Not trivial !
Use of verification software, e.g. MET by HMC Russia
FMI online verification package “tuned” for Sochi Input (fc + obs) provided at common ftp site Only some did
Formal verification period : 15 January – 15 March = 2 monthsOlympics : 7-23 February + Paralympics : 7-16 March
XXII Olympic Winter Games
10
FROST-2014 : Weather parameters - ThresholdsTemperature ( OC ) T < -20 -20 ≤ T < -5 -5 ≤ T < -2 -2 ≤ T < 0 0 ≤ T < 2 2 ≤ T < 5 T ≥ 5
Wind speed ( m/s ) WS ≥ 3 WS ≥ 4 WS ≥ 5 WS ≥ 7 WS ≥ 11 WS ≥ 15 WS ≥ 19
Horizontal visibility ( m ) V < 100 V < 300 V < 1000 V < 10 000
Precipitation amount RR < 0.3 RR ≥ 0.3 RR ≥ 1.0 RR ≥ 5.0 RR ≥ 10.0 RR ≥ 15.01-hr and 24-hr (mm)
Photo © johnny9s on Flickr Photo © Prentice Korea on Flickr
High-impact What is hi-impact ? Thresholds
Valley(c. 600m)
Ski Jump
SkiBiathlon (Laura, top 1400m)
Alpine Ski(Rosa Khutor, top 2300m)
FMI Verification User InterfaceAdapted for Sochi 2014 Olympics
14
All stations Determ. models
XXII Olympic Winter Games
TMAE
15
HARMONIE best during daytime Sports events HARMONIE worst during nighttime ”who cares” ? Lo-res models, COSMO 7km + ARPA, worst
00UTC forecast runs (available for Olympics forecasters in the morning) used in verification statistics
Coastal stations Determ. models
Hi-res model, GEM 0.25km, strangely worst at coast
XXII Olympic Winter Games
16
Mountain cluster Determ. models
HARMONIE best during day, worst during night is the feature in the mountains
COSMO worst
TMAE
00UTC forecast runs (available for Olympics forecasters in the morning) used in verification statistics
XXII Olympic Winter Games
17
Determ. models
GEM 0.25km and JOINT best HARMONIE ”in the middle” Lo-res model, COSMO 7km, worst
Mountain clusterWSSEDI
00UTC forecast runs (available for Olympics forecasters in the morning) used in verification statistics
EPS models
Calibrated + hourly T-updated GLAMEPS best Raw GLAMEPS worst
Mountain clusterNB: Actual Olympics 2-wk period
XXII Olympic Winter Games
18
PoTROC
EPS models
Calibrated + hourly T-updated GLAMEPS best More noisy statistics due to sample size (3 stations)
Ski stadiumNB: Actual Olympics 2-wk period
All forecast runs used in the EPS verification statistics
XXII Olympic Winter Games
Subjective Evaluation
19
Model Overall usefulness
Forecast accuracy
Visualization (appearance)
Timeliness and reliabilityTemp Precip Wind Gusts Visib
COSMO-RU 7 km 1.9 2.3 2.3 2.1 x 2.9 2.9
COSMO-RU 2 km 2.7 2.3 2.3 2.3 2.1 x 2.9
COSMO-RU 1 km 1.5 2.0 2.0 2.3 x
NMMB 1 km
GEM 250 m
2.0
2.2
2.0
2.2
1.3
2.0
1.3
2.0
2.0
1.8
Harmonie 1 km 2.3 2.3 2.0 2.0 ( * )
Forecast quality assessment : 10 HMC Russia forecasters / team leadersScale : 0 = Not useful / 1 = Partly useful / 2 = Useful / 3 = Excellent
Acknowledgement : Michael Tsyrulnikov, HMC Russia
( * ) : Harmonie DID produce (good) visibility forecasts which were not used by Sochi forecasters(not disseminated as XML)
XXII Olympic Winter Games
20
HARMONIE beats GEM 0.25kmduring first 15 hrs
3 Alpine stationsVIS
SEDI
00UTC forecast runs (available for Olympics forecasters in the morning) used in verification statistics
XXII Olympic Winter Games
Sochi 2014 Road Weather Pilot: End-user FeedbackScale : 1 = Fully disagree / 2 = Disagree / 3 = Neutral / 4 = Agree / 5 = Fully agree
DateFriction forecast
gave a good estimate of slipperiness
Storage terms (water, snow, ice)
are realistic
Road weather forecast was
usefulNotes (free text)
16.2.2014 5 4 Precipitation, however road surface dried towards Adler
20.2.2014 5 5 2 Dry road surface, i.e. No real need for RW forecast
21.2.2014 5 3 Variable road surface humidity
23.2.2014 5 5 2 Dry road surface, i.e. No real need for RW forecast
End-user = Finnish skiing service team, comments :• ” HARMONIE was superior ”• ” HARMONIE visibility forecasts were excellent ”• ” HARMONIE precipitation and cloud height forecasts were highly useful ”• ” HARMONIE five-panel user interface was really good ”
FMI speciality : Testing of road weather forecasts for Sochi roads under externally funded “CoMoSeF” project
21
The Impact Issue
XXII Olympic Winter Games
22
Future actions and activities Re-run of statistics after full final data sets available
Comprehensive diagnostic verification
Comparison with others’ verification results
Detailed look at the most interesting cases vs. forecast performance
Societal aspects The Impact Issue WMO SERA interests ?
Joint reporting-publishing-presentations, with WMO FROST-2014 contributors WWOSC i.e. here; FROST-2014 October meeting; EMS; AMS etc…
Do same at 2018 Winter Olympics in S-Korea (?)
