Producing Guidance and...
Transcript of Producing Guidance and...
![Page 1: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/1.jpg)
Exercise Part 1 - Producing Guidance and Verification -
1
![Page 2: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/2.jpg)
Objectives
• To understand how to produce guidance for seasonal temperature and precipitation.
• To think rationally and find possible predictors.• To find more effective predictors.• To verify the guidance based on deterministic
and probabilistic methods.
2
![Page 3: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/3.jpg)
Contents
1. Single Regression Model2. Multiple Regression Model3. Probabilistic Forecast4. Verification 4.1 Deterministic Verification 4.2 Probabilistic Verification
3
![Page 4: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/4.jpg)
Preparations
• Observation data (You have prepared.)• Predictors (init. 28th Sep. for DJF)
– GPV data over your stations “GPVdata.xls”– Indices such as NINO3SST, Asia Monsoon, etc.“indices.xls”
• Excel software for making guidance“ProducingGuidance.xls”
• Excel software for verification“Verification.xls” 4
![Page 5: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/5.jpg)
IndicesSST
RAIN
Z500
Thickness
![Page 6: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/6.jpg)
1. Single Regression Model• Open the ProducingGuidance.xls.• Paste observation data into the Temperature/Precipitation worksheet.
Temperature worksheet
Paste observation data.
In this case, the predictand is temperature and the station is Tokyo.
6
“1981 DJF” means DJF of 1980/1981.
![Page 7: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/7.jpg)
A graph of time series of the observation data has been automatically produced.
1. Single Regression Model
7
![Page 8: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/8.jpg)
• Calculate the normal from 1981 to 2010.
1. Single Regression Model
Input “=AVERAGE(C4:C33)” at C34.
8
![Page 9: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/9.jpg)
1. Single Regression Model• Rank the data from 1981 to 2010 in descending order (preparation for probabilistic forecast).
Input “=RANK(C4,$C$4:$C$33,0)” at D4.
9
![Page 10: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/10.jpg)
1. Single Regression Model• Rank the data from 1981 to 2010 (preparation for probabilistic forecast).
Copy D4 and paste it into D5:D33.
10
![Page 11: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/11.jpg)
1. Single Regression Model• Calculate the upper and lower limits of around normal (preparation for probabilistic forecast).
Calculate the mean value of the rank 10th and 11th data (C36)
Calculate the mean value of the rank 20th and 21st data (C35)
11
![Page 12: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/12.jpg)
1. Single Regression ModelOriginal line number
Year MeanTemp
Rank
The upper limit of around normal can be calculated as the average of C23” and “C33”.
The lower limit of around normal can be calculated as the average of “C17”(“C11) and “C24” (“C31)”.
12
![Page 13: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/13.jpg)
1. Single Regression Model• Open GPVdata.xls and indices.xls.• Select a predictor and paste the data into E column.
Surface temperature GPV around at the station as a predictor.
Confirm the anomaly correlation coefficient (the absolute value) at E39.
In case of Tokyo, the anomaly correlation is 0.32 (not so large). I will try other predictors.
Predictors
13
![Page 14: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/14.jpg)
1. Single Regression Model• Try other predictors.
Select the IOBW SST (SST in 20S-20N, 40-100E) and paste the data into E column.
Correlation coefficient has increased to 0.36.
Please try to look for the most effective predictor. 14
![Page 15: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/15.jpg)
1. Single Regression Model• Calculate the forecasts using single regression equation.
Y = a x + bInput “=$E$37 $E4+$E$38” at H4.
15
![Page 16: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/16.jpg)
1. Single Regression Model• Calculate the forecasts using single regression equation.
Copy H4 and paste it into H5:H33.
16
![Page 17: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/17.jpg)
The time series of the forecast data has been automatically added to the graph.
1. Single Regression Model
I will ask you the following questions at the end of this session:1. “What predictor did you select?”2. “Could you get an accurate guidance?”
17
![Page 18: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/18.jpg)
1. Single Regression ModelHow about precipitation?
The histogram of temperature is generally normal (Gaussian) distribution, but that of precipitation is usually Gamma distribution and has gaps from Gaussian distribution. The error distribution of regression model is assumed normal distribution, so normalization is necessary for precipitation. JMA seasonal forecast guidance uses the 1/4-power transformation as the simplest method.
The histogram of raw precipitation data has gaps from Gaussian distribution (left).The histogram of the power of 1/4 of precipitation values is similar to Gaussian distribution (right).
18
![Page 19: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/19.jpg)
1. Single Regression Model
Temperature worksheet Precipitation worksheet
the 1/4-power transformation
the 4th-power transformation
Precipitation worksheet has two additional columns related with “normalization”. Please see the appendix slides to know the detail.
Temperature worksheet Precipitation worksheet
How about precipitation?
19
![Page 20: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/20.jpg)
2. Multiple Regression Model• Try to look for most effective combination of predictors.
Select the most effective combination of predictors after trial and error.
The ITACS will help you find them.
Correlation coefficients (the absolute values)
20
“1981 DJF” means DJF of 1980/1981.
![Page 21: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/21.jpg)
2. Multiple Regression Model
Select REGRESSION_COEFFICIENT
Upload the observation data(In this case, TokyoDJFTemp.csv)
ITACS
Select UPLOAD_TXT
Drawing: CONTOUR
Graphic Option
![Page 22: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/22.jpg)
However, it is necessary to confirm the seasonal forecast model’s skill about them.
2. Multiple Regression Model
ITACS
This is the relationship between OLR and DJF temperature in Tokyo.The OLR (convective activity) in Maritime Continent and that in the eastern Indian Ocean look possible predictors.
![Page 23: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/23.jpg)
2. Multiple Regression ModelPrediction skill evaluated by 30 years hindcast
Anomaly correlation between the observed and forecasted DJF rainfall amounts in Maritime Continent is more than 0.6. Prediction skill is good.
http://ds.data.jma.go.jp/tcc/tcc/products/model/hindcast/cgcm03_vrf/shisuu_10mem12mon_tcc.html
23
![Page 24: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/24.jpg)
2. Multiple Regression Model• Calculate the forecasts using multiple regression equation.
Input“=$E$41*$E4+$F$41*$F4+$G$41*$G4+$E$42”at H4.
Y = a1 x1 + a2 x2 + a3 x3 + b
24
![Page 25: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/25.jpg)
2. Multiple Regression Model• Calculate the forecasts using multiple regression equation.
Copy H4 and paste it into H5:H33.
Multiple Anomaly Correlation Coefficient
25
![Page 26: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/26.jpg)
The time series of the forecast data has been automatically updated.
I will ask you the following questions at the end of this session:1. “What predictors did you select?2. “Is the multiple regression model guidance more accurate
than the single regression model guidance?”
2. Multiple Regression Model
26
![Page 27: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/27.jpg)
3. Probabilistic Forecast• Calculate squares of regression errors. • Calculate the root mean square error.
Input “=($H4-$C4)^2” at I4 to calculate square error.
27
“1981 DJF” means DJF of 1980/1981.
![Page 28: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/28.jpg)
3. Probabilistic Forecast• Calculate squares of regression errors. • Calculate the root mean square error.
Copy I4 and paste it into I5:I33.
28
![Page 29: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/29.jpg)
3. Probabilistic Forecast• Calculate squares of regression errors. • Calculate the root mean square error.
Input “=SQRT(AVERAGE(I4:I33))” at I34 to calculate root mean square error.
29
![Page 30: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/30.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of below-normal.
Input
“=NORMDIST($C$35,$H4,$I$34,TRUE)“ at J4.
TRUE: Value of cumulative distribution function for the first argument (i.e. $C$35).FALSE: Value of probability density function for the first argument (i.e. $C$35).
![Page 31: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/31.jpg)
3. Probabilistic Forecast• Calculate probability of below-normal.
Copy J4 and paste it into J5:J33.
31
![Page 32: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/32.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of above-normal.
Input
“= 1-NORMDIST($C$36,$H4,$I$34,TRUE)“
at L4.
TRUE: Value of cumulative distribution function for the first argument (i.e. $C$36).FALSE: Value of probability density function for the first argument (i.e. $C$36).
Very important!
![Page 33: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/33.jpg)
3. Probabilistic Forecast• Calculate probability of above-normal.
Copy L4 and paste it into L5:L33.
33
![Page 34: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/34.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of around-normal.
Input “=1-J4-L4“ at K4.
![Page 35: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/35.jpg)
3. Probabilistic Forecast• Calculate probability of around-normal.
Copy K4 and paste it into K5:K33.
35
![Page 36: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/36.jpg)
The time series of the probabilistic forecast data has been automatically added.
I will ask you the following questions at the end of this session:1. “How did the guidance predict the hottest/coldest/wettest/driest year in your country?”2. “Are there any essential differences of the probability between the temperature and precipitation?”
3. Probabilistic Forecast
36
![Page 37: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/37.jpg)
•Anomaly Correlation•Root Mean Square Error (RMSE)•Time Series Chart•Scatter Plot
4-1. Deterministic Verification
37
![Page 38: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/38.jpg)
•Anomaly Correlation•Root Mean Square Error (RMSE)•Time Series Chart•Scatter Plot
4-1. Deterministic Verification
Select “Insert” Select “Scatter plot”Select C4:C33 Select H4:H33(with pushing “ctrl” key)
Observation
Forecast
38
![Page 39: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/39.jpg)
3
1
2
1
)(21
m
mi
mi
N
iop
NBS
Brier score measures mean squared error of the probability forecasts.
Brier skill score measures skill relative to a reference forecast (usually climatology).
referenceBSBSBSS 1
Range: 0 to 1. Perfect score: 0
Range: minus infinity to 1. BSS=0 indicates no skill when compared to the reference forecast. Perfect score: 1.
4-2. Probabilistic VerificationBrier skill score (3-category forecast)
39
![Page 40: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/40.jpg)
4-2. Probabilistic Verification
Brier skill score
limlim
lim 10 cc
c
BSBS
BSBSBSBSS
Perfect score: 1BSS>0 : better than the climatology. BSS=0 indicates no skill when compared to the climatology.BSS<0 : worse than the climatology.
= the relative skill of the probabilistic forecast to the climatology
1/3
Brier skill score (3-category forecast)
40
![Page 41: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/41.jpg)
4-2. Probabilistic VerificationBrier skill score (3-category forecast)
Paste observation data (*)
into C2:C31.
Paste observation data (*)
into C32:C61.
Paste observation data (*)
into C62:C91.
•Open the Verification.xls.In this case, the predictand is temperature and the station is Tokyo.
* These values should be copied from C4:C33 in the temperature sheet of the ProducingGuidance.xls.…
…
…
41
“1981 DJF” means DJF of 1980/1981.
If you are busy, please use BriefVerification.xls in place of Verification.xls.*
![Page 42: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/42.jpg)
4-2. Probabilistic Verification
Paste the values of the lower and upper limits of around normal (*) into C93 and C94, respectively.
* These values should be copied from the C35 and C36 cells in the temperature sheet of the ProducingGuidance.xls
Brier skill score (3-category forecast)
…
…
…
42
![Page 43: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/43.jpg)
4-2. Probabilistic Verification
Paste the values of the probabilities of below normal (*) into D2:D31.
Paste the values of the probabilities of around normal (**) into D32:D61.
Paste the values of the probabilities of above normal (***) into D62:D91.
* J4:J33 in the temperature sheet of the ProducingGuidance.xls
** K4:K33 in the temperature sheet of the ProducingGuidance.xls
*** L4:L33 in the temperature sheet of the ProducingGuidance.xls
Brier skill score (3-category forecast)
…
…
…
…
…
…
43
![Page 44: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/44.jpg)
4-2. Probabilistic Verification
2)1(2 mipE
2)0(2 mipE
2)0(32 mipE
2)1(32 mipE
Input “=IF(C2<=$C$93,(D2-1)^2,D2^2)” at E2 (*) and paste it into E3:E31.
If not so (around or above normal),
* If the observed value is below normal,
Input “=IF(C32<=$C$93,D32^2,IF(C32>$C$94,D32^2, (D32-1)^2))”
at E32 (**) and paste it into E33:E61.** If the observed value is below or above normal,
If not so (around normal),
2)1(62 mipE
2)0(62 mipE If not so (below or around normal),
*** If the observed value is above normal,
Input “=IF(C62>$C$94,(D62-1)^2,D62^2)” atE62 (***) and paste it into E63:E91.
Brier skill score (3-category forecast)
…
…
…
…
…
…
…
…
…
44
If you use BriefVerification.xls in place of Verification.xls, the procedure in this slide is unnecessary.
*
![Page 45: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/45.jpg)
4-2. Probabilistic VerificationBrier skill score (3-category forecast)
Input “=AVERAGE(E2:E91)*(3/2)” at E92 (*).3
1
2
1
)o(21
m
mi
mi
N
i
pN
BS
Input “=((1-1/3)^2+(0-1/3)^2+(0-1/3)^2)/2” atF92 (**).
**
*
Input “=(F92-E92)/F92” at G92 (***).
***lim
lim0 c
cBSBSBSBSS
…
…
…
…
…
…
…
…
…
45
If you use BriefVerification.xls in place of Verification.xls, the procedure in this slide is unnecessary.
*
![Page 46: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/46.jpg)
4-2. Probabilistic VerificationReliability Diagram
•Continue using the Verification.xls.
Input “=ROUND(D2,1)” at F2.
* 10% 10%, 7% 10%, 24% 20%, and so on.
Copy F2 and paste it intoF3:F91.
…
…
…
…
…
…
…
…
…
…
…
…
46
If you use BriefVerification.xls in place of Verification.xls, the procedure in this slide is unnecessary.
*
![Page 47: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/47.jpg)
…
…
…
4-2. Probabilistic VerificationReliability Diagram
Input ”=IF($C2<=$C$93,"1","0")” at G2 (*) and paste it into G3:G31.
Input “=IF(C32<=$C$93,"0",IF(C32>$C$94,"0","1"))”
at G32 (**) and paste it into G33:E61.
Input “=IF(C62>$C$94,"1","0")” at G62 (***) and paste it into G63:G91.
* If the observed value is below normal, G2=1.
If not so (around or above normal), G2=0.
** If the observed value is below or above normal, G32=0.
If not so (around normal), G32=1.
*** If the observed value is above normal, G62=1.
If not so (below or around normal), G62=0.
…
…
…
…
…
…
…
…
…
…
…
…
47
If you use BriefVerification.xls in place of Verification.xls, the procedure in this slide is unnecessary.
*
![Page 48: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/48.jpg)
4-2. Probabilistic VerificationReliability Diagram
…
…
…
Copy G2:G91, and paste their values into H2:H91 (functions values).
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
48
![Page 49: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/49.jpg)
4-2. Probabilistic VerificationReliability Diagram
* XXX: 0,10, …,100
(1) Frequency of forecast XXX%(2) Hit frequency of
forecast XXX% =COUNTIF($F$2:$F$91,“XXX%")
=SUMPRODUCT(($F$2:$F$91=J$2)*($H$2:$H$91=$H$1))
(3) Hit rate of forecast XXX%(observed probability)
=J4/J3(4)(3)
(4) Ratio of forecast XXX% compared to the total number of the forecast
=J3/90
49
![Page 50: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/50.jpg)
4-2. Probabilistic VerificationReliability Diagram
Ratio of Forecast XXX% (XXX: 0,10, …, 100)Observed Probability
•The reliability diagram plots the observed probability(Y-axis) against the forecast probability(X-axis).•The diagonal broken line indicates perfect reliability (observed probability equal to forecast probability).•Points below (above) the diagonal line indicate overforecasting (underforecasting).
50
![Page 51: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/51.jpg)
QuestionsSingle Regression Model1. “What predictor did you select?”2. “Could you get an accurate guidance?”Multiple Regression Model1. “What predictors did you select?2. “Is the multiple regression model guidance more accurate
than the single regression model guidance?”Probabilistic Forecast1. “How did the guidance predict the hottest/coldest/wettest/driest year in your country?”2. “Are there any essential differences of the probability between the temperature and precipitation?”Verification1. Is your guidance skillful from the viewpoint of deterministic forecast? (Anomaly Correlation, Root Mean Square Error, Time Series Chart, and Scatter Plot)2. Is your guidance skillful from the viewpoint of probabilistic forecast? (Brier skill score and Reliability Diagram)
Please answer these questions as many as possible in your presentation time (17 minutes) after the Exercise Part 2. 51
![Page 52: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/52.jpg)
Appendix
52
![Page 53: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/53.jpg)
1. Single Regression Model• Open the ProducingGuidance.xls.• Paste observation data, calculate the normal from 1981 to 2010, and calculate the
power of 1/4 of precipitation values.
Precipitation worksheet
(1) Paste observation data.
In this case, the predictand is precipitation and the station is Tokyo.
(2) Input “=AVERAGE(C4:C33)
“ at C34.
(4) Copy D4 and paste it into D5:D34.
(3) Input “=C4^0.25“ at D4.
Appendix: Producing Guidance for Precipitation
53
“1981 DJF” means DJF of 1980/1981.
![Page 54: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/54.jpg)
1. Single Regression Model• Rank the data from 1981 to 2010 in descending order and calculate the lower and
upper limits of around normal (preparation for probabilistic forecast).
(1) Input “=RANK(C4,$C$4:$C$33,0)” at E4.
(2) Copy E4 and paste it into E5:E33.
(3) Calculate the mean value of the rank 20th
and 21st data (C35).
(4) Calculate the mean value of the rank 10th
and 11th data (C36).
(5) Copy D34 and paste it into D35:E36.
Appendix: Producing Guidance for Precipitation
54
![Page 55: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/55.jpg)
1. Single Regression Model• Open GPVdata.xls and indices.xls.• Select a predictor and paste the data into F column
(1) In this case, select IOBW SST and paste the data into F column.
(2) Confirm the anomaly correlation coefficient (the absolute value) between the power of the 1/4 of precipitation value and the selected data at F39.
Please try to look for the most effective predictor.
Appendix: Producing Guidance for Precipitation
55
![Page 56: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/56.jpg)
1. Single Regression Model• Calculate the power of 1/4 of the precipitation value using single regression equation.• Calculate the 4th powers of them. (That is to say, forecasted precipitation values.)
(1) Input “=$F$37 $F4+$F$38” at I4.
(2) Copy I4 and paste it into I5:I33.
(3) Input “=I4^4” at J4.
(4) Copy J4 and paste it into J5:J33.
Appendix: Producing Guidance for Precipitation
56
![Page 57: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/57.jpg)
1. Single Regression ModelAppendix: Producing Guidance for Precipitation
57
![Page 58: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/58.jpg)
2. Multiple Regression Model• Try to look for most effective combination of predictors. after trial and error. The
ITACS will help you find them (see the slides 21-23).
(1) In this case, select IOBW, Z2030, and NINOWEST SST. Paste them to F4:F33, G4:G33, and H4:H33, respectively.
(2) Check the correlation coefficients (the absolute values).
Appendix: Producing Guidance for Precipitation
58
“1981 DJF” means DJF of 1980/1981.
![Page 59: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/59.jpg)
2. Multiple Regression Model• Calculate the power of 1/4 of the precipitation value using multiple regression equation.• Calculate the 4th powers of them. (That is to say, forecasted precipitation values.)
(1) Input “=$F$41*$F4+$G$41*$G4+$H$41*$H4+$F$42”at I4.
(2) Copy I4 and paste it into I5:I33.
(3) Input “=I4^4” at J4.
(4) Copy J4 and paste it into J5:J33.
Appendix: Producing Guidance for Precipitation
59
![Page 60: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/60.jpg)
2. Multiple Regression ModelAppendix: Producing Guidance for Precipitation
60
![Page 61: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/61.jpg)
3. Probabilistic Forecast• Calculate squares of regression errors of the powers of 1/4 of precipitation values . • Calculate the root mean square error.
(2) Copy K4 and paste it into K5:K33.
(1) Input “=(I4-D4)^2” at K4 to calculate the square error.
(3) Input “=SQRT(AVERAGE(K4:K33))” at K34 to calculate root mean square error.
Appendix: Producing Guidance for Precipitation
61
“1981 DJF” means DJF of 1980/1981.
![Page 62: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/62.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of below-normal.
(1) Input
“=NORMDIST($D$35,$I4,$K$34,TRUE)“at L4.
TRUE: Value of cumulative distribution function for the first argument (i.e. $C$35).FALSE: Value of probability density function for the first argument (i.e. $C$35).
(2) Copy L4 and
paste it into L5:L33.
Appendix: Producing Guidance for Precipitation
62
![Page 63: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/63.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of above-normal.
(1) Input
“= 1-NORMDIST($D$36,$I4,$K$34,TRUE)”
at L4.
TRUE: Value of cumulative distribution function for the first argument (i.e. $C$36).FALSE: Value of probability density function for the first argument (i.e. $C$36).
Very important!
(2) Copy N4 and
paste it into N5:N33.
Appendix: Producing Guidance for Precipitation
63
![Page 64: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/64.jpg)
Forecast probability density function (PDF)
3. Probabilistic Forecast• Calculate probability of around-normal.
(1) Input “=1-L4-N4“ at M4.(2) Copy M4 and
paste it into M5:M33.
Appendix: Producing Guidance for Precipitation
64
![Page 65: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/65.jpg)
3. Probabilistic ForecastAppendix: Producing Guidance for Precipitation
65
![Page 66: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/66.jpg)
Anomaly Correlation (AC)
Range: -1 to 1. Perfect score: 1.
• AC measures correspondence or phase difference between forecast and observation, subtracting out the climatological mean at each point.
N
iii
N
iii
N
iiiii
COCF
COCFAC
1
2
1
2
1
)()(
))((
AC <<AC
Appendix: AC and RMSE
66
![Page 67: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/67.jpg)
Root Mean Square Error (RMSE)
Range: 0 to infinity, Perfect score: 0.
• RMSE measures absolute magnitude of the forecast error.
• It does not indicate the direction the error.
N
iii OF
NRMSE
1
21
sizesample:nobservatio
forecast
NOF
::
RMSE largeRMSE small
Appendix: AC and RMSE
67
![Page 68: Producing Guidance and Verificationds.data.jma.go.jp/tcc/tcc/library/library2013/10_ProducingGuidanceA… · Select UPLOAD_TXT Drawing: CONTOUR Graphic Option. However, it is necessary](https://reader036.fdocuments.us/reader036/viewer/2022071021/5fd56560a47373648647d37e/html5/thumbnails/68.jpg)
AC positive ( 1) RMSE large
AC positive( 1)RMSE small
AC negativeRMSE large
AC negativeRMSE small
Appendix: AC and RMSE
68