D2.2.1: Methodology for dense high-‐resolution EO time series, gap filled WP2.2-‐ Time Series of satellite data from multiple satellites in near real
time
Guido D’Urso, Carlo De Michele (Ariespace)
with inputs from Francesco Vuolo, BOKU and Jesús Garrido, UCLM
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 633945.
Ref. Ares(2015)5467399 - 30/11/2015
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
2 | 28
Document Information
Grant Agreement Number 633945 Acronym FATIMA Full Title of Project Farming Tools for external nutrient inputs and water Management Horizon 2020 Call SFS-‐02a-‐2014: External nutrient inputs (Research and innovation Action) Start Date 1 March 2015 Duration 36 months Project website www.fatima-‐h2020.eu Document URL (insert URL if document is publicly available online) REA Project Officer Aneta RYNIAK Project Coordinator Anna Osann Deliverable D2.2.1 Methodology for dense high-‐resolution EO time series, gap filled
Work Package WP2.2 – EO for monitoring plant status and yield
Date of Delivery Contractual 30 November 2015 Actual 30 November 2015 Nature R -‐ Report Dissemination Level PU Lead Beneficiary 04_ARIESPACE Lead Author Guido D’Urso (ARIESPACE) Email durso@unina,it
Contributions from
internal Reviewer 1 Ali Gul (EA-‐TEK) Internal Reviewer 2 Nicos Spyropulos (SIGMA) Objective of document To describe the methodology for deriving dense time series from multi-‐
sensors (operated as in a virtual constellation of available satellites). E.O. data and related products (from Vegetation index to canopy parameters) to reduce the impact of noise, cloud cover, missing data (including Landsat7ETM+SLC-‐off) and to derive smooth curves at regular temporal intervals
Readership/Distribution All FATIMA Regional Teams; All WP leaders and other FATIMA team members; European Commission / REA
Keywords Forecast, EO time series, gap filling, remote sensing, monitoring data, crop growth model
Document History
Version Issue Date Stage Changes Contributor 1.0 3/8/2015 draft Main structure and contents G. D’Urso
1.1 16/9/2015 draft New contents F.Vuolo
1.2 18/9/2015 draft Figure and bibliography adjustments
G. D’Urso
2.0 19/11/2015 draft Text revise for crop growth models G. D’Urso
2.1 24/11/2015 draft Text revised for reviewers comments
G. D’Urso
2.2 27/11/2015 Final draft Contribution from Garrido included G.D’Urso
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
3 | 28
Disclaimer
Any dissemination of results reflects only the authors’ view and the European Commission is not responsible for any use that may be made of the information it contains.
Copyright
© FATIMA Consortium, 2015 This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or
both. Reproduction is authorised provided the source is acknowledged. Creative Commons licensing level
Executive summary
This Deliverable aims at describing the methodology for the derivation of dense time series from multi-‐
sensors Earth Observation (EO.) data and related products (from Vegetation Index to canopy parameters)
to reduce the impact of noise, cloud cover, missing data (including Landsat7ETM+SLC-‐off) and to derive
smooth curves at regular temporal intervals. The procedure to be implemented in FATIMA should take into
account current (e.g., Landsat, Spot, Deimos, RapidEye, Formosat, WorldView-‐2) and new platforms
(including Sentinel-‐2). In an operational system, some of the main constraints are the amount of data to
deal with and the management of cloud-‐cover. Previous experience has shown that even with a five-‐ days
period of return, some areas won’t will not receive enough data to follow crop development. Gap filling,
interpolation and above all, extrapolation, while the system is waiting for incoming data, are needed. We
will examine the various aspects of data lacks and gaps and implement procedures to fill them.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
4 | 28
Table of Contents Executive summary ........................................................................................................................................... 3
1 Problem statement ................................................................................................................................... 6
2 Methodologies for gap filling and interpolation of existing E.O. data and related products .................... 7
3 Methodologies for the extrapolation of E.O. data and related products (forecast) ............................... 22
3.1 Forecast based on curve fitting ....................................................................................................... 22
3.2 Forecast based on crop growth model ............................................................................................ 23
References ...................................................................................................................................................... 26
List of Tables Table 1 : Summary of methods considered in the present study ..................................................................... 8 Table 2 -‐ Parameters of Whittaker Smoother function .................................................................................. 14
List of Figures Figure 1 -‐ By combining data from different sensors, differently affected by various deteriorating processes, we aim at creating a radiometrically uniform multi-‐spectral product with proper spectral signatures and realistic time profiles. ............................................................................................................... 7 Figure 2 -‐ Test for the interpolation procedure in Castilla-‐La Mancha [31] ............................................... 9 Figure 3 -‐ Comparison between actual and interpolated values of NDVI for an irrigated summer crop and for alfalfa [adapted from 31]. ................................................................................................................... 10 Figure 4 -‐ Dense time serie of NDVI derived with INTERPOLA for an irrigated summer crop [31]. ......... 10 Figure 5 -‐ Temporal profiles of forest raw MODIS NDVI data over 8-‐yeas ............................................... 11 Figure 6 -‐ TIMESAT software with sample-‐time series .................................................................................... 12 Figure 7 -‐ Asymmetric Gaussian function ........................................................................................................ 12 Figure 8 -‐ Savitzky – Golay Filter ..................................................................................................................... 13 Figure 9 -‐ Number of iteration for the upper envelope fitting on MODIS NDVI data (Whittaker filtering by means of R package). ...................................................................................................................................... 16 Figure 10 -‐ Scatter plots of Landsat CDR and surface reflectance data (corrected at BOKU) corresponding to Landsat bands 1–5 and 7 (Blue, Green, Red NIR, SWIR-‐1 and SWIR-‐2, respectively) for a set of satellite observations acquired over different seasons over the Austrian pilot area. The broken lines show the ordinary least squares linear regression fits [26]. ........................................................................................... 17 Figure 11 -‐ Two examples of pixel-‐based Landsat time series for the near-‐infrared spectral band. The red dots represent the high quality observations; the green crosses represent the lower quality observation; the red line is the resulting filtered and gap filled series. ............................................................................... 17 Figure 12 -‐ Times series of smoothed and gap filled data of Marchfeld (Austria) for the year 2009. The temporal resolution between gap-‐filled output data is 15-‐days. ................................................................... 18 Figure 13 -‐ Times series of smoothed and gap filled data of Barrax (Castilla la Mancha, Spain) for the year 2003. The temporal resolution between gap-‐filled output data is 15-‐days. ................................................... 19
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
5 | 28
Figure 14 -‐ Example of a time series (2009-‐2015, 15-‐days) for four vegetation indices (left) calculated from the smoothed/gap-‐filled reflectance (right, only near-‐infrared band is shown); Austrian pilot area of Marchfeld, one pixel randomly selected representing a forest land cover type. ........................................... 20 Figure 15 -‐ An example of a time series (2002-‐2015, 15-‐days) for four vegetation indices (left) calculated from the smoothed/gap-‐filled reflectance (right, only near-‐infrared band is shown); Spanish pilot area. one pixel randomly selected representing agricultural land cover type ................................................................ 21 Figure 16 -‐ Crop growth and NDVI for wheat (Calera, presentation at Mammamia45 conference, Enschede (NL), June 2015). ............................................................................................................................................. 22 Figure 17 – Linear extrapolation of the crop coefficient Kc(NDVI) and prediction of the reference ET0 from air temperature data [32]. ................................................................................................................................... 23 Figure 18 -‐ Crop growth model schematisation .............................................................................................. 24 Figure 19 -‐Ensemble LAI forecasted with a lead time of 5 days compared with observed values indirectly estimated from VIS-‐NIR satellite images [16] ................................................................................................. 25
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
6 | 28
1 Problem statement This activity in FATIMA includes the development of the methodology to derive dense time series from
multi-‐sensors (operated as in a virtual constellation of available satellites). E.O. data and related products
(from Vegetation Index to canopy parameters) can be filtered to reduce the impact of noise, cloud cover,
missing data (including Landsat7ETM+SLC-‐off) and to derive smooth curves at regular temporal intervals.
The procedure to be implemented in FATIMA should take into account current (e.g., Landsat, Spot, Deimos,
Geoeye, RapidEye, Formosat, World View 2) and new platforms (including Sentinel-‐2). In an operational
system, some of the main constraints are the amount of data to deal with and the management of cloud-‐
cover. Previous experience has shown that even with a five-‐day period of return, some areas will not
receive enough data to follow crop development. Gap filling, interpolation and above all, extrapolation,
while the system is waiting for incoming data, are needed. The present document will examine various
aspects of data lacks and gaps and implement procedures to fill them.Two main different problems can be
distinguished:
1 gap filling and interpolation of existing E.O. data and related products;
2 extrapolation of E.O. data and related products (forecast).
The problem of type 1), illustrated in Figure 1, has been extensively studied and there are several useful
methodologies that just need to be evaluated in each particular case, ranging from mosaicking to more
complex pixel-‐based compositing and data fusion [17, 18, 19, 20, 22, 23, 24, 25]. The techniques have been
mainly developed for smoothing time series of vegetation indexes such as NDVI, but they can be easily
applied also to surface reflectance data and crop related products, such as LAI, Kc, fractional vegetation
cover. In this case, the same techniques are applied for smoothing and for gap-‐filling problems. In addition,
available techniques do not differ when applied to reflectance data or derived products. However, most of
them have been tested for low-‐medium spatial resolution data i.e. AVHRR, MODIS, Spot Veg., but few
applications exist for finer resolution data (Landsat-‐like or better),
Diversely, the problem of type 2) needs to some extent a forecast of crop growth, and for this reason it can
be more easily afforded by looking at the possible evolution of crop parameters, based on crop growth
models. Forecasting surface reflectance or vegetation indexes is certainly more difficult and inaccurate.
One major output of these activities will be the assessment of uncertainties associated with the elaboration
of each product and the applicability conditions.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
7 | 28
Figure 1 -‐ By combining data from different sensors, differently affected by various deteriorating processes, we aim at creating a radiometrically uniform multi-‐spectral product with proper spectral signatures and realistic time profiles.
2 Methodologies for gap filling and interpolation of existing E.O. data and related products
There are several methods of interpolation for time-‐series of vegetation indexes or surface reflectance
values. Four major classes of methods (Table 1) might be considered:
1. slope methods, including the best index slope extraction technique (BISE);
2. filter-‐based methods, including the Savitzky-‐Golay filter technique and its variants, and the mean
value iteration filter;
3. function fitting methods, such as the Asymmetric Gaussian fitting and the harmonic analysis of time
series (HANTS);
4. smoothing techniques, i.e. the Whittaker smoother.
Comparisons of these techniques have been carried out in several case-‐studies, by using different
indicators of performance. Each method has its own advantages and drawbacks, see for example [12]. New
techniques have been proposed in recent years, and in many cases there is not a rigorous comparative
analysis with other techniques. Almost all comparisons have been based on one sensor.
Besides the choice of the algorithm, it is important in the context of FATIMA to consider which tools are
available for performing these analyses. For this reason, within the categories given in Table 1, the
following techniques are considered for operational implementation in the context of FATIMA:
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
8 | 28
a) Filter techniques based on the Savitzky-‐Golay algorithm and its variants [3];
b) The Whittaker smoother [2].
These methods can be implemented in Matlab and in the open-‐source software package R; a software with
GUI is also available, named TIMESAT, developed by Lund University.
Table 1 : Summary of methods considered in the present study
Category Method Description Reference
1) Slope Interpol. Best-‐Index
slope extraction technique
Compares the current term value with the previous and the next term within a predefined sliding window, and replaces these values with the mean value of the previous and the next values if the percentage difference is greater than a predefined threshold (20%).
[13]
2) Filter based
Savitzky-‐Golay and its variants
Local polynomial fitting of the upper envelope of data series, based on two parameters: the length of the temporal window used and the order of the polynomial. As proposed by Chen et al. (2004), the values of these parameters have to be optimized for each case to get the best match between observations and reconstructed values. In newer variants, the temporal window may be asymmetric and variable in length.
[3]
Mean value iteration
Iteratively compares each date with the average of the dates before and after it, replacing the date with this average if the difference is above a certain threshold. The maximum difference date value will be removed in an iteration process. Iteration will stop when all differences are less than the threshold.
[14]
3) Function
fitting
Asymmetric Gaussian fitting
Fits local, nonlinear functions at intervals around the local maxima and minima, then merges these into a global function describing the full NDVI time series.
[10]
Fast Fourier and Harmonic analysis (HANTS)
Time series are decomposed into sum of sinusoidal functions; once derived phase and amplitudes, these parameters are used for reconstructing and analyzing the data set. [15]
4) Smoothing Whittaker smoother
Based on “penalized” least squares regression, it fits a discrete series to discrete data and penalizes the roughness of the smooth curve. In this way, it balances the reliability of the data and roughness of the fitted data.
[2]
The procedure “INTERPOLA” developed at the University of Castilla-‐La Mancha by Garrido et al. [31] is an
example of the first class of methods “Slope interpolation”. The algorithm is based on the following
equation:
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
9 | 28
( )( )
( )001
010 xx
xxyyyy −
−
−+=
where : § y: interpolated image pixel value § y0: pixel value on the Julian day x0 (first valid acquired image) § y1: pixel value on the Julian day x1 (second valid acquired image) § x: Julian day of the interpolated image § x0: Julian day of the first valid acquired image § x1: Julian day of the second valid acquired image.
This algorithm can be used to replace clouds or shadows, or an interely missing acquisition, and it can be
applied either to reflectance either to derived products (i.e. Kc). The procedure has been validated by using
Landsat images over an area in Castilla-‐La Mancha, falling within the overlap between the orbits paths no.
199 and 200 (row 33). Hence an interpolated image has been generated between two consecutive Landsat
acquisitions (path 199) and compared, for the overlapping portion, with the acquisition over path 200
(fig.2); hence the time difference (x1-‐x0) was 16 days, and the interpolated image was in the middle of this
interval.
Figure 2 -‐ Test for the interpolation procedure in Castilla-‐La Mancha [31]
The results of the comparison are shown in the plots of fig.3, and they are particularly satisfactory for
irrigated crops like maize (R2=0.90); diversely, cuttings falling between the two acquisitions on days x0 and
x1 introduce a large error in the interpolation. For crops which growth follows a monotonic curve, the
interpolation procedure is able of producing series of images at interval of 7 days or better, as shown in
fig.4 for a typical irrigated summer crop.
yoxo
x
x1
Tiem
po
y1
Análisis y evaluación
y
Imagen real -- (199-033)Imagen real -- (199-033)Imagen interpolada -- (199-033)Imagen real -- (200-033)Zona de solape y validación
yoyoyoxo
x
x1
Tiem
po
xo
x
x1
Tiem
po
y1y1
Análisis y evaluación
yyy
Imagen real -- (199-033)Imagen real -- (199-033)Imagen interpolada -- (199-033)Imagen real -- (200-033)Zona de solape y validación
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
10 | 28
Figure 3 -‐ Comparison between actual and interpolated values of NDVI for an irrigated summer crop and for alfalfa [adapted from 31].
Figure 4 -‐ Dense time serie of NDVI derived with INTERPOLA for an irrigated summer crop [31].
In the case of long time series, or for crop growth curves which may divert from a monotonic behaviour it is
needed to adopt more complex procedures, included in the categories 2) to 4) of Tab.1.
To this aim, considering the complexity of operations to be performed, two different software packages
can be considered as candidates for utilization in FATIMA.
The first one is TIMESAT, software originally intended for handling noisy time-‐series of AVHRR NDVI data
and to extract seasonality information from the data. The current version of the program has the capability
of handling different types of remotely sensed time-‐series, e.g. data from Terra/MODIS at different time
resolutions. The data analysis can be carried out by means of Savitzky-‐Golay filter (Chen et al., 2004). The
link for downloading the software (registration required) and further descriptions can be found at:
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
Regadío de Verano
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
para
el 0
1-07
-04,
en
tre la
s im
ágen
es d
el 2
6-06
-04
y 10
-07-
04
Media NDVI, Imagen Real del 01-07-04Media NDVI, Imagen Real
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
Regadío de Verano
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
para
el 0
1-07
-04,
en
tre la
s im
ágen
es d
el 2
6-06
-04
y 10
-07-
04
Media NDVI, Imagen Real del 01-07-04Media NDVI, Imagen Real
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
Alfalfa
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
para
el 0
1-07
-04,
en
tre la
s im
ágen
es d
el 2
6-06
-04
y 10
-07-
04
Media NDVI, Imagen Real del 01-07-04Media NDVI, Imagen Real
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
Alfalfa
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
para
el 0
1-07
-04,
en
tre la
s im
ágen
es d
el 2
6-06
-04
y 10
-07-
04
Media NDVI, Imagen Real del 01-07-04Media NDVI, Imagen Real
Med
ia N
DV
I, Im
agen
Inte
rpol
ada
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
0 50 100 150 200 250 300 350DoY
Val
ores
de
ND
VI
NDVI Real - R.VeranoNDVI Interpolado - R.Verano
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
11 | 28
http://web.nateko.lu.se/timesat/timesat.asp
It is possible to load into TIMESAT time series of image data and plot temporal profiles pixel by pixels as
shown in Figure 5.
Figure 5 -‐ Temporal profiles of forest raw MODIS NDVI data over 8-‐yeas
The interface of TIMESAT appears as follows; the first step is the removal of cutoffs (spikes) has shown in
the Figure 6. Spikes and outliers removal is important to avoid seriously degrading in the final function fits.
In the interface box, the low amplitudes means that only time series with an amplitude higher than a
certain value are processed. This makes sure that uninteresting time series with low variation were not
processed.
The Seasonal fit means the choice of the function or filter technique. They are the Double Logistic or
asymmetric Gaussian (AG) functions (Figure 7) and the Savitzky-‐Golay (SG) filter (Figure 8).
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
12 | 28
Figure 6 -‐ TIMESAT software with sample-‐time series
Figure 7 -‐ Asymmetric Gaussian function
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
13 | 28
Figure 8 -‐ Savitzky – Golay Filter
Envelope adaption involves the fitting of the data. No. of fitting steps means the adaption to the upper
envelope, which is realistic because most noise in NDVI data is, even for clear data, negatively biased. It was
set to ‘3’. Adaption strength is set to ‘1’ and the SG window size: ‘4 5 6’ (the higher the numbers the greater
the gliding window, meaning an increased smoothing but probably less accuracy).
It should be noted that the filtering parameters, once defined, are fixed for the entire image.
The second procedure is based on the R software package, which is a programming environment for
statistical computing and graphics, providing also a wide variety of techniques, including filtering and
smoothing. R is available as Free Software under the terms of the Free Software Foundation’s GNU General
Public License in source code form. The installation package can be downloaded from the following link:
http://www.inside-‐r.org/download/cran
R includes among others the Whittaker smoother. Reference and sample script can be found at:
http://www.inside-‐r.org/packages/cran/pracma/docs/whittaker
It has been shown that Whittaker performs well against a number of other filters based either on curve
fitting or Fast Fourier transform (FFT) [5] and it can be considered as the method allows filtering data
without information on pixel quality [1,2]. In [1], it was shown that this filter permits a significant increase
in the signal-‐to-‐noise ratio (SNR) of coarse resolution VI time series. Besides VI usefulness information, the
smoother also takes into account land/water mask layers of the VI Quality Assessment Science Data Set [9].
This method is based on two assumptions [3]:
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
14 | 28
i. that the time-‐series of vegetation index follows an annual cycle of growth and decline as the index
is primarily related to vegetation density and plant vigor;
ii. that clouds and poor atmospheric conditions produce a negative bias in the vegetation index
values, requiring that sudden drops in vegetation index, which are not compatible with the gradual
process of vegetation change, are regarded as noise and will be removed.
The Whittaker smoother family was firstly presented by Whittaker in 1923 for life tables, based on
penalized least squares. These ideas were revived by Paul Eilers, Leiden University, in 2003. This approach is
also known as Whittaker-‐Henderson smoothing.
Whittaker smoother is based on penalized least squares, fits a discrete series to discrete data and penalizes
the roughness of the smooth curve. In this way, it balances the reliability of the data and roughness of the
fitted data. The smoother takes a time series of observations together with some parameters and outputs
the filtered time series. Whittaker filter for smoothing multi-‐temporal satellite sensor observations with the
ultimate purpose of deriving an appropriate annual vegetation growth cycle and estimating phenological
parameters reliably.
Diversely from other methods, the Whittaker adapt the filtering to each single pixel within the image,
thus providing the maximum adaptability to image itself.
Table 2 -‐ Parameters of Whittaker Smoother function
Lambda smoothing parameter. Allowed are (non-‐integer) values > 0 (scalar)
Weights weights (0 ... 1) used for weighting the time series to be filtered
n°iter scalar, indicating the number of iterations to be performed
Order scalar (integer) order of differences (default = 2).
min val scalar, indicating which values in the input time series are valid
max val scalar, indicating which values in the input time series are valid
min length
minimum number of valid observations in for that a filtering is performed
miss val scalar, used to indicate which values in the input time series are invalid
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
15 | 28
BOX 1: Main Parameters of the Whittaker Smoother in R
The smoothing parameter (λ)
The smoothing parameter (λ) of the Whittaker smoother determines the roughness of the smoothed curve. For the back-‐processing can be fixed, after some trial-‐and-‐error tests, to a constant value for the entire study region; acceptable considering balancing fidelity to the input EO data with the roughness of the resulting curve.
Quality Flags
As already applied with MODIS NDVI data, the operational filtering procedures of high spatial resolution data can take advantage of the quality flag during smoothing, since this information is available for Landsat Level-‐2A product (Landsat 4-‐5 Thematic Mapper (TM), Landsat 7 Enhanced Thematic Mapper Plus (ETM+) and Landsat 8 Operational Land Imager (OLI)) generated by the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS)) and for Landsat Level-‐1T product (Landsat 8). Pixels having a VI usefulness value lower than three were considered to be acceptable and assigned a weight of one (very good to good quality), while a VI usefulness larger than seven was excluded from further processing with a weight of zero (not acceptable). VI usefulness values between three and seven were linearly scaled between one and zero.
Numbers of Iterations
Indicating the number of iterations to be performed. To further reduce the possible impact of undetected clouds and poor atmospheric conditions, three filtering iterations were performed to fit the upper envelope of the VI. Multiple filter runs were for example recommended by [8,10].
Similar to [11], two filtering iterations can be performed to fit the VI data to the upper envelope. Iterative filtering to the upper envelope is recommended, as undetected clouds and poor atmospheric conditions decrease the observed VI.
Order difference
This parameter of the Whittaker smoother can be set after some tests to two. Based on the order difference, the smoother calculates the roughness of the smoothed curve [1].
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
16 | 28
An example of application of the Whittaker filter is shown is fig.9 for MODIS NDVI time series. Preliminary
analyses were also carried out with surface reflectance data and they showed satisfactory results. In
particular, the filter was applied on a time series of the Landsat Surface Reflectance Climate Data Record
archive for some test sites in Austria, Spain and Italy. Landsat CDR is a Landsat Level-‐2A product generated
by the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS) [21]. The data set includes
atmospherically corrected (BOA) Landsat 4-‐5 Thematic Mapper (TM), Landsat-‐7 Enhanced Thematic
Mapper Plus (ETM+) and Landsat-‐8 Operational Land Imager (OLI) data at global level. These sensors have
identical spatial resolutions and a comparable spectral resolution. The Landsat CDR data have
demonstrated to be as accurate as a currently available atmospherically corrected surface reflectance
products that can be obtained using an industrial-‐standard radiative transfer model of the atmosphere (e.g.
ATCOR-‐2) in combination with a detailed study, performed by a trained operator, of the atmospheric
conditions at the time of each satellite acquisition [26]. In Figure 10 a comparison is shown between
Landsat CDR and surface reflectance data independently corrected by using a manual fine-‐tuning of ATCOR-‐
2 parameters to reach the highest possible accuracy.
A similar procedure can be adopted for other sensors and for time series of different sensors, once a cross-‐
calibration has been performed and all image have the same geographical projections, geometrical
resolution and dimensions (number of rows and columns).
Training session might be organised within FATIMA in order to practice the proposed procedures and to
standardise them for the different datasets.
Figure 9 -‐ Number of iteration for the upper envelope fitting on MODIS NDVI data (Whittaker filtering by means of R package).
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
17 | 28
Figure 10 -‐ Scatter plots of Landsat CDR and surface reflectance data (corrected at BOKU) corresponding to Landsat bands 1–5 and 7 (Blue, Green, Red NIR, SWIR-‐1 and SWIR-‐2, respectively) for a set of satellite observations acquired over different seasons over the Austrian pilot area. The broken lines show the ordinary least squares linear regression fits [26].
Figure 11 shows two examples of a pixel-‐based filtering of surface reflectance of Landsat CDR data for the
near-‐infrared band for the period 2009-‐2015.
Figure 11 -‐ Two examples of pixel-‐based Landsat time series for the near-‐infrared spectral band. The red dots represent the high quality observations; the green crosses represent the lower quality observation; the red line is the resulting filtered and gap filled series.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
18 | 28
The approach was applied to all spectral bands to derive RGB colour composites and various vegetation
indices. An example of RGB (Near-‐infrared, Red, Green) is shown in Figure 12 for the Austrian test site of
Marchfeld for the year 2009 (temporal resolution of gap-‐filled data is 15-‐days) and in Figure 13 for the
Spanish test site of Barrax for the year 2003. Tested vegetation indices included NDVI, NDWI, fAPAR and
Tasselled Cup Transformation (TCB). The temporal profile for two exemplary pixels is shown in Figure 14
(from the Austrian dataset) and Figure 15 (from the Spain dataset ), along with the correspondent
reflectance (raw and smoothed/gap-‐filled data) in the near-‐infrared band only.
Figure 12 -‐ Times series of smoothed and gap filled data of Marchfeld (Austria) for the year 2009. The temporal resolution between gap-‐filled output data is 15-‐days.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
19 | 28
Figure 13 -‐ Times series of smoothed and gap filled data of Barrax (Castilla la Mancha, Spain) for the year 2003. The temporal resolution between gap-‐filled output data is 15-‐days.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
20 | 28
Figure 14 -‐ Example of a time series (2009-‐2015, 15-‐days) for four vegetation indices (left) calculated from the smoothed/gap-‐filled reflectance (right, only near-‐infrared band is shown); Austrian pilot area of Marchfeld, one pixel randomly selected representing a forest land cover type.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
21 | 28
Figure 15 -‐ An example of a time series (2002-‐2015, 15-‐days) for four vegetation indices (left) calculated from the smoothed/gap-‐filled reflectance (right, only near-‐infrared band is shown); Spanish pilot area. one pixel randomly selected representing agricultural land cover type
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
22 | 28
3 Methodologies for the extrapolation of E.O. data and related products (forecast)
3.1 Forecast based on curve fitting
As mentioned in the introduction, in the context of applications regarding crop management, the
extrapolation of data (forecast) is possible with regards to the crop parameters i.e. crop coefficients Kc,
fractional vegetation cover, height, Leaf Area Index, based on crop growth models. Crop growth is usually
described by smooth curves of known shape (Figure 16). In particular, it is well known that NDVI follows the
same shape of the crop coefficient Kc. It is possible to establish from actual EO data the current point along
the ideal Kc curve, and forecast the possible evolution for the considered crop. This approach can be used in
conjunction with forecast of reference ET0 to predict crop water requirements for the incoming 5-‐7 days.
In case of a plant growth diverting from the ideal curve, due to water or nutrient stresses, and such
occurrence is evidenced by the first available acquisition, it would be needed to adopt appropriate scaling
procedure of the ideal Kc curve in order to continue correctly the same fitting method. Alternatively,
complex growth model might be applied (see following section). In this case, assimilation or forcing
techniques to integrate E.O data products into the model should be adopted.
Figure 16 -‐ Crop growth and NDVI for wheat (Calera, presentation at Mammamia45 conference, Enschede (NL), June 2015).
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
23 | 28
The method of linear extrapolation of Kc(NDVI) has been applied in a case-‐study in the Castilla La-‐Mancha
region [32]. In this case, the reference ET0 was predicted by using forecast air temperature data for a
period of 7 days and the Hargraves-‐Samani formula. A reanalysis carried out with actual meteorological
data on a weekly basis, a very good agreement was found between the predicted ET0 and the
corresponding calculated by means of the standard FAO Penman-‐Monteith equation (Figure 17).
Figure 17 – Linear extrapolation of the crop coefficient Kc(NDVI) and prediction of the reference ET0 from air temperature data [32].
3.2 Forecast based on crop growth model
An alternative -‐ physically based -‐ procedure could be represented by the implementation of a crop model,
to predict its growth. A simplified crop growth model with the aim to assess the biomass and LAI growth at
daily time step scale can be used for this purpose. In this class of models the main biophysical processes are
conceptualized by a set of simplified analytical relations. A typical approach is the 3PG [30] where the net
primary production (NPP) is modeled according to a light-‐use efficiency approach with a constant carbon
use efficiency factor, similarly to other popular biomass growth models. In short, the LAI dynamics is based
on temperature and leaf dry matter supply, driven by the development stage of the crop. The conceptual
scheme is depicted in Figure 18. In most crop growth models, the main biophysical processes are
conceptualized by a set of simplified analytical relations, with the aim to assess the biomass and LAI growth
with a given time step i.e. daily. The net primary production (NPP) is modelled according to a light-‐use
efficiency approach; the LAI dynamics is based on temperature and leaf dry matter supply, driven by the
development stage of the crop.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
24 | 28
Figure 18 -‐ Crop growth model schematisation
Within the crop growth model, there is an uncertainty related the spatial distribution of soil properties,
initial soil conditions, crop parameters, meteorological forcings. This uncertainty majorly influences the
simulation of two important physiological processes:
1 the simulation of crop canopy development, which determines light interception and
photosynthetic potential;
2 the simulation of soil moisture content, which determines the actual evapotranspiration and the
reduction of photosynthesis as a result of drought stress.
Because the crop growth process has inherent errors, including the errors on initial and boundary
conditions, the forward simulation of the crop model would result in an increasingly enlarged differences
between the simulated and observed results.
E.O. derived LAI can be coupled with crop growth models according to different strategies, i.e. by means of
an Ensemble Kalman Filter. In this case, a recursive Bayesian ensemble-‐based filter estimates the state
variable of a dynamic system from a series of noise corrupted measurements, to mitigate modeling
uncertainty, i.e. to update model state predictions. In this way the model state variables are continuously
updated when remote sensing information is available.
An example of application of a stochastic model is shown in Figure 19, where the assimilation has been
implemented for simulating the growth of the crop above ground biomass and LAI, starting from its seeding
[16]. In Figure 19 it is also possible to notice that the uncertainty on LAI decreases when more observations
are available, thus improving the estimates for the next 5 days.
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
25 | 28
Figure 19 -‐Ensemble LAI forecasted with a lead time of 5 days compared with observed values indirectly estimated from VIS-‐NIR satellite images [16]
Other more sophisticated model can be used while maintaining the same approach, for example:
• CGMS: The Crop Growth Monitoring System (operational crop yield forecasting)
• WOFOST: Within CGMS a version of the WOFOST crop model is implemented, which has been
adapted to the applications at European scale in the Agri4cast EU action.
• CropSyst: A multi-‐crop model for growth simulations, currently applied e.g. in climate change
impact studies
• STICS: internationally recognised as a dynamic, generic and robust model aiming to simulate the
soil-‐crop-‐atmosphere system, developed by INRA [27, 28 29]
• EPICS: crop model developed by Texas A & M Univ. initially to to estimate soil productivity as
affected by erosion and further on expanded to predict effects of management decisions on soil,
water, nutrient and pesticide movements.
The utilisation of these models is required to adequately consider the management of nutrients. For the
implementation and available software, the reader should follow the hyperlinks provided above.
100 150 200 250 3000
0.5
1
1.5
2
2.5
3
3.5
DOY
LAI
Observation5th day forecast
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
26 | 28
References
[1] Atzberger, C. and Eilers, P. H. C., "Evaluating the effectiveness of smoothing algorithms in the absence of ground reference measurements", International Journal of Remote Sensing, 32(13), 3689-‐3709, 2011.
[2] Eilers P. H. C., “A perfect smoother,” Analytical Chemistry, vol. 75, no. 14, pp. 3631–3636, 2003.
[3] Chen, J.; Jönsson, P.; Tamura, M.; Gu, Z.H.; Matsushita, B.; Eklundh, L. A simple method for reconstructing a high-‐quality NDVI time-‐series data set based on the Savitzky–Golay filter. Remote Sens. Environ. 2004, 91, 332–344.
[4] Mattiuzzi ,M., Verbesselt ,J., Hengl ,T., Klisch , A., Evans,B. & Lobo ,A., 2012b: MODIS download and processing package. Processing functionalities for (multi-‐temporal) MODIS grid data. R package version 0.5-‐3/r276. http://R-‐Forge.R-‐project.org/projects/modis (2.7.2014).
[5] P. M. Atkinson, C. Jeganathan, J. Dash, and C. Atzberger, “Inter-‐comparison of four models for smoothing satellite sensor time-‐series data to estimate vegetation phenology,” Remote Sensing of Environment, vol. 123, pp. 400–417, Aug. 2012.
[6] C. Atzberger and P. H. C. Eilers, “A time series for monitoring vegetation activity and phenology at 10-‐daily time steps covering large parts of South America,” International Journal of Digital Earth, vol. 4, no. 5, pp. 365–386, 2011.
[7] Vuolo, F.; Mattiuzzi, M.; Klisch, A.; Atzberger, C. Data service platform for MODIS vegetation indices time series processing at BOKU Vienna: Current status and future perspectives. Proc. SPIE 2012, 8538, doi:10.1117/12.974857.
[8] Beck P. S. a., C. Atzberger, K. A. Høgda, B. Johansen, and A. K. Skidmore, “Improved monitoring of vegetation dynamics at very high latitudes: A new method using MODIS NDVI,” Remote Sensing of Environment, vol. 100, no. 3, pp. 321–334, Feb. 2006.
[9] Solano, R.; Didan, K.; Jacobson, A.; Huete, A. MODIS Vegetation Index User’s Guide (MOD13 Series); Version 2.00, May 2010 (Collection 5); The University of Arizona: Tucson, AZ, USA, 2010.
[10] Jönsson, P.K.; Eklundh, L. Seasonality extraction by function fitting to time series of satellite sensor data. IEEE Trans. Geosci. Remote Sens. 2002, 40, 1824–1832.
[11] Rembold, F.; Atzberger, C.; Savin, I.; Rojas, O. Using low resolution satellite imagery for yield prediction and yield anomaly detection. Remote Sens. 2013, 5, 1704–1733
[12] Hird, J.N., McDermid, G.J. (2009): Noise reduction of NDVI time series: An empirical comparison of selected techniques. Remote Sensing of Environment 113, 248–258.
[13] Viovy, N.; Arino, O.; Belward, A. The Best Index Slope Extraction (BISE): A method for reducing noise in NDVI time-‐series. Int. J. Remote Sens. 1992, 13, 1585–1590
[14] Ma, M.M.; Veroustraete, F. Reconstructing pathfinder AVHRR land NDVI time-‐series data for the Northwest of China. Adv. Space Res. 2006, 37, 835–840
[15] Roerink, G.J.; Menenti, M.; Verhoef, W. Reconstructing cloudfree NDVI composites using Fourier analysis of time series. Int. J. Remote Sens. 2000, 21, 1911–1917
[16] Pelosi A., Falanga S., De Michele C., Medina Gonzalez H., Villani P., D’Urso G., Chirico G.B. 2015. A stochastic ensemble-‐based model to predict crop water requirements from numerical weather
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
27 | 28
forecasts and VIS-‐NIR high resolution satellite images in Southern Italy. EGU General Assembly 2015 , Vienna
[17] Bielski, C., Grazzini, J., & Soille, P. (2007). Automated morphological image composition for mosaicing large image data sets. In International Geoscience and Remote Sensing Symposium (IGARSS) (pp. 4068–4071). doi:10.1109/IGARSS.2007.4423743
[18] Eivazi, A., Kolesnikov, A., Junttila, V., & Kauranne, T. (2015). Variance-‐preserving mosaicing of multiple satellite images for forest parameter estimation: Radiometric normalization. ISPRS Journal of Photogrammetry and Remote Sensing, 105, 120–127. doi:10.1016/j.isprsjprs.2015.03.007
[19] Griffiths, P., van der Linden, S., Kuemmerle, T., & Hostert, P. (2013). A Pixel-‐Based Landsat Compositing Algorithm for Large Area Land Cover Mapping. Selected Topics in Applied Earth Observations and Remote Sensing, IEEE Journal of, PP(99), 1–14. doi:10.1109/JSTARS.2012.2228167
[20] Hermosilla, T., Wulder, M. a., White, J. C., Coops, N. C., & Hobart, G. W. (2015). An integrated Landsat time series protocol for change detection and generation of annual gap-‐free surface reflectance composites. Remote Sensing of Environment, 158, 220–234. doi:10.1016/j.rse.2014.11.005
[21] Masek, J. G., Vermote, E. F., Saleous, N. E., Wolfe, R., Hall, F. G., Huemmrich, K. F., Lim, T. K. (2006). A landsat surface reflectance dataset for North America, 1990-‐2000. IEEE Geoscience and Remote Sensing Letters, 3(1), 68–72. doi:10.1109/LGRS.2005.857030
[22] Roy, D. P., Ju, J., Kline, K., Scaramuzza, P. L., Kovalskyy, V., Hansen, M., Zhang, C. (2010). Web-‐enabled Landsat Data (WELD): Landsat ETM+ composited mosaics of the conterminous United States. Remote Sensing of Environment, 114(1), 35–49. doi:10.1016/j.rse.2009.08.011
[23] Vanonckelen, S., Lhermitte, S., & Van Rompaey, A. (2013). The effect of atmospheric and topographic correction methods on land cover classification accuracy. International Journal of Applied Earth Observation and Geoinformation, 24(1), 9–21. doi:10.1016/j.jag.2013.02.003
[24] Xu, L., Li, B., Yuan, Y., Gao, X., & Zhang, T. (2015). A Temporal-‐Spatial Iteration Method to Reconstruct NDVI Time Series Datasets. Remote Sensing, 7(7), 8906–8924
[25] Zhu, Z., Wang, S., & Woodcock, C. E. (2015). Improvement and expansion of the Fmask algorithm: cloud, cloud shadow, and snow detection for Landsats 4–7, 8, and Sentinel 2 images. Remote Sensing of Environment, 159, 269–277. doi:10.1016/j.rse.2014.12.014
[26] Vuolo, F., Mattiuzzi, M., & Atzberger, C. (2015). International Journal of Applied Earth Observation and Geoinformation Comparison of the Landsat Surface Reflectance Climate Data Record ( CDR ) and manually atmospherically corrected data in a semi-‐arid European study area. International Journal of Applied Earth Observations and Geoinformation, 42, 1–10. doi:10.1016/j.jag.2015.05.003
[27] Brisson N., Mary B., Ripoche D., Jeuffroy M.H., Ruget F., Gate P., Devienne-‐Barret F., Antonioletti R., Durr C., Nicoullaud B., Richard G., Beaudoin N., Recous S., Tayot X., Plenet D., Cellier P., Machet J.M., Meynard J.M., Delecolle R., 1998, STICS: a generic model for the simulation of crops and their water and nitrogen balance. I. Theory and parametrization applied to wheat and corn. Agronomie 18, 311-‐346
D2.2.1: Methodology for dense high-resolution EO time series, gap filled
27/11/2015 V2.2
28 | 28
[28] Brisson N.,Ruget F.,Gate P., Lorgeou J., Nicoullaud B., Tayot X., Plenet D., Jeuffroy M.H., Bouthier A., Ripoche D., Mary B., Justes E., 2002, STICS: a generic model for the simulation of crops and their water and nitrogen balances. II. Model validation for wheat and corn. Agronomie, 22, 69-‐93
[29] Brisson N., Gary C., Justes E., Roche R., Mary B., Ripoche D., Zimmer D., Sierra J., Bertuzzi P., Burger P., Bussiere F., Cabidoche Y.M., Cellier P., Debaeke P., Gaudillere J.P., Maraux F., Seguin B., Sinoquet H., 2003, An overview of the crop model STICS. Eur. J. Agron. 18, 309-‐332
[30] Landsberg J.J., Waring R.H., 1997. A generalised model of forest productivity using simplified concepts of radiation-‐use efficiency, carbon balance and partitioning. Forest Ecology and Management, 95: 209–228
[31] Garrido-‐Rubio, J., Campos, I., Villa, G., Plaza, N., Calera, A., 2011. INTERPOLA. Herramienta para la generación de imágenes sintéticas y sustitución de valores reales con presencia de nubes-‐sombras por valores interpolados, XIV Congreso de la Asociación Española de Teledetección, Mieres, Asturias, Spain, p. 4
[32] Garrido-‐Rubio, J., González, L., Madurga del Cura, C., Arellano, I., Navarro Comalrena de Sobregau, M., López Tapia, J., Calera, A., 2015. Predicción de las necesidades hídricas con una semana de antelación mediante teledetección y meteorología en la finca experimental CENTER para cultivos de Maíz, XXXIII Congreso Nacional de Riegos, Valencia, p. 10
Top Related