A comparison of three learning methods to predict N20 fluxes and N leaching

A comparison of three learning methods topredict N2O fluxes and N leaching

Nathalie Villa-Vialaneix

http://www.nathalievilla.org

Institut de Mathématiques de Toulouse &

IUT de Carcassonne (Université de Perpignan)

March 18th, 2010

UC - Climate Change Unit, Ispra

1 / 25Comparison of metamodels

N

Sommaire

1 Basics about machine learning for regression problems

2 Presentation of three learning methods

3 Methodology and results [Villa-Vialaneix et al., 2010]


N

Basics about machine learning for regression problems

Sommaire





N


Regression

Consider the problem where:

we want to predict Y ∈ R from X ∈ Rd ;

a learning set of N i.i.d. observations of (X ,Y) is already available:(x1, y1), . . . , (xN , yN);

new values of X are available from which we expect to evaluate thecorresponding Y : xN+1, . . . , xM .

Example: Predict N2O fluxes from PH, climate, concentration of Nin rain, fertilization for a large number of HSMU . . .


N


Two different approaches

1 Linear model: A linear relation between X and Y is supposed:

yi = βT xi + ε, i = 1, . . . ,N

β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .

2 Non parametric model: No particular relation is supposed betweenX and Y :

yi = Φ(xi) + ε, i = 1, . . . ,N

where Φ is unknown.

From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.


N


Two different approaches

1 Linear model: A linear relation between X and Y is supposed:

yi = βT xi + ε, i = 1, . . . ,N

β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .

2 Non parametric model: No particular relation is supposed betweenX and Y :

yi = Φ(xi) + ε, i = 1, . . . ,N

where Φ is unknown.From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.


N


Desirable properties for a machine

The adequacy of ΦN is measure through an error function, L .Example: Mean squared error

L(y, y) = (y − y)2

Hence, ΦN should be:

accurate for the learning data set:∑N

i=1 L(yi , ΦN(xi)) has to be

small;

able to predict well new data (generalization ability):∑Mi=N+1 L(yi , Φ

N(xi)) has to be small (but here, yi are unknown!).

These two objectives are somehow contradictory and a tradeoffhas to be found (see [Vapnik, 1995] for further details).


N


Purpose of the work

We focus on methods that are universally consistent. Thesemethods lead to the definition of machines ΦN such that:

EL(ΦN(X),Y)N→+∞−−−−−−→ L∗ = inf

Φ:Rd→RL(Φ(X),Y)

for any random pair (X ,Y).

1 multi-layer perceptrons (neural networks): [Bishop, 1995]

2 Support Vector Machines (SVM): [Boser et al., 1992]

3 random forests: [Breiman, 2001] (universal consistency is notproven in this case)


N

Presentation of three learning methods

Sommaire





N


Multilayer perceptrons (MLP)

A “one-hidden-layer perceptron” is a function of the form:

Φw : x ∈ Rd →

Q∑i=1

w(2)i G

(xT w(1)

i + w(0)i

)+ w(2)

0

where:

the w are the weights of the MLP that have to be learned from thelearning set;

G is a given activation function: typically, G(z) = 1−e−z

1+e−z ;

Q is the number of neurons on the hidden layer. It controls theflexibility of the machine. Q is a hyper-parameter that is usuallytuned during the learning process.


N


Symbolic representation of MLP

INP

UTS

x1

x2

. . .

xd

w(1)11

w(1)pQ

Neuron 1

Neuron Q

OU

TPU

TS

φw(x)

w(2)1

w(2)Q

+w(0)Q 10 / 25

Comparison of metamodelsN


Learning MLP

Learning the weights: w are learned by a mean squared errorminimization scheme :

w∗ = arg minw

N∑i=1

L(yi ,Φw(xi)).

Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.

Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .


N


Learning MLP

Learning the weights: w are learned by a mean squared errorminimization scheme penalized by a weight decay to avoidoverfitting (ensure a better generalization ability):

w∗ = arg minw

N∑i=1

L(yi ,Φw(xi))+C‖w‖2.

Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.

Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .


N


SVM

SVM is also a error loss with penalization minimization:1 Basic linear SVM for regression: Φ(w,b) is of the form

x → wT x + b with (w, b) solution of

arg minN∑

i=1

Lε(yi ,Φ(w,b)(xi)) + λ‖w‖2

whereλ is a regularization (hyper) parameter (to be tuned during thelearning process);Lε(y, y) = max{|y − y | − ε, 0} is an ε-insensitive loss function

See ε-insensitive loss function

2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.

Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)

Tϕ(xj).

Common kernel: Gaussian kernel

Kγ(u, v) = e−γ‖u−v‖2

is known to have good theoretical properties both for accuracy andgeneralization.


N


SVM

SVM is also a error loss with penalization minimization:

1 Basic linear SVM for regression

2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.

Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)

Tϕ(xj).





N


SVM

SVM is also a error loss with penalization minimization:

1 Basic linear SVM for regression

2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)

Tϕ(xj).





N


Learning SVM

Learning (w, b): w =∑N

i=1 αiK(xi , .) and b are calculated by anexact optimization scheme (quadratic programming). The only stepthat can be time consumming is the calculation of the kernel matrix:

K(xi , xj) for i, j = 1, . . . ,N.

The resulting ΦN is known to be of the form:

ΦN(x) =N∑

i=1

αiK(xi , x) + b

where only a few αi are non zero. The corresponding xi are calledsupport vectors.

Tuning of the hyper-parameters, C = 1/λ, ε and γ: simplevalidation has been used. To limit waste of time, ε has not beentuned in our experiments but set to the default value (1) whichensured 0.5N support vectors at most.


N


From regression tree to random forest

Example of a regression tree

|SOCt < 0.095

PH < 7.815

SOCt < 0.025

FR < 130.45 clay < 0.185

SOCt < 0.025

SOCt < 0.145

FR < 108.45PH < 6.5

4.366 7.10015.010 8.975

2.685 5.257

26.26028.070 35.900 59.330

Each split is made such thatthe two induced subsets havethe greatest homogeneity pos-sible.The prediction of a final nodeis the mean of the Y value ofthe observations belonging tothis node.


N


Random forest

Basic principle: combination of a large number of under-efficientregression trees (the prediction is the mean prediction of all trees).

For each tree, two simplifications of the original method areperformed:

1 A given number of observations are randomly chosen among thetraining set: this subset of the training data set is called in-bagsample whereas the other observations are called out-of-bag andare used to control the error of the tree;

2 For each node of the tree, a given number of variables arerandomly chosen among all possible explanatory variables.

The best split is then calculated on the basis of these variables andof the chosen observations. The chosen observations are thesame for a given tree whereas the variables taken into accountchange for each split.


N


Random forest

Basic principle: combination of a large number of under-efficientregression trees (the prediction is the mean prediction of all trees).For each tree, two simplifications of the original method areperformed:

1 A given number of observations are randomly chosen among thetraining set: this subset of the training data set is called in-bagsample whereas the other observations are called out-of-bag andare used to control the error of the tree;

2 For each node of the tree, a given number of variables arerandomly chosen among all possible explanatory variables.

The best split is then calculated on the basis of these variables andof the chosen observations. The chosen observations are thesame for a given tree whereas the variables taken into accountchange for each split.


N


Learning a random forest

Random forest are not very sensitive to hyper-parameters(number of observations for each tree, number of variables foreach split): the default values have been used.

The number of trees should be large enough for the mean squarederror based on out-of-sample observations to stabilize:

0 100 200 300 400 500

02

46

810

trees

Err

or

Out−of−bag (training)Test


N


Learning a random forest

Random forest are not very sensitive to hyper-parameters(number of observations for each tree, number of variables foreach split): the default values have been used.The number of trees should be large enough for the mean squarederror based on out-of-sample observations to stabilize:

0 100 200 300 400 500

02

46

810

trees

Err

or

Out−of−bag (training)Test


N

Methodology and results [Villa-Vialaneix et al., 2010]

Sommaire





N


Data

5 data sets corresponding to different scenarios coming from thegeobiological simulator DNDC-EUROPE.

S1: Baseline scenario (conventional corn cropping system): 18 794HSMU

S2: like S1 without tillage: 18 830 HSMU

S3: like S1 with a limit in N input through manure spreading at 170kg/ha y: 18 800 HSMU

S4: rotation between corn (2y) and catch crop (3y): 40 536 HSMU

S5: like S1 with an additional application of fertilizer in winter:18 658 HSMU

2 outputs to estimate from the inputs:N2O fluxes;

N leaching.


N


Data

11 input variablesN FR (N input through fertilization; kg/ha y);

N MR (N input through manure spreading; kg/ha y);

Nfix (N input from biological fixation; kg/ha y);

Nres (N input from root residue; kg/ha y);

BD (Bulk Density; g/cm3 );

SOC (Soil organic carbon in topsoil; mass fraction);

PH (Soil PH);

Clay (Ratio of soil clay content);

Rain (Annual precipitation; mm/y);

Tmean (Annual mean temperature; C);

Nr (Concentration of N in rain; ppm).

2 outputs to estimate from the inputs:N2O fluxes;

N leaching.


N


Data

2 outputs to estimate from the inputs:

N2O fluxes;

N leaching.


N


Methodology

For every data set, every output and every method,

1 The data set has been split into a training set and a test set (on a80%/20% basis);

2 The machine has been learned from the training set (with a fullvalidation process for the hyperparameter tuning);

3 The performances have been calculated on the basis of the testset: for the test set, predictions have been made from the inputsand compared to the true outputs.


N


Numerical results

MLP RF SVM LMN2O Scenario 1 90.7% 92.3% 91.0% 77.1%

Scenario 2 80.3% 85.0% 82.3% 62.2%Scenario 3 85.1% 88.0% 86.7% 74.6%Scenario 4 88.6% 90.5% 86.3% 78.3%Scenario 5 80.6% 84.9% 82.7% 42.5%

N leaching Scenario 1 89.7% 93.5% 96.6% 70.3%Scenario 2 91.4% 93.1% 97.0% 67.7%Scenario 3 90.6% 90.4% 96.0% 68.6%Scenario 4 80.5% 86.9% 90.6% 42.5%Scenario 5 86.5% 92.9% 94.5% 71.1%

20 / 25Comparison of metamodelsN


Graphical results

Random forest (S1, N2O)

●●●●●●●●●● ●●●

●

●

●●

●●

●

●

●

●

●●

●

●●

●

●

●●●

●●●

●●●

●●●●

●●

●●

● ●●●

●

●●

●

●

●

●●●●

●

●●●

●●

●●●●●●

●

●

●●●

●

●●

●●

●

●

●

●●●

●

●●●

●

●●●

●●●

●

●

●●

●

●

●

●● ●

●

●

●

●

●

●

●

●

●

●

●●●●●

●●●●●

●●

●

● ●●

●●●●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

● ●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●●●●●

●

●● ●

●

●

●●●

●

●

●●●

●

●●● ●

●

●

●

●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●●●

●

●

●

●

●●

●●

●

●●

●●●

●

●●

●●●

●●

●

●

●

●

●●●

●

●

●

●●●

●

●●●●●●

●

●

●

●●●●●●

●

●

●

●

●●

●

●

● ●

●●

●●●

●

●

●

●●●

●

●●●

●

●

●

●

●

●

●●●●●

●

●

●

●

●●

● ●

●●●

●

●

●

●

●

●●

●●

●●● ●

●●●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●●●

●●●●

●

●●

●

●●●

●●

●

●●

●

●●

●●

●

●●

●●●

●●

●●

●

●●

●●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●●●●

●

●●●

●

●

●

●

●●

●●●●

●●●

●●

●

●

●

●

●

●●

●

●

●●

●●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●●●

●●

●●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●● ●

●

●

●●

●●

● ●●

●

●●

●●●●

●●

●

●●

●

●

●

●●●

●

●

●

●●

●

●

●

● ●

●

● ●

●

●

●

●

●

●

●●●●●

●●●

●

●

●

●

●●

●

●●●

●

●

●

●●●●●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●●

●

●

●

●

●●

●●

●

●●

●

●●

●

● ●●

●

● ●

●●

●

●

●

●

●

●

●

●

●●●●

●●

●●

●●

●

●

●●●●

●●●

●

●

●

●●

●

●●●

●●●●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●●

●

● ●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●● ●

●

●

●

●●

●

●

●

●●●

●●

●

●

●●

●●

●

●

●

●

●

●

●●

●●●

●●●

●●● ●●

●

●

●●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●●● ●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●●

●

●

●●

●

●

●

●●

●●

● ●

●●

●●

●

●

●

● ●

●●●

●●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●●●

●

●●

● ●

●

●

●●

●

●

● ●

● ●

●●

●

●

●

●

●

●●

●

●

●

●●●●

●

●●

●

●

●●

●

●●

●●

●

●●●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●●

●

●●

●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●●

●●

●

●

●

●●●

●

●

●

●

●●

●●●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●●

●●

●●●●

●

●●

●

●

●●

●●●●

●

●●●●

●

●●●

●●●

●

●●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

● ●

●

●

●●

●

●

●

●●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●● ●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●●●

●

● ●

●

●

●

●

●

●●●●●

●●

●

●

●

●

●●

●

●●●●●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●●●●●

●●●

●

●●●

●

●●●●●●●

●●

●

●

●

●

●

●

●●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●●●●

●● ●●●

●

●

●

●●

●

●

●●●

●

●

●

●●

●

●●●●●●●●●●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●●●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

● ●

●●

●●

●

●●

●

●●

●

●●

●

●●

●

●

●

●●

●

●●●●

●

●●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●●

●●

●

●●

●

●

●

●

●

●●●

●●●

●

●

●

●●

●●●●

●

●

●

●

●● ●

●

●

●

●●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●●●

●

●

●●

●

●

●

●

●●

● ●

●

●

●

●

●●

●

●●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

● ●

●

●

●

●●

●●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●●●

●●

●●●●

●

●

● ●●●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●●

●●

●

●

●●

●

●●

●

●

●●

● ●

● ●

●●

●●

●

●●

●

●●

●

●

●●●●●

●

●

●

●

●●

●

●

●

●●●

●●

●

●●

●

● ●●

●●

● ●●

●

●

●

●●

●

●

●●

●

●●

●●

●

●●●

●●

●●

●

●●

●●

●

●

● ●

●●●

●

●●

●

●●●●

●●●

●

●

●●

●●

●

●●

●

●● ●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●●

●●

●

●

● ●

●

●

●

●

● ●

●

●

●●

●

●●

●

●

●●●

●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●●

●

●●

●●●●●● ●

●●

●●●●

●●●

●●●●●●●

●●●●

●

●●

● ●

●

●

●

●●●●●

●

●

●

● ●●

●● ●

●

●●

●●●●●●

●

●

●

●●●●

●●

●

●

●●●●

●

●

●

●●

●●●

● ●●

●

●

●●●

●

●

●

● ●●

●●●

●

●●●

●●

●

●●●●●

● ●

●●

●●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●● ●

●●●●

●

●

●●●●●●

●●●●

●●

●●

●

●

●●●●●●●●●

●●

●●

●

●

●

●●●●●

●

●●●●

●●

●●●

●

● ●●●

●

●●

●

●●●●●●●

●●

●

●●●●●

● ●●

●●●●●●●●●

●●● ●

●●●●●●

●●●●

●●

●

●●

●●

●

●

●

●

●●

●

●●

●

●●●

●●●

●●●●

●●●●●●

●

●

●

●●●●

●

●●

●

●●

●●

●●

●●

●

●●●●●

●●

●

●●

●

●●●●●

●

●

●●

●●

●

●

●

●●●●

●

●

●●●●

●

●●

●●●●

●●

●●

●

● ●●

●

●

●

●●

●

●●

●

●●

●

●●

●

●●●

●●

●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●●●●●●●●●

● ●●●

●

●

●

●●

●●●●

●●

●

● ●

●

●

●

●●

●

●

●● ●

●

●

●●

●

●

●●

●●

●

●

●●

●

●

●●

●●●

●

●

●

●

●●●

●

●●

●

●●

●●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●●●

●●

●● ●●

●

●

●

● ●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●●

●●●

●●

●

●

●●●●

●

●

●

●

●●

●●

●●●●●

●

●●

●

●

●

●

●●

●

●●

●●●

●

●●●

●

●

●●

●●●

●

●●●

●

●

●

●●

●●●●

●●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●●

●

●●

●●●

●●

●●●

●

●●●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●●

●●●●

●

●●

●

●

●

●●●●

●

●

● ●

●●

●

●

●

●●●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●●

●●●●

●●●●

●

●

●●●

●●

●●

●

●

●

●●●

●

●

●●

●●

●

●

●●

●●

●●

●

●●●●●

●●●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●●●

●●●

●●●

●

●

●●

●

●●●●●●●●●●●

●●●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●●

●

●●

●

●●

●

●

●●

●

●●

●●●●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

● ●

●

●

●●

●

●●

●

●

●●

●

●●

●●

●

●

●●

●

●●●

●●●●●

●

●

●●

●●

●

●●

●●

●

●

●●●●●●●

●●

●

●●●

●●

●

●●●●

●

●●

●

●

●

●

●●●

●●●

●●

●●

●●

●

●

●

●

●

●

●

●●●

●

●●

●●

●

●●

●

●

●●

●●

●●

●

●

●

●●

●

●

●

●●●

● ●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●●

●●●●●●●

●

●●

●

●●●●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

● ●●

●

●●

●

●

●

●

●

●●

●

●●●●

●

●

●●

●

●

●●

●

●●●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●●●

●●

●●● ●

●●●

●

●●

●

●

●

●●

● ●

●

●●

●

●

●

●

●●●

●

●●● ●

●

●

●

●

●

●●

●●

●

●

●●

●●●

●

●

●

●●

●

●

●●

●●

●

●●

●

●

●●●●●●

●

●●●●

●

●●

●●

●

●●●●

●●

●

●●●

●

●●●●

●

●

●

●●

●

●

●●

●

● ●

●●

●

●

●

●●

●

●

●

●

●

●

●●●●

●

●

●

●

●●

●●

●

● ●●

●

●●

●

●

●

●

●

●

●

●

●

●●●●

●●● ●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

● ●●

●

●● ●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●●●

●●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

● ●●

●●

●

●

●

●

●●

●

●

●●

● ●●

●

●

●

● ●

●

●●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

● ●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●

●

●●

●

●

●

●●

●

●

●●●●

●

●

●●

●

●●●

●

●

●●

●●●

●●●

●

●

●

●

●

●

●

●●●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●● ●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

● ●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●● ●

●

●

●●

●

●

●

● ●

●

●●

●

●

●

●

●

●

●

●

●●●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●● ●

●●●

●● ●●●

●●

●●●

●●

●● ●●

●

● ●●

●

●

●● ●

●

●

●●

●●

●

●●

●●

●

●●

●●

●● ●●●●●●

●●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

● ●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●●●

●

●●

●●

●

●

●

●●

●

●

●●

●●

●●

●

●●

●●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●●●●

●

●●

●

●

●●

●●

●●

●●

● ●●

●●

●

●

●●●

●

●

●

●

●●

●

●

● ●

●

●

●

● ●

●

●

●

●●

●

●●

●●

●

●

●

●●●

●

●

●●●

●

●

●

●●

●●●

●

●

●

●●●●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●●●●

●●

●

● ●●

●

●

●

●

●

●●

●

●●●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●●

●

●●

●

●

●

●

● ●

●●

●

●

●

●

●

●●

●

●●

●●

●●

●

●●

●

●

●

●

●●

●●

●

●●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

● ●

●●●

●●

●●

●

●●●

●

●●

●

●●●

●

●●●

●

●●●

●

●●

●● ●

●●

●●

●●

●

●

●

●●

●

●●●

●●●●

●●

●

●●

●

●

●

●●●

●

●

●

●●

●●●

●●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●●●

●

●●

●●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

● ●

●●●

●

●

●

●

●●● ●●

●

●

●

●●

●

●

●

●

●●

●●●●●●

●

●●

●●

●

●●

●●●●

●●

●●●●●●

●●●●

●

●

●

●●

●

●●●

●●

●●●

●

●●

●

●●●●

●

●●●●

●

●●

●●

●

●

●

●

●

●

●●●

●●●●

●

●

●●●

●●●●●

●

●●●

●●●

●●●

●

●●●●

●●

●

●●

●●●●

●

●●

●●●●●

●●●●●●

●●●

●

●●

●●●●

●

● ●●●●

●●●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●

●

●

●●●

●●●●●

●●

●

●●●●●●

●●●●●●●●

● ●●●●●

●

●●●

●●●●

●

●

●●

●●

●●●

●

●●●●

●

●

●●

●●●

●

●●●

●

●●

●

●

●●

●●

●●

●●

●

●

●

●●●

●●

●●●●● ●●

●●●●●●●

●●●●

●●●

●

●●●

●●●●

●

●

●●●

●●●

● ●

●

●●●●●

●●

●

●●

●

●●

●●

● ●●●●●●

●●●●

●●●

●

●●

●

●●●●●

●●●

●●

●●

●●

●●

●

●●

●

●● ●●

●

●●●

●●

●

●

●

●

●

●●

●

●●●●

●

●●

●

●●● ● ●

●

●

●●●●

●●

●

●●●

●●

●

●

●●

●

●●●

●

●●●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●●●

●●●

●●●

●

●

●●●●

●●●●

●

●●

●●

●●●

●●

●●●●●●●

●●●●

●●●●●●●

●●●

●●●●●●

●●●●

●

●

●

●●

●●●

●

●

●●

●

●

●

●● ●

●●

●●

●

●●

●

●●●●

●●

●●●●

●

●

●●●

●●

●

●

●

●

●

●●●

●●

●

●

●

●●

●

●

●●

●

●

●●

●●●

●●

●●●●●●

●

●●

●●●

●

●

●●

●●●●

●

●●

●

●

●

●

●

●●●

●

●

●●

●●●

●

●●●●

●●

●●●●

●●●●●

●●

●●

●●

●

●●●●

●●

●●

●

●

●

● ●

●●●●

●

●●●

●

●●●

●●●●●●●●

●

●●●

●●

●

●

●●

●●●●

●

●

●●

●

●●●

●

●●●●●

●●●●

●●

●

●●

●●●

●

●●●●●●●

●●●●●●

●●●

●

●●

●

●●

●

●

●

●●●●

●●●

●●●

●●●

●●

●●●

●

●

●

●●

●

●

●

●●● ●

●

●●●●●●●

●

●●●● ●

●●

●

●●● ●

●

●

●

●

●

●● ●●●

●

●

●●●●

●●●●●●●●●●●●●●●●●●●●●

●●●●●

●

●

●●●●●●●

●

●

●●●●●

●●●

●

●

●

●

●

●●

●●

●

●

●●●

●

●

●

●

●●●

●

●

●●●●

●

●●

●

●●

●●●

●

●●

●●

●●

●●●

●●●●

●

●

●●●

●●●●●

●●

●

●

●●

●

●●

●●●●●●

●

●●●●●●●●●●●●

● ●●●

●●●

●●●●●

● ●●●

●●

●●

●●●●●●

●

●●

●●

●●●

● ●●●

●

●●●●

●

●●

●

●

●●●●

●●

● ●

●●

●

●●

●

●

●●●●●●●●●●●●●

●●

●●● ●

●●

●

●●●●

●●●●

●

●

●●● ●●●

●●●●●

●●

●●●●●

●●

●●●

●●

●●

●●●

●

●

●

●

●

●●

●●

●●●

●●

●

●●●

●

●

● ●●●

●

●●●●●●●●●●

●

●●●●●

●●●●

●●●●●

●●●●●

●●●

●●

●●●● ●

●

●●●●● ●

●

●

●●●●

●●●●●

●

●

●

●●●

●●●●

●

●●

●●

●●●●●●

●●

●●

●●

●

●

●●●

●

●●

●

●●●

●●●

●

●

●

●●●●●●●

●●●

●●●

●●●●●●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●●●

●

●

●●

●●●●●●●●●●●●●

●●

●

●●

●

●●●●●

●

●●●●●

●

●

●

●●

●●●

●

●

●

●●●●●●

●●

●●

●●●●●●●●

●

●

●

●●●●●

●

●

●

●●

●●●

●●●●

●

●●

●●

●

●●●

●●●●●●●●●

●

●

●●

●

●●

●

●●

●●

●●

●

●●

●

●

●●●●

●●●●●

●●●

●●●●

●

●

●●●●●

●●●

●●●●●

●●

●

●

●●

●

●

●●●●

●●

●●

●●

●●

●

●●●

●●●●●●●●●●●●●

●●●●●

●●●●

●●

●

●●●●●●

●●

●

●●●●●

●

●

●

●

●

●●●

●●

●●●●

●

●

●●●

●

●●●●●●●

●

●

● ●●●●

●

●●

●

●●●●●●●●

●●●

●●

●

●●

●●●●●

●●

●

●●●

●●

●

● ●

●

●●●●●●●

●

●●●●●●●

●●●●●●●●●

●

●●

●●●●

●●●●●●

●●

●

●●

●●●●●

●

●

●●●●●

●

●●

●

●●

●

●●

●●

●●

●●●

●●

●

●●

● ●

●●●●●

●●●●●●

●●● ●●

●●

●

●

●●●●●

●●

●●●

●●

●

●

●

●

●●●●●●

●●

●●●●●●

●●

●

● ●

●

●●

●

●

●●●

●

●

●●

●

●●● ●

●

●●●●

●

●●●●

●

●

●

●●●●●●

●

●

●●●●●●●●●●●●●●●●●●●●

●●

● ●

●●●●

●●●

●

● ●

●

●

●●●●

●

●

●●●● ●

●●●●

●●●

●●●●●

●●

●●●●●●●●●

●●

●●

●● ●●●●●●●

●●

●●

●●●●●

●

●●●●●

●●

●●

●

●

●●●

●

●●●●●●●

●

●●

●●

●●

●●

●●●●●●

●

●●●

●

●

●

●●

●

●●

●

●

●●●

●

●

●●

●

●

● ●

●

●●

●

●●

●●●

●

●●

●●

●●

●●

●●●

●

●

●

●

●

●●

●●●●●

●●●●●●

●●●

●

●●●●●

●

●

●

●●●

●

●●

●●● ●●

●●●●●●●●

●●●●●●

●

●

●●● ●●●

●●●●

●●

●

●●●● ●●

●●●●●●

●

●●

●●

●●

●●●

●

●

●●

●

●●

●

●●

●●●

●●

●●

●

●●●●

●●● ●●

●●●●

●●

●●●●

●●●

●

●

●

●●

●●●●●

●

●●●

● ●●●●●●●●●●●●●●

●

● ●●●●●●●●

●●●

●●●●●●●

●●

●

●●

●

●●

●

●●●●●●●●●●

●●●●●●

●●●●●●●●

●●

●

●●●●●●●

●

●●●

●

●●●

●●

●●

●●●●

●●

●●●●●

●●●●

●

●

●●

●●●●●●

●●

●

●

●●

●●

●●●

●●

●

●

●●●●

●

●●

●

●●●●

●●

●

●●

●

●●

●

●●●●●●

●●●● ●

●

●●●●●

●

●

●●●

●●

●●

●●●●●

●●●

●●●

●

●●●●

●●●

●

●●●●●

●●●●●

●●●

●●●

●●●

●

●●●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●●●●

●●●●●

●

●●

●●

●

●

●

●●

●

●●

●●●●

●

●●

●

●

●

●●

●●●●●●●

●●

●●●

●●

●●●●●

●●

●●

●●●

●

●●

●

●●●

●●

●●

●●

●

●●●

●●●

●

●●

●

●

●

●●●●

●●●●●●●

●●●●●●●

● ●●●●

●●●●●

●

●●

●●

●●●●●●●●●●●●

●●●

●

●●

●●

●●

●

●●

●

●●

●

●●

●

●

●

●●●●

●●

●●●

●●

●●

●●

●●

●●

●●

●

●●

●

●●●

●

●●

●●

●●

●

●

●●

●●

●●●●

●

●

●

●●

●

●●●●●●●

●●●

●●

●●

●

●●●●

●

●

●

●

●

●●

●●●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●●●

●

●●

●●●●

●●●

●

●●●

●●

●●●

●

●●

●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●

●●●●●●●

●●

●●●●●●●●●●●●●

●

●●

●

●

●

●●●●●●●

●

●●

●●●●●

●

●●●●

●

●

●●

●●

●●●

●●●

●●

●●●●●

●●

●●●●

●●

●

●●

●

●●

●

●●

●●●

●

●●

●●

●

●

●

●

●●

●●●●●●●●●

●

●

●●●●●

●

●

●●●

●●●●●

●

●●●●

●

●

●

●●

● ●●●●●●

●●●●

●

●

●

●

●

●

●●●●

●

●

●●●●●●

●

●●

●

●

●

●●●

●

●

●

●●●

●

●●●

●

●

●

●

●

●●●

●●●●

●

●

● ●●●●

●

●●●

●

●

●●

●

●

●

●

●

●●●●

●

●

●●

●

●

●

●

●●●●

●

●●

●

●●

●●

●●●

●

●●●●●●●

●●●

●

●

●

●

●●●●

●

●●

●

●

●

●

●●

●

●

●●●●

●●

●●●●

●

●●

●

●●●

●●

●●●

●

●●●●

●

●●●●

●●● ●●●●●●

●●

●

●

●●●

●●●●●●●●

●●●●

●●

●

●●

●

●

●●

●

●●

●

●

●●

●

●

●

●●●

●●

●●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●●●

●●●

●

●

●●●●

●

●

●●

●

●

●●

●

●

●●●

●●

●

●●

●●●

●●●●

●●●● ●●

●●

●●

●

●

●

●

●

●●

●●

●

●

●●●● ●●

●

●●●

●●

●

●●

●

●●

●●

●

●

● ●●●

●●●

● ●

●

●

●

●●●

●

●

●●●●●

●

●●●

●●●

●●

●●●

●

●

●

●

●

●

●●●●

●

●

●

●●●●●●●●●●●●

●●●

●●●●●●●

●●

●

●●

●●●

●

●

●

●

●●

●

●●

●

●

●●●

●

●

●

●

●●●

●●●

●

●

●●

●

●

●●

●●●

●

●●●●●●

●●●●●●●●●

●

●●

●●●●●●●●

●●

●●●●●

●●●

●●●●●

●●●

●

●●●●●●

●●●●

●●

●● ●

●●●●

●●●●

●

●

●●

●

●●

●●

●

●●

●

●

●

●●●

●

●●

●●

●●

●●●●●

●

●

●

●●

●●

●

●●●

●●●●●●●●●

●

●●

●●

●

●

●●●●●

●

●●●●

●●●

●

●●●

●●

●●●

●● ●

●●●●

●●

●●

●●

●●

●

●●●

●●●

●●●

●

●

●

●

●●●●●

●● ●

●●●●●●

●

●

●

●

●●●

●●●●

●

●●●●

●

●●

●

●

●●●

●●

●●●●

● ●

●

●●

●

●●●

●

●●

●●●

●●●●●●●

●

●

●

●●●

●

●● ●●●●●

●●● ●

●●●●

●●●

●●

●

●●●

●

●●

●●

●

●

●●●●

●

●●

●

● ●

●●

●

●●

●●●

●●

●●

●●●●●●

●●●

●

●●●●●●

●●●●

●●●●●

●●

●

●

●●

●

●●●

●

●

●

●●

●

●●

●

●

●●

●●●●

●

●

●

●●

●

●●●●

●

●●●●●●

●●

●

●

●●

● ●

●●●

●

●

●

●

●

●●

●● ●

●●●

●

●●●

● ●●●

● ●●●●

●●

●

●

●●

●●

●

●●●●●●

●

●●

●●

●●

●●●●●●●

●

●

●

●●

●

●

●

●●

●

●●

●●●

●

●

●●●

●

●●

●

●

●

●●

●●

●

●

●●●●

●

●●

●

●●●

●

●●●●

●●

●

●●●

●●●

●●

●●

●

●

●●●●

●●

●

●

●

●

●●

●●●●●● ●●

●●●●

●●●●●●●

●●

●

●●●

●●●●

●

●●●

●

●

●

●●●

●●

●

●●

●●●

●●●●●●

●

●●

●

●

●

●●●

●

●●●

●

●●

●

●●●●●

●

●

●

●

●

●●●●

●●

●

●●

●●

●

● ●

●

●●●●

●●●

●

●

●

●●

●

●

●

●●●

●●

●●

●●

●●

●

●

●

●

●●

●

●

●

●●

●

●●●

●

●

●●●●

●

●

●

●● ●●

●●

●

●●

● ●

●

●

●

●●

●

●

●●●

●

●●●

●●●

●●

●●●

●●●

●

●●

●

●●●

●●

●●●

●

●●

●●●●

●●●●

●

●

●●

●●●●●●

●

●●●●●●●●

●●●●

●

●●●

●●●●

●

●

●●

●●

●●●

●

●●

●

●●●●●●●

●●●

●●

●●

●

●

●

●

●●●

●

●●●

●●

●

●

●

●●●●

●

●

●●●●●●

●●●●●

●

●

●●●

●●●●●●●●●

●

●

●

●●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●●●●●●●

●●●●●●●

●●

●●

●●●

●●●

●●

●

●

●

●●●

●●

●

●

●

●

●

● ●

●●●

●

●

●

●

●

●●●

●●

●●●

●

●●●●●●●

●

●●

●●

●●●

● ●

●●●

●

●●●●

●

●

●

●

●●●●

●●●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●●●●●●

●

●

●●●●

●

●

●●●●●

●●

●●●●●

●

●●

●

●●

●

●

●●●●●●●●

●●●

●

●●

●

●●

●●●●● ●●

●

●●

●●●

●

●●●●●●●

●

●

●

●●●

●

●

●●

●

●●●●●●●●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●

●●

●

●

●●●

●●

●●

●

●●

●●

●

●●

●

●

●●●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●● ●●

●●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

● ●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●●●●

●●

●

●●

●●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●●

●

●●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●●●

●

●●

● ●●

●

●●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●●

●

●

●●

●

●● ●●

●●●●●

●●●

●●

●

●●

●

●

●●

●

●

●●

●

●

●●●

●●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

● ●●●

●

●

●

●●●

●

●●

●

●●

●

●

●●

●●

●

●●●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●●●

●●●

●●●●●

●

●

●

●●

●

●●

●

●

●

●

●

●●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●●●

●●

●

●

●●●

●●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●●●

●●●

●●●

●●●●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●●

●●

●●

●

●

●

●

●●●

●●

●●●

●●●●●●●●●

●

●●●●

●●●

●

●

●

●

●●●●

●●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●●

●

●●●●●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●● ●

●

●

●

●●●

●●●

●●

●●

●

●

●

●●

●●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●●●●●

●●●●

●

●●●●●●

●●●

●●

●

● ●

●

● ●

●

●●

●

●●

●

● ●

●

●

●●

●●

●●

●

●

●

●

●●●●

●●●●●●●

●

●

●

●

●

●

●● ●●

●

●●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●●●

●

●●

●●

●

●

●●

●●

●

●●●

●●

●●●●●●●●●

● ●

●

●

●●●

●

●

● ●●●

●● ●●●

●

●

●●

●

●●●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●●

●●

●●

●

●

●●

●●●●

●

●●●●●●●

●●

●

●

●●●

●●

●

●

●●●

●

●●

●

●

●●

●●

●●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●●●●

●

●

●●●

●●●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●

●●●●●

●●

●

●

●

●

●

●

●●

● ●

●●

●

●

●

●●●

●●

●

●

●●

●

●●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●●●●

●●●

●

●●●

●●●

●●●

●

●●●●

●

●

●●

●

●●●●

●●●

●

●●

●

●

●●●

●●●●

●

●●

●

●●●●●

●

●●

●

●

●●

●

●

●

●

●

●●●●●●●

●

●

●

●

●

●●

●

●●●

●●●

●

●●●●

●●●●●●●●●●●

●

●

●

●

●

●●●●

●

●●●●●●

●

●

●

●●

●●●●●

●

●●●●

●●●

●

●

●

●●●

●●●●●●●●

●●●

●●●●●●●●●●●●●●●

●●

●●

●●

●

●

●

●●●●●●

●●

●

●●

●

●

●

●

●

●

●

●●●

●●●

●●

●

●●●

●

●●

●

●●●●

●●●

●

●●●

●

●●●●

●●

●

●

●

●●

●

●●●●

●

● ●

●

●●●

●●●●●●●●●●●●●●

●●

●

●

●●●

●

●

●

●●●

●●

●●●

●●

●

● ●

●

●

●●●

●

●

●●●

●●●

●●

●

●

●●●

●

●

●

●

●●●●

●

●●

●

●

●●● ●●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●●●●●●●

●●●●●●

●

●●●

●

●

●●●

●●

●

●

●●

●

●

●

●

●

●

●

● ●●

●●

●

●

●

●●●●

●

●

● ●●

●

●●●●

●

●

●●●

●

●

●

●

●

●

●●

●●

●

●●

●●

●

●●●●

●●●

●

●

●●

●

●●●● ●●●

●●●

●●

●● ●●

●●

●●●

●

●●●●●●

●●

●●●●

●●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●●

●

●

●●●●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●● ●

●

●●●

●

●

●●

●●●

●

●●●

●●●

●●●●●●

●

●

●●

●●

●

●●●

● ●

●

●

●

●

●

●●

●●

●●●

●

●

●●

●

●●●●

●●●

●

●

●

●

●

●●●

●●

●

●●●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●●●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

● ● ●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●●

●

●●

●●

● ●

●

●

●

● ●

●●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●●●●●

●●

●●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

● ●●●

● ●

●

●● ●

●●

●●

●

●●

●●

●

●

●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●●●●●

●

●●

●

●●

●

●

●●

●

●

●●

●●

●

●

●●●

●

●

●

● ●●●

●●

●

●

●

●

●●●

●

●●

●

●

●●

●

●●

●●

●

●●●●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●

●

●● ●

●

●●●●●

●

●

●●●

●

●●

●

●

●●●●

●●

●

●

●

●●

●

●●

●

●

●

●●●

●●

●●

●

●

●●●

●

●

●●●●●

●

●●●

●●

●●

●

●●●●●

●

●

●●●●

●

●

●

●●

●

●

●

●

●●●

●

●●

●

●

●●

●

●●

●

●

●●

●

●

●

●●

●

●●

●

●●

●

●

●●

●

●

●●

●●

●●

●●

●●

●

●

●

●●●●●●

●●

●

●●●●●●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●●

●

●

●●

●●●

●●

●●

●●●

●●

●

●●

●

●

●●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●

●●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●

●●●

●●●●●

●●●●●●

●●

●●

●

●

●●

●

●

●●

●●

●●

●

●●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●● ●

●●●●

●

●●●

●●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●●●●●●

●

●

●

●

●

●●

●

●●●●●●●●●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●●●●●

●

●

●

●●●●

●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●●

●●●

●

●●

●

●●

●●

●

●●

●

●

●

●

●●●●

●

●●

●●●

●

●●●●●

●

●

●

●●

●

● ●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●●●

●●

● ●

●●

●

●●●

●

●●

●

●

●

●●●

●●

●

●

●●●●●

●

●●

●

●

●●●

●●

●

●

●

● ●●●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

● ●

●

●

●●

●

●● ●

●

● ●

●

●

●●

●

●

●

●●●

●●

●

●

●

●●

● ●

●

●●

●●●

●●●

●●

●

●

●●

●●●●●

●

●

●

●

●

●

●●

●

●

●●●●

●

●●

●●

●

●

●

●●●

●

●●

●

●

●●●

●

● ●

●●

●●●●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●●

●●●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●●

●

●●

●

●●●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●●

●

● ●

●

●●

●

●●●●●●

●●●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●●●

●●●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●●

●

●

● ●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●●

●

●●

●

●

●

●●●●

●

●

●●

●●

●●

●

●●

●●

●

●●

●

●

●●

●

●●●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●●

●●

●●

●

●

●

●

●●

●●●●

●●●●

●

●●

●●●●●

●

●

●●●

●●●●

●

●

●●

●

●●●●●●

●

●

●●

●

●●

●

●

●

●●

●

●●

●

●

●●●

●●

● ●

●

●●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

● ●●

●

●

●

●

●●●

●

●

●

●●

●●

●●●

●

●

●

●●

●

●●

●

●

●●

●

●●●

●

●●

●●

●

●●●

●●

●●

●

●

●

●

●

●●

●

●●

●

●

●●●

●

●

●

●●●

●

●

●●●●●

●

●●

●

●●●●

●●

●●

●●

●●●

●

●●

●●

●

●

●

●

●●

●●●

●

●

●

●●●●●

●●

●

●

●●●●●●●

●

●●

●

●

●

●

●●●

●

●●●●

●

●

●●

●

●

●●

●

●●●●●

●

●●

●●●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●● ●●

●

●

●●

● ●●

●●

●●●

●●

●●●

●

●●●

●●

●

●●●●

●●

●

●

●

●

●

●

●●●

●●●●●

●●

●

●●

●●

●

●

●

●●●●

●●

●

●

●

●

●

●●

●

●

●●●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●●●

●●●

●

●●

●

●

●●

●●

●

●

●

●●

●

●●

●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●●

●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●●●●

●

●

●

●

●

● ●

●

●

●●●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●●● ●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●

●

●●●

●●●

●

●●

●●

●●●

●●

●●●●

●

●●

●

●

●●

●●

●●●

●

●●

●

●

●●

●

●

●

●

●

●●●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●●●●

●●●

●●●●●

●●

●●

●

● ●

●●

●

●●

●

●

●

●

●●

●●

●

●●

●

●

●●●●●

●● ●

●●●

●●●

●●

●

●●

●●

●

●●

●

●●●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●●

●●●

●●

● ●

●

●

●

●●

●

●

●

●●

●

●●

●

●●●

●

●●

●

●

●●●

●

●●

●

●

●

●●

●●

●

●

●

●●

●

●

●●

●●

●

●

●●●

●

●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●

●●

●●

●

●● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●●

●

● ●

●

●

●●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●●

●●●●●

●

●

●●

●

●

●

●●

●

●

●●●

●

●

●●

●

●●●

●●

●● ●

●

●●●●●

●

●●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●●

●

●

●● ●

●

●●

●

●●●

●

●●

●●

●●●●

●

●

●

●● ●

●●

●

●

●●

●●●

●●

●

●

●●●●●●●●

●

●●

●

●

●

●

●●●

●●

●●●●●●●

●

●●●

●

●

●●●

●● ●

●

●●

●● ●●

● ●●

●

●●●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●

● ●●

●

●

●●

●

●●

●●

●

●

●

●

●●

●

●●

●

●●●●●

●

●●●

●

●●

●●

●

●●

●●

●

●

●●●●●●

●●

●

●●

●

●●●

●

●●

●●

●●

●

●

●

●●●●

●

●●

●●●●

●

●

●

●●

●

●

●

●●

●

●●●

●●

●

●●

●

●

●

●●

●

●

●●●

●●●

●●

●

●

●●

●●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●●●●●●●

●

●

●

●●

●

●

●

●

●

●●●●

●

●●●

●

●

●

●

●●

●

●●

●

●●

●●

●

●

●

●

●●

●●●

●

●●●

●●●

●● ●

●

●

●

●●

●

●

●

●

●●

●

●●●

●

●

●

●

●●

●

●●●

●●●●●●●●●

●●●

●●

●●●

●

●

●

●

●

●

●● ●

●●

●●

●

●

●

●

●

●

●●●

●●

●

●

●●●●●

●●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●●

●

●●●

●●●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●● ● ●

●●●●

●

●●

●

●

●

●●

●●●●●

●

●●

●

●

●●

●

●●

●

●

●

●●●

●

●●

●●

●●●

●

●

●

●

●

●

●●

●●●●●

●●

● ●●

●●

●●●

●●●

●●

●

●

●

● ●●

●●

●

●● ●●● ●

●●

●

●

●

●●●

●

●

●●

●

●● ●●

●●

●●●●

●

●●●

●

●

●●●●

●●

●

●

●●●

●

●●

●●

● ●

●● ●

●

●

●●

●

●● ●●

●

●

●

●

●

●

●

●● ●

●

●

●

●

●

●

●●

●●●

●

●

●●

●

●●●●

●

●

●●

●

●

●●● ●

●

●

●

●

●●

●●●●

●●●

●

●

●●●

●

●

●

●●

●

●●

●

●●●

●●

●

●●

●

●●

●

●●

●

●

●●

●

●

●

●●

●

●●●●

●

●●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●●

●●●

●

● ●●

●●●●●●●

●●

●

●

●●●●●●●

●●

●

●●

●

●

●

●

●●●●●

●●

●●

●●●

●

●●●●

●●

●●

●●

●

●●

●

●●●

●●

●●

●

●●●●

●●

●

●●●●

●●●

●

●●

●

●

●●

●

● ●●

●●●

● ●

● ●

●

●

●●

●

●

●●

●●

●

●●●

●

●●●●●

●

●

●

●●

●

●●●●

●●●

●●

● ●

● ●●●

●●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●

●●

●●

●

●

●

●

● ●

●●

●

●

●●●●●●●

●

●●●

●●

●

● ●

●●●

●

●●●●●

●●

●●

●

●

●

●●

●●

●●

●●

●

●●

●●

●

●

●●

●●●●●●

●●

●●

●

●

●

●

●●●

●

●●

●

●

●●

●●●●●●●

●●●●●

●

●●●

●●

●

●●●●●●●●●●●●●●

●●●●●● ●●●●●●

●●

●●●●

●

●

●●●●

●●

●

●

●

●

●

●●●

●●●●●

●●

●

● ●●

●● ●

●

●

●●

●●●●●●●●

●●

●●●●●●●●●●●

●

●●●●●

●●● ●

●●●●

●

●

●

●

●

●

●●● ●●●●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●●

●

●●

●

●●●●

●

●●

●

●

●

●●

●

●

●

●●●

●●●

●●

●

●●

●

●●

●

●

●

●

●●

●●●●

●

●●

●

●

● ●●●

●

●

●

●●

●

●

●

●

●●

●

●●●

●

●●●

●●

●

●

●●

●●

●

●

●●●●

●●

●

●

●

● ●

●

●●● ●●

●●

●●●●

●●●●●

●

●

●●

●●●●

●

●

●

●●

●●●

●●

●

●

●

●●●●●●●●●

●●

●

●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

● ● ●

●●●●

●

●

●●●●

●

●

●

●●●●●●

●●

●●

●

●

●

●

●

●●●

●

●●●●

●

●

●

●

●

●

●

●●●

●●

●●

●

●●

●

●●

●

●●

●

●●

●●

●

●●

●●

●●●

●

●

●

● ●

●

●

●

●

●

●●

●

●●●

●●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●

●

●

●●●●●●●

●●

●●●

●

●

●

●

●

●

●

●●

●

●

●●●

●●●

●●●

●● ●●●

●●

●

● ●

●●

●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●●●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●● ●

●●

●

●

●

●●

●

●

●

●●●

●

●●●

●

●●●●

●

●●

●●●

●

●

●

●●●●●

●●

●●

●

●

●

●●●

●●

●

●

●

●

●

● ●

●

●●

●

●

●●

● ●

●

●●●

●

●

●

●

●

●

●

●

●

●●●● ●●

●

●●

●

●

●● ●●

●●●

●

●●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●●●

●●●

●

● ●

●

●

●

●

●

●

● ●

●●●

●

●

●

●

●

●

●●

●● ●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●●

●

●●

●

●●

●

●●

●●●

●

●

●

●

●

●

●●

● ●

●●

●

●

● ●●

●●

●

●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●●●

●

●

●

●●

●●●

●

●

●●●●●●

●●

●●

●

●

●●

●

●●●

●

●●●●●

●

●●●●

●●

●●●

●

●●●●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●●

●●

●

●●

●

● ●●●

●

●

●

●●●

●

●

●●

●

●

●●●

●●●●●● ●●

●●

●

● ●●●

●●

●

●

●

●

●

●

●

●

●●

●

● ●●

●

●● ●

●●

●

●

●

●

●

●

●

●

●●

●●

● ●

●

●

●

●

●

●

●●●

●

●●

●

●●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●●

●

●●

●●

●●

●

●●●

●

●●●

●

●

●●●●●●●●

●

●●●

●●●●●

●●●●●●●●●●●●●

●●●●●

● ●

●

●●

●

●

●●●

●

●

●

●●●

●

●

●

●

●●

●

●

●●

●

●

●

●●●

●

●

●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●●

●●

●

●●

●●●●●●

●●●

●

●●

●

●●

●

●●●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

● ●

●

●●●●●

●

●

●

●●

●● ●● ●●●

●

●

●●

●●●●

●

● ●●

●●● ●

●●

●

●

●

●●

●

●●

●

●●

●

●●●

●

●●●

●●●●●

●●●●

●

●●●●●●●

●

●

●●

● ●

●●●

●

●●●●●

●●

●

●●

● ●

●●

●●

●

0 20 40 60 80

010

3050

70

True value (train)

Pred

icted

val

ue (t

rain

)

●● ●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●●●

●

●

●

●

●

● ●

●

●●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●●

●●●●

●●●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●● ●

●

●

●●

●

●●

●

●

●

●●●

●

●

● ●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●●●

●

●●

●●

●

●

●

●●●

●

●

●

●●

● ●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●●●

● ●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●●●

●

●

●

●

● ●

●

●

●

●

●

●

●

●●●

●

●

● ●

●

●

●

●

●

●●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●●

●●

●

●

●● ●●

●

●●

●●

●

●●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●●

●

●

●

●

●●

●●

●

●●

●●

●

●●

●●●●

●

●

●●

●●

●

●

●

●

●●

●

●●● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●● ●

●

●

●

●● ●

●●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●●

●●

●

●●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●●

●

●

●● ●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●● ●

●●

●

●● ●

●

●●●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

● ●

●

●

●●

●

●

●

●

●●

●

●

●● ● ●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●●

●

●

●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●●●

●

●●

●●●

●●

●

●

●●●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

● ●●●●●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●●

●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

● ●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●

●●●

●

●●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●●●● ●

●

●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

● ●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●●●●

●

●

●●●

●

●●●●

●

●

●

●

●

●●

● ●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●●

● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●

●

● ●

●

●

●

●● ●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●●

●●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

● ●●

●

●●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●●

●

●

●

●

● ●●

●

●●

●●

●

●

●●

●

● ●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

● ●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●●● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●● ●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●●

●

●

●●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

● ●●●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●● ●●

●●

●

●

●

●

●

●

●

●

●

●●●●

● ●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●● ●

●

● ●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●

●●

●

●●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●●●

●

●

●●

●

● ●

●

●●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

● ●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

● ●●

●●●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●●

●● ●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●●

●

●

●●

●

●

0 10 20 30 40 50 60 70

010

3050

True value (test)

Pred

icted

val

ue (t

est)

Linear model (S1, N2O)

0 20 40 60 80

010

2030

4050

True value (train)

Pred

icted

valu

e (tr

ain)

0 10 20 30 40 50 60 70

010

2030

4050

True value (test)

Pred

icted

valu

e (te

st)


N


Further comparisons

Time for training Time for using once the model is trainedRF + (∼ 15 minutes) = (∼ 5 seconds)SVM = - (∼ 20 seconds)

sensitive to NMLP - + (∼ 1 second)

sensitive to d

Times correspond to the prediction for about 20 000 HSMU on aDell Latitude D830, Intel Core 2DUO 2.2GHz, 2GO RAM, OSUbuntu Linux 9.10 (and, for RF, to the training of about 15 000HSMU) with R.Time for DNDC: around 200 hours by using a desktop computerand around 2 days by using cluster 7!


N


Understanding which inputs are impor-tant

Importance: A measure to estimate the importance of the inputvariables can be defined by:

for a given input variable randomly permute the input values andcalculate the prediction from this new randomly permutated inputs;

compare the accuracy of these predictions to accuracy of thepredictions obtained with the true inputs: the increase of meansquared error is called the importance.

Example (S1, N2O, RF):

●

●

●●

●●

●●

●●

●

1 2 3 4 5 6 7 8 9 10 11 12

510

1520

2530

Rank

Impo

rtan

ce (

mea

n de

crea

se m

se)

SOC

PH

Nr N_MRNfixN_FR

clay Nres TmeanBD rain

The variables SOC and PH are the most important for accuratepredictions.


N



Importance: A measure to estimate the importance of the inputvariables can be defined by:

for a given input variable randomly permute the input values andcalculate the prediction from this new randomly permutated inputs;

compare the accuracy of these predictions to accuracy of thepredictions obtained with the true inputs: the increase of meansquared error is called the importance.

This comparison is made on the basis of data that are not used todefine the machine, either the validation set or the out-of-bagobservations.


●

●

●●

●●

●●

●●

●

1 2 3 4 5 6 7 8 9 10 11 12

510

1520

2530

Rank

Impo

rtan

ce (

mea

n de

crea

se m

se)

SOC

PH

Nr N_MRNfixN_FR




N




●

●

●●

●●

●●

●●

●

1 2 3 4 5 6 7 8 9 10 11 12

510

1520

2530

Rank

Impo

rtan

ce (

mea

n de

crea

se m

se)

SOC

PH

Nr N_MRNfixN_FR




N


Understanding the relation between agiven input and the output

Evolution of N2O in function of the value of SOC

0.00 0.05 0.10 0.15 0.20 0.25

510

1520

25

SOC

N2O

pre

dict

ed

All the other variables are set to their mean values (S1, RF).


N

Bishop, C. (1995).Neural Networks for Pattern Recognition.Oxford University Press, New York.

Boser, B., Guyon, I., and Vapnik, V. (1992).A training algorithm for optimal margin classifiers.In 5th annual ACM Workshop on COLT, pages 144–152. D. Haussler Editor, ACM Press.

Breiman, L. (2001).Random forests.Machine Learning, 45(1):5–32.

Vapnik, V. (1995).The Nature of Statistical Learning Theory.Springer Verlag, New York.

Villa-Vialaneix, N., Follador, M., and Leip, A. (2010).A comparison of three learning methods to predict N2O fluxes and N leaching.Submitted to MASHS 2010 http://mashs2010.free.fr.


N

http://mashs2010.free.fr


ε-insensitive loss function

Go back


N

A comparison of three learning methods to predict N20 fluxes and N leaching

Science

Transcript of A comparison of three learning methods to predict N20 fluxes and N leaching