Investigating the Local-Meta-Model CMA-ES forLarge Population Sizes

Zyed Bouzarkouna1,2 Anne Auger2 Didier Yu Ding1

1IFP (Institut Francais du Petrole)

2TAO Team, INRIA Saclay-Ile-de-France, LRI

April 07, 2010

Statement of the Problem

Objective

To solve a real-world optimization problem formulated in ablack-box scenario with an objective function f : Rn 7→ R.

f may be:

multimodal non-smoothnoisy non-convexnon-separable computationally expensive. . .

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 2 of 15

A Real-World Problem in Petroleum Engineering

History Matching

The act of adjusting a reservoir model until it closely reproducesthe past behavior of a production history.

A fluid flow simulation takes several minutes to several hours !!

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 3 of 15

Statement of the Problem (Cont’d)

Difficulties

Evolutionary Algorithms (EAs) are usually able to cope withnoise, multiple optima . . .

Computational cost

build a model of f , based on true evaluations ;

use this model during the optimization to save evaluations.

⇒ How to decide whether:

the quality of the model is good enough to continueexploiting this model ?

ornew evaluations on the “true” objective function should be

performed ?

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 4 of 15

Table of Contents

1 CMA-ES with Local-Meta-ModelsCovariance Matrix Adaptation-ESLocally Weighted RegressionApproximate Ranking Procedure

2 A New Variant of lmm-CMAA New Meta-Model Acceptance Criterionnlmm-CMA Performance

3 Conclusions

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 5 of 15

Covariance Matrix Adaptation-ES

CMA-ES (Hansen & Ostermeier 2001)

Initialize distribution parameters m, σ and C, set population sizeλ ∈ N.while not terminate

Sample xi = m + σNi (0,C), for i = 1 . . . λ according to amultivariate normal distribution

Evaluate x1, . . . , xλ on f

Update distribution parameters(m, σ,C)← (m, σ,C, x1, . . . , xλ, f (x1), . . . , f (xλ))

where

m ∈ Rn: the mean of the multivariate normal distribution

σ ∈ R+: the step-size

C ∈ Rn×n: the covariance matrix.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 6 of 15

Covariance Matrix Adaptation-ES (Cont’d)

Moving the mean

m =µ(=λ

2 )∑i=1

ωixi :λ.

where xi :λ is the i th ranked individual:

f (x1:λ) ≤ . . . f (xµ:λ) ≤ . . . f (xλ:λ) ,

ω1 ≥ . . . ≥ ωµ > 0,µP

i=1ωi = 1.

Other updates

Adapting the Covariance Matrix

Step-Size Control

⇒ Updates rely on the ranking of individuals according to f andnot on their exact values on f .

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 7 of 15

Locally Weighted Regression

q ∈ Rn: A point to evaluate

⇒ f (q) : a full quadratic meta-model on q.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 8 of 15

Locally Weighted Regression

A training set containing m points with their objective functionvalues (xj , yj = f (xj)) , j = 1..m

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 8 of 15

Locally Weighted Regression

We select the k nearest neighbor data points to q according toMahalanobis distance with respect to the current covariance matrixC.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 8 of 15

Locally Weighted Regression

h is the bandwidth defined by the distance of the kth nearestneighbor data point to q.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 8 of 15

Locally Weighted Regression

Building the meta-model f on q

mink∑

j=1

[(f (xj , β)− yj

)2ωj

], w.r.t β ∈ R

n(n+3)2

+1.

f (q) = βT(q2

1 , · · · , q2n, · · · , q1q2, · · · , qn−1qn, q1, · · · , qn, 1

)T.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 8 of 15

Approximate Ranking Procedure

Every generation g , CMA-ES has λ points to evaluate.

⇒ Which are the points that must be evaluated with:

the true objective function f ?

the meta-model f ?

Approximate ranking procedure (Kern et al. 2006)

1 approximate f and rank the µ best individuals

2 evaluate f on the ninit best individuals

3 for nic := 1 to“λ−ninit

nb

”do

4 approximate f and rank the µ best individuals

5 if (the exact ranking of the µ best individuals changes) then

6 evaluate f on the nb best unevaluated individuals

7 else

8 break

9 fi

10 od

11 adapt ninit depending on nic

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 9 of 15

A New Meta-Model Acceptance Criterion

Requiring the preservation of the exact ranking of the µ bestindividuals is a too conservative criterion to measure the quality ofthe meta-model.

New acceptance criteria (nlmm-CMA)

The meta-model is accepted if it succeeds in keeping:

the best individual and the ensemble of the µ best individualsunchanged

or

the best individual unchanged, if more than one fourth of thepopulation is evaluated.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 10 of 15

nlmm-CMA Performance

Success Performance (SP1):

SP1 = mean (number of function evaluations for successful runs)ratio of successful runs .

Speedup (algo) = SP1(algo)SP1(CMA−ES) .

0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 11 of 15

nlmm-CMA Performance

4 nlmm-CMA � lmm-CMA

fSchwefel fSchwefel1/4 fNoisySphere

(2, 6) (4, 8) (8, 10) (16, 12)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 6) (4, 8) (5, 8) (8, 10)0

2

4

6

8

Sp

eed

up

(Dimension, Population Size)(2, 6) (4, 8) (8, 10) (16, 12)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

fRosenbrock fAckley fRastrigin

(2, 6) (4, 8) (5, 8) (8, 10)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 5) (5, 7) (10, 10)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 50) (5, 140)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

⇒ nlmm-CMA outperforms lmm-CMA, on the test functions investigated

with a speedup between 1.5 and 7.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 11 of 15

nlmm-CMA Performance for Increasing PopulationSizes

4 nlmm-CMA � lmm-CMADimension n = 5

fSchwefel1/4 fRosenbrock fRastrigin

8 16 24 32 48 960

1

2

3

4

5

Population Size

Sp

eed

up

8 16 24 32 48 960

1

2

3

4

5

Population Size

Sp

eed

up

70 140 2800

1

2

3

4

5

Population Size

Sp

eed

up

⇒ nlmm-CMA maintains a significant speedup,between 2.5 and 4, when

increasing λ while the speedup of lmm-CMA drops to one.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 12 of 15

Impact of the Recombination Type

nlmm-CMA

a default weighted recombination type

ωi = ln(µ+1)−ln(i)µ ln(µ+1)−ln(µ!) , for i = 1 . . . µ.

nlmm-CMAI

an intermediate recombination type

ωi = 1µ , for i = 1 . . . µ.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 13 of 15

Impact of the Recombination Type (Cont’d)

4 nlmm-CMA � nlmm-CMAI (with equal RT)

fSchwefel fSchwefel1/4 fNoisySphere

(2, 6) (4, 8) (8, 10) (16, 12)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 6) (4, 8) (8, 10)0

2

4

6

8

Sp

eed

up

(Dimension, Population Size)(2, 6) (4, 8) (8, 10) (16, 12)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

fRosenbrock fAckley fRastrigin

(2, 6) (4, 8) (8, 10)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 5) (5, 7) (10, 10)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

(2, 50) (5, 140)0

2

4

6

8

(Dimension, Population Size)

Sp

eed

up

⇒ nlmm-CMA outperforms nlmm-CMAI .

⇒ The ranking obtained with the new acceptance criterion still has an amount

of information to guide CMA-ES.Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 14 of 15

Summary

CMA-ES with meta-modelsThe speedup of lmm-CMA with respect to CMA-ES drops to one when the populationsize λ increases.

⇒ The meta-model acceptance criterion is too conservative.

New variant of CMA-ES with meta-modelsA new meta-model acceptance criterion: It must keep:

the best individual and the ensemble of the µ best individuals unchanged

the best individual unchanged, if more than one fourth of the population

is evaluated.

nlmm-CMA outperforms lmm-CMA on the test functions investigated with aspeedup in between 1.5 and 7.

nlmm-CMA maintains a significant speedup, between 2.5 and 4, when

increasing the population size on tested functions.

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 15 of 15

Thank You For Your Attention

Zyed Bouzarkouna, Anne Auger, Didier Yu Ding page 16 of 15

Investigating the Local-Meta-Model CMA-ES forLarge Population Sizes

Zyed Bouzarkouna1,2 Anne Auger2 Didier Yu Ding1

1IFP (Institut Francais du Petrole)

2TAO Team, INRIA Saclay-Ile-de-France, LRI

April 07, 2010