Comparison of Contemporary Evolutionary Algorithms on theRotated Klee-Minty Problem

Michael HellwigVorarlberg University of AppliedSciences, Research Center PPE

Dornbirn, Austriamichael.hellwig@fhv.at

Patrick SpettelVorarlberg University of AppliedSciences, Research Center PPE

Dornbirn, Austriapatrick.spettel@fhv.at

Hans-Georg BeyerVorarlberg University of AppliedSciences, Research Center PPE

Dornbirn, Austriahans-georg.beyer@fhv.at

ABSTRACTThe Rotated Klee-Minty Problem represents an advancement ofthe well-known linearly constrained Klee-Minty Problem that wasintroduced to illustrate the worst case running time of the Sim-plex algorithm. Keeping the linearity as well as the complexity ofthe original Klee-Minty Problem, the Rotated Klee-Minty Problemremedies potential biases with respect to Coordinate Search andturns out to be a suitable constrained test environment for ran-domized search heuristics. The present paper is concerned withthe comparison of recent evolutionary algorithm variants for con-strained optimization on this respective test bed. The consideredalgorithm variants include the most successful participants of theCEC Competition on Single Objective Real-parameter optimizationand other selected strategies that are not directly applicable to theCEC test suite. Taking into account the diverse constraint handlingapproaches, the performance results of all search heuristics are in-terpreted. It turns out that most strategies that have been successfulin the CEC competitions do have severe problems on the RKMP.

CCS CONCEPTS• Theory of computation → Optimization with randomizedsearch heuristics; Bio-inspired optimization; Linear programming;

KEYWORDSRandomized Search Heuristics, Constrained Optimization, Bench-marking, Rotated Klee-Minty Problem

ACM Reference Format:Michael Hellwig, Patrick Spettel, and Hans-Georg Beyer. 2019. Compari-son of Contemporary Evolutionary Algorithms on the Rotated Klee-MintyProblem. In Genetic and Evolutionary Computation Conference Companion(GECCO ’19 Companion), July 13–17, 2019, Prague, Czech Republic. ACM,New York, NY, USA, 9 pages. https://doi.org/10.1145/3319619.3326805

1 INTRODUCTIONThe development and the assessment of Evolutionary Algorithms(EA) widely rely on benchmarking due to the limited availability of

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic© 2019 Association for Computing Machinery.ACM ISBN 978-1-4503-6748-6/19/07. . . $15.00https://doi.org/10.1145/3319619.3326805

theoretical performance results. One goal of such benchmark envi-ronments is supporting the selection of that algorithm best suitablefor a given real-world applications. Other aims would be gatheringexperimental insights into the working principles of algorithmicideas and fostering the algorithm development for specific tasks.Further, benchmarks can be used to verify theoretically derivedperformance results.

Regarding single-objective constrained optimization benchmarksfor randomized search heuristics, the most elaborated test suiteshave been provided in the context of the IEEE Congress on Evolu-tionary Computation (CEC) competitions (2006, 2010, and 2017) onconstrained real-parameter optimization. These test functions aswell as the associated rules for algorithm evaluation are commonlyused to promote novel algorithmic ideas in the EA community.However, these benchmark sets mainly consist of very complexnon-linear constraint and objective functions. Yet, rather simpleconstrained problem formulations still may present a difficult taskfor EAs. Additionally, the CEC benchmarks are missing a clearstructure that would allow for allocating specific problem classesto especially beneficial algorithmic ideas in retrospect. While otherconstrained test function sets do exist, the respective problemsdo not come with properly defined standards for algorithm com-parison and usually do not scale with the search space dimension.An exception is the COCO BBOB-constrained test suite [4] whichis currently under development. It is based on the unconstrainedCOCO suite and makes use of 8 distinct objective functions. Yet,it is restricted to varying numbers of (almost) linear inequalityconstraints (i.e. the boundary of the feasible set is subject to smallnonlinear perturbations).

Compared to the vast number of distinct constrained problemfeatures, only a small number of well-designed benchmarks exist.This makes it hard to determine those features that do make a con-strained optimization problem hard for a specific search algorithm.Hence, a benchmark suite needs to take into account consistentsubgroups of conceivable problems. Based on these considerations,the Rotated Klee-Minty Problem (RKMP) was proposed in [7]. Itscales with the search space dimensionality, and, by construction,causes the Simplex-Algorithm to exhibit an exponential worst-caserunning time. While EAs usually cannot compete with custom-built linear problem (LP) solvers, the RKMP represents a reasonablycomplex problem even for certain LP solvers (e.g. interior-pointmethods (IPM) or basis-exchange pivoting algorithms). Interest-ingly, by realizing comparable or even improved precision, someEA variants for constrained optimization are able to compete withLP solvers on the RKMP [7, 13].

GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic Hellwig et al.

The present paper is concerned with the assessment and theperformance comparison of selected EA algorithms for constrainedoptimization. Interpreting the gathered data, we come across theinteresting observation that most of the algorithms that are con-sidered to be successful in the CEC competitions exhibit severeproblems on the RKMP. Furthermore, the comparison gains in-sights into the working principles of those strategies using theso-called ϵ-level constraint handling [15]. This constraint handlingtechnique is part of the recently most successful algorithms forconstrained optimization, but it slows down the algorithm perfor-mance in the context of the RKMP. The present paper comes upwith an explanation of this ϵ-level constraint handling drawback.

The rest of this paper is organized as follows: We recap theRotated Klee-Minty Problem and the comparison methodology inSec. 2. Section 3 then introduces the algorithms considered in thisstudy. The algorithm results are presented and discussed in Sec. 4.The paper is concluded in Sec. 5.

2 BENCHMARK FORMULATIONThis section recaps the Rotated Klee-Minty benchmark definitionintroduced in [7] and states the corresponding benchmarking con-ventions as well as the criteria considered for algorithm assessment.The source code of the Rotated Klee-Minty Problem is available ina separate Github.com branch [14].

2.1 The Rotated Klee-Minty ProblemThe Rotated Klee-Minty Problem represents a scalable linear con-strained optimization problem [7]. The optimization problem isdetermined by

miny∈RN

d⊤y

s.t. A · R(y − t) ≤ b,

y ≤ y ≤ y,

(1)

with linear objective function given by d = (0, 0, . . . , 0, 1)⊤ ∈ RN .Note that the terms objective function value and fitness are usedsynonymously, as essentially evolutionary methods are consideredin this study.

The feasible region of (1) represents a perturbed unit hyper cubein theRN which is defined by a matrix A and a vector b as

A =

(B + IB − I

)∈ R2N×N , and b =

(10

)∈ R2N×1, (2)

with identity matrix I ∈ RN×N ,

B =

©­­­­­­­­«

0 0 . . . 0 0 0ϵ 0 . . . 0 0 00 ϵ . . . 0 0 0....... . .

. . .......

0 0 . . . ϵ 0 00 0 . . . 0 ϵ 0

ª®®®®®®®®¬∈ RN×N , and ϵ = 1/10. (3)

By application of the translation vector t = N 3 ·(1, 1, . . . , 1)⊤ ∈ RN

and the rotation matrix R ∈ RN×N (det(R) = 1), the feasible regionof problem (1) is relocated within the N -dimensional domain.1 For1The deterministically chosen motions represent a trade-off between problem hardnessfor LP solvers and unbiasedness of EAs.

Figure 1: Translation and rotation of the 2-dimensionalKlee-Minty cube. The contour lines of the objective function areindicated by the color (or grey-scale) gradient.

the definition of the rotation matrix R refer to [7]. The optimalsolution is moved to y∗ = N 3 · 1 with f (y∗) = d⊤y∗ = N 3. Therelocation of the optimal solution away from the origin removesundesired problem characteristics that would prevent reasonablealgorithm comparisons. An illustration of the transformation in the2-dimensional case is provided in Fig. 1.

Additionally, we provide component-wise lower and upper pa-rameter bounds, or box-constraints, to constitute the domain ofeligible input values for black-box optimization strategies and tofacilitate the generation of initial populations. These are defined byy = 0 ∈ RN and y = 5N 3 · 1 ∈ RN .

2.2 Algorithm AssessmentThis section introduces the benchmarking principles as well as theperformance indicators used in the context of the Rotated Klee-Minty Problem. The proposed benchmark problem takes into ac-count search space dimensions N ∈ {2, 3, 5, 10, 20, 40}. Accordingly,six distinct constrained functions are considered. For each dimen-sion, the following three performance indicators are derived from15 independent algorithm runs:

i) The algorithm runtime is measured with respect to the func-tion evaluations consumed. This information is displayed byuse of empirical cumulative distribution functions (ECDF),or performance profiles [11], respectively.

ii) We assess the effectiveness of each algorithm by taking intoaccount its feasibility ratio (FR) as well as the deviation of itsmedian fitness realization (among all final feasible solutions)from the known optimal function value.

iii) Effectiveness is also evaluated in the search space by tak-ing into account the combination of the FR and the meandeviation of all feasible solutions from the known optimizer.

Depending on the search space dimension, the total budget of func-tion evaluations per run is predefined as 2 · 104 · N . One singlefunction evaluation is consumed regardless of whether the objec-tive function, a single constraint function, or the whole constrainedproblem is evaluated for a single parameter vector y at a time.

The final candidate solutions realized in different algorithm runsare compared by use of a lexicographic ordering ⪯lex that takes

EA Comparison on the Rotated Klee-Minty Problem GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

into account the objective function value f (y) as well as the corre-sponding amount of constraint violation ν (y). The respective orderrelation is defined by

y ⪯lex z ⇔

{f (y) ≤ f (z), if ν (y) = ν (z),ν (y) < ν (z), else. (4)

In the context of the RKMP, the objective function value corre-sponds to f (y) B d⊤y. The constraint violation value ν (y) can bemeasured as the sum of the deviation over all inequality constraints

ν (y) BN∑i=1

max{0,(ARy −ARt − b

)i}. (5)

The lexicographic order relation permits to define the quality indica-tors that can be used to assess and compare algorithm performanceon problem (1). Note that the lexicographic order is also known asDeb’s method or as the Superiority of Feasibility (SOF) principle [5].

Taking into account different search space dimensions, the ef-fectiveness indicators (ii) and (iii) of each algorithm are presentedin a table. Facilitating the comparison of multiple algorithms, weuse a graphical representation. To this end, the indicators are plot-ted against the dimension. Note that, due to the limitation of thepage number, the tabular presentation of the individual algorithmresults is omitted in the paper but can be found in the supplemen-tary material. Regarding performance indicator (i), the runningtime of meta-heuristic algorithms can be directly identified withthe number of function evaluations needed to satisfy a number ofpredefined targets. This definition is adopted from [11]. It is usedin the context of the COCO BBOB benchmarks. A more detailedexplanation of the ECDF construction and interpretation is pro-vided in [6]. We also apply a bootstrapping procedure to obtain

Algorithm 1 Pseudo-code for the bootstrapping.

1: Initialize the maximum number of tries K for sampling a suc-cessful run and the number of bootstrapped runs S to generate

2: for all targets t do3: if at least one of the 15 runs reached target t then4: for s in {1, . . . , S} do k ← 05: success← false6: rl ← 07: while ((not success) and k < K ) do8: Sample one run r of the 15 runs

with replacement uniformly at random9: rl ← rl + fevals of run r to reach target t10: if r was a successful run then11: success← true12: end if13: k ← k + 114: end while15: if success then16: Runlength for the bootstrapped run s

considering target t is rl17: end if18: end for19: end if20: end for

more accurate performance profiles while minimizing the bench-mark execution time. The pseudo code of the bootstrapping used isdisplayed in Alg. 1. Yet, the target definition used for the RotatedKlee-Minty Problem is different. Instead of defining targets only forthe feasible region of the search space, we allocate about half ofthe targets to the infeasible region. This supports the illustration ofalgorithm behavior within the infeasible region. Refer to Sec. 4 forthe illustration of the algorithm results. In total, 103 target valuesare defined.

The 52 targets in the infeasible region are defined with respect tothe constraint violation of a candidate solution. They are uniformlydistributed between 104 and 10−6 as well as 0. Realizing a candidatesolution with constrained violation below a target definition forthe first time, a target value is considered to be hit.

The logarithmically equidistant targets in the feasible regionrange from 100 to 10−8 and take into account the fitness valuerealized by a strategy. The targets are reached after having realizeda feasible candidate solution with an objective function value thatdeviates from the known optimal value by a number smaller thana certain target value. For example, target ti , (i ∈ 1, . . . , 51) is hitafter the algorithm generates a candidate solution y which satisfies| f (y) − f (y∗)| < ti for the first time. The ECDF plots display thepercentage of the targets reached in all 15 algorithm runs for anynumber of function evaluations. Accordingly, an algorithm realizesa 100% target ratio if and only if it manages to yield a feasible finalsolution with higher accuracy than specified by the final targetprecision 10−8 in all 15 runs. This way they provide a notion ofalgorithm performance: Smaller upper left areas indicate fasteralgorithm running times [11].

3 METHODSThis section introduces the eight search strategies considered forcomparison on the Rotated Klee-Minty Problem. Among the fiveconsidered Differential Evolution (DE) variants, the selection ofalgorithms comprises the CEC competitionwinners of 2010 (ϵDEag),2017 (LSHADE44), and 2018 (iUDE), respectively. The other twoDE variants, and the ϵMAg Evolution Strategy (ES), also turned outto be successful on many CEC benchmark problems. In particular,the ϵMAg-ES showed superior performance with regard to high-dimensional search spaces of the CEC 2018 competition. The otherES variants have not been developed to solve the CEC benchmarkproblems. Both instead originate from theoretical considerations onconstrained optimization problems. For example, the Active-Set-ESand the lcCMSA-ES assume that linear constraints are provided inexplicit form. As all the algorithms treat the RKMP problem as ablack-box problem, a preprocessing step that determines the RKMPlinear constraint system is necessary to compare those two with theother strategies. The remainder of this section briefly introduces allsearch algorithms considered in the present study and provides theirreferences. Due to the space restrictions, this paper is confined tothe comparison of EA variants on RKMP. We refer to [7] for resultsof the Octave interior point LP-solver glpk. That respective paperalso indicates very poor performance of Random Search on RKMP.Hence, a comparison of the EA variants with Random Search isomitted in this paper.

GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic Hellwig et al.

ConSaDE – The Self-adaptive Differential Evolution Algorithmfor Constrained Real-Parameter Optimization (ConSaDE) has beenproposed in [9]. It incorporates constraint handling into the Self-adaptive Differential Evolution (SaDE) algorithm. The two centralideas of the SaDE algorithm are the use of multiple mutation oper-ators and self-adaptive parameter adaptation. The mutation opera-tors DE/rand/1, DE/current-to-best/2, DE/rand/2, and DE/current-to-rand/1 are used together with binary crossover. Each of themis applied with equal probability in the beginning. The probabilityis then adapted from one generation to the next according to thenumber of successful individuals created by the respective mutationoperators. Additionally, local search (Sequential Quadratic Program-ming) is used every 500 generations for convergence speedup. Theextension ConSaDE uses the lexicographic ordering for handlingconstraints the selection step.

ECHT-DE – The algorithm Differential Evolution with Ensem-ble of Constraint Handling Techniques (ECHT-DE) [10] combinesmultiple constraint handling methods into a single DE algorithmwith the idea that some constraint handling approaches are bettersuited for some problems than others. The constraint handling ap-proaches used are Superiority of Feasiblility, Self-Adaptive Penalty,ϵ-level constraint handling, and Stochastic Ranking. Every of thoseconstraint handling approaches has its own population with a set ofparameters and generates offspring. For the computation of the nextgeneration’s population of a specific constraint handling method,the offspring populations of the other constraint handling meth-ods are considered in addition to the parental population of theconstraint handling method under consideration.

ϵDEag – The algorithm Constrained Optimization by the ϵ Con-strained Differential Evolution with an Archive and Gradient-BasedMutation (ϵDEag) has been presented in [16]. It integrates fourmain ideas into a DE algorithm. First, an archive of individualsis kept throughout the optimization. Individuals from the archiveform a part of the trial vector generation with the intention tomaintain diversity. Second, if an offspring is not better than itsparent, the corresponding parental individual is used to generate asecond offspring. Third, the ϵ-level constraint handling approachis used for dealing with constrained optimization problems. Andfourth, gradient-based mutation is performed to mutate in a waythat reduces the constraint violation.

L-SHADE44 – L-SHADE with Competing Strategies Applied toConstrained Optimization (LSHADE44) [12] is an algorithm thatenhances the Success History based DE with linear population sizereduction (L-SHADE) [17]. It incorporates four different combi-nations of mutation and crossover operators that compete witheach other for the creation of a trial point. Specifically, DE/current-to-pbest/1/bin, DE/current-to-pbest/1/exp, DE/randrl/1/bin, andDE/randrl/1/exp are used. For the constraint handling, the strategyuses the lexicographic ordering approach.

iUDE – The Unified Differential Evolution (UDE) algorithmfor constrained optimization problems has been proposed in [18].An improved version of this algorithm, the iUDE, represents thewinner strategy of the IEEE CEC competition on single objectivereal-parameter optimization. While the respective paper was notyet published in the context of the CEC, a technical report [19] aswell as theMatlab source code of iUDE are available on the CEC2018

competition website2. The algorithm combines a multitude of ideasfromDE algorithms. It makes use of the three trial vector generationapproaches DE/rand/1, DE/current-to-rand/1, and DE/current-to-pbest/1. A dual population approach with strategy adaptation isused. For constraint handling, a combination of lexicographic andϵ-level constraint handling is used.

ϵMAg-ES – Based on the Matrix Adaptation (MA) ES [3], theϵMAg-ES [7] incorporates basically three constraint handling tech-niques. It uses a reflection approach for the box-constraints and theϵ-level constraint handling to rank the individuals of a current pop-ulation. Additionally, it occasionally makes use of a repair approachthat is based on estimated gradients for equality and inequality con-straints in order to prevent premature convergence. The ϵMAg-ESachieved the second over-all rank in the CEC 2018 competition. Inparticular, it showed superior performance in high-dimensionalsearch spaces.

Active-Set-ES – The Active-Set-ES [1] is a (1+1)-ES interleavedwith an evolution of the active set of constraints. In every generationone feasible offspring is generated. It is then projected either ontothe whole feasible region or onto a reduced search space. The choiceis done uniformly at random with a fixed probability. The reducedsearch space is created by making all the inequality constraintsequality ones that are active at the parent. It replaces the parent ifits objective function value is better.

lcCMSA-ES –The Linear Constraint CMSA-ES (lcCMSA-ES) [13]is an interior point method that specializes on linear constraints. Itis based on the Covariance Matrix Self-Adaptation (CMSA) ES [2]and extends it with constraint handling. It turns a set of linear(in)equality constraints into a linear equality constraint systemwith a non-negativity constraint. Then, the main idea of the ap-proach is to evolve in the null space of the constraint matrix. Bydoing this, the mutated individuals do not violate the linear equalityconstraints. Possible violation of the non-negativity constraint isdealt with using repair by projection onto the non-negative orthant.

The last two ES variants need information about the problemspecific constraints. As the RKMP is regarded to be a black-boxoptimization problem, this information needs to be gathered in apreprocessing step and the associated function evaluations con-sumed need to be taken into account appropriately to ensure a faircompetition. The preprocessing step reconstructs the matrix/vectorform of the RKMP constraints by querying the black-box constraintfunction. For this, the constraints are evaluated for a given num-ber of randomly sampled vectors in the search space to create a(possibly overdetermined) system of equations. Solving it (possi-bly using the pseudoinverse) allows computing the original RKMPlinear constraint system. With this preprocessing, the last two ESvariants are able to evolve in such a way that the individuals arealways feasible. They start with an initially feasible solution whichis obtained by projecting onto the feasible region.

The original algorithm codes are used for our comparisons wherepossible. These are partly available on the CEC competition web-site or are published by the authors themselves. Only the codeof the ϵDEag algorithm needed to be transferred from a C to a

2http://web.mysites.ntu.edu.sg/epnsugan/PublicSite/Shared%20Documents/CEC-2018/Constrained

EA Comparison on the Rotated Klee-Minty Problem GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

Matlab implementation. In order to establish a fair comparison,we slightly adjust the termination criteria to fit the Rotated Klee-Minty framework. That is, the search is stopped after the maximalfunction evaluation budget is consumed or after having found afeasible candidate solution the fitness of which deviates from theoptimal fitness value by less than 10−8. The strategy parametersrecommended by the algorithm developers have been used.

4 RESULTSHaving introduced the algorithms of interest, the correspondingtest results obtained on the Rotated Klee-Minty benchmark are pre-sented and discussed in Sec. 4.1. Interestingly, in comparison to theoutcome of the past CEC competitions, one observes very differentperformance results of those strategies that use the so-called ϵ-level constraint handling method. The reason for this performancedeterioration is examined in Sec. 4.2.

4.1 Algorithm ComparisonAiming at a performance comparison on the rotated Klee-MintyProblem (1), Fig. 2 displays the performance profiles (ECDF plots) ofthe eight algorithms under consideration (Sec. 3) for the dimensionsN ∈ {2, 3, 5, 10, 20, 40}. Each plot displays the percentage of targetsreached in all 15 runs for each number of function evaluations.We use two distinct black-rimmed markers (▽▽▽ and △△△) to illustratethe transition from the infeasible region to the feasible region foreach algorithm. The markers must be interpreted as follows: The ▽▽▽marker displays the expected number of function evaluations forwhich an algorithm has reached a single feasible target for the firsttime in at least one of the considered runs. Whereas, the△△△ indicatesthe expected number of function evaluations after which at leastone feasible solution was found in each algorithm run. Hence, the△△△ maker indicates the expected number of function evaluations torealize a feasibility ratio of one.

Taking into account the respective markers (▽▽▽ and △△△), one ob-tains a clear ranking of all algorithms in each dimension.3 The effectof the preprocessing for lcCMSA-ES and for Active-Set-ES is clearlyvisible. In the implementation for the experiments, an overdeter-mined system consisting of 10 · N equations is used to reconstructthe RKMP linear constraint system. Because the lcCMSA-ES andthe Active-Set-ES then project onto this system, the step observedin Fig. 2 is the result.4 That is, on average those two algorithmsreach the feasible region earliest. Among the other strategies, Con-SaDE appears to realize feasible solutions first (especially in highdimensions). However, the algorithm faces difficulties to satisfy allfeasible targets. Except for Active-Set-ES and iUDE, all algorithmscan be considered rather resilient in terms of their search behaviorin the infeasible region. In contrast, due to numerical approxima-tion errors, the Active-Set-ES struggles to find feasible solutionsin each run (△△△) for higher dimensions. That is, many runs of theActive-Set-ES faces stagnation in the vicinity of the boundary ofthe feasible region.

3Note that the close proximity (in terms of function evaluations) of both data pointsfor most algorithms is augmented by the logarithmic scaling and by the resolution ofthe x-axis.4A number of further evaluations (constant w.r.t. N ) are used up in the preprocessingfor tasks such as determining the number of constraints. Hence, the step observed inFig. 2 at around 101 is slightly above 101 for the smaller dimensions.

Regarding the infeasible targets for the remaining algorithms, thetransitioning into the feasible region occurs rather rapidly. For di-mension 2, the LSHADE44 reaches the feasible region first, followedby the ConSaDE, the ECHT-DE, the ϵDEag, the ϵMAg-ES, and theiUDE. For dimension 3, it is very similar, except that the ϵMAg-ESreaches the feasible region before the ϵDEag. In dimensions 5 and 10the order is ConSaDE, LSHADE44, ECHT-DE, ϵMAg-ES and ϵDEag,however, iUDE does not reach it. The situation for dimensions 20and 40 is very similar, except that ECHT-DE performs worse thanfor small N .

Considering also the targets in the feasible region, the ConSaDEreaches all targets in dimensions 2, 3, 5, 20, and about 90% of thetargets for dimensions 10 and 40. The ECHT-DE reaches all targetsfor the dimensions 2, 3, and 5. For the dimensions 10, 20, and 40,about 70%, 60%, and 50% of the targets are reached, respectively. Inhigher dimensions, it takes considerably more function evaluationsto reach the same number of targets. The ϵDEag reaches all thetargets for dimensions 2 and 3. In dimensions 5, 10, 20, and 40 thestrategy satisfies about 98%, 70%, 58%, and 55% of the targets, respec-tively. The LSHADE44 reaches all the targets in all dimensions. TheActive-Set-ES reaches all the targets for the dimensions 2, 3, 5, and10. It reaches about 78% and 72% of the targets for the dimensions40 and 20, respectively. The lcCMSA-ES reaches all the targets forthe dimensions 2, 3, 5, and 10. It reaches about 73% and 67% of thetargets for the dimensions 20 and 40, respectively. Considering theϵMAg, it reaches all the targets in all the dimensions. The iUDEreaches all the targets only for dimensions 2 and 3. This is verysurprising as it was the winner of the IEEE CEC 2018 competition.Further investigations in this regard revealed that the ϵ constrainthandling is a reason for this behavior on the RKMP (see Sec. 4.2 fora more detailed discussion of this aspect).

In overall, the LSHADE44, the ϵMAg-ES, and the ConSaDE arethe best performing algorithms when considering all the dimen-sions and default parameters on the RKMP (1). For dimensions 2, 3,5, and 10, the lcCMSA-ES and the Active-Set-ES can also competewith the LSHADE44 and the ϵMAg-ES, but they require more func-tion evaluations to reach all the targets. However, they are able tobeat the ConSaDE for dimension 10. For the dimension 40, onlythe LSHADE44 and the ϵMAg-ES are able to reach all the targets.Regarding the strategies that make use of the ϵ-level constrainthandling, the ϵMAg-ES exhibits the best performance on the RKMPin all dimensions N ≥ 3.

Figs. 3 and 4 show plots comparing the errors in the objectivefunction space and the search space, respectively, for the differentalgorithms and the dimensions considered. From them, the sameobservations as for the ECDF plots can be concluded, however, froma different perspective. The median objective function error plotshows that the LSHADE44 and the ϵMAg-ES reach all the targetsfor all the dimensions in the median (as they are on the dashedblack line). Apart from that, one can see that the median error tothe optimum in the objective function increases with increasingdimension for the other algorithms. This also holds for the meannorm error in the parameter space. Note that only the algorithmsthat achieve feasible solutions in all the runs are considered in thosetwo figures. Information on the feasibility ratio of the other strate-gies and on the performance indicators ii) and iii) corresponding

GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic Hellwig et al.

Figure 2: Comparison of the performance profiles (ECDF plots) of the eight algorithms presented in Sec. 3 obtained on prob-lem (1) in different dimensions. The black-rimmed markers are obtained based on the bootstrapped data. The ▽▽▽ markerdisplays the event of hitting the first feasible target, and △△△ indicates number of function evaluations for which all runs havereached the feasible region, respectively.

EA Comparison on the Rotated Klee-Minty Problem GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

Figure 3: Comparison of themedian fitness value deviationsfrom the known optimal objective function valueN 3. The al-gorithm realizations are plotted against the search space di-mension N . The final target defines a precision of 10−8. Datapoints for each algorithm are only shown if all runs of thatalgorithm resulted in feasible solutions for the given dimen-sion. In N = 2 and N = 3 all data points do interleave. Onlythese two data points are obtained for the Active-Set-ES. No-tice that all the data points obtained by ϵMAg-ES and byLSHADE44 overlap with the final target for all dimensions.

to Fig. 3 and Fig. 4 are available in the supplementary material. Assuggested in [7], the results are presented in tabular form.

In order to precisely distinguish two similar performing algo-rithms, particular events may be examined for significant differ-ences by making use of statistical tests, e.g. differences in theamount of function evaluations needed to reach the final target.Depending on the number of compared algorithms and the testobjective, the Wilcoxon rank-sum test or the Friedman test arewell-known non-parametric hypothesis tests for this purpose.

4.2 Additional InvestigationsIn this section we compare only those algorithms that incorpo-rate the ϵ-level constraint handling approach [15]. This constrainthandling represents a relaxation of the lexicographic ordering pre-sented in (4). The ϵ-level constraint handling softens the principlethat feasible solutions are always considered superior to infeasiblesolutions. It enables the algorithm to treat infeasible candidate so-lutions with constraint violation below a specific ϵ (д) threshold asfeasible. The threshold ϵ (д) is continuously reduced to zero with thenumber of iterations д. Hence, the strategy is able to move outsidethe feasible region within the early phase of the search processwhich can potentially support the convergence to the optimizer y∗in some cases. This constraint handling technique turned out tobe especially successful in the context of the CEC competitions. Infact, the LSHADE44 algorithm (CEC 2017) is the only competitionwinner that refrained from using the ϵ-level constraint handling.

Preliminary experiments suggest that the performance deteri-oration on the Rotated Klee-Minty Problem (1) can at least partly

Figure 4: Comparison according to the mean distance devia-tion from the known optimal solution of (1). The algorithmresults are plotted against the search space dimension N .Data points are only shown if all runs of a particular algo-rithm resulted in feasible solutions for the given dimension.

be attributed to the use of ϵ-level constraint handling. In orderto support this claim, we execute the following experiment. Theperformance profiles of iUDE, ϵDEag, and ϵMAg-ES, are comparedto those ECDF plots that are obtained by running the exact samealgorithms after turning off the ϵ-level constraint handling method.To this end, the ϵ threshold is initially set to zero in all algorithmruns, i.e. the ϵ-level ordering is replaced with the lexicographic or-dering (4). The resulting algorithm variants are denoted as lexiUDE,lexDEag, and lexMAg-ES, respectively. Due to the space limitation,the comparison of the corresponding ECDF plots is only illustratedfor dimension N = 40 in Fig. 5. Yet, the results in lower dimensionslook very similar. One observes in Fig. 5 that setting the ϵ thresholdto zero initially improves the original counterparts for the algo-rithms considered. The lexDEag transitions into the feasible regionearlier but is not able to reach more feasible targets than the ϵDEag.The lexMAg-ES reaches all targets with less evaluations comparedto the ϵMAg-ES. The most surprising result is achieved by the lex-iUDE. Whereas the iUDE is not even able to reach the infeasibletargets, its lexicographic variant lexiUDE performs similarly to thelexMAg-ES5.

In order to further investigate the influence of ϵ , an additionalexperiment is conducted. While Fig. 5 shows the two extremes ofsetting ϵ to zero initially and using the original ϵ decay, Figure 6 dis-plays the results obtained by accelerating the ϵ reduction within theiUDE. An acceleration of the ϵ decay is expected to result in a perfor-mance in between those exhibited by the two extremes (see Fig. 5).Considering the three corresponding iUDE variants (iUDE, lexiUDE,and iUDE (with faster ϵ decay)), the empirical results substantiateour proposition. As intuitively expected, the lexiUDE achieves thebest performance, the original iUDE performs worst, and the iUDEwith faster ϵ threshold reduction shows a performance that is in5Note that, in terms of the ECDF plots, the lexMAg-ES performance and the lexi-UDE performance, respectively, turn out to be slightly superior to the LSHADE44performance on the RKMP.

GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic Hellwig et al.

Figure 5: Investigation of the ϵ-level constraint handlingimpact on the algorithm performance. The black-rimmedmarkers display the first (▽▽▽) observation of hitting a feasi-ble target, and the number of function evaluations when all(△△△) runs have reached the feasible region, respectively.

Figure 6: Investigation of the ϵ-level constraint handling in-fluence on the algorithm performance of the iUDE.

between the former two variants. Searching for an explanationof this weakness, we come to the conclusion that the benefit ofusing the ϵ-level constraint handling is strongly connected to theanalytical structure of the objective function. The objective func-tion of problem (1) is a linear function and thus strictly monotonic.Hence, the optimal solution is always located on the boundary ofthe feasible region. In such situations, a certain relaxation of theconstraints that define the boundary of the feasible set will resultin the relocation of the optimal solution. For simplicity, the emerg-ing optimal solution is referred to as the relaxed optimal solution.Accordingly, when starting with comparably large ϵ (0)-threshold, astrategy employing the ϵ-level constraint handling will approachtowards the relaxed optimal solution. By gradually reducing the

ϵ (0) with the number of iterations, the relaxed optimal solution willapproach the real optimal solution. However, this process mightalready consume a large amount of function evaluations that wouldhave been needed at the end of the algorithm run for the searchwithin the feasible region, or even for the detection of the feasibleregion, respectively.

Hence, boundary relaxation techniques like ϵ-level constrainthandling can be disadvantageous in similar environments, i.e. onconstrained problems with strictly monotonic decreasing objectivefunctions in at least one dimension of the search domain.

5 CONCLUSIONIn this paper, the results of evaluating multiple randomized searchmethods on the RKMP [7] have been presented. To this end, thebenchmark function has been recapped (1) and a selection of promi-nent algorithms for constrained optimization have been described.Regarding the corresponding empirical algorithm results yields aninteresting observation: search algorithms that rely on the ϵ-levelconstraint handling technique can exhibit a comparably weak per-formance on the RKMP. This is in contrast to the results obtained inthe context of the CEC competitions on single objective constrainedreal-parameter optimization, where these strategies commonly per-formwell. Whereas the ϵ constraint handling is beneficial in solvingthe CEC constrained benchmark problems, it is disadvantageousfor the considered Rotated Klee-Minty Problem.

The RKMP still represents a quite new benchmark proposal.Taking into account the survey in [8], the design of elaboratedtest suites for benchmarking randomized search heuristics on con-strained optimization problems still is in its infancy. Hence, thebenchmarking conventions and the considered performance in-dicators must be regarded as first suggestions. For example, thequestions of which performance indicators need to be capturedto provide a full picture of the algorithm performance, how theseshould be parameterized, and how the results should be presentedbest in order to provide easy performance interpretations are notconclusively answered. They should to be thoroughly discussedand examined theoretically if possible.

However, due to the number of distinct constrained problemfeatures, the work towards a sophisticated algorithm selection forreal-world problems requires an extension of the available bench-mark environments. The RKMP contributes to this line of researchand the present paper supplies a first comparison of selected EAson this linearly constrained problem class. In future studies, weaim to explore the incremental progression of the RKMP towardsnonlinear objective functions and disconnected feasible regions (byadding multiple hyper cubes to the search domain).

Finally, an elaborated ranking approach that determines the bestalgorithm in a certain dimension is still missing. To this end, thequestion of how to combine the individual performance indicatorsin an unbiased way still needs to be answered.

ACKNOWLEDGMENTSThe authors thank all algorithm developers for making their codeopenly available. This work was supported by the Austrian ScienceFund FWF under grant P29651-N32.

EA Comparison on the Rotated Klee-Minty Problem GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

REFERENCES[1] Dirk V. Arnold. 2016. An Active-Set Evolution Strategy for Optimization with

Known Constraints. In International Conference on Parallel Problem Solving fromNature. Springer, 192–202.

[2] Hans-Georg Beyer and Bernhard Sendhoff. 2008. Covariance Matrix AdaptationRevisited – The CMSA Evolution Strategy –. In Parallel Problem Solving fromNature – PPSN X: 10th International Conference, Dortmund, Germany, September13-17, 2008. Proceedings. Springer, Berlin, Heidelberg, 123–132.

[3] Hans-Georg Beyer and Bernhard Sendhoff. 2017. Simplify Your CovarianceMatrixAdaptation Evolution Strategy. IEEE Transactions on Evolutionary Computation21, 5 (Oct 2017), 746–759.

[4] Dimo Brockhoff, Nikolaus Hansen, Tea Tušar, O. Mersmann, Phillipe RodriguesSampaio, Anne Auger, and Asma Atamna et al. 2016. COCO DocumentationRepository. (2016). http://github.com/numbbo/coco-doc/tree/gh-pages/docs.

[5] Kalyanmoy Deb. 2000. An efficient constraint handling method for geneticalgorithms. Computer Methods in Applied Mechanics and Engineering 186, 2(2000), 311 – 338. https://doi.org/10.1016/S0045-7825(99)00389-8

[6] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, Dejan Tusar, and Tea Tusar.2016. COCO: Performance Assessment. CoRR abs/1605.03560 (2016). http://arxiv.org/abs/1605.03560

[7] Michael Hellwig and Hans-Georg Beyer. 2018. A Linear Constrained Opti-mization Benchmark for Probabilistic Search Algorithms: The Rotated Klee-Minty Problem. In Theory and Practice of Natural Computing, David et al.Fagan (Ed.). Vol. 11324. Springer International Publishing, Cham, 139–151.https://doi.org/10.1007/978-3-030-04070-3_11

[8] Michael Hellwig and Hans-Georg Beyer. 2019. Benchmarking evolutionaryalgorithms for single objective real-valued constrained optimization âĂŞA criticalreview. Swarm and Evolutionary Computation 44 (2019), 927 – 944. https://doi.org/10.1016/j.swevo.2018.10.002

[9] Vicky Ling Huang, A Kai Qin, and Ponnuthurai N Suganthan. 2006. Self-adaptiveDifferential Evolution Algorithm for Constrained Real-Parameter Optimization.In IEEE Congress on Evolutionary Computation (CEC). IEEE, 17–24.

[10] Rammohan Mallipeddi and Ponnuthurai Nagaratnam Suganthan. 2010. Differ-ential Evolution with Ensemble of Constraint Handling Techniques for SolvingCEC 2010 Benchmark Problems. In IEEE Congress on Evolutionary Computation(CEC). IEEE, 1–8.

[11] Jorge J Moré and StefanMWild. 2009. Benchmarking derivative-free optimizationalgorithms. SIAM Journal on Optimization 20, 1 (2009), 172–191.

[12] Radka Polakova and Josef Tvrdík. 2017. L-SHADE with competing strategiesapplied to constrained optimization. In 2017 IEEE Congress on Evolutionary Com-putation, CEC 2017, Donostia, San Sebastián, Spain, June 5-8, 2017. 1683–1689.

[13] Patrick Spettel, Hans-Georg Beyer, and Michael Hellwig. 2018. A Covari-ance Matrix Self-Adaptation Evolution Strategy for Optimization under Lin-ear Constraints. IEEE Transactions on Evolutionary Computation (2018), 1–1.https://doi.org/10.1109/TEVC.2018.2871944

[14] Patrick Spettel and Michael Hellwig. 2019. EA comparison on the Rotated Klee-Minty Problem. Source code. (2019). Retrieved April 18, 2019 from https://github.com/patsp/RotatedKleeMintyProblem/tree/ea_comparison

[15] Tetsuyuki Takahama and Setsuko Sakai. 2006. Constrained Optimization bythe ϵ Constrained Differential Evolution with Gradient-Based Mutation andFeasible Elites. In IEEE International Conference on Evolutionary Computation,CEC 2006, part of WCCI 2006, Vancouver, BC, Canada, 16-21 July 2006. 1–8. https://doi.org/10.1109/CEC.2006.1688283

[16] Tetsuyuki Takahama and Setsuko Sakai. 2010. Constrained optimization by theepsilon constrained differential evolution with an archive and gradient-basedmutation. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC2010, Barcelona, Spain, 18-23 July 2010. 1–9. https://doi.org/10.1109/CEC.2010.5586484

[17] Ryoji Tanabe and Alex S. Fukunaga. 2014. Improving the search performance ofSHADE using linear population size reduction. In IEEE Congress on EvolutionaryComputation (CEC). 1658–1665. https://doi.org/10.1109/CEC.2014.6900380

[18] Anupam Trivedi, Krishnendu Sanyal, Pranjal Verma, and Dipti Srinivasan. 2017.A unified differential evolution algorithm for constrained optimization problems.In 2017 IEEE Congress on Evolutionary Computation, CEC 2017, Donostia, SanSebastián, Spain, June 5-8, 2017. 1231–1238. https://doi.org/10.1109/CEC.2017.7969446

[19] Anupam Trivedi, Dipti Srinivasan, and Nimagna Biswas. 2018. An ImprovedUnified Differential Evolution Algorithm for Constrained Optimization Prob-lems. Technical Report. http://web.mysites.ntu.edu.sg/epnsugan/PublicSite/Shared%20Documents/CEC-2018/Constrained/Improved_Unified_Differential_Evolution_CEC_2018_Report.pdf (Technical Report accessed on Mar 11, 2019).

Comparison of Contemporary Evolutionary Algorithms on the

Rotated Klee-Minty Problem

Supplementary Material

This supplementary material accompanies the GECCO 2019 workshop paper of the same title andthe same authors. It contains additional experimental data in tabular form. In particular, the qualityindicators illustrated in Fig. 3 and Fig. 4 of the workshop paper are provided.

The indicators aggregate the results obtained from 15 independent algorithm runs by making useof the lexicographic ordering relation �lex, refer to Eq. (4). The following indicators are displayedtogether with the optimal fitness value fopt = N3 for each search space dimension considered (N ∈{2, 3, 5, 10, 20, 40}). For a more detailed examination of the mutual algorithm performance, thesesignificant differences among these indicators can also be examined by use of statistical hypothesistesting, e.g. by using the Wilcoxon singed-rank test.

fbest fitness of the best solution found in all 15 runs

fmed fitness of the median solution

νmed constraint violation of the median solution

|fmed − fopt| absolute fitness error of the median solution

FR feasibility rate FR defined as the quotient of those runs that re-alized a feasible solution and the total number of algorithm runs

‖y − yopt‖ mean deviation of all final and feasible algorithm realizations yfrom the known optimal solution y∗ over 15 algorithm runs

meanFevals mean number of function evaluations consumed until termination

The final algorithm results corresponding to the comparison of eight algorithms considered inSec. 4.1 of the workshop paper are presented in Tables 1 to 8. The presentation style is inspired bythe CEC competitions on constrained real-parameter optimization. It allows for comparing differentalgorithms with respect to realized median objective function values. Further, algorithm performancein the search space is measured by taking into account the mean deviation of the best found candidatesolution from the optimal solution. An illustration of these quality indicators is given in Fig. 3 and4 of the workshop paper (provided that the respective algorithm realizes feasible solutions in each ofthe 15 runs).

The results displayed in Tables 9 to 11 correspond to the algorithm variants that replace the ε-levelconstraint handling with the lexicographic ordering approach. These algorithms are taken into accountwithin Sec. 4.2 (Fig. 5). Table 12 provides the final results of the iUDE variant that establishes a fastreduction of the ε-threshold. This algorithm variant is considered in Fig. 6 of the workshop paper.

1

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

01.5

737563

0e-

08

9727.6

73

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.7

539392

8e-

08

18444.3

35

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.0

0000000e-

09

1.0

04.3

024276

9e-

08

47156.9

310

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

1.6

0731406e-

06

1.0

06.8

130123

0e-

06

102038.2

020

8.00

e+03

8.00

0000

00e+

038.0

0000000e+

03

0.0

0000000e+

00

4.5

4973815e-

07

1.0

02.7

061341

2e-

06

210502.5

340

6.40

e+04

6.40

0000

00e+

046.4

0000000e+

04

0.0

0000000e+

00

3.1

6998921e-

06

1.0

02.4

542484

1e-

05

447974.3

3

Tab

le1:

Con

SaD

E

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

01.5

085114

7e-

08

25041.0

03

2.70

e+01

2.70

0000

00e+

012.

70000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.4

262004

5e-

08

43254.3

35

1.25

e+02

1.25

0000

00e+

021.

25000000e+

02

0.0

0000000e+

00

9.3

1471789e-

09

1.0

04.2

474729

4e-

08

92134.3

310

1.00

e+03

1.00

0000

12e+

031.

00000041e+

03

0.0

0000000e+

00

4.0

6482002e-

04

1.0

02.8

818374

0e-

03

200001.0

020

8.00

e+03

8.00

0017

03e+

038.

00004713e+

03

0.0

0000000e+

00

4.7

1343116e-

02

1.0

02.7

209238

0e-

01

400001.0

040

6.40

e+04

6.40

0028

05e+

046.

39589826e+

04

1.6

4980009e+

02

4.1

0174116e+

01

0.3

33.5

655726

3e+

02

800001.0

0

Tab

le2:

EC

HT

-DE

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

01.3

9381656

e-08

27024.2

73

2.70

e+01

2.70

0000

00e+

012.

70000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.2

2144282

e-08

47078.9

35

1.25

e+02

1.25

0000

00e+

021.

25000000e+

02

0.0

0000000e+

00

5.4

5501848e-

08

1.0

02.5

7066580

e-07

100018.0

010

1.00

e+03

1.00

0000

68e+

031.

00000122e+

03

0.0

0000000e+

00

1.2

2198641e-

03

1.0

07.0

3545152

e-03

200056.1

320

8.00

e+03

8.00

0064

33e+

038.

00007671e+

03

0.0

0000000e+

00

7.6

7098748e-

02

1.0

05.0

2152648

e-01

400220.1

340

6.40

e+04

6.40

0030

83e+

046.

40003322e+

04

0.0

0000000e+

00

3.3

2211704e-

01

1.0

02.1

2962362

e+00

800450.8

7

Tab

le3:εD

Eag

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

||y−yopt||

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.0

090742

0e-

08

5498.5

33

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.6

216486

4e-

08

10045.0

75

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.4

8705292e-

09

1.0

04.2

856149

7e-

08

24115.8

010

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

9.4

4476142e-

09

1.0

05.3

435716

8e-

08

79201.0

020

8.00

e+03

8.00

0000

00e+

038.0

0000000e+

03

0.0

0000000e+

00

9.5

7061275e-

09

1.0

05.8

793150

9e-

08

217896.6

740

6.40

e+04

6.40

0000

00e+

046.4

0000000e+

04

0.0

0000000e+

00

9.8

2254278e-

09

1.0

01.0

805410

2e-

07

576247.2

0

Tab

le4:

LS

HA

DE

44

2

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

||y−yopt||

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

0.7

34.0

477529

9e-

01

28490.0

03

2.70

e+01

2.70

0000

00e+

012.

72976481e+

01

2.0

1738622e+

00

2.9

7648143e-

01

0.4

02.2

947574

3e+

00

49282.2

75

1.25

e+02

1.25

0000

00e+

021.

17966924e+

02

8.6

5296186e+

01

7.0

3307641e+

00

0.0

76.6

972954

9e+

01

97186.8

010

1.00

e+03

0.00

0000

00e+

000.

00000000e+

00

1.0

1542661e+

03

1.0

0000000e+

03

0.0

01.0

154266

1e+

03

200050.0

020

8.00

e+03

0.00

0000

00e+

000.

00000000e+

00

8.1

2341290e+

03

8.0

0000000e+

03

0.0

08.1

234129

0e+

03

400100.0

040

6.40

e+04

0.00

0000

00e+

000.

00000000e+

00

6.4

9873032e+

04

6.4

0000000e+

04

0.0

06.4

987303

2e+

04

800200.0

0

Tab

le5:

iUD

E

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

||y−yopt||

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

06.0

858866

8e-

02

15266.4

03

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.5

679894

3e-

08

23636.0

05

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.0

0000000e-

09

1.0

03.8

060858

2e-

08

39102.4

010

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

9.0

0000000e-

09

1.0

05.1

866663

2e-

08

83248.7

320

8.00

e+03

8.00

0000

00e+

038.0

0000000e+

03

0.0

0000000e+

00

9.4

6511136e-

09

1.0

06.0

548766

0e-

08

194422.1

340

6.40

e+04

6.40

0000

00e+

046.4

0000000e+

04

0.0

0000000e+

00

9.6

6974767e-

09

1.0

02.5

134986

3e-

06

544200.4

7

Tab

le6:εM

Ag-E

S

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

||y−yopt||

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000000e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.0

0100915

e-06

2764.2

03

2.70

e+01

2.70

0000

00e+

012.

70000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

06.8

9578984

e-08

28082.8

75

1.25

e+02

1.25

0000

00e+

021.

25000000e+

02

0.0

0000000e+

00

3.1

6959785e-

07

0.8

03.8

4001175

e-01

66712.3

310

1.00

e+03

1.00

0000

04e+

031.

00015893e+

03

0.0

0000000e+

00

1.5

8927151e-

01

0.5

32.4

5406562

e-01

200002.0

020

8.00

e+03

8.00

0020

14e+

038.

00000764e+

03

1.6

0812363e-

08

7.6

4449326e-

03

0.1

31.1

0411738

e+00

400002.0

040

6.40

e+04

6.40

0000

01e+

046.

40000147e+

04

3.4

3804173e-

06

1.4

7123790e-

02

0.0

72.2

8827143

e+00

800002.0

0

Tab

le7:

Act

ive-

Set

-ES

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

||y−yopt||

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000000e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

09.0

000000

0e-

09

740.6

03

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

09.0

000000

0e-

09

1771.4

05

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.0

0000000e-

09

1.0

04.1

493137

6e-

02

26196.0

010

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

9.4

9069090e-

09

1.0

08.6

306825

8e-

02

88248.6

020

8.00

e+03

8.00

0000

01e+

038.0

0000043e+

03

0.0

0000000e+

00

4.2

6878859e-

04

1.0

01.0

195023

2e-

02

400031.0

040

6.40

e+04

6.40

0001

22e+

046.4

0000398e+

04

0.0

0000000e+

00

3.9

7965899e-

02

1.0

06.2

170243

2e-

01

800102.0

0

Tab

le8:

lcC

MS

A-E

S

3

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

01.2

293895

5e-

08

1707.4

73

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.8

828156

2e-

08

5284.8

05

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.0

0000000e-

09

1.0

03.4

994004

3e-

08

9775.8

710

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

9.0

1491148e-

09

1.0

05.3

060634

9e-

08

24109.6

720

8.00

e+03

8.00

0000

00e+

038.0

0000000e+

03

0.0

0000000e+

00

9.5

6424628e-

09

1.0

06.4

551218

4e-

08

83333.6

740

6.40

e+04

6.40

0000

00e+

046.4

0000000e+

04

0.0

0000000e+

00

9.7

3523129e-

09

1.0

06.6

694347

8e-

08

345932.8

0

Tab

le9:

lexM

Ag-E

S

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

01.4

1485366

e-08

4925.6

73

2.70

e+01

2.70

0000

00e+

012.

70000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.4

3434093

e-08

11050.2

75

1.25

e+02

1.25

0000

00e+

021.

25000000e+

02

0.0

0000000e+

00

9.0

0608654e-

09

1.0

04.5

7978724

e-08

29606.6

710

1.00

e+03

1.00

0000

00e+

031.

00000000e+

03

0.0

0000000e+

00

9.3

1811428e-

09

1.0

04.9

4928378

e-08

146899.6

020

8.00

e+03

8.00

0001

90e+

038.

00000292e+

03

0.0

0000000e+

00

2.9

2135866e-

03

1.0

01.8

9821384

e-02

402093.3

340

6.40

e+04

6.40

0026

51e+

046.

40002867e+

04

0.0

0000000e+

00

2.8

6711667e-

01

1.0

01.8

9367521

e+00

804557.4

0

Tab

le10:

lexD

Eag

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.0

0000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

1.0

02.2

651518

2e-

08

1766.0

03

2.70

e+01

2.70

0000

00e+

012.7

0000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

1.0

04.3

978082

7e-

08

3910.6

75

1.25

e+02

1.25

0000

00e+

021.2

5000000e+

02

0.0

0000000e+

00

9.5

5236601e-

09

1.0

01.0

275713

3e-

07

10058.4

010

1.00

e+03

1.00

0000

00e+

031.0

0000000e+

03

0.0

0000000e+

00

9.2

2818799e-

09

1.0

01.9

144862

4e-

07

34216.6

720

8.00

e+03

8.00

0000

00e+

038.0

0000000e+

03

0.0

0000000e+

00

9.5

3059498e-

09

1.0

03.4

649462

8e-

07

119100.0

040

6.40

e+04

6.40

0000

00e+

046.4

0000000e+

04

0.0

0000000e+

00

9.8

2981874e-

09

1.0

05.7

728640

3e-

07

439373.3

3

Tab

le11:

lexiU

DE

Nf o

pt

f best

f med

ν med

|fmed−f o

pt|

FR

‖y−

yopt‖

mea

nF

evals

28.

00e+

008.

0000

0000

e+00

8.00000001e+

00

0.0

0000000e+

00

9.0

0000000e-

09

0.9

31.6

949407

5e-

01

12264.6

73

2.70

e+01

2.70

0000

00e+

012.

70000000e+

01

0.0

0000000e+

00

9.0

0000000e-

09

0.8

04.3

167465

2e+

00

23897.4

05

1.25

e+02

1.25

0000

00e+

027.

43233365e+

01

1.2

2044477e+

02

5.0

6766635e+

01

0.4

01.5

402737

4e+

02

69922.2

010

1.00

e+03

1.00

0000

00e+

031.

00000000e+

03

0.0

0000000e+

00

9.5

1149559e-

09

1.0

01.8

874922

7e-

07

54303.3

320

8.00

e+03

8.00

0000

00e+

038.

00000000e+

03

0.0

0000000e+

00

9.8

2163328e-

09

1.0

03.3

892972

0e-

07

147326.6

740

6.40

e+04

6.40

0000

00e+

046.

40000000e+

04

0.0

0000000e+

00

9.8

9530236e-

09

1.0

07.2

461803

3e-

07

474493.3

3

Tab

le12:

iUD

E(f

ast

erε

dec

ay)

4