On the BMDGAs and Neural Nets

hlarco Carpentieri

Abstract- We analyze a bivariate marginal distribution genetic model in case of infinite populations and provide relations between the associated infinite population genetic system and the neural networks. A lower bound on population size is exhibited stating that the behaviour of the finite population system. in case of sufficiently large sizes. can be suitably approximated by the behaviour of the corresponding infinite population system for a number of transitions exponentially greater than that suggested by Vose's analysis. The infinite population system is analyzed by showing that. conversely to what happens in the univariate case. the fitness is not a Lyapunov function for its asynchronous variant. The attractors (with binary components) of the infinite population genetic system are characterized as equilibrium points of a discrete (neural network) system that can be considered as a variant of a Ho~field's network; it is shown that the fitness is a Lyapunov function for the variant of the discrete Hopfield's net. The genetic algorithm based on the proposed infinite population system is experimentally compared with the (neural) network algorithm for the Max - Cut problem. Our main result can be summarized by stating that the relation between marginal distribution genetic systems and neural nets is much more general than that already shown elsewhere for the univariate models.

T. Introduction Genetic algorithms are probabilistic search algorithms

inspired by mechanisms of natural selection and genetics. introduced by John Holland in the 1970s. They have received considerable attention because of their many applications to several research fields such as optimiza- tion. adaptive control and others [ I 01. [ I 51. [I 61. [19].

A classic way to describe the behaviour of genetic algor ~ thms is obtained by means of homogeneous Marltov chains [9], [ll]. [29] whose states encode populations and are multi-sets of binary strings. General theoretical results were introduced for infinite populations by Vose [36], [37] who showed how to use them to perform the qualitative analysis of the behavior of the finite populations models. In particular. in [36]. [37]. a dy- namical system model is introduced for which simulation is computationally difficult. It is worth noting that the original formalism presented by %se mas intended to model situations in which recombination of genetic material is obtalned through the crossover of the chro- mosomes of two mating parents selected with probability in proportion to their fitness. Thus the intractability of simulating any general system (such as Vose's) is due

hI;nl~ls(.ript r(v:(.i\,cd Now:~r~b(.r, 4. 2006. hlar(.i, Carprntirri is with thr Dipartimrnt~ (li Fisica E. R .

Caianirllo (DF); I-nivn-sitA ( l~gl i Stu(li (li Salrl-n(~ (Sa.), Italy

to the fact that it keeps track of every chromosome [3S]. To avoid exponential complexity one may consider approximate models, or. alternatively. restrict the type of fitness functions (see for example [31]). different approach consists in changing one's mind about what is being modeled, thus, representing some other related genetic system. This last alternative is followed by the present paper (along with restriction of the fitness to some classes of polynomial functions) following the guidelines of references [2]. [3]: in such papers a model has been proposed in which the recombination of the genetic material is obtained by means of the bit-based simulated crossover operator [33]. This rule. as well as the gene pool recombination introduced by Miihlenbein and l;oight [35]. [24] maintains an Infinite population in linkage equilibrium: the genotype frequencies are the product of marginal frequencies. In this context. a fre- quency vector with exponentially many components can be reconstructed by the vectors of marginal frequencies that in the univariate marginal distribution algorithms [2]. [3], [27]. [24]. [35], [38] are I-component vectors. where throughout by 1 we mean the chromosome length. Other models based on the marginal frequencies are presented in [I], [13], [20]. Related work can be also found in [3S] in which it is analyzed a recombination- mutation-selection genetic algorithm that uses gene pool recomblnation. In particular \.\/right et al. show that in case of linear fitness functions there is a single stable fixed point for their univariate marginal distribution genetic algorithm. Moreover. readers interested in exact mathematical analysis of simple genetic algorithms and their use as an alternative approach to combinatorial optimization are referred to [31].

In this paper. we consider the problem of extending the analysis of univariate marginal distribution genetic algorithms for infinite populations (UMDGAs) to the bivariate case (BMDGAs). IVe review and analyze the genetic model based on simulated crossover of fixed sequences of two bit genes introduced in [5]. Such a model represents an instance of the Random Heuristic Search (as defined in chapter 3 of reference [37]) and can be considered as an extension of the model presented in [33]. [35]. [2]; the main characteristic of the system that %ve shall consider is that the recombination of the genetic material is obtained by performing a weighted average of the alleles along each fixed two-bit locus and using such statistics to produce offspring whose alleles in distinct l oc~ are independently generated. The model 5ve propose is more tightly connected to the classic Holland's framexvorlt with respect to the models that have been presented in the literature follommg the more recent development of marginal distribution genetic algorithms (we mean for example the area of the Estimation of

149

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

1-4244-0703-6/07/$20.00 ©2007 IEEE

Distribution 4lgorithms [23] [26]. [30]. [39] [do]) In fact. accordrng to the Holland s the015 the fittest individuals chromosomes are formed by classlc genetlc algonthms merglng short definition length and small specificit) order allele schemata. %\hose fitness remalns above the average fitness of the populatlons generated by the genetic cycles (Holland [lo]. [15]. [16]) This central result has introduced the concept of separabihty of the fitness functions with respect to short chromosomic traits. that is recognized [8]. [28]. [32] as a basic propert) required to justrfy the appllcatlon of a (classic) genetic alzorrthm The main idea of the model me Diesent IS " that of proceeding in bottom-up fashion in such a way to compute the (average) fitness associated to some of the chromosome building-blocks (alleles) and translate such information into statistics that are used to generate nem offspring The may in mhich such stat~strcs are used 1s almed at preserving positions of the alleles. Thus. the reader can understand that the sense in which our bivarrate model is an extensron of the unrvanate marginal distnbutlon genetic algonthms is drfferent from that proposed to sol-ce problems that contain significant dependencies and that cannot be classified as linear or decom~osable ~roblems 1301. Note that the rnteiest

L ,

in devising marginal distribution genetic models lies not only in the fact that they consent efficient (state transition) implementation for infinite populations. In- deed. in case of univariate marginal distributions. they have been used to construct approximation algorithms to solve hard combinatorial problems for which error bounds can be theoretically estimated. Moreover. the stability analysis of such systems has evidenced an interesting relation mith neural networlts (in particular with Hopfield's Networlts [2]. [3]).

f\/e exhibit the following results about the genetic system based on simulated crossover of fixed sequences of two bit genes.

1) we provide an exponential lower bound on the con- centration probability (viewed as function of the population size n ) stating that for sufficiently large population sizes the finite population stochastic system can be considered as an approximation of the infinite population deterministic system the result is interpreted stating that the behaviour of the finite population system. in case of sufficiently large sizes n. can be suitably approximated by the behaviour of the corresponding infinite population system for a number of transitions (considered as a function of n) exponentially greater than that suggested by l:ose's analysis in [37];

2) the infinite population system is analyzed by showing that, conversely to what happens in the univariate case [4]. the fitness is not a Lyapunov function for its asynchronous variant; the at-

31 "

tractors in ( 0 . 1 ) ~ ( 1 even) are characterized as equilibrium points of a discrete (neural network) system that can be considered as a variant of a Hopfield's network;

3) it is shown that the fitness is a Lyapunov function for the discrete neural networlt system

4) the genetic algorithm based on the proposed infi- nite population system is experimentally compared with the (neural) network algorithm for the Max-

Cut problem; the results show that the perfor- mances are comparable being those obtained by the neural network algorithm slightly worse.

11. Tlie LnIodel \Ve revrew the model [5] on which the genetic system

is based and introduce the technical formalism useful to define states and dynamics. ,4 population P of individ- uals is represented by a multi-set of n E N I-length binary strings (throughout the paper we suppose 1 even) from the set

Each population P is associated with its frequency vector F = (F,,. . . . . FY2] ) specifying the propor tion of the strings in 0 contained in P. where F,, = and ni, is the number of occurrences of the string in P. u .. Let ii,, denote the set of the freauencv vectors that

L d

represent populatlons of n ~ndlvrduals Each individual is evaluated by his fitness that is measured by means of a fitness functlon f C2 -+ Rf that associates a positive real value to each chromosome. Throughout the paper. let 1 = (00.01.10 11 ) and B = i - (00) The strings in the populations represent chromosomes and each chromosome is divided into a sequence of genes that can assume four distinct forms or alleles For k = 1. . $ and a = a , a2 E 4. consider funct~ons xi, [a] i2 -+ (0.1) defined by

1. if al . are in positions 2k - 1.2k in W ; xi, [a1 =

0. otherwise.

In the rest of the paper we shall use notation

to mean the expectation of function X 0 -4 R con- sidered as a random variable along with the stochastic vector P = (pl.. . . .p,i).

Starting from an inltial population Po, if at time t the state of the (genetic) system is the population P. represented by its frequency vector F. then the population at time t f 1 is obtained as follo~vs

1) for every k = 1; . . . , and a E A compute

2) generate a new population P' of n I-length binary strings. denoted by

mith probability ~ [ a ] of obtaining a 1 . a ~ in positions 2k - 1.2k independently from r , and k for 1 < k < ; a n d l < i < n .

Steps 1. and 2. describe the way rn which recombination of the genetic material is obtained: 11y performing a weighted average of the alleles along each fixed two-bit position and using such statistics to produce offspring

150

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

\\hose alleles in distlnct loci are independent11 generated. The stochastic rule described In 2 1s the b~t-based simulated crossover ~ntr oduced In [33]. [2] extended to fixed sequences of tno-bit genes By definition of the recombmat~on process descr bed In 2 , ~f P 1s a populat~on at t ~ m e t and F ~ t s frequency vector. the population at time t + l is obtained by selecting n strings mlth probab~llty dlstrlbutlon

where the probability @(F),J of generating the string w, = d l 1 . . . w,,! is

111. Probability Coilceiltratioil Results The following theorem states a probability concentra-

tion result on the Marltov chains describing the stochas- tic genetic system in Section TI. The result of the theorem allows to derive an iterative deterministic system with states in [O. 11'" that represents the behaviour of the infi- nite population system. More particularly. for sufficiently large population sizes n. we prove that (provided that the current state of the systems is F ) a transition taltes the finite population stochastic genetic system near the next state of the infinite populat~on deterministic genetic system with probability close to one. The reader should notice that this fact has been shown in greater generality for anv instance of the Random Heuristic Search 1371:

L ,, ho~vever: in case of the considered instance. we are able to provide an exponential lower bound on the concentration probability (viewed as function of n ) that is much better than that exhibited in the more general case in [36]. [37]. Other results concerning this topic. that can be considered as special cases, can be found in [2]. [3], [3].

Theorem 1 : Let E . 6 E (0.11 and

where M is the maximum value that the fitness function can assume: if at time t the system is in the state F. then the state F' at time t f 1 is such that for all k = 1.. . . , q and a E B it results

with probability at least 1 - 6.

Proof It is based on Hoeffding's inequality (see [A] and [GI).

To provide a comparison with Vose's analysis [37]. setting

and assuming that the fitness is lower bounded by some positive constant K > 0. 5ve can state the following.

Theorem 2 Let t. 6 E (0.11 and

where the fitness assumes values in [K. MI; if the system is in state F, at time T for 0 5 T 5 t. then it holds that

Proof The guidelines of the proof can be found in [GI).

For example. the bound in Theorem 2 holds with K = 1 for positive integer fitness functions that are widely used in the applications. Moreover. note that the bound pro- vided by Vose in [37] states that the probability that the infinite population system is a suitable approximation of the finite population system for t state-transitions is bounded by 1 - 5. In practice. the result of Theorem 2 can be interpreted 11y saying that the behaviour of the finite population system, In case of sufficiently large sizes n. can be suitably approximated by the behaviour of the corresponding infinite population system for a number of transitions (considered as a function of n) exponentially greater than that suggested by Vose's analysis.

TV. Fitness Functions First of all. m-e briefly review the topic of efficient im-

plementation in case of arbitrary finite fitness functions f : n - R' and for infinite populations. For ; E B and k = 1.. . . , q. one has

By (3) we observe that efficient implementation depends. not only on the dimension of the involved states. but also on the type of fitness funct~on. In this regard, it 1s well Itnown that f can be expressed in terms of multivariate polynomials and this fact can be (naturally) used to characterize classes of functions for which we are able to perform efficient implementation in the sense that state transitions can be computed in time polynomial in I . Let, now.

be a multivariate polynomial defined on [o. 11' and coincident with f on 0. Notice that since Pf is a polynomial of degree at most one in each variable its global maximum is on elements in 0. Denote. for u = I . . . . . q and a E A1. by ?,,[a] the product of zt,l), . z t , ~ ) , that is:

151

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

Polynomial (4) can be rewritten in the form

where

and bi. ,,(%) does not depend on the variables x2i.-1. x ~ k for every a E A and k = 1.. . . . q. Calculating the expectations. being ~ i . ~ [ 0 0 ] = 1 -CnER QX ~ [ a ] for each k = 1, . . . .q, me get

by linearity of the mean and by independence. Moreover. since

EWF) [xi; [Z]i!k [a]] Qli F [z] if z = a

= ( 0 otherwise

(a. z E -4). one has for I; = 1.. . . . q and z E B that

E * ( ~ ) [xi. Is] Pf 1 = 1 [yk [z]?L [a]] EaCF) [bk .] a t 4

= O A , F [ ; ] ~ L , - ( ~ ~ ) .

Therefore. in case of infinrte populatrons Theorem 1 implies that as n --- x the stochastic genetic system con- verges to an Infinite population detei minlstic system the states of such a determinlst~c system are 3q-component vectors

in [O. lIiq. with CoER ~ i . , ~ < 1 (1 < k < q) and the dynamics is described. for z E B and k = 1.. . . , q. by the equations

where

Note that TJ~. - (t) in Equation (6) represents the proba- brlity of having z as the k-th allele aftei t transitions.

The state space of the iterative deterministic genetic system for rnfinite populations is a subset A(3q) C [O. 11'" of 3q-component vectors 2: such that CntB ~ i , , ~ ( t ) 5 1 (1 5 k 5 q). Moreover. by76) it 1s clear that. to be able to perform (state transition) efficient implementatron. the terms bi, 3 ( ~ ( t ) ) for all k = 1. . . . . q and s E B must be computrd In tlme polynomial in the chromosome length 1. ln such class of funct~ons theie are Important tvnes of fitness functions such as auadratic ones that

d .

a.re useful to model hard ontimization ~roblems: a more general class consists of the functions than can be expressed as sums of monomials (products) of at most O(1og I' ) variables, where c > 0 is constant.

V. Analysis By equations (6) it follows that

(t)

- 1 , ( ( t ) ) - ( ( t ) ) ) ] (7) o E R - ( )

for k = 1 . , q and z E B \\/e first, remark that the fitness 1s not a Lyapunov functlon for the asynchronous variant of the system as in the case of the univariate (infinite population) genetlc system introduced in [2]. [A] (by asynchronous me mean that the components of the state vector are updated one at a time in a predefined order) in case of the blvariate (rnfinite population) system. me are only able to state necessary cond~t~ons such that this property holds In the other cases. it is quite srmple to find fitness functions and states ~ ( t ) for mh~ch state-updating In a slngle locus and for ;single allele produces a decrease of the fitness

Theorem 3 If Z / L - (t + 1) = ui. - (t) + AVI. - (t) (z E B) and uh . ( t + l ) = Z/L (t) for a E B - (2). then cond~t~ons

imply

where H S1(), HS() are heavy-side functions defined by

and

Proof The result follo~vs by quite straightforward manipulations of A P f ( l ~ ( t ) ) and by inspection of the conditions on the bi.,,,.&,: polynomials implying that the fitness does not decrease.

152

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

Denote by ,%-Ai, = {x E (0.1)'" : Co,,zx,c, < 1 for k = 1.. . . . q} the subset of the state space composed by the vectors with components each assumlng values in (0.1). A11 points in XI\{, are fixed points of System (6) as stated by the follo~ving lemma.

Lemma 1. Tf x t Ni\?,. then it holds that

Ay~,,=(t)l,(~)=, = 0, for each I; = 1.. . . .q and r E B.

Proof It is straightforward by (7).

Set XI, = (xi.01.x~ IO.ZI , 1 1 ) (1 5 k 5 q). The next theorem states sufficient and necessary conditions in order that a point x t Yil i, is an attractor of the System (6).

Theorem 4 The polnt x E lYAi, IS an attractor of the System (6) if and only if for every k = 1.. . . , q ~t holds that xi, = 0 and

n H S ( ~ L ,oo (x) - b ~ , , (x)) = I n E R

or XI,. # 0 and

n HS(bx :(x) - bx,, (x)) = X A z

o E A - { z )

for z E B, where H S() is the heavy-side function defined by

Proof First, we get the lineal approximation of the infinite population genetic system in the neighbourhood of the point x E L%A+,. Then ive notlce that x IS

an attractor if and only ~f the Jacobian matrix of the linearized system has eigenvalues mith modulus less than one. Since the Jacobian of the linearized system can be decomposed as the dlrect sum of three- order blocks. the result follows by using the Laplace's expansion and examining separately each of the cases XI, = (0.0.0). (0.0.1). (0.1.0). (1.0.0).

VT. Neural Network By Theorem 4 we are able to design a discrete deter-

ministic netmoi k m hose equilibr ium points, for arbitrary (positive) fitness. Include the attractors of System (6). This alm has been already obtained in [2]. [3]. [4] for a simpler univarlate genetic system whose attiactors have essentially been proved to be the equ~libr~um polnts of a discrete Hopfield s netmorlc with sequential updating and havlng as energy function the fitness In this context the equivalence seems to be less satisfactory in the sense that for some unstable equilibrium points x t ,% 44, mlth x~ = 0 (for some I; E (1. .q}). the behaviour of the associated discrete netivork cannot accordingly be denved by the result in Theorem 4 Hoivetter as far as me are interested in maximizing the fitness. it is reasonable to take as updatlng rule 5% hat locally maximizes the fit- ness Thus. we define the discrete deterministic network associated to the infinite population bivariate marginal distribution genetlc system having as states the points x E ?1121, and 5% lth sequentla1 updating equations

~ i , . ~ ( t + 1) = n H S(bi,,z(x) - b k ; , (x)) for r E B

if it holds that

for some z E A 1 and

otherwise. where xi; E ((0: 0: 1): (0; 1: 0); (1; 0.0)) is the choice that locally maximizes the fitness in x(t) with xk(t) = xi, (1 < k < q). The following theorem states that the fitness function is a L,yapunov function for the iterative discrete network; consequently. i e are able to derive an approximation algorithm to maximize the fitness over NA3v

Theorem 5: For every x(t) E NAs, it holds that

if x( t + 1 ) is obtained by (8) and A Pf (x(t)) >- 0 otherwise (x(t + 1 ) is computed by (9)).

Proof The result folloivs by lnspectlon of the cases (XA ( t+l ) = O)A(XI. (t) = 0). (XI. ( t+ l ) = O)A(XA (t) Z 0). (XI, (t + 1) # 0) A (XI, (t) = 0) and eventually (XI, (t + 1) # 0) A (xx (t) # 0).

VII. Application to the Ma.x-Cut Problem The topic of designing approximation algorithms

based on infinite population genetlc models (mith sim- ulated crossover of one-bit genes) has been studied in [2]. [3]. [4] for hard problems. Tn summary. the available maln theoretical results about such models evidence that the fitness function becomes a Lyapunov function for asynchronous variants of the corresponding iterative dynamical systems. As a consequence of thls property. it is conceivable to design univariate marginal distribution genetic algorithms that can be used as approximation algorithms to solve combinatorial optimization problems. Other results connecting such genetic systems with Hopfield's Networks (well known local optimizers on which some approximation algorithms are based) can be found in [2]. [3]. The usual uay of using the infinite population systems to get local optimization of the fitness 1s to initialize with (slightly perturbed) equally likely marginal probability distributions for each allele (see also [2]. [3] and In particular [A]). ln the expenments. the synchronous valiants of the consideled systems have exhibited convergence and optimization properties very similar to those shown by the asynchronous systems. Follo~ving the guidelines of references [2]. [3], [4], [5]. System (6) has been implemented to solve (in the sense of an approx~mation algorithm) the Max-Cut problem. namely. that of partitioning the vertices of an arbitrary graph G In two other subsets 1'1 and i ' z in such a way that the number d ( l ~ ) of edges mith one endpoint in 1/1 and the other in 1; is maximized. IVe remind that the decision version of the Max-Cut problem 1s NP-complete [17]. In the genetic algorithm. we have considered the quadratic fitness f . 0 -+ N+ defined by

153

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

\\here the rneights w~,, are set w z , = 1 ~f the input graph G has edge { a , 3 ) and u, , = 0 other rnise f\/e i emar k that initial equally lrl<ely marginal probabihty drstiibutrons for each allele are fixed points. in case of undirected graphs, for the system based on recombrnation of se- quences of one-bit genes, hornever. in [2]. [3] it has been shonn that such points are not asymptoticallj stable. Conversely. for the system based on simulated crossover of sequences of two-bit genes, equally lilcely marglnal d~stributions are not fixed points. but they initialize trajectones that do not converge toxvaids states rn Q. Never theless. the Initial shght perturbations (wi, - (0) = f f ?. %here 2 = lo-' for k = I . . . q and 7 E B) consent to solve this problem In our experrments the genetrc algorithm wrth simulated crossover of trno-b~t genes exhibited con-cergence characteristics verj similar to those held by the univariate maiglnal drstiibutrons genetrc algorithm

In Table 1/11 1 there are the experrmental results intended to compare the performance of the algorithm 2BGSC based on srmulated crossover of sequences of two-bit eenes wrth the 2DNN based on the coires-oond- - ing discretized (neural netmorlt) system initialized mith random points and with sequential updating. In the table it is reported the mean size of the cuts found by the two algorithms for p-random graphs mith p = $ and p = f. In the table it is also reported the expected number of edges (row Edges) of the p-random graphs. The performance of the algorithm 2BGSC is slightly better (in average) than that of the 2DNN algorithm conversely to what happened in the univariate case [2] in which the infinite population genetic algorithm. in case of p-random graphs for p = $.. i. exhibited performance slightly worse than that of an Hopfield's netmorlt [Is]. By completeness. in Table VfT.1 we report the results of the simulations intended to compare the performance of the algorithm 2BGSC with the univariate l B G S C based on recombination of sequences of binary genes introduced in [2]. As 5ve have already noticed elsewhere. the algorithm 2BGSC consents to improve (in average) the performance of the algorithm 1BGSC. Moreover. by giving a look at the mean sizes of the cuts obtained by choosing the best ones. for a same p-random graph. found by the two algorithms (rows 12BGSC). we note that the performance %vas dependent on specific generated p-random graphs t h ~ s fact suggests. as a conjecture. that mutations in positions and/or lengths of genes could be helpful to obtain better results and to speed convergence. In t h ~ s regard, even ~f our models are merely computational. such mutations could be explained as having the precise aim of adaptat~on to environment mod~fications that. in case of evolutronary processes. are very slow.

VTTT. Coiiclusion

In this paper. me have reviewed and analyzed a brvariate marginal distribution genetic model. Our aim is oriented to extend the analysis of univariate marginal distribution genetic algorithms for infinite populations to the bivariate frame~vorlc. The choice of the bivariate model is for sake of conciseness and simplicity both in the exposition and in technical details. By a first preliminary analysis. we conjecture that the results exhibited in

T a h l e V I I . I : S lean size of t h e c u t s f o ~ ~ n d hy t h e a l g o r i t h m s

2DNN 61.8 75.6 9 1 9 107.2 126.9 lBGSC 62.2 76.4 92.6 108.2 127.3

1 2BGSC 63.3 76.7 93.1 108.7 128.2 - 7 12BGSC 63.7 7 7 . 94.1 109.5 128.8

E(1grs 80 1 100.4 125. 0 147.9 175.0

2DNN 96.2 118.9 145.0 175.5 201.0 lBGSC 96.7 119.8 145.3 176.1 201.3 2BGSC 97.2 120.6 146.6 177. 1 204.2 l2BGSC 97.9 121.4 147.2 178.5 205.9

Edg('s 1L0.3 175.8 215.3 258.8 306.3

this paper can quite straightforwardly be translated into the multivariate framework. However. further research is reauired to better understand how to ~rovide a suitable more general model for marginal distribution genetic algorithms (some more recent insigths are addressed in the Estimation of Distr ibution Algorithms literature and the related research lines. see [26]. [30] for an introduc- tion). \Ve are convinced that an important topic in this scenario is constituted by the possibility of extending the relation found by (infinite population) univariate marginal distribution genetic systems and discrete Hop- field's networks. Such extension is considered to be of particular relevance to answer to some crucial questions such as the meaningfulness of genes codification. how to improve performance and. in particular. the design of new models that could be more convenient to try to solve hard optimization problems (in this regard the reader is also refer red to the area of classic approximation algorithms with special attention to the Goemans and Williamson results [12]).

Acknowledgment This paper is memory of my father Mario Carpentieri.

[ I ] S. Bal~l,jt~ and R. C'ar~larra; R('rliovirrg L ~ I ( . Ck~n~c>Lirs Trorr~ thr Stan(lar(1 Grnrtic i4lgi~l-ithrns. Pl-(~(.~r(l ings i,f the Tw~l f th Intrl-na.tiona1 Confrl-rncr on h l ach in~ I,rasning, pp. 38-46. h1orgtin1 Ka~~frritiri, St111 Frtinlris(:o; C't~lilorllia.

I % ] 11. B(>~Lorri. P. Canripad(~lli, hI. Carp(:riti(\ri, C+. C+rossi, 14 Grnrtic: h lo( l~1 and thr Hilpfirl(1 Krtworks. I(:ann96: Iritrrnatii,nal C!onfrrrn(:r on i4rtific:ial Krl~ral Nrt~vorks, I,r(:t~~l-rs Kutrs in Cornp~~ t r r Sc:irn(:r; pp. 4ti3 - 468. Sprirrg(:r-V(:rl;rg; B(3rliri; C+c.rrr~tn~y: 1996.

131 11. B(\~Lorri. P. C:anripad(~lli, hI. C:arp(:riti(\ri, G. C+rossi, !111trlysis of a C+(.ri(\Li(: hIod(.l: IC'C+A97: Pro(.(~(vlirigs of tIrc> S(~v(\rrLki l ~ ~ t ~ ~ r n ~ ~ r L i ( ~ r ~ t i l C'on~I(~c~ri(:c> on1 C+(\ri(:li(: i\lgorithms, pp. 12 1 - 126. h'lorgaii-Ka1lfrr1a~r1~1. San hla- t ~ o ; California, 1997; (also prrsrntr(1 a t Nips96: Nrl~ral Iriforrriatiori Pro(:c>ssin~g Syst(~l~rs Worksh(~l~; Sn~ownriass; Colort1(10. D(~ : (~ r r~h(~ r . 1996).

141 14. Rrrtoni; P. Cainpa(lrlli, hl. C!arprnti~ri, G. Grc~ssi, 14 Grnrtic: hli,(lrl: i4nalysis a.ml i4ppli(:atii1n to hlAXSAT, Evi,l~ltii,na.ry C(~inpl~ta. t i(~n, pp. 291 - 309. \;,I. 8. n. 3. hiassa(:l~~ls(~tLs I r i s t i t ~ ~ l ( ~ (11 T(Y.IIII(IIO~Y. 1Iit Prc>ss, Carrlbridg(\, hil l , 2000.

154

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)

151 hl. Cal-prritirri, A Rivariatr hlarginal Distril-~l~tion Gr- nrtic hloclrl. C'EC20Otj: 200ti IEEE Cc~ngl-rss (111 Evil- l 111 i11 t r C I I I I I I ~ ~ ~ ~ O I . 1 0 - 1022. \'ar11:i111\vr, C'arladn, 2006.

[GI hI. Carpc.rlli(:ri, !\ C;~>rrc.ti(. S y s t c ~ ~ Btrs~:d OII Crosso\,(.r of T\vt~-Rit Grnrs, Thrc~l-rti~:al Cornplrtrr S(.irn(:r, liol. 367. r1. 3. pp. X24 - 335. 200ti (a.va.ila.k)lr orilir~r a t \t~~t~\t~.s(:irr~(:r(lil-r(;t.(:(~rr~).

[7] S. !I. C'ook; Ttw Corr~pl(~xily of T I I I Y ~ ~ ( > I I ~ Provillg PI.OI:P~II~PS, Prcx:rrclings of thr Third Syrnpi~sillrn of thr !\C!hl o n thr Th~ol-y of C!ornp~lting, pp. 15 1 - 158. flCM Pr(:ss; h c ~ Sork, NY. 1071.

[8] L. David<lr Epistasis \'trrinr~(:v: 11 \'i~wpoirrt 1111 C;h- Ht~sdll~\ss: Foilrldatiorls of C+(3~~(.li~: l \ lgOri~h~l l~; C:. 1. E. R.awlings Ecli t~r, pp. 23 - 35. San hlatrc~; Ci4: hlilrgan Ka~~frnann, 1991.

191 A. E. Eil-)rn, E. H. I,. !\arts, K. hl. va.n Hrr. Glc~hal C'orlv(:rgc>r~:c> of C:c>rl(:li~: Algorit1111ls: a hItrrko\~ Ctlairr Allalysis: Ptrrall(4 Probl(.rr~ Solvillg frorll hatilr(:, pp. 4 - 12. H. P. S(:h\vrfrl and R. h.liinnrr E(litill-s; Rrrlin and Hri(1rhrrg: Spr ing~r . R~r l in ; 199 I.

1101 D. E. C:ill(ll-)n-g; Grnr t i~ : Algorithms in Srar(:h; Op- tillriztrtioll arrd 1In(:lli111\ L(~tlsllirrg, Add i so~~-W(~s l~~g , R(>tvlillg, M A . 1080.

1 1 I ] D. E. C:ol(lt)rrg; P. Srgl-rst. Finitr h.lask(~v C!ha.in Analy- sis ilf Cknrtic Algoritlirns. Pl-il(:~r(lings ( ~ f thr Sr(:(~n(l In- trrnational Confrl-rn(:r o n Grnrti(: Algorithms and ttirir Applictrtiorls, pp. 1-8. J . C:r(~f(~r~st(~lt(~ Editor; La\vr(~11(:1\ Erlbtrilrll l \ ~ ~ < l ( : i a l < ~ ~ . Carrlbridgc., hIassa(:hi~s~~tls, 1087.

1 1 21 hl. X. C+orrna.ns; D. P. bl'illiainson. Irnprovr(1 i\pproxi- mation !\lgoritlirns for hlaxirnllm Cut and Satisfiak~ility Probl(:llrs L sirlg S(~rr~idc~Grlit(~ P r o g r a r r ~ ~ ~ r i ~ ~ g , Joilrrltrl of t t r ~ ,,\sso~:ialiorl for Corr~p~~lirrg hl;~~:trirl(~r,y~ \/ol. 42. 11. 6. 1993.

1131 G. R. Harik, F. G. 1,1lk)o1 D. E. Gill(lhn-g, T h r C!ornpa(:t Grnrtic i\lgi~l-ithrn; Tr(:hni~:al Rrport 9700ti. Illint~is C:cw(\li[: l\lgOri~h~ll laboralory (illiC:l\L), I!rli\v:rsit,y of Illir~ois, I ! L ~ ) ~ I I ~ lllir~ois, 1007.

1141 .I. Hastar1,Sorn~ Optirnal Inapproxiinability Rrslllts; Pl-il~:~r(lings ( ~ f thr Tlv~nty-Ninth i\Chl Syrnpctsillin o n thr Throl-y of C(~rnpl~ta t i t~n, 1111. 1 - 10. ACRI Pl-rss, h(.w Y'11l.k; M 1'. 1007.

1151 .J. H. Hollt111~1; l r r ( l i~ (~ t io~~ , P~o(:vss~>s of I I I~ (~~(~I I ( : ( : , L(~rr11- ing and Dis(:(~vrry; M I T Prrss. Ca.rnl-)ri(lgr. 1989.

[ I t ? ] .I. H. HoIIarid; i\(la.pta.tiori in Na.t~rl-al and i\rtifi<:ial Systrrns; M I T Prrss. Clarnl-~ridgr. 1992.

1171 J . E. Hop<:rofl and J . D. Llhrlarl; lrltr<~di~c:tii~rr t < ~ Ail- torna.ta Tlirory; I,anglra.grs and C(~inpl~ta.tion. !\(l(lison- \hrslry; 1979.

1 1 81 .I. .J. Hopfirlcl. Nmral hr t~vi~l -ks and Physic:al Systrrns wit11 E~t~:rgc>rll C:ollc><:liv~\ C ' o r r ~ ~ ~ i ~ t ~ i t i o ~ ~ t r l Ak)iliti~>s. Pro- (:(\(~(lirrgs of 1111> N~itior~trl l \(~~r~1~1rly of S(:i(:ll(:(:s? pp. 2554 - 2558. 1982.

1191 .I. R. Kilxa. Cknrti(: Progl-a.inrning; hllT Prrss. ra in- k)ri(lgr; AdA. 1992.

1201 P. Ltrrranlaga. J . 11. Lozarlo; Estillrtrtioll i l l Distrib~~tiorr ~ \ I g ( ~ r i t t ~ ~ ~ r s : a hew, T(tol E1~o~ilti~111~1ry Op t i r r~ i za t io~~~ Kl~iwrr 14c:adrrnic. P~~bl ishrrs , Bostotl: 2001.

1211 S. Lips~:hi~lz; T t ~ ~ ) r y arid Probl(\rrls of Li11c.ar Algvbra; hIr C:ra~,-Jill Books C'orllpa11.y: 1068.

1221 K. hlarton, i\ Ciln(:rntrat io~~-of-hlras~lrr I n rq~ra l i t~~ for C'orltrtl(:ti~lg h1trrkov Chair~s. C;(\(ll~lc>tri(. t111c1 Fi~r~(:t ior~~il Allalgsis: Vol. 6. pp. 556 - 571. Bi~khilw:r; Bawl, 1006.

1231 A. hI(~rrdibilr~~: J . '4. Lozarro, J . hIigi~c>l-Alor~so, Ptlr- allrl Irnplrrnrntatior~ ilf EDAs Rasrcl (In Pl-ok)a.l-lilistic: Graphical hlo(1rls; I E E E Tl-ansa(:tii~ns (In E\v~l~~t i (~nal -y C ' o r r ~ ~ ~ ~ ~ t t ~ t i ( t r ~ , \'ol. 0. 11. 4. 2005.

1241 H. ~ I ~ u I I ~ ~ ~ I I ~ I ( ~ ~ I ~ , H. h1. \;oigllt, C;PII~> Pool R(>(.or~lk~illt~- tin11 irl Ckwc.ti(. Alg<~ritlnrrs: Pro(.(.cdillgs of the hIvta-

hrllristi(:s Irltn-nati(1na.l Ci~nfn-rnw, I. Osrnaii and .I. Krlly E(1itors. Kl~rwrl-, Roston, hlassar:h~lsrtts, 199ti.

[25] H. hliihl~~rrbc~ir~, The: Eqilaliorls for 111(. Rvsporls(: to S(\lc>~:liorl a11(1 its I!sv for Pr~~~li(: t ioll? Evoli~tior~trr,\~ C:OIII- p~ltatiorl, 1'01. 5 (3). pp. 300 - 346. 1008.

1261 H. hliihlrnk)rin, R.. Hijns, Tlir Estimation ilf Distrik~lr- tioiis and thr hlinirnllrn Rrlati\rr Er~t ro~>y Priri~:iplr, E\,oli~tiol~ary Corr~piltatiorr: \/01. 13. 11. 1. pr). 1 - 27. 2005.

1271 H. hIi i l~k~r~b(~irl , T . hlatnlig: E\ rOl l l~ iOl la~~ hlgoril!nrls: fl-urr~ R~(:oirlt)ir1atioT1 tu S~al-<:I1 Distrih~ltictns. Thrilrrti- (:a1 i'\spr(:ts ( ~ f Evctll~tionary C(~inpl~ta.tion. pp. 137- 176. L. Knllc.1: B. haildts and A. Rogc.rs Editors; Sprirlgcr- \'(:rlag; 2000.

1281 B. Naiwlts and L. Knllc.1 A C ( t ~ ~ r p a r i s ~ ~ r ~ ( t f Pr~:di(:tiir(: hlrasl~rrs i ~ f Prohlrin Diffi~:lllty in Ew)l~~tionary i\lgo- rithrns, IEEE Transar:tiilns 011 ~~~~~~~~~~~~~~~y Cornpllta- t i I . 4 ( 1 ) . 1 - 15. 2000.

[20] A. Nix arid hi. D. \ji1sc.; hIod~\lillg C;~>rrc.ti(. ,4lgorittlrrls whit hiarkov Chaills, Arlrlals of hIath(:~~rtrti(.s trrld Arti- ficial Iritrlligrn~:~, 1'01. 5. pp. 79 - 88. 1992.

[:<ill hl. Prlikan, H. hliihlrnl- in, Tlir Rivariat~ hlal-ginal Dislrik)~~lior~ C:(~rl(:tic: !\lgorittlrr~~ !\(lvtrrl~:c>s in Soft C:or~~pi~tir~g - Erlgirr(~~~rirlg D(>sigrr ti1111 r r ~ t ~ ~ ~ ~ ~ f ; i ~ : l ~ ~ r i r r g , Sprirrg(:r-\i(:rltrg; Lorldorl. pp. 521 - 535. 1000.

[:<I1 Y. R.a.hync,\ri(:h, !\. Wig(lrrs(~n~ A11 i\rlalysis ( ] fa Sirriplr Cknrtic: i\lgorithm. Pro~:r~( l ings of thr Follrth Irltrr- ~ l t ~ t i o ~ l a l C ~ I I ~ I ~ ~ I : I I ( . ( : 011 C;~\rr(>ti(: !\lgorithr~~s~ pp. 215 - 221, Rick Bc.1c.w and Lashor1 boo kc^ Editors. hiorgall K ~ I I ~ I I I ~ U I . San lhrt(:o, C.4. 1001.

1321 S. Ro(:li~t; Er~istasis ill C:rr~rti(: Algorithriis Rrvisitrrl, Iriforrna.tion S(:irn(:rs; \ill. 102. pp. 1:3:3 - 155. 1997.

[33] C:. Sys\v(~rda, Sirrlillat~d Crosso\vr irl C+1:111\ti(. 1ilgo- rit1111ls. Proc(n1irlgs (11 t t r ~ S(:(:(JI~(~ \4:orkstlop (11 FOIIII- (lati(1ns of Grnrtic: i\lgilrithrns, D. Wtiitlry E(liti1l-, ~ I P . 2:39 - 253, Morgan Kallfrnann, San hfatro; California, 199:3.

[34] \'. \'aprlik, T11c. hatil~c. i l l Stalisti(.trl Lc>arrlillg TIr(>nry, Sprirrg(:r \'('rlag, h c ~ Sork, 1005.

1:351 H. \:eight; hliihlrnhrin, Grnr P(101 Rr(a~inhina.tioii a.nd thr Iltilization ( ~ f C(~vasian(:rs f(1l- thr Rrrrclrr C+rnrti~: !\lgorithrn, Pl-t~(:rr(lings of thr Sr(:(~n(l 1 E E E Intrl-na- tiorla1 C(III~(:~C>III:C> 011 E\ ,o l i~ t io~~;r~y C ~ ~ I I I } ) I L ~ ~ ~ ~ ~ < I I I , pp. 172 - 177. Z. hli~:llal(:\vit(:tr Edilor, IEEE Press; h(:\v Yol-k. 1995.

[:36l hl. D. VOS~, hloclrlirig Sirnplr Cknrtic: Algt~rithrns, Evilll~tiilna.ry C!ornrxrtation, 1'01. :3. 453 - 472. 1996.

[37] hi. D. \'ow:, The Sirrlplc. Ckw\ti(: 1\1gorill1111: Ttl(~)r,y tl11(1 Foi~rldtrtio~~s, AflT Pr(.ss: C'arllbridg~.. M.4. 1000.

[:<XI .I. H. Rclwr. R. Poli; C. R. Strphrris, i\ Fixrcl Puint 14nalysis ( ~ f a C+rnr Pct(1l <:A with h.ll~ta.ti(~n; Pro(:r~(lings of GECCO, PP. 642 - 649. hlorgari-Ka~lfma.nii; 2002.

[30] Q. Zharlg; 011 t h ~ Co11v1~rgcw1:c> of a Class of Estillrtrtioll of Disi,rik)i~lior~ ~ \ lgo r i l I~~ t l s~ IEEE Trtrl1s~i~:tiolls (111

Evol~~tiilna.ry C!ornrxrtatictn; 1'01. 8. 11. 2. 2004. 1401 Q. Zhang, On Stability i~ f Fixrcl Points of I,iinit hl(~(lrls

of I-nivariatr marginal Distl-il-)~~tioii i\lgilrithrn and Fa(:- lorizcd Distrib~~tiorl Algorithrll; IEEE Trnrlstu:tiorw OII

E\,olutiolrary Cu~rlputatiolr, Vul. 8. 11. 1. 2004.

155

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence (FOCI 2007)