Transient hidden chaotic attractors in a Hopfield neural system

Marius-F. Dancaa,b,∗, Nikolay Kuznetsovc,d

aDept. of Mathematics and Computer Science, Avram Iancu University of Cluj-Napoca, RomaniabRomanian Institute of Science and Technology, Cluj-Napoca, Romania

cDepartment of Applied Cybernetics, Saint-Petersburg State University, RussiadDepartment of Mathematical Information Technology, University of Jyvaskyla, Finland

Abstract

In this letter we discuss the existence of transient hidden coexisting chaotic attractors, a new kind of hiddenattractors, in a simplified Hopfield neural network with three neurons.

Keywords: Hopfield neural network; Transient hidden chaotic attractor; Limit cycle

1. Introduction

A Neural Network (NN) is a mathematical or computational model inspired by biological neural networksand consists of interconnected groups of neurons. Without chaotic behavior neural systems cannot beadequately addressed and fully understood [1]. Neurobiological chaos, omnipresent in the brain, points outseveral possible avenues at understanding how the brain works and is demonstrably so, in the somatosensoryand the olfactory cortices [2]. Many NNs, such as discrete time NNs, or continuous (time-delayed) NNS,may behave chaotically.

The roles of chaos in these kind of systems have been investigated in many papers in the last years[3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17].

Hopfield Neural Networks (HNN) are constructed from artificial neurons and represent particular casesof NNs inspired by spin systems. Even it is not easy to find, chaos and hyperchaos has been identified inmany HNNs [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31].

On the other side, the origin of transient chaos is well known: it is due to nonattracting chaotic saddlesin phase space [38, 39, 32, 33, 34, 35, 36, 37]. Transient chaos is a common phenomenon of many engineering,physical and biological systems. Compared wit chaos, which is characterized as a long-term behavior, thetransient chaos, is a phenomenon which appears when a nonlinear system behaves chaotically during sometransient time interval after which falls into a periodic attractor. These systems may initially exhibit anaperiodic behaviour and sensitivity to initial conditions (i.e. “chaos”) and after a period of time, settledown on a periodic orbit or fixed point (for example, for the Lorenz system, transient chaos exists e.g. fort ∈ [0, T ∗], with T ≈ 11 sec). In applications it is often desirable to sustain transient chaos [40].

Such phenomena were observed in radio circuits [41], hydrodynamics [42], neural networks [43], standardmodels of nonlinear systems such as Rossler system [44], Lorenz system [45], experiments [46], maps [47] andso on.

Transient chaos can be quite disastrous, as in situations of voltage collapse or species extinction. Thereforeanticontrol (in the sense of maintaining) of transient chaos can be welcome in some cases [40].

In the recent paper [49] it is revealed a new phenomenon of transient chaos, fundamentally different fromthe hyperbolic and nonhyperbolic transient chaos reported in the existing literature, and which appears inmany systems (chemical reactions, binary star behavior, etc.) and are likely far less predictable than hasbeen previously thought: doubly transient chaos.

In this paper we consider the case of the following 3-neuron simplified HNN modeled by the followingODEs

∗Corresponding author

Preprint submitted to Elsevier April 14, 2016

xi = −xi +

3∑j=1

wijf(xj), i = 1, 2, 3, (1)

with f(xj) = tanh(xj) and with the weight matrix [22]

W =

w11 w12 w13

w21 w22 w23

w31 w32 w33

=

2 −1.2 01.9995 1.71 1.15−4.75 0 1.1

.and we show numerically that the system admits a new kind of coexisting hidden attractors, transient hiddenattractors.

With above values of weights, the system (1) reads

x1 =− x1 + 2 tanh(x1)− 1.2 tanh(x2),

x2 =− x2 + 1.9995 tanh(x1) + 1.71 tanh(x2) + 1.15 tanh(x3),

x3 =− x3 − 4.75 tanh(x1) + 1.1 tanh(x3).

(2)

From computational perspective point of view, it is natural to suggest the following attractors classifica-tion, based on the connection of their basins of attraction with equilibria in the phase space [50, 51, 52, 53,54, 55, 56]

Definition 1. An attractor is called a self-excited attractor if its basin of attraction intersects with any openneighborhood of a stationary state (an equilibrium); otherwise, it is called a hidden attractor.

Self-excited attractors can be visualized numerically by a standard computational procedure, in whichafter a transient process, a trajectory starting from a point of a neighborhood of unstable equilibrium isattracted to the attractor, while the basin of attraction for a hidden attractor is not connected with anyequilibrium. Therefore, for the numerical localization of hidden attractors it is necessary to develop specialanalytical-numerical procedures.

If the problem is formulated to find all of the transition chaotic sets, then we can classify them as self-excited (can be found by watching the trajectories from equilibria vicinities, e.g. unstable manifolds ofequilibria) and hidden (formed by trajectories not from equilibria vicinities, e.g. from infinity). If in thephase space of a muldimensional dissipative system (i.e. there is a global absorbing set) there is a stablelimit cycle, then some trajectories from the infinity may be attracted to this limit cycle in a complex wayfor a rather long time.

Hidden attractors are attractors in systems with no-equilibria or in multistable systems with only sta-ble equilibrium. Coexisting self-excited attractors in multistable systems can be found using a standardcomputational procedure, whereas there is no regular way to predict the existence or coexistence of hiddenattractors in a system (for various examples of multistable engineering systems refer to [57]).

To verify numerically that a chaotic attractor is hidden, one has to check that all trajectories starting insmall neighborhoods of unstable equilibria, either are attracted by stable attractors, or diverge to infinity.

2. Hidden transient attractors of HNN

The HNN system (2) is symmetrical with respect to the origin and has the following equilibria

X∗0 = (0, 0, 0), X∗1,2 = ±(0.493, 0.366,−3.267).

The Jacobian is

J =

1− 2 tanh2(x1) −1.2 + 1.2 tanh2(x2) 01.9995− 1.9995 tanh2(x1) 0.71− 1.71 tanh2(x2) 1.15− 1.15 tanh2(x3)−4.75 + 4.75 tanh2(x1) 0 0.1− 1.1 tanh2(x3)

,and the eigenvalues of X∗0 are λ1 = 1.942 and λ2,3 = −0.066 ± 1.879i while the eigenvalues of X∗1,2 areλ1 = −0.987 and λ2,3 = 0.538± 1.286i. Therefore, equilibria are unstable: one attracting focus saddle (X∗0 ),and two repelling focus saddles (X∗1,2).

2

The numerical integration of the HNN (2) is done with the Matlab differential solver ode45 with optionopts = odeset(′RelTol′, 1e−9,′AbsTol′, 1e−9) which yields 8 decimals accurate results, over the time interval[0, T ] with T = 850.1

In Hopfield like systems it is common to find transient chaos (see e.g. [22]). Generally, the duration of achaotic transient behavior of many trajectories is rather short before they settle down on some periodic stableattractor [35]. However, in the case of the HNN, our numerous numerical tests reveal a relative long transienttime [0, T ∗], with T ∗ = 500, over which the system behaves chaotically along two coexisting transient chaoticattractors H1 and H2 (see Fig.2 where H1,2 are obtained with initial conditions ±(1.9, 3, 1))2.

For the initial conditions chosen inside small δ-vicinities of the unstable equilibria, with δ = 1.5E−4, theunderlying trajectories are attracted by one of the two stable limit cycles C1 (red plot) and C2 (blue plot)respectively as shown by Figs. 3 and 4 where, for simplicity, in the detailed images only 50 trajectories areconsidered.

Trajectories starting from a δ-vicinity VX∗0

of X∗0 (Fig. 3), tend either to C1 (light red plot), or to C2

(light blue plot). This happens because X∗0 belongs to the separatrix of the basins of attractions of C1 andC2.

Fig. 4, where only the case of stable limit cycle C2 is considered, reveals 50 trajectories starting insidea vicinity VX∗

2of equilibrium X2, which all are attracted by the stable cycle C2. Similarly, all trajectories

from small vicinities of X∗1 are attracted by C1.The shape of the trajectories starting within VX∗

0and VX∗

1,2are consistent with the equilibria type: the

trajectories from vicinities VX∗1,2

exit by scrolling equilibriaX1,2 in the unstable two-dimensional manifold (see

detailed image in Fig. 3), while the trajectories from the vicinity of X∗0 leave VX∗0

along the one-dimensionalunstable manifold of X∗0 (detail in Fig. 4).

Concluding, by following intensive numerical tests, the underlying numerical analysis leads to a conclu-sion that the transient chaotic attractors H1,2, are very likely to be hidden and could be called transienthidden chaotic attractors. For the numerical localization of hidden attractors, a special analytical-numericalprocedure has been designed (see e.g. [54]). In this paper the initial conditions for transient hidden attractors±(1.9, 3, 1), have been found by trial and errors.

Note that the coexistence of transient trajectories which start from VX∗0

and VX∗1,2

and reach the stable

cycles C1,2 with the stable cycles C1,2, are ensured by the entrainment of limit cycles by chaos (see [61], wherethe replication of sensitivity and the existence of infinitely many unstable periodic solutions were rigorouslyproved and [62], where this result is applied in Hopfield systems). Based on this result, the transient hiddenchaotic attractors H∗1,2 differ from the transients to C1,2. Moreover, they have different attraction basins:H1,2 are generated starting from initial conditions ±(1.9, 3, 1), while C1,2 are obtained with initial pointsclose to equilibria X∗0 and X∗1,2. For t > T ∗, T1,2 vanish.

The transients of the HNN system seems to be deformed by the form of the coexisting limit cycles andunstable equilibria, thus H1 and H2 have a complex structure. So one may say that the behavior is rathernatural here.

Since chaotic behavior in neural activity seems to be unavoidable, chaos control and anticontrol of thesetransient hidden chaotic attractors is an unexplored theme yet and offer an exciting subject for futureresearch.

References

[1] C.A. Skarda, W.J. Freeman, How brains make chaos in order to make sense of the world, Behav. BrainSci. 10 (1987) 161-195.

[2] W.J. Freeman, Neurodynamics: An exploration in mesoscopic brain dynamics, Berlin, Springer 2000.

1Matlab implicitly uses default values RelTol = .001 and AbsTol = .000001 and the approximate error at each step ek isensured to be: ek ≤ max(RelTol × xk, AbsTol), for all k, where xk is the value calculated at the node tk (see e.g. [58] for theused Runge-Kutta and other numerical methods utilized by Matlab). In [59] it is suggested RelTol = 10−(m+1) for m precisedigits of the required solution.

2Related to length of the time interval, precautions should be considered since a too large time interval could lead toinaccurate numerical solutions (see e.g. [60] for the case of Lorenz system).

3

[3] G. Dror, M. Tsodyks, Chaos in neural networks with dynamic synapses, Neurocomputing, 32-33 (2000)365-370.

[4] A. Babloyantz Brain chaos and computation, Int. J. Neural Syst. 07 (1996) 461-471.

[5] K. Aihara, T. Takabe M. Toyoda, Chaotic Neural Networks. Phys. Lett. A 144 (1990) 333-340.

[6] S. Nara, P. Davis, M. Kawachi, H. Totsuji, Chaotic memory dynamics in a recurrent neural networkwith cycle memories embedded by pseudoinverse method, Int. J. Bifurcation Chaos 5 (1995) 1205-1212.

[7] Q. Liu, S. Zhang, Adaptive lag synchronization of chaotic Cohen-Grossberg neural networks withdiscrete delays, Chaos 22 (2012) 033123.

[8] W. Lu, T. Chen, Synchronization of coupled connected neural networks with delays, IEEE Trans.Circuits Syst. I: Regul. Pap. 51 (2004) 2491-2503.

[9] J. Cao, J. Lu, Adaptive synchronization of neural networks with or without time-varying delay, Chaos16 (2006) 013133.

[10] W. Yu, J. Cao, W. Lu, Synchronization control of switched linearly coupled neural networks withdelay, Neurocomputing 73 (2010) 858-866.

[11] K. Aihara, T. Takebe, M. Toyoda, Chaotic neural networks, Phys. Lett. A 144 (1990) 333-340.

[12] K. Aihara, G. Matsumoto, Chaotic oscillations and bifurcations in squid giant axons, in: A.V. Holden(Ed.), Chaos, Manchester University Press, Manchester, UK, 1986, pp. 257-269.

[13] I. Tsuda, Chaotic itinerancy as a dynamical basis of hermeneutics in brain and mind, World Futures32 (1991) 167-184.

[14] W.J. Freeman, Tutorial on neurobiology: from single neurons to brain chaos, Int. J. Bifurcation Chaos2 (1992) 451-482.

[15] X. Wang, Period-doublings to chaos in a simple neural network: an analytical proof, Complex Syst. 5(1991) 425-441.

[16] J. Guckenheimer, R.A. Oliva, Chaos in the Hodgkin?Huxley model, SIAM J. Appl. Dyn. Syst. 1 (2002)105-114.

[17] Q. Ke, B.J. Oommen, Logistic neural networks: their chaotic and pattern recognition properties,Neurocomputing 125 (2014) 184-194.

[18] Hopfield J.J., Neurons with graded response have collective computational, Proc. Nat. Acad. Sci. USA81 (1984) 3088-3092.

[19] H. Bersini, P. Sener, The connections between the frustrated chaos and the intermittency chaos insmall Hopfield networks, Neural Networks 15, (2002) 1197-1204.

[20] H. Bersini, The frustrated and compositional nature of chaos in small Hopfield network, Neural Net-works 11 (1998) 1017-1025.

[21] Q. Li, X.S. Yang, F. Yang, Hyperchaos in Hopfield-type neural networks. Neurocomputing 67 (2005)275-280.

[22] X.S Yang, Q. Yuan Chaos and transient chaos in simple Hopfield neural networks, Neurocomputing69 (2005) 232-241.

[23] X.S. Yang, Y. Huang, Complex dynamics in simple Hopfield neural networks, Chaos 16 (2006) 033114.

[24] J. Li, F. Liu, Z.-H. Guan, T. Li, A new chaotic Hopfield neural network and its synthesis via parameterswitchings, Neurocomputing 117 (2013) 33-39.

4

[25] W.-Z. Huang, Y. Huang, Chaos, bifurcation and robustness of a class of Hopfield neural networks, Int.J. Bifurcation Chaos 21 (2011) 885-895.

[26] P.C. Rech, Chaos and hyperchaos in a Hopfield neural network, Neurocomputing 74 (2011) 3361-3364.

[27] Q. Yuan, Q. Li, X.-S. Yang, Horseshoe chaos in a class of simple Hopfield neural networks, ChaosSolitons Fractals 39 (2009) 1522-1529.

[28] X.-S. Yang, Q. Yuan, Chaos and transient chaos in simple Hopfield neural networks, Neurocomputing69 (2005) 232-241.

[29] W.-Z. Huang, Y. Huang, Chaos of a new class of Hopfield neural networks, Appl. Math. Comput. 206(2008) 1-11.

[30] Q. Li, X.-S. Yang, F. Yang, Hyperchaos in Hopfield-type neural networks, Neurocomputing 67 (2005)275-280.

[31] H. Alonso, T. Mendon, P. Rocha, Hopfield neural networks for on-line parameter estimation, NeuralNetw. 22 (2009) 450-462.

[32] J.L. Kaplan, J. A. Yorke, Pre-turbulence: a regime observed in a fluid flow model of Lorenz, Commun.Math. Phys. 67 (1979) 93-108.

[33] J.A. Yorke and E.D. Yorke, Metastable chaos: the transition to sustained chaotic oscillations in theLorenz model, J. Stat. Phys. 21 (1979) 263-277.

[34] G. Pianigiani J.A. Yorke, Expanding maps on sets which are almost invariant. Decay and chaos, Trans.Am. Math. Soc. 252 (1979) 351-366.

[35] H.E. Nusse, J.A. Yorke, A procedure for finding numerical trajectories on chaotic saddles, Physica D36 (1989) 137-156.

[36] G.-H. Hsu, E. Ott, C. Grebogi, Strange saddles and the dimensions of their invariant manifold, Phys.Lett. A 127 (1988) 199-204.

[37] C. Grebogi, E. Ott, and James A. Yorke, Critical exponent of chaotic transients in nonlinear dynamicalsystems, Phys. Rev. Lett. 57 (1986) 1284-1287.

[38] C. Grebogi, E. Ott, J. A. Yorke, Chaotic attractors in crisis, Phys. Rev. Lett.48 (1983) 1507.

[39] H. Kantz and P. Grassberger, Repellers, semi-attractors, and long-lived chaotic transients, Physica D17 (1985) 75-86.

[40] M. Dhamala, Y.-C. Lai, Controlling transient chaos in deterministic flows with applications to electricalpower systems and ecology, Phys. Rev. E 59 (1999) 1646-1655.

[41] L. Zhu, A. Raghu, Y.-C. Lai, Experimental observation of superpersistent chaotic transients, Phys.Rev. Lett. 86 (2001) 4017-4020.

[42] G. Ahlers, R.W. Walden, Turbulence near Onset of Convection, Phys. Rev. Lett. 44 (1980) 445-448.

[43] X.-S. Yang, Q. Yuan, Hyperchaos in Hopfield-Type Neural Networks, Neurocomputing 69 (2005) 232-241.

[44] M. Dhamala, Y.-C. Lai, E. J. Kostelich, Analysis of transient chaotic time series, Phys. Rev. E 61(2000) 6485.

[45] P. Vadasz, Analytical prediction of the transition to chaos in Lorenz equations, Appl. Math. Lett. 23(2010) 503-507.

[46] L. Zhu, A. Raghu, Y.-C. Lai, Experimental Observation of Superpersistent Chaotic Transients, Phys.Rev. Lett. 86 (2001) 4017-4020.

5

[47] G.B. Astaf’ev, A.A. Koronovskiı, A.E. Hramov, Behavior of Dynamical Systems in the Regime ofTransient Chaos, Tech. Phys. Lett.+ 29 (2003) 923-926.

[48] P. Vadasz, Analytical prediction of the transition to chaos in Lorenz equations, Appl. Math. Let. 23(2010) 503?507.

[49] A.E. Motter, M. Gruiz, G. Karolyi, and T. Tel, Doubly transient chaos: Generic form of chaos inautonomous dissipative systems, Phys. Rev. Lett. 111 (2013) 194101.

[50] N.V. Kuznetsov, Hidden attractors in fundamental problems and engineering models. A short sur-vey, AETA 2015: Recent Advances in Electrical Engineering and Related Sciences, Lecture Notes inElectrical Engineering, 371 (2016) 13.

[51] G. Bianchi, N.V. Kuznetsov, G.A. Leonov, M.V. Yuldashev, R.V. Yuldashev, 20157th International Congress on Ultra Modern Telecommunications and Control Sys-tems and Workshops (ICUMT), Limitations of PLL simulation: hidden oscilla-tions in MATLAB and SPICE, 79-84 (2015) http://arxiv.org/pdf/1506.02484.pdf,http://www.mathworks.com/matlabcentral/fileexchange/52419-hidden-oscillations-in-pll.

[52] G.A. Leonov, N.V. Kuznetsov, Hidden attractors in dynamical systems. From hidden oscillations inHilbert-Kolmogorov, Aizerman, and Kalman problems to hidden chaotic attractors in Chua circuits,Int. J. Bifurcation Chaos 23 (2013) 1330002.

[53] G. Leonov, N. Kuznetsov, T. Mokaev, Homoclinic orbits, and self-excited and hidden attractors in aLorenz-like system describing convective fluid motion. Eur. Phys. J. Special Topics 224 (2015) 1421-1458.

[54] G.A. Leonov, N.V. Kuznetsov, V.I. Vagaitsev, Hidden attractor in smooth Chua systems, Physica D241 (2012) 1482-1486.

[55] G.A. Leonov, N.V. Kuznetsov, V.I. Vagaitsev, Localization of hidden Chua’s attractors. Physics LettersA 375 (2011) 2230-2233.

[56] A.A. Andronov, E.A. Vitt, S.E. Khaikin, Theory of Oscillators (in Russian). ONTI NKTP SSSR (1937)(English transl. 1966, Pergamon Press).

[57] A. Pisarchik, U. Feudel, Control of multistability, Physics Reports 540 (2014) 167-218.

[58] L.F. Shampine, I. Gladwell, and S. Thompson, Solving ODEs with MATLAB, Cambridge Univ. Press,Cambridge, 2003.

[59] K.E. Brenan, S.L. Campbell, and L.R. Petzold, Numerical Solution of Initial-Value Problems inDifferential-Algebraic Equations (SIAM Classics in Applied Mathematics, 14) SIAM, Philadelphia,1996.

[60] B. Kehlet, A. Logg, Quantifying the computability of the Lorenz system, arXiv:1306.2782v1 [math.NA]12 Jun 2013 http://arxiv.org/abs/1306.2782

[61] M.U. Akhmet, M.O. Fen, Entrainment by chaos, J. Nonlinear Sci. 24 (2014) 411-439.

[62] M. Akhmet, M.O. Fen, Generation of cyclic/toroidal chaos by Hopfield neural networks, Neurocom-puting 145 (2014) 230-239.

6

Figure 1: Transient hidden chaotic attractors H1 and H2 of HNN.

7

Figure 2: Time series of the first component x1. (a) Transient hidden chaotic attractor H1, for t ∈ [0, T ∗], starting from(1.9, 3, 1). For t > T ∗, chaos vanishes. (b) Stable cycle C1.

8

Figure 3: Stable cycles C1 (red plot) and C2 (blue plot. Trajectories starting from vicinity VX∗0

(detail), are attracted either

by C1 (light red plot), or C2 (light blu plot).

9

Figure 4: Trajectroies starting from vicinity VX∗2

(light blue plot) are attracted by C2 (blue plot).

10