Neural Network Boolean Factor Analysis and Application

DUSAN HUSEKICS AS CR, v.v.i.Pod Vodárenskou

věžı́ 2,182 07 Prague 8Czech Republic

ALEXANDER FROLOVInstitute of HigherNervous Activity

and NeurophysiologyRussian Academy of Sciences

Butlerova 5a,117 485 Moscow

Russia

PAVEL POLYAKOVInstitute of Optical

Neural TechnologiesRussian ASVavilova 44,

119 333 MoscowRussia

VACLAV SNASELVSB TU Ostrava

Departmet ofComputer Science17 listopadu 15,70 833 OstravaCzech Republic

Abstract: The recurrent Neural network capable to provide the Boolean factor analysis of the binary data sets ofhigh dimension and complexity is applied to roll-call voting problem. The method of sequential factor extraction,based on the Lyapunov function is discussed in deep. Efficiency of this attempt is shown on simulated data and onreal data from Russian parliament as well.

Key–Words: Hopfield Neural Network, Boolean Factor Analysis,Unsupervised Learning, Dimension Reduction,Data Mining

1 Introduction

Theoretical analysis and computer simulations per-formed in (5) revealed that Hopfield-like neural net-works are capable of performing boolean factor anal-ysis signals of high dimension and complexity. Factoranalysis is a procedure which maps original signalsinto the space of factors. The principal component(PC) analysis is a classical example of such mappingin the linear case. Linear factor analysis implies thateach original case can be presented as

X = FS + ε (1)

where F is a matrix N×L of factor loadings, S isa vector of factor scores and ε an error. Each compo-nent of S gives contribution of corresponding factorin the original signal. Columns of loading matrix Fgive vectors presenting corresponding factors in thesignal space. Below namely these vectors are termedfactors. Mapping of original space to the factor spacemeans that signals are represented by vectors S in-stead of original vectors X. Dimensionality of vec-tors S is much less than dimensionality of signals X.Thereby factor analysis provides high compression oforiginal signals.

Boolean factor analysis implies that a complexvector signal has a form of the Boolean sum ofweighted binary factors:

X =∨

Slf l. (2)

In this case, original signals, factor scores andfactor loadings are binary and mapping of original sig-nal to the factor space means citation of factors thatwere mixed in the signal. The mean number of factorsmixed in the signals we term signal complexity C.

For the case of large dimensionality and com-plexity of signals it was a challenge (5) to utilize forBoolean factor analysis the Hopfield-like neural net-work with parallel dynamics. Binary patterns X of thesignal space are treated as activities of N binary neu-rons (1 - active, 0 - nonactive) with gradually rang-ing synaptic connections between them. During thelearning stage patterns Xm are stored in the matrixof synaptic connections J′ according to correlationalHebbian rule:

J ′ij =M∑

m=1

(Xmi − qm)(Xmj − qm), i�=j, J ′ii = 0, (3)

where M is the number of patterns in the learning

set and bias qm =N∑

i=1Xmi /N is the total activity of

the m-th pattern. This form of bias corresponds to thebiologically plausible global inhibition being propor-tional to an overall neuronal activity.

Additionally to N principal neurons of Hopfieldnetwork described above we introduced one specialinhibitory neuron activated during the presentation ofevery pattern of the learning set and connected with allprincipal neurons by bidirectional connections. Pat-

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 29

terns of the learning set are stored in the vector J′′ ofthe connections according to Hebbian rule:

J ′′i =M∑

m=1

(Xmi − qm) = M(qi − q), (4)

where qi =M∑

m=1Xmi /M is a mean activity of i-th

neuron in the learning set and q is a mean activity ofall neurons in the learning set. It is also supposed thatexcitability of the introduced inhibitory neuron de-creases inversely proportional to the size of the learn-ing set being 1/M after storing of all its patterns.

Due to the Hebbian learning rule (3), neuronswhich represent one factor and therefore tend to firetogether, become more tightly connected than neuronsbelonging to different factors, constituting attractor ofnetwork dynamics. This property of factors is a baseof the used two-run procedure of factor search. Itsinitialization starts by presentation of random initialpattern Xin with kin = rinN active neurons. Activ-ity kin is supposed to be much smaller than activityof factors. On presentation of Xin, network activityX evolves to some attractor. The evolution is deter-mined by the parallel dynamics equation in discretetime. At each time step:

Xi(t + 1) = Θ(hi(t) − T (t)), Xi(0) = Xini ,i = 1, · · ·, N,(5)

where hi are components of the vector of synapticexcitations

hi(t) =N∑

j=1

J ′ijXj(t) − (1/M)J ′′iN∑

j=1

J ′′j Xj(t), (6)

Θ is step function, and T (t) is activation thresh-old. The first term in (6) gives synaptic excitationsprovided by the principal neurons of Hopfield networkand the second one by the additional inhibitory neu-ron. The use of the inhibitory neuron is equivalent tothe substraction of (1/M)J ′′i J

′′j = M(qi − q)(qj − q)

from J ′ij . Thus (6) can be rewritten as hi(t) =N∑

j=1JijXj(t) where J = J′ − MqqT, q is a vector

with components qi − q and qT is a transposed q.As shown in (8) the replacement of common connec-tion matrix J by J, first, completely suppressed twoglobal attractors which dominate in network dynam-ics for large signal complexity C, and second, made

the size of attractor basins around factors to be inde-pendent of C.

At each time step of the recall process the thresh-old T (t) was chosen in such a way that the level of thenetwork activity was kept constant and equal to kin.Thus, on each time step kin ”winners” (neurons withthe greatest synaptic excitation) were chosen and onlythey were active on the next time step. To avoid uncer-tainty in the choice of winners when several neuronshad synaptic excitations at the level of the activationthreshold, small random noise was added to the acti-vation threshold of each individual neuron. The am-plitude of the noise was put to be less than the smallestincrement of the synaptic excitation given by formula(6). This ensured that neurons with the highest exci-tations were kept to be winners in spite of the randomnoise be added to the neurons’ thresholds. Noise toindividual neurons was fixed during the whole recallprocess to provide its convergence. As shown in (5),this choice of activation thresholds allows for stabi-lization of the network activity in point or cyclic at-tractor of length two.

When activity stabilizes at the initial level of ac-tivity kin, kin + 1 neurons with maximal synaptic ex-citation are chosen for the next iteration step, and net-work activity evolves to some attractor at the new levelof activity kin + 1. Then level of activity increases tokin + 2, and so on, until number of active neuronsreaches the final level rfN with rf > p. Thus, onetrial of the recall procedure contains (rf − rin)N ex-ternal steps and several steps inside each external stepto reach some attractor for fixed level of activity.

At the end of each external step the relative Lya-punov function was calculated by formula

Λ = XT (t + 1)JX(t)/(rN), (7)

where XT (t+1) and X(t) are two network statesin cyclic attractor (for point attractor XT (t + 1) =X(t) ). The relative Lyapunov function is a meansynaptic excitation of neurons belonging to some at-tractor at the end of the external step with k = rNneurons.

Attractors with the highest Lyapunov functionwould be obviously winners in the most trials of therecall process. Thus, more and more trials are re-quired to obtain new attractor with relatively smallvalue of Lyapunov function. To overcome this prob-lem the dominant attractors should be deleted fromthe network memory. The deletion was performedaccording to Hebbian unlearning rule by substractionΔJij , j �= i from synaptic connections Jij where

ΔJij =η

2J(X)[(Xi(t − 1) − r)(Xj(t) − r) +

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 30

+(Xj(t − 1) − r)(Xi(t) − r), (8)

J(X) is the average synaptic connection betweenactive neurons of the attractor, X(t− 1) and X(t) arepatterns of network activity at last time steps of itera-tion process, r is the level of activity, and η is an un-learning rate. For point attractor X(t) = X(t−1) andfor cyclic attractor X(t − 1) and X(t) are two statesof attractor.

There are three important similarities between thedescribed procedure of Boolean factor analysis andlinear PC analysis. First, PC analysis is based on thesame covariation matrix as connection matrix in Hop-field network. Second, factor search in PC analysiscan be performed by the iteration procedure similar tothat described by (5) and (6) but binarization of synap-tic excitations by the step function must be replacedby their normalization:Xi(t+1) = hi(t)/|h(t)|. Theniteration procedure starting from any random stateconverges to the eigenvector f1 of covariation matrixwith the largest eigenvalue Λ1. Just this eigenvectoris treated as the first factor in PC analysis. Third, toobtain the next factor the first factor must be deletedfrom the covariation matrix by the substraction ofΛ1f1fT1 , and so on. The substraction is similar to Heb-bian unlearning (8).

However, Boolean factor analysis by Hopfield-like network has one principal difference from linearPC analysis. Attractors of iteration procedure in PCanalysis are always factors while in Hopfield-like net-works the iteration procedure can converge to factors(true attractors) and to spurious attractors which arefar from all factors. Thus, two main questions arise inview of Boolean factor analysis by Hopfield-like net-work. First, how often would network activity con-verge to one of the factors starting from random state?Second, is it possible to distinguish true and spuriousattractors when network activity converges to somestable state? Both these questions are answered in thenext Section.

There are many examples of data in the scienceswhen Boolean factor analysis is required (1). In ourprevious papers (6),(7) and (8) we performed it foranalysis of textual data. Here we apply our method tothe data for Russian parliament voting. The results aredescribed in Section 3.

1.1 Artificial SignalsTo reveal peculiarities of true and spurious attrac-tors we performed computer experiments with artifi-cial signals. Each pattern of the learning set is sup-posed to be a Boolean superposition of exactly Cfactors and each factor is supposed to contain ex-actly n = pN 1-s and (1 − p)N 0-s. Thus, each

factor f l∈ BNn and for each pattern of the learningset, vector of factor scores S ∈ BLC where BNn ={X|Xi ∈ {0, 1},

N∑i=1

Xi = n}

. We supposed factor

loadings and factor scores to be statistically indepen-dent. As an example, Fig. 1 demonstrates changes ofrelative Lyapunov function for N = 3000, L = 5300,p = 0.02 and C = 10. Recall process started atrin = 0.005.

0,01 0,02 0,030,2

0,4

0,6

0,8

1,0

l

r

Fig. 1 Fig.1. Relative Lyapunov function λ in dependence on the

relative network activity r. The data were normalized to the mean

of Lyapunov function for true attractors at r = p.

Trajectories of network dynamics form two sepa-rated groups. As shown in Fig. 2, the trajectories withhigher values of Lyapunov function are true and withlower ones are spurious. This Figure relates values ofLyapunov function for patterns of network activity atpoints r = p to maximal overlaps of these patternswith factors. Overlap between two patterns X1 andX2 with p active neurons was calculated by formula

m(X1,X2) =1

Np(1 − p)N∑

i=1

(X1i − p)(X2i − p)

By this formula overlap between equal patterns isequal to 1 and mean overlap between independent pat-terns is equal to 0. Patterns with high Lyapunov func-tion have high overlap with one of the factors, whilepatterns with low Lyapunov function are far from allthe factors. It is shown that true and spurious trajec-tories are separated by the values of their Lyapunovfunctions. In Figs 1 and 2 the values of Lyapunovfunction are normalized by mean value of this func-tion over true attractors at the point r = p.

The second characteristic feature of true trajecto-ries is the existence of a kink at a point r = p wherethe level of network activity coincides with that in fac-tors (see Fig. 1). When r < p the increase of r re-sults in almost linear increase of the relative Lyapunovfunction. Increase of r occurs in this case due to join-ing of neurons belonging to the factor that are strongly

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 31

0,0 0,2 0,4 0,6 0,8 1,00,6

0,8

1,0

l

Ov

Fig. 2 Values of normalized Lyapunov function in relation to over-

laps with the closest factors. It is shown that true and spurious

attractors can be easily separated due to a large gap between dis-

tributions of values of Lyapunov function.

connected with other neurons of factor. Then join-ing of new neurons results in proportional increase ofmean synaptic excitation to the active neurons of fac-tor that is just equal to their relative Lyapunov func-tion. When r > p the increase of r occurs due to join-ing of some random neurons that are connected withfactor by week connections. Thus, the increase of therelative Lyapunov function for true trajectory sharplyslows and it tends to the values of Lyapunov functionfor spurious trajectories.

The third characteristic feature of true trajectorieswhich allows for their distinction from spurious tra-jectories, lies in the behavior of their activation thresh-olds. Activation thresholds T at the final states of ex-ternal steps are shown in Fig. 3 for the same networkparameters as in Fig. 1.

Initially activation thresholds for true attractorsare larger in comparison with spurious ones. Howeverat the point r = p they sharply drop to the level of spu-rious attractors. This behavior is originated from thewell known fact (see, for example, (8) ) that during ac-tivation of the fragment of one of the stored patternsthe distribution of synaptic excitations has two sepa-rated high and low modes: high - for neurons belong-ing to the pattern and low - for neurons not belongingto it. The mean of the high mode increases propor-tionally to the size of the fragment and the mean of thelow mode is close to zero. When r < p fragments ofstored patterns are activated along the true trajectories.Then activation thresholds are inside the high mode.

0,01 0,02 0,0310

20

30

40

50

T

r

Fig. 3 Fig.3. The change of activation threshold T along the recall

process. Notice the drop of activation threshold at the point r =

p. r = p.

The mean of high mode increases when r increasessince r is the relative size of activated fragment of thestored pattern. Consequently the activation thresholdsincrease with r. However when the stored pattern isactivated totally (r = p), the activation threshold hasto jump to the low mode to activate additional neuronsnot belonging to the pattern as r continues to increase.The use of these three features of true trajectories pro-vides reliable recognition of factors. This statementwas clearly confirmed in our previous papers (6), (7)and (8) where our method was performed for analy-sis of textual data. Here we apply it for analysis ofparliament voting.

2 Analysis of Parliament Voting

The analysis was performed for results of roll-callvotes in Russian parliament in 2004 (9). Each voteis considered as binary vector with component 1 ifthe correspondent deputy voted affirmatively and 0negatively. The number of deputies (consequentlythe dimensionality of signal space and network size)amounts 450.

The number of votes during the year amounts3150. Fig. 4 shows Lyapunov function along trajec-tories starting from 1500 random initial states. Allthese states converge to four trajectories. Two of themhave obvious kinks and therefore were identified astwo factors. The factor with highest Lyapunov func-tion contains 50 deputies and completely coincideswith the fraction of Communist Party (CPRF). An-other factor contains 36 deputies. All of them be-long to fraction of Liberal-Democratic Party (LDPR)which contains totally 37 deputes. Thus one of themembers of this fraction fell out of the correspondingfactor. Pointed kinks at the corresponding trajectoriesgive evidence that these fractions are the most disci-

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 32

pline and their members vote coherently.

0 20 40 60 800

50

100

150

200

l

number of active neurons

Fig. 4 Relative Lyapunov function λ for parliament data in depen-

dence on the number of active neurons. Thick points are klinks of

the first and second factors.

Fig. 5 demonstrates trajectories after deleting ofthe two factors. Starting from 1500 initial states theyconverge to only two trajectories. One of them hasa kink but it is not so strict as for CPRF and LDPRfactors. We supposed that the point where the secondderivative of Lyapunov function by k has maximumby absolute value is the third factor. The factor con-tains 37 deputies. All of them belong to the fraction”Rodina” which contains totally 40 deputes. Thus 3of its members fell out of the factor. The fuzziness ofkink at the trajectory gives evidence that this fractionis not so homogeneous as two first ones and actuallythe fraction split at two fractions in 2005.

0 20 40 60 800

50

100

150

l

number of active neurons

Fig. 5 The same as in Fig. 4 after deleting two first factors. Thick

point is klink of third factor.

Matching of neurons along the second trajectoryin Fig. 5 with the list of deputes showed that it cor-responds to the fraction ”United Russia” (UR). Thisfraction is the largest and contains totally 280 deputesbut it is less homogeneous. Therefore Lyapunov func-tion along the trajectory is low and it has no kink atall.

0 20 40 60 800

50

100

l

number of active neurons

Fig. 6 The same as in Figs. 4 and 5 after deleting three first fac-

tors.

Fig. 6 shows trajectories of neurodynamics afteradditional deleting the third factor from the network.The remaining two trajectories contain only membersof UR. The states of different trajectories do not in-tersect, so the UR fraction is evidently divided to twosubfractions.

3 Conclusion

Hopfield-like neural network is capable of perform-ing Boolean factor analysis of the signals of high di-mension and complexity. In this case analysis of avoting. But it is suitable for much complicated taskanalysis. In our previous papers we showed its highefficiency in analysis of textual data. Here its abilityis demonstrated in the field of politics. More detailedhere used method description can be found in work(2; 3; 4). The idea of the method is to exploit thewell known property of the network to create attrac-tors of the network dynamics by the tightly connectedgroups of neurons. We supposed, first, that each topicis characterized by the set of specific words (oftencalled concepts) which appear in the relevant docu-ments coherently and, second, that different conceptsare presented in documents in random combinations.

Acknowledgements: The research was fundedby the Center of Applied Cybernetics (grant No.1M6840070004) and by the Institutional ResearchPlan AV0Z10300504 Computer Science for the Infor-mation Society: Models, Algorithms, Applications,by the National Research Program of the Czech Re-public (project No. 1ET100300414) and by of theGrant Agency of the Czech Republic (Grant No.201/05/0079).

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 33

References:

[1] De Leeuw J. (2003) Principal component analy-sis of binary data. Application to roll-call anal-ysis. http://gifi.stat.ucla.edu

[2] Frolov AA, Husek D, Muraviev IP, PolyakovPY. 2007. Boolean factor analysis by attractorneural network. IEEE Tran on Neural Networks,18,3:698-707.

[3] Frolov AA, Husek D, Polyakov PY. 2008. Re-current neural network based Boolean factoranalysis procedure and its application to auto-matic papers categorization, IEEE Tran on Neu-ral Networks, Submited 2007

[4] Frolov AA, Husek D, Polyakov PY. 2008. Originand elimination of two global spurious attractorsin Hopfield-like network performing Booleanfactor analysis. JMLR, Submited 2007

[5] Frolov A.A., Sirota A.M., Husek D., MuravievI.P., Polyakov P.J. (2004). Binary factorizationin Hopfield-like neural networks: single-step ap-proximation and computer simulations. NeuralNetworks World, 14, 139-152.

[6] Frolov A.A., Husek Dusan, Polyakov P.A.,Rezankova H., Snasel Vaclav (2004) BinaryFactorization of Textual Data by Hopfield-LikeNeural Network. In: Computational Statistics.(Ed.: Antoch J.) - Heidelberg, Physica Ver-lag 2004, pp. 1035-1041 (ISBN:3-7908-1554-3) Held: COMPSTAT 2004. Symposium /16./,Prague, CZ, 04.08.23-04.08.27

[7] Husek D., Frolov AA., Rezankova, H., Snasel,V., Polyakov, P. (2005) Neural Network Non-linear Factor Analysis of High Dimensional Bi-nary Signals. Yaound 27.11.2005 - 1.12.2005.In: SITIS 2005. Dijon : University of Bour-gogne, 2005, s. 86-89. ISBN 2-9525435-0.

[8] Polyakov P.Y., Frolov A.A., Husek D. (2006) Bi-nary factor analysis by Hopfiled network andits application to automatic text classification.Neuroinformatics-2006, Russia.

[9] http://www.indem.ru/indemstat

6th WSEAS Int. Conference on Computational Intelligence, Man-Machine Systems and Cybernetics, Tenerife, Spain, December 14-16, 2007 34