I 3

Title:

Author@):

Submitted to:

LIMITED WEIGHTS NEURAL NETWORKS: VERY TIGHT ENTROPY BASED BOUNDS

Valeriu Beiu Sorin Draghici

SOCO ‘97 Second International ICSC Symposium on Soft Computing, Fuzzy Logic, Artificial Neural Networks, Genetic Algorithms September 17-1 9, 1997 Nimes, France

DISCLAIMER

This report was prepared as an account of work sponsored by an agency of the United States - - Government. Neither the United States Gavernment nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsi- bility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Refer- ence herein to any specific commercial product, process. or service by trade name, trademark, manufacturer, or otherwise docs not necessarily constitute or imply its endorsement, ream- mendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the

ESSSSE r-

gP = E - - - - - - - United States Government or any agency thereof. - -

Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by the University of California for the U S . Depattment of Energy under contract W-7405-ENG-36. By acceptance of this article, the publisher recognizes that the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or to allow others to do so, for US. Government purposes. The Los Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U S . Department of Energy.

FomrNo.836R5 ST 2629 ?om

Title of the conference: SOCO’97

Title of the proposed paper:

Limited Weights Neural Networks: Very Tight Entropy Based Bounds

Authors name:

Valeriu Beiu and Sorin Draghici

Afiliutions and addresses:

Los Alamos National Laboratory, Division NIS-1, MS D466 Los Alamos, New Mexico 87545, USA

Vision and Neural Networks Laboratory Department of Computer Science, Wayne State University

431 State Hall, Detroit, Michigan 48202, USA

Corresponding author:

Valeriu Beiu

Phone: +I-505-667 2430

E-mail: beiu@lanl.gov Fax: +1-505-665 7395

Topics:

neural networks limited (integer) weights entropy bounds (number of bits) complexity of neural networks classification problems

This paper is a perfect fit for the following workshop/tutorial mentioned on the home page of SOC0’97 at “http://www.compusmart.ab.ca/icsc/soco97.h~#Workshop~utori~s”

Workshopflutorial:

Information Theory and Artificial Neural Networks

1

+ Limited Weights Neural Networks: Very Tight Entropy Based Bounds

soc097 .

1. Statement of the problem

Being given a set of m examples (i.e., data-set) from IRn belonging to k different classes, the problem -'i , is to compute the required number-ofbifs (Le., entropy) for correctly classifying the data-set. Very tight upper and lower bounds for a dichotomy (i.e,, k = 2 ) will be presented, but they are vaiid for the general case.

2. Results achieved

1 '

2

The paper presents an upper bound (tighter than the ones previously known [4, 83) o f

#bits < mn~log(D/d)+0.51111/2

4 nOg ( D I ~ + 0.51 111 if lweightsl < A tight lower bound will also be detailed staring from the bound presented in [lo]. In this case

lweightsl < p (i.e., integer weights in the range [-p, +PI). We improve on the bound detailed there, and show that:

#bits > I [log ( D / d ) - 1.5359 + (logn) / n l 2

which clearly gives us:

mn mn 2 2 - - . r i o g ( ~ / d ) - 1.53591 < #bits < - - r i o g ( ~ / ~ + o . 5 i i i i

3. Significance

The bounds are proven in a constructive way, Although they do not lead to complexity reductions, they should be judged in the context of lowering certain constants for very difficult (i.e.. NP-complete or NP-hard) problems [91. An interesting aspect is that a constructive algorithm based on the upper bound has already been designed and used to generate both classical Boolean circuits and threshold gate circuits, or a mixture of them [2, 3, 61. Work is in progress for designing a constructive algorithm based on the lower bound.

4. Comparison with previous work

A recent result has shown that [4] #bits < mnrlog(D/d)+2.04711 with weights bounded as

weights < 4 2r'0g(D/4+2~04711 . This upper bound has been very recently [8] improved to:

#bits < mnrlog(D/d)+ 1.83961/2

with weights bounded as weights < ,/ 2rlog(D/d)+1.8396i

A lower bound (but not an absolute one) has been recently [lo] detailed for the case when the weights are integers in the range [-p,+p]:

#bits > mn [log (2pD)1/2

This bound is consistent with the upper bounds presented in [4,8] as in this case it was proven [lo] that d = 1 / 2 p , which gives:

Valeriu M u & Sorin Draghici 2

4 Limited Weights Neural Networks: Very TlgM Entropy Based Bounds * soc0'97

Appendix: selected references

r31

141

[71

E.B. Baum. On the Capabilities of Multilayer Perceptrons. J. of Complexity 4, 193-215, 1988. V. Beiu and J.G. Taylor. VLSI-Optimal Neural Network Learning Algorithm. In D.W. Pearson, N.C. Steele and R.F. Albrecht (eds.): Artificial Neural Nets and Genetic Algorithms (ICAN- NGA'95, AI&, France), Springer-Verlag, Vienna, 61-64, 1995. V. Beiu and J.G. Taylor. Direct Synthesis of Neural Networks. Proc. MicroNeuro'96 (Lausanne, Switzerland), IEEE Comp. SOC. Press, Los Alamitos, CA, 257-264, 1996.

V. Beiu. Entropy Bounds for Classification Algorithms. Neural Network World, 6(4), 497-505, 1996.

V. Beiu. Digital Integrated Circuit Implementations. Chapter E1.4 in [13].

V. Beiu. Optimization of Circuits Using a Constructive Learning Algorithm. To appear in Proc. Engineering Applications of Neural Networks (EANN'97, Stockholm, Sweden), June 16-18, 1997. V. Beiu. VLSI Complexity of Discrere Neural Networks. Gordon and Breach, Newark, 1997 (in press). V. Beiu and T. de Pauw. Tight Bounds on the Size of Neural Networks for Classification Prob- lems. Submitted for publication, January 1997.

V. Beiu. When Constants Are Important. Submitted for publication, January 1997. S . Draghici and I.K. Sethi. On the Possibilities of the Limited Precision Weights Neural Networks for Classification Problems. Submitted for publication, January 1997. V.P. Roychowdhury, K.-Y. Siu and A. Orlitsky (eds.). Theoretical Advances in Neural Compu- tation and Learning. Kluwer Academic, Boston, 1994.

K.-Y. Siu, V.P. Roychowdhury and T. Kailath. Discrete Neural Computation: A Theoretical Foun- dation. Prentice Hall, Englewood Cliffs, 1994.

E. Fiesler and R. Beale (eds.). Handbook of Neural Computation. Oxford University Press and the Institute of Physics Publishing, NY, 1996.

M.H. Hassoun. Fundamentals of Artifcia1 Neural Networks. MIT Press, Cambridge, MA, 1995.

S.-C. Huang and Y.-F. Huang. Bounds on the Number of Hidden Neurons of Multilayer Percep- trons in Classification and Recognition. IEEE Trans. on Neural Networks 2(1), 47-55, 1991.

A.V. Krishnamoorthy, R. Paturi, M. Blume, G.D. Linden, L.H. Linden and S.C. Esener. Hardware Tradeoffs for Boolean Concept Learning, Proc. World Con$ on Neural Networks '94 (WCNN'94, San Diego), Lawrence Erlbaum & INNS Press, Hillsdale, vol. 1, 551-559, 1994.

P. Raghavan. Learning in Threshold Networks: A Computational Model and Applications. Tech. Rep. RC 13859, IBM Res., 1988. Also in Proc. 1st Workshop on Computational Learning Theory (Cambridge, MA), ACM Press, NY, 19-27, 1988.

R.C. Williamson. &-entropy and the Complexity of Feedforward Neural Networks. In R.P. Lippmann, J.E. Moody and D.S. Touretzky (eds.): Advances in Neural Information Processing Systems (NIPS*90, Denver, CO), Morgan Kaufmann, San Mateo, CA, 946-952, 1991.

V W u Beiu k Sorin Draghki 3