Fast Image Vector Quantization Using a Modified CompetitiveLearning Neural Network Approach

Robert Li, Earnest Sherrod, Jung Kim, Gao Pan

Department of Electrical Engineering, North Carolina A&T State University, Greensboro, NC 27411

ABSTRACT: The basic goal of image compression through vector generates the address of the codevector specified by Q(x) ; andquantization (VQ) is to reduce the bit rate for transmission or data a decoder, which uses this address to generate the codevector y .storage while maintaining an acceptable fidelity or image quality. The The signal-noise-ratio (SNR) is usually used to measure the fi-advantage of VQ image compression is its fast decompression by delity or quality of recovered image [4,15]. One would like totable lookup technique. However, the codebook supplied in advance maximize the SNR value given bymay not handle the changing image statistics very well. The needfor online codebook generation became apparent. The competitivelearning neural network design has been used for vector quantization. SNR(db) Å 10 logS∑

n

jÅ1

x 2j Y∑

n

jÅ1

(xj 0 yj)2D (1)However, its training time can be very long, and the number of outputnodes is somewhat arbitrarily decided before the training starts. Ourmodified approach presents a fast codebook generation procedure

where n is the number of input vectors and yj is the output vectorby searching for an optimal number of output nodes evolutively. Theresults on two medical images show that this new approach reduces that is closest to xj in the codebook. By using SNR as a qualitythe training time considerably and still maintains good quality for criterion, one can compare the performance of different codingrecovered images. q 1997 John Wiley & Sons, Inc. Int J Imaging Syst schemes with the same codebook size. An important advantageTechnol, 8, 413–418, 1997 of VQ image compression is its fast decompression by table

lookup technique. However, the codebook supplied in advanceKey words: image compression; vector quantization; neural network;may not handle the changing image statistics very well. In addi-competitive learningtion, certain conventional code generation methods such as theLinde-Buzo-Gray (LBG) algorithm are taking an excessivelylong time in their iterative approach [5]. As a result, the needI. INTRODUCTIONfor a fast on-line codebook generation algorithm has become

Image compression techniques can be used in areas of high infor-apparent [6,7] . In this article, we use competitive learning neural

mation volume to reduce the data rate to within the channelnetworks to perform the vector quantization. In particular, we

capacity or storage requirement. Usually, large amounts of com-compare a standard competitive learning algorithm (CLA) and

pression can be obtained with lossy compression techniques. Onea modified version of it, the modified competitive learning algo-

of the most important lossy compression approaches is the vectorrithm (MCLA). It will be shown that MCLA has the following

quantization (VQ) method [1–3]. The VQ approach is basicallyadvantages: (a) It is an evolutive algorithm using a threshold

a clustering method. The vectors are obtained from image dataparameter to resolve the adaptation/creation issue in the design

by extracting nonoverlapping blocks and arranging the pixels inof neuron units. (b) It has a much shorter training time than the

each block in a line-by-line order. The VQ method builds up aCLA. (c) Its recovered image usually has a superior image quality

codebook, a dictionary of a few representative vectors (codevec-in terms of SNR value. The article is organized in the following

tors) . Then, each block in the image is coded with the indexway. In Section II, the basic CLA model is explained. In Section

value of the closest codevector in the codebook. The index valueIII, the new MCLA is presented in details. In Section IV, compari-

will take log2M bits, where M is the number of codevectors insons with other related VQ methods are described. In Section V,

the codebook. Assume that we use 8 bits /pixel for the image andwe discuss the experimental results of applying these techniques

the size of each block (vector) is N . Then, the compression ratioto two different medical images. Section VI contains a summary

of 8*N / log2M can be achieved.of the whole article.

The fundamental goal of image vector quantization is to re-duce the bit rate for transmission or data storage while main-

II. CLA FOR VQtaining an acceptable fidelity or image quality. A vector quantizerNeural network approaches have been used for pattern classifica-can be defined as a mapping, Q , from N-dimensional Euclideantion and data clustering [8–10]. It is possible to apply the trainingspace R into a finite subset Y of R . Y is the set of codevectorsalgorithm of neural networks to the design of an appropriatecontaining M vectors. It can also be seen as a combination ofcodebook which maximizes the SNR values of a reconstructedtwo functions: an encoder which takes the input vector x andimage. One approach is based on the CLA. The basic idea is thatonly one output unit of the neural network is active (on) at a

Correspondence to: R. Y. Li given time. The output units compete for the active status and

q 1997 John Wiley & Sons, Inc. CCC 0899–9457/97/040413-06

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST

are often called winner-take-all units. The model can be used to III. MODIFIED CLA (MCLA) NETWORKcategorize or cluster the input data [17,20,25]. This leads to its We propose an improved approach by modifying the CLA modelusefulness in image VQ [11–13]. In our implementation, a given and call this new model the MCLA. The basic ideas of choosingimage is first segmented into vectors of the same size. The ele- winning nodes based on Euclidean distance and weight adjustingments of those vectors are gray levels of the pixels in the image. based on input samples are retained. However, an evolutive pro-A neural network is set up with an input and output layer. For cedure is proposed here by generating output nodes and associ-the input layer, its number of input nodes equals to the dimensions ated weights adaptively from the input samples. The steps areof image vectors. For the output layer, its number of output nodes listed as below:equals the size of the codebook.

The training of the network is outlined in the following steps: 1. Set the proper threshold value:A threshold u is set to specify the expected maximum

Step a. Initialize the connection weights from N input nodes distance or minimum correlation between an input patternto the M output nodes using small random values. and its cluster center.

Step b. New input sample vectors are presented. Compute 2. Initialize the network parameters:distances dj between the input and each output nodes j Set the weights between the first output node and inputusing the Euclidean distance formula: nodes to the feature values of the first input pattern. Let

Wi 1( t) Å Xi ( t) (4)dj Å ∑N01

iÅ0

(Wij( t) 0 Xi ( t))2 (2)

Xi ( t) is the input sample vector with dimension N(X1 , X2 ,rrr XN) corresponding to N input nodes at time t . Wi 1( t)where Xi ( t) is the input to node i at time t and Wij( t) isis the weight from input node i to the output node 1. Also,the weight from input node i to output node j at time t .set C1 , the number of input patterns assigned to outputStep c. Select the node j* as that output node with the mini-node 1, equal to one. In general, Cj represents the numbermum value dj .of input patterns assigned to output node j . The index jStep d. Weights are updated for the node j* selected fromwill increment one by one as we add more output nodesStep c. In other words, the weights Wij between input unitduring the process.i and output unit j are adjusted by the following formula:

3. Present the new input pattern.4. Compute Euclidean distances to all existing nodes.Wij( t / 1) Å Wij( t) 0 Gain(Wij( t) 0 Xi ( t)) (3)

Compute the Euclidean distances, dj , between the inputvector Xi ( t) and each existing output node j whose location

where Gain is a parameter that can be adjusted as a functionis represented by Wij( t) .

of time.Step e. Go to Step b to obtain a new sample vector for another

iteration. After all samples have been presented in a given dj Å ∑N

iÅ1

(Wij( t) 0 Xi ( t))2 (5)cycle, an average of all DWij obtained in this cycle iscomputed to see if it is smaller than a prespecified number

where Xi ( t) is as stated before, and Wij( t) is the weight1. If that is the case, the network is reaching its stable state,from the input node i to the output node j at time t .and its training stops or converges. Otherwise, another cy-

5. Select the output node with minimum distance.cle for all the samples will start at Step b. When the processSelect node j with the minimum dj as computed in Stepconverges, the values of those final weights will serve as

4, as the winning output node.the desired code words. This process is essentially the gen-6. Decide adaptation/creation of output uniteralized Lloyd algorithm.

Compare dj’s magnitude for the winning node with thatIn this process, several parameters must be decided,of the threshold u. If dj is smaller than u, go to Step 7 forsuch as the dimension of the image vector and the size ofthe adaptation of the winning node. On the other hand, ifthe codebook. Given the dimension of image vectors, thedj is larger than or equal to u, create or set up a new outputentire image can be divided into a certain number of samplenode by going to Step 8.vectors. For a given image, it is important to find an appro-

7. Update network weights.priate codebook size, to give a good SNR value whileThe input pattern is grouped or assigned to the clusterkeeping the training time as short as possible. For the CLA

j , and the corresponding counter Cj is incremented by 1.network, the number of output nodes or codebook size isThe weights between output node j and input layer nodesdecided arbitrarily. If the number of output nodes chosenare updated byis too high, a higher bit rate will be required. If the number

of output nodes chosen is too low, a poor image will beWij( t / 1) Å Xi ( t) /Cj( t) / Cj( t 0 1)Wij( t) /Cj( t) (6)reconstructed owing to the low SNR value. In addition, the

training time for the CLA network can be exceedingly highowing to the large number of iterations usually required Essentially, new weights are set to the new center of gravity

for a given node j . Old weight Wij( t) contributes a propor-during its training process. In many cases, some of theoutput nodes specified may never get used, and become tion of Cj( t 0 1)/Cj( t) , while the newly assigned input

pattern x( t) contributes a proportion of 1/Cj( t) . Cj( t 0 1)so-called ‘‘dead units.’’ These are some of the difficultiesencountered in the original CLA approach [14]. contains the content of the counter Cj before the current

414 Vol. 8, 413–418 (1997)

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST

Figure 1. The network architecture for CLA or MCLA. For CLA, thenumber of output nodes, M , is fixed. For MCLA, M is not fixed be-

Figure 2. A general procedure for image compression system usingcause output nodes are generated adaptively from input vectors.either CLA or MCLA methods.

increment; therefore, it is one less than the content of Cj( t) . IV. COMPARISON WITH OTHER COMPRESSIONNext, go to Step 3 for more input patterns if not all the METHODSpatterns have been presented. Otherwise, go to Step 9. There are some other VQ techniques using neural network or

8. A new output unit is being created, with the initial weights other related techniques. For instance, the Kohonen self-organiz-being assigned to the values of the current input vector and ing feature map (KSFM) approach can obtain some improvementthe corresponding Cj set to one. Then go to Step 3 for more over the situation of underutilized output units by making adjust-input training patterns if not all the patterns have been ments within a neighborhood area [25]. Ahelt [16], Desientopresented. Otherwise, go to Step 9. [26], and Xu [27] proposed methods using ideas related to the

9. Check to see if the number of output nodes has increased. frequency-sensitive competitive learning (FSCL) concepts. TheyIf it has, repeat Steps 3–8 for another cycle before returning essentially resolved the problem of dead units by giving priorityto this step. If it has not, the process has stabilized and can to the underutilized units. However, these approaches are compu-be terminated. tational-intensive and do not alleviate the slow convergence prob-

lem of the competitive learning algorithms. Yair [18] and Zeger[19] both presented enhancement versions of the CLA method byFor the MCLA algorithm, every output node represents a class

to which a certain number of input patterns belong. There is no introducing the concept of stochastic relaxation. Their approachesprovide better performances in terms of error analysis, but do notwaste of output nodes. Figure 1 shows a general illustration. In

this figure, N is the dimension for the input sample, and M , the solve the dead unit issue and still require intensive computerpower. Martinetz [22] used the concept of ‘‘neighborhood rank-number of output nodes, will depend on the training result. In

other words, M is fixed for the CLA algorithm, but not fixed for ing’’ of the reference vector to obtain smaller distortion at theexpense of high computational effort. Huang [21] used principalthe MCLA algorithm. A threshold is needed to start its training

process. A big threshold would generate a small codebook which components to reduce design complexity for VQ, but no specificdata are given for speed comparison. All of the approaches men-may not recover images satisfactorily. On the other hand, a small

threshold would generate a big codebook, which in turn increases tioned so far have not clearly demonstrated the ability to speedup the convergence process, which is a very important issue forthe bit rate and training time. The initial threshold is estimated

as an average intraclass distance for all clusters, based on previous any fast real-time application.Grossberg’s ART algorithm [17] and Xu [23] proposed usingexperience. One usually starts with a large threshold to get coarse

results. If the quality of recovered image is not good, one can threshold parameter for the decision on adaptation/creation issue.Our MCLA shares some of the concepts of ART algorithm,reduce the threshold to obtain more output nodes/codevectors,

which usually lead to better quality for recovered images. Figure namely, the need for stability-plasticity balance. However, ourMCLA model is more flexible in that it does not constrain itself2 shows a general procedure for image compression/decompres-

sion using either the CLA or MCLA method. It should also be with the setup of the weight vectors and the limitation of thenumber of output units that is available.noted that distance measure other than Euclidean distance can be

considered [28]. In comparison with JPEG, some preliminary study has been

Table I. Vector quantization results for liver images.

Codebook Compression CPU TimeAlgorithm Size Bit Rate Ratio SNR (dB) (s)

CLA 115 0.43 18.6 26.8 33.8MCLA 115 0.43 18.6 26.9 16.9CLA 245 0.5 16.0 28.1 107.1MCLA 245 0.5 16.0 28.5 45.4CLA 480 0.56 14.3 29.8 853.0MCLA 480 0.56 14.3 30.5 90.1CLA 993 0.63 12.7 31.8 5050.0MCLA 993 0.63 12.7 33.5 153.0

Vol. 8, 413–418 (1997) 415

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST

Figure 5. Vector quantization result of liver image by MCLA method.Figure 3. Original liver image.Codebook size Å 993; SNR Å 33.5; time Å 153.0 s.

done. The results show that with the same compression ratio, theV. EXPERIMENTAL RESULTSMCLA model has better SNR values or image quality than thatA digital magnetic resonance imaging (MRI) liver image of sizeof JPEG. Of course, the two approaches are different, since JPEG256 1 256 was first used to test the algorithms discussed above.is a transform-based approach [4,24]. We are doing some newBefore the training starts, they were segmented into blocks of 4research by combining the transform and VQ approaches, re-1 4. Therefore, every image was divided into 4096 vectors ofsulting in complexity reduction through truncating high-fre-dimension 16. To make comparisons, both CLA and MCLA ap-quency components. We hope to report the results in anotherproaches have been used for image vector quantization. From thearticle in the near future.results in Table I, one can observe the benefits from using the

Figure 4. Vector quantization result of liver image by CLA method.Codebook size Å 993; SNR Å 31.8; time Å 5050.0 s. Figure 6. Original brain image.

416 Vol. 8, 413–418 (1997)

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST

Figure 7. Vector quantization result of brain image by CLA method. Figure 8. Vector quantization result of brain image by MCLAmethod. Codebook size Å 972; SNR Å 31.2; time Å 211.6 s.Codebook size Å 972; SNR Å 25.7; time Å 1180.0 s.

modified self-organizing network. For example, the MCLA ap-the MCLA method has reduced the time requirement by a factorproach uses a threshold of 70 to perform the vector quantizationof more than 30.on the liver image. The training algorithm generates 115 output

Next, a digital MRI brain image of size 256 1 256 was usednodes using 16.9 s, and the SNR value for the recovered imageto test the algorithms discussed. Figure 6 shows the original brainis 26.9 dB. On the other hand, the CLA algorithm generates 115image. Figures 7 and 8 show the images recovered for a codebookoutput nodes using 33.8 s, with the SNR value for the recoveredsize of 972 for both CLA and MCLA approaches, respectively.image being 26.8 dB. It turns out that the quality of recoveredThe complete results for different codebook size are shown inimage is not good for 115 output nodes, so we need to generateTable II. The results show again that for a fixed codebook sizemore output nodes or larger codebook size. First, smaller thresh-and similar SNR ratio, the MCLA approach takes much less timeolds are used to generate 245, 480, and 993 nodes for the modifiedto complete the codebook construction than the CLA approach.approach. Next, the CLA approaches are also performed using

the same number of nodes and similar SNR values. From theseVI. SUMMARYresults shown in Table I, it is clear that as the number of nodes

increases, the CLA approach takes much more time to train. For The above discussion and experimental results for the VQ oftwo-dimensional images lead us to the following observations.example, in the case of 993 nodes, the CLA approach takes

5050.0 s and the MCLA approach only takes 153.0 s. Figure 3 MCLA has the following advantages: (a) It is an evolutive algo-rithm, using a threshold parameter to resolve the adaptation/shows the original liver image. Figure 4 shows the recovered

image by the CLA method with 993 nodes. Figure 5 shows the creation issue in the design of neuron units. This approach com-pletely resolves the problem of dead units. (b) It has a muchrecovered image by the MCLA method, also with 993 output

nodes and similar SNR values. As one can see from the figures, shorter training time than the CLA, a critical requirement for anyfast real-time image compression. (c) Its recovered image usuallyboth recovered images have excellent SNR measurements, but

Table II. Vector quantization results for brain images.

Codebook Compression CPU TimeAlgorithm Size Bit Rate Ratio SNR (dB) (s)

CLA 121 0.43 18.6 20.6 53.5MCLA 121 0.43 18.6 21.5 18.5CLA 245 0.5 16.0 22.1 303.7MCLA 245 0.5 16.0 23.4 33.5CLA 498 0.56 14.3 24.0 774.7MCLA 498 0.56 14.3 26.6 69.1CLA 972 0.63 12.7 25.7 1180.0MCLA 972 0.63 12.7 31.2 211.6

Vol. 8, 413–418 (1997) 417

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST

upon the Kohonen self-organization feature maps,’’ in Proceedingshas a superior image quality in terms of SNR value than that ofof IEEE International Conference of Neural Network, San Diego,the CLA method. Although there are many other VQ methodsIEEE, NJ, 1988, pp. 1101–1107.designed to reduce distortion or error, they usually suffer from

14. Y. H. Pao, Adaptive Pattern Recognition and Neural Networks (Addi-being computationally intensive, and thus are not suitable forson-Wesley, New York), 1989, pp. 26–50.real-time application. Since the image source can change continu-

15. T. Stockham, Jr. ‘‘Image processing in the context of a visual model,’’ously, the need for a fast on-line codebook design is apparent. Proc. IEEE 60, 828–842 (1972).One can send the new codebook along with compressed codes for 16. S. Ahalt, A. Krishnamurthy, P. Chen, and D. Melton. ‘‘Competitivetransmission. These advantages indicate that the MCLA method learning algorithm for vector quantization,’’ Neural Networks 3, 277–should be given serious consideration. 290 (1990).

17. J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory ofREFERENCES Neural Computation: Vol. 1 of Santa Fe Institute Lecture Notes (Ad-

dison-Wesley, New York), 1991.1. A. Gersho and R. Gray, Vector Quantization and Signal Compression18. E. Yair, K. Zeger, and A. Gersho. ‘‘Competitive learning and soft(Kluwer Academic, Boston), 1992.

competition for vector quantizer design,’’ IEEE Trans. Signal Pro-2. A. K. Jain. ‘‘Image data compression: A review,’’ Proc. IEEE 69,cess. 40, 294–309 (1992).349–389 (1981).

19. K. Zeger, J. Vaisey, and A. Gersho. ‘‘Globally optimal vector quan-3. A. N. Netravali and J. O. Limb. ‘‘Picture coding: A review,’’ Proc.tizer design by stochastic relaxation,’’ IEEE Trans. Signal Process.IEEE 68, 366–406 (1980).40, 310–322 (1992).4. R. Gonzalez and R. Woods, Digital Image Processing (Addison-

20. R. Jarvis and E. Patrick. ‘‘Clustering using a similarity measure basedWesley, New York), 1992.on shared near neighbors,’’ IEEE Trans. Comput. C-22, 1025–10345. Y. Linde, A. Buzo, and R. M. Gray. ‘‘An algorithm for vector quanti-(1973).zation design,’’ IEEE Trans. Commun. com-28, 84–95 (1980).

21. S. Huang and Y. Huang. ‘‘A constrained vector quantization scheme6. X. M. Wang and S. M. Shende, ‘‘Online compression of video se-for real-time codebook retransmission,’’ IEEE Trans. Circuits Syst.quences using adaptive VQ codebooks,’’ in IEEE Data CompressionVideo Technol. 4, 1–7 (1994).Conference ’94, Snowbird, Utah, IEEE, Los Alamitos, CA, 1994, pp.

22. T. Martinetz, S. Berkovich, and K. Schulten. ‘‘Neural-gas network185–194.for vector quantization and its application to time-series prediction,’’7. C. M. Huang and R. W. Harris. ‘‘A comparison of several vectorIEEE Trans. Neural Network 4, 558–569 (1993).quantization codebook generation approaches,’’ IEEE Trans. Image

23. L. Xu. ‘‘Adding learned expectation into the learning procedure ofProcess. 2, (January 1993).self-organizing maps,’’ Int. J. Neural Systems 3, 269–283 (1990).8. R. P. Lippman. ‘‘An introduction to computing with neural nets,’’

24. R. Hunter and A. H. Robinson. ‘‘International digital facsimile codingIEEE ASSP, April, 4–21 (1987).standards,’’ Proc. IEEE 68, 854–867 (1980).9. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Pro-

25. T. Kohonen, Self-Organization and Associative Memory, 3rd ed.cessing, Vol. 1, (MIT Press, Cambridge, MA), 1986.(Springer-Verlag, New York), 1989.10. John P. Guiver, NeuralWorks, Neural Ware, Inc., Sewickleg, PA,

26. D. DeSieno, ‘‘Adding a conscience to competitive learning,’’ in Pro-1988.ceedings of IEEE International Conference on Neural Networks, San11. T. Kohonen, G. Barna, and R. Chrisley, ‘‘Statistical pattern recogni-Diego, IEEE, NJ, 1988, pp. 1117–1124.tion with neural networks: benchmarking studies,’’ in Proceedings of

27. L. Xu, A. Krzyzak, and E. Oja. ‘‘Rival penalized competitive learningIEEE International Conference of Neural Network, San Diego, IEEE,for clustering analysis, RBF net, and curve detection,’’ IEEE Trans.NJ, 1988, pp. 61–68.Neural Networks 4, 636–649 (1993).12. T. A. Kangas, T. K. Kohonen, and J. T. Laaksonen, ‘‘Variants of

28. J. Mao and A. K. Jain. ‘‘A self-organizing network for hyperellipsoi-self-organizing maps,’’ IEEE Trans. Neural Network March, 93–99dal clustering (HEC),’’ IEEE Trans. Neural Network 7, 16–29(1990).

13. N. Nasrabadi and Y. Feng, ‘‘Vector quantization of images based (1996).

418 Vol. 8, 413–418 (1997)

8407 4432/ 8407$$4432 07-17-97 13:51:40 ista W: IST