homepages.cae.wisc.eduhomepages.cae.wisc.edu/~ece539/project/f03/carey.doc · Web viewUsing...
Transcript of homepages.cae.wisc.eduhomepages.cae.wisc.edu/~ece539/project/f03/carey.doc · Web viewUsing...
Associative MemoriesA Morphological Approach
by
Jason M. Carey
Semester Project: CS53919 December 2003
Table of ContentsIntroduction 2
Motivation 2Morphological Associative Memories 3
Work Performed 5Experiment 5Data Collection 6Data Setup for Associative Memory 6Training the Associative Memory Models 7Testing the Associative Memory Models 8User Application 9
Results 9Effects of Distortion and Memory Size on Average Recall Rate 9Effects of Memory Size and Letter Font on Average Recall Rate 11
Discussion of Results 12
Conclusion 13
References 14
Appendix: Tables of Results 15Results of Memory Performance with 5 Images 15Results of Memory Performance with 10 Images 15Results of Memory Performance with 26 Images 16Results of Memory Performance with 52 Images 17
1
Introduction
Motivation
Throughout the course of evolution, humans have acquired the ability to retrieve
information from applied associated stimuli. Amazingly, this ability is not hindered by
the perturbation of the original information-stimuli pair (i.e. recalling one’s relationship
with another after not seeing them for several years despite the other’s physical changes
such as aging, growing a beard, etc.). How the human brain efficiently organizes and
stores such vast amounts of information and is able to recall it given partial or incomplete
stimuli has brought forth much interest. Specifically, researchers have developed the
theoretical neural network model of the associative memory. One of the earliest variants
of this model was the linear associative memory. Despite its simplistic formulation, the
memory was severely limited in flexibility, robustness, and capacity. The primary
defects inherent in the memory were its requirement for orthogonality between stored
patterns and the condition for perfect recall only under negligible pattern distortion and a
constrained memory capacity of no more than the length of the memory [1].
In recent years, these limitations have been improved by the development of the
fully connected, feed-forward Hopfield network. Using this model, the capacity of the
associative memory has been shown to be , where n is the length of the memory [3].
Although the memory capacity of this model is less than that of the linear associative
model, there are no necessary conditions placed on the input for perfect recall and,
moreover, given substantial input distortion recall is still possible. Today, the Hopfield
network is one of the most popular methods for binary associative memories [2].
Unfortunately, even though the Hopfield model as an associative memory drastically
improved the performance of the linear associative memory, the limited capacity of this
network still made associative memories impractical.
This paper will attempt to push the limits of associative memories still further by
using the radically different model of the morphological neural network (MNN).
Historically, neural networks have assumed the biological phenomena that the strength of
2
the electric potential of a signal traveling along an axon is the result of a multiplicative
process where the mechanism of the postsynaptic membrane of a neuron adds the various
potentials of electrical impulses to determine activation [4]. However, MNNs are
formulated on the belief that the strength of the electric potential is an additive process
where the postsynaptic membrane only accepts signals of a certain maximum strength
[4]. Mathematically, this causes a change from the historically used ring structure
consisting of operators: addition and multiplication to the ring structure using operators:
addition and minimum/maximum. Ultimately, the power of MNNs as a model for
associative memories lies in their ability to store as many patterns as can be represented
(2n in the binary case) with the ability to recall partial or incomplete patterns in one step
convergence [5].
Morphological Associative Memories
An associative memory is a system such that when given input x produces output
y. That is, the memory associates x with y. An auto-associative memory is an
associative memory such that . Furthermore, a binary associative memory is an
associative memory containing strictly binary values. Using neural networks, associative
memories are able to recall the desired information given partial or incomplete inputs [5].
The following is a brief formulation to the morphological associative memory taken from
[5].
Morphological associative memories require the basic morphological operations of
max product and min product. The max product, , is defined for matrices
A and B as:
,
where is the maximum operator. Similarly, the min product, , is defined
for matrices A and B as:
,
3
where is the minimum operator. Using these operations, a morphological associative
memory can be constructed using two separate memories, and . Here, memory is
used for input patterns that are dilated versions of a trained pattern. Informally, dilation
means to expand the image. That is, for a black and white image, dilation would cause
the image to contain more black pixels. Memory associating input pattern matrix X
with output pattern matrix Y, is defined as follows:
where xi and yi are the ith patterns of the pattern matrices X and Y, respectively. In
contrast, memory is used for input patterns that are eroded versions of a trained
pattern. Informally, erosion means to contract the image. Again, in the case of a black
and white image, erosion would cause the image to contain less black pixels. Memory
associating input pattern matrix X with output pattern matrix Y, is defined as follows:
where xi and yi are the ith patterns of the pattern matrices X and Y, respectively.
However, even with memories and the morphological memory is not useful. This
is because patterns are generally distorted both by dilation and erosion, which neither
or can sufficiently handle. The recent solution to this is to construct a kernel matrix
which in conjunction with memories and would allow morphological associative
memories to recall general distorted patterns. The kernel matrix, is defined as a matrix
satisfying the following conditions:
A sufficient theorem for determining a kernel for binary associative memories is given by
Ritter and Sussner [5]:
Theorem:
Let X, Y, Z be binary patterns with Z X. If
4
where means the ith bit of the xth pattern, then Z is a kernel for X and Y.
Putting together morphological associative memories are able to recall general distorted inputs in the following manner:
.
Work Performed
Experiment
The ultimate goal of the experiment was to create a binary auto-associative
memory to recall patterns of capital alphabetic letters (i.e. A, B, … , Z) in either
Microsoft San Serif font or Math5 font (see Fig. 1 for illustration of fonts). The
alphabetic patterns were represented as a bitmap using 50 x 50 black and white pixels. A
morphological associative memory was implemented, tested, and compared to the
popular discrete Hopfield network with synchronous updating, which served as the
baseline model. Both models were trained on various sized memories consisting of 5, 10,
26, and 52 patterns. For memory sizes of 5, 10, and 26, two memories were constructed.
The first memory contained patterns strictly using Microsoft San Serif font while the
second memory contained the same patterns using Math5 font. When the memory size
was less than 26, letters were randomly selected from the alphabet. In the case with 52
stored patterns, the memory consisted of the 26 letters from both fonts. Note that this is
the only case were the memory contained letters using both fonts.
After training the models for the various memories, they were tested by applying each
trained pattern 5 times at each distortion setting of 0%, 2%, 4%, 8%, 10%, 15%, 20%,
and 25%. Distortion at X% implied that each pixel had X% probability of flipping
(i.e. or ). Correct recall was defined
as perfect recall in an attempt to eliminate the subjective or heuristic based judging of
recall performance. Lastly, the performance measure was defined as the average recall
rate per memory size given its trained input patterns at the above distortion settings. All
training, testing, and implementation were completed using Matlab 6.5.1.
5
6
Data Collection
Data collection for the experiment required several steps. First, a pattern set had
to be determined. As mentioned above, the pattern set was decided to be capital
alphabetic letters in either Microsoft San Serif font or Math5 font. Capital alphabet
letters were chosen for patterns because they were abundant, easy to generate, and, most
importantly, could be reduced to 50 x 50 pixel bitmaps without severe loss of image
quality. Microsoft San Serif was chosen as a font setting because it most closely
represented simplistic block letters. This was hypothesized to be one of the easiest fonts
for the models to recall because the letters are readily distinguishable by the human eye.
Math5 font was chosen for comparison and exhibited a more complicated representation
of the letter. Differences in these fonts are illustrated below:
Fig. 1
After determining the pattern set, the images had to be generated. First, a word
document consisting of the 26 letters of the alphabet was created for each font. After
generating the two documents, each was scanned using an HP ScanJet 4570c scanner.
Lastly, Adobe Photoshop 7.0 was used to crop out each letter in the scanned document
and convert them to jpeg format.
Data Setup for Associative Memory
Even though the data was collected and generated, it was still necessary to
preprocess the data before applying it to the associative memories. Preprocessing
consisted of two obstacles. First, the letter images needed to be converted to a
7
representation suitable for associative memories in Matlab. This included importing the
images into Matlab, scaling them down to 50 x 50 pixel bitmaps, and converting them to
a 2500 by 1 vector. The importing and scaling were accomplished using Matlab’s image
processing toolbox while the conversion to a vector was accomplished by concatenating
each column in the bitmap. These steps can be found in the makeAlphabet.m file.
The second challenge was to be able to perturb the image to a desired distortion so
that partial trained patterns could be applied to the memories. For this, two probabilistic
distortion methods were chosen. The first method distorted each bit of the image with a
fixed probability (saltAndPepper.m) whereas the second method distorted each bit based
on its value (probSaltAndPepper.m). In other words, in the second method white pixels
and black pixels were distorted with individual probabilities. Only the first method was
used for the experiment, but the user application utilizes both methods. The effects of
distortion using the first method are illustrated below:
Fig. 2
Training the Associative Memory Models
Before testing could begin, both the Hopfield associative memory and the MNN
associative memory needed to be trained. A 3.0 GHz Pentium 4 processor with 1 GB
memory was used for training. The Hopfield model was iteratively trained using J.J.
Hopfield’s proposed learning algorithm [1]. Note that the Hopfield model uses values in
the set {-1, 1} instead of {0, 1}, so it was necessary to first convert the memories to this
format. Training this memory resulted in a single 2500 x 2500 memory, and took
approximately 20 minutes for the tested memory sizes (see HOP_Training.m).
Training for the MNN associative memory was more involved. First, the MNN
model required the construction of a kernel matrix for each memory. The kernel matrix
8
is essentially a reduction of the memory so that each image is mapped to a unique subset
of itself. This is illustrated below with the letters A and B.
Fig. 3 Fig. 4
During the research phase, no algorithm for developing a general kernel was listed in
the literature. To bypass this obstacle, a randomizing algorithm was implemented to
satisfy Ritter and Sussner’s Theorem mentioned above (see getKernel.m). It is worth
noting that a genetic algorithm was also considered for this task, but the preliminary
implementations were inefficient compared to the randomizing algorithm because
generating a gene pool of 2500 x 2500 matrices demanded a large computational cost. A
more suitable representation may exist for a generic algorithm to complete this task.
Using the determined kernels, the MNN model was trained, resulting in memories
and . Both of these were of size 2500 x 2500. Also, training for both and took
approximately 10 minutes for the tested memory sizes (see
MNN_Training.m).
Testing the Associative Memory Models
Once the models were trained for all memory sizes, testing began. As mentioned
above, for each memory size, each trained pattern was presented 5 times at each
distortion setting of 0%, 2%, 4%, 8%, 10%, 15%, 20%, and 25%. The Hopfield
associative memory was allowed a maximum of 1000 iterations for convergence given an
input pattern (HopAA.m). On the other hand, only one step was necessary for the MNN
model using the following method:
.
9
Model performance was measured by the average recall rate per memory size given the
above applied patterns, where recall was correct if and only if it was perfect. Lastly, after
recording the results of the 5 trials for each input, the average recall rate was computed
for each distortion setting.
User Application
A user application, RunAM.m, was developed to test any of the following claims
(read the Readme.txt before running the program).
Results
** Complete results can be found in the Appendix. The below illustrations graphically summarize the results, and the following abbreviations are used:
MNN means Morphological Neural NetworkHOP means Hopfield Neural NetworkMSS means Microsoft San Serif FontM5 means Math5 Font
Below is a graphical description of the effects of image distortion on average
recall rate for a memory consisting of 5 images.
10
Fig. 5
Below is a graphical description of the effects of image distortion on average
recall rate for a memory consisting of 10 images.
Fig. 6
Below is a graphical description of the effects of image distortion on average
recall rate for a memory consisting of 26 images.
11
Fig. 7
12
Below is a graphical description of the effects of image distortion on average
recall rate for a memory consisting of 52 images.
Fig. 8
Below is a graphical description of the effects of memory size (5, 10, and 26) and
letter font on average recall rate for the morphological model.
Fig. 9
13
Discussion of ResultsThe above results reveal several observations. A minor observation regarding
training is that the morphological model always took at least half the time compared to
the Hopfield model. This may be the result of implementation. As for performance
observations, in either the Hopfield model or the morphological model, the results clearly
illustrate that average recall performance degrades as the memory size increases. This is
most noticeable with the Hopfield model. When a memory contained 5 images, this
model had a perfect recall rate at all tested distortion levels with images using Microsoft
San Serif font while the recall rate was between 45% and 100% for images using Math5
font. However, when the memory contained more than 5 images, it never correctly
recalled a pattern. Similarly, but far less dramatically, the morphological model’s
performance degraded with memory size.
Another observation is that both models were worse in recalling images using
Math5 font. This observation is most evident in the morphological model’s performance
illustrated in figure 9, since the Hopfield model’s performance degraded so quickly that it
can only be seen in figure 5. It is worth noting that this observation supports the initial
hypothesis that letters in Math5 font are more difficult to distinguish.
A very important observation is that at 0% distortion (i.e. the original image)
regardless of the memory size, the morphological associative memory always recalled
perfectly. This is not a surprise because the formulation of this model guarantees this
property. The implications of this are amplified by comparing it to the Hopfield network
with memories containing more than 5 images. In these cases, the Hopfield network was
scarcely able to correctly recall an image. Ultimately, given the above results, it is clear
that the associative memory using a morphological model outperformed the memory
using a Hopfield model in the areas of robustness, capacity, and training algorithm
complexity.
14
Conclusion
The ability for humans to retrieve information from associated stimuli continues
to elicit great interest among researchers. Much progress has been made in pushing the
limitations of associative memories. One major milestone in this progression came with
the advent of the Hopfield network as an underlying model for associative memories.
Although this did not improve storage capacity, it did alleviate the orthogonality
constraint placed on the input by linear associative memories and also allowed for greater
input distortion. This paper has shown that progress has not stopped for associative
memories by comparing the Hopfield associative memory model to the radically different
morphological model. Specifically, the morphological associative memory has improved
multiple limitations inherent in the Hopfield model. This includes expanding the storage
capacity from in the Hopfield case to as many patterns as can be represented
(2n in the binary case) [3, 5]. In addition to this, an improvement in robustness and time
complexity in both training and testing were illustrated. Further research into
morphological memories includes constructing kernels for non-binary matrices and
increasing the memory’s robustness. Although the determined results suggest that it is
still premature to add associative memories as a mainstream soft computing technique,
their continual improvement do indicate that this may one day be the case.
15
References
[1] Y. H. Hu. Associative Learning and Principal Component Analysis. Lecture 6 Notes, 2003
[2] R. P. Lippmann. An Introduction to Computing with Neural Nets. IEEE Transactions of Acoustics, Speech, and Signal Processing, ASSP-4:4-22, 1987.
[3] R. McEliece and et. al. The Capacity of Hopfield Associative Memory. Transactions of Information Theory, 1:33-45, 1987.
[4] G. X. Ritter and P. Sussner. An Introduction to Morphological Neural Networks. In Proceedings of the 13th International Conference on Pattern Recognition, pages 709-711, Vienna, Austria, 1996.
[5] G. X. Ritter, P. Sussner, and J. L. Diaz de Leon. Morphological Associative Memories. IEEE Transactions on Neural Networks, 9(2): 281-293, March 1998.
16