Pattern representation & the future of pattern recognition Lev Goldfarb ETS group Faculty of...

Pattern representation Pattern representation & &

the future of pattern recognitionthe future of pattern recognition

Lev Goldfarb

ETS groupFaculty of Computer Science

UNBFredericton, Canada

Outline

1. The wisdom of modern physicists (3 slides)

2. The maturity of a science (1 slide)

3. The currently prevailing wisdom in our field (1 slide)

4. Why should this be the guiding wisdom? (4 slides)

5. Are we mature enough for the task? (2 slides)

6. The (social) reason for the status quo (1 slide)

7. The forgotten history: syntactic pattern recognition (8 slides)

8. Syntactic pattern recognition: the unrealized hopes (4 slides)

9. How should we apply the wisdom of physicists? (2 slides)

10. ETS formalism: its inspiration (2 slides)

11. ETS formalism: temporal information (3 slides)

12. ETS formalism (13 slides)

13. ETS formalism: representational completeness (1 slide)

14. ETS formalism: the intelligent process (1 slide)

15. Learning without representation? (2 slides)

16. Conclusion (3 slides)

Lev Goldfarb, ICPR, Aug. 2004 3

[ The wisdom of modern physicistsFrom: Freeman Dyson, Innovations in Physics, Scientific American, September 1958:

A few months ago two of the great historical figures of European physics, Werner Heisenberg and Wolfgang Pauli, believed that they had made an essential step forward in the direction of a theory of elementary particles. Pauli happened to be passing through New York, and was prevailed upon to give a lecture explaining the new ideas to an audience that included Niels Bohr, who had been mentor to both . . . in their days of glory thirty years earlier when they made their great discoveries. Pauli spoke for an hour, and then there was a general discussion during which he was criticized sharply by the younger generation. Finally, Bohr was called on to make a speech summing up the argument. “We are all agreed,” he said, “that your theory is crazy. The question which divides us is whether it is crazy enough to have a chance of being correct. My own feeling is that it is not crazy enough.”


The wisdom of modern physicists

The objection that they are not crazy enough applies to all the attempts which have so far been launched at a radically new theory of elementary particles. It applies especially to crackpots. Most of the crackpot papers that are submitted to the Physical Review are rejected, not because it is impossible to understand them, but because it is possible. Those that are impossible to understand are usually published. When the great innovation appears, it will almost certainly be in a muddled, incomplete, and confusing form. To the discoverer himself it will be only half-understood. To everybody else it will be a mystery. For any speculation that does not at first glance look crazy, there is no hope.

(the quote continues)


The wisdom of modern physicists ]

Why did two of the 20th century leading physicists behave so “childishly”?

Their wisdom is this:

based on the past experience, a radical novelty of the proposed physical model (of a unified field theory) is a necessary prerequisite for it to be a serious “contender”.

6

The maturity of a science

Why did I begin the talk with the above quote?

I want to draw your attention to one very important informal fact:

the maturity of a science is reflected in the ability of its practitioners to estimate the quality of the match between the reality and its model.

This is one of the main messages I would like you to keep in mind, and I hope we will discuss it in this workshop.

How is our field doing in this respect?


The currently prevalent wisdom in our field

• The main (subconscious?) postulate: Completely rely on statistical models (and, therefore, on the vector space formalism).

• In accordance with the above postulate, expect that there exist some new, statistically “profound” models/algorithms that would do a satisfactory job.

• Innovative, by taking an apparently structural representation, convert it to a numeric one (to “tame” its structural elements), and reduce the problem to the more familiar statistical setting (in the process, misleading yourself and others that one is actually dealing with the structural representation).

8

[ Why should this be the guiding wisdom?

Indeed, why should we confine ourselves to the statistical framework?

In their 1974 book, similar (however somewhat rhetorical) doubts were expressed by two of the present leaders in the field of statistical pattern recognition, Vapnik and Chervonenkis:

In those days it appeared that the pattern recognition problem carried within itself the beginnings of some new idea, which is inno way based on the system of old concepts; researchers wanted tofind new formulations, not to reduce the problem to already knownmathematical schemes. In this sense the reduction of the patternrecognition problem to the scheme of average risk minimizationrouses some disappointment. True, there are attempts to understand the problem in a more complex formulation . . . . As yet, however, such attempts are extremely rare.


Why should this be the guiding wisdom?

Now, many years after the period of ‘pattern recognition romantics’, it is difficult to estimate what this problem formalization brought. It is possible that the desire to find a rigorous formulation led scientists to restrict the meaningful problem solution of which was attempted at the beginning of the ‘60s.

(the quote continues)


Why should this be the guiding wisdom?

From the preface of Probability, Statistics and Truth (1957), by one of the 20th century pioneers of modern probability and statistics, Richard von Mises:

The stated purpose of these [mentioned earlier] investigations is to create a theory of induction or ‘inductive logic’. According to the basic viewpoint of this book, the theory of probability in its application to reality is itself an inductive science; its results and formulas cannot serve to found the inductive processes as such . . ..


Why should this be the guiding wisdom? ]

From: A. N. Kolmogorov, Logical basis for information theory and probability theory, IEEE IT-14 (1968) (one of the founders of modern probability theory):

The proceeding rather superficial discourse should prove two general theses:

(1) Basic information theory concepts must and can be founded without recourse to the probability theory . . . . (2) Introduced in this manner, information theory concepts can form the basis of the concept random, which [would then] naturally suggest that the random is the absence of periodicity.


[ Are we mature enough for the task?

Why are the present day physicists feel compelled to venture (and on a big scale) into such highly speculative theories as “string theories”, while we are infatuated with the “good old” statistics that simply cannot address the qualitative side of modeling, i.e. the structure of the model itself?

Do we understand that an adequate modeling of information processes cannot succeed in the same manner as, for example, the modeling of a flight has succeeded (i.e. without capturing the essence of the corresponding biological processes)?

In particular, do we understand that without producing reasonably good models of the information processes in nature we will not succeed in developing satisfactory information systems?


Are we mature enough for the task? ]Are we mature enough for the task?

God forbid if we are not:

from the very beginning of our science, we are faced with modeling of much more abstract processes then physicists have ever been

=> when modeling information processes, we need even greater imagination than physicists do (who, as I mentioned above, are ahead of us in many ways).

It seems quite obvious to me that without some radically new insights we are not going to get to any “promised land”.


The (social) reason for the status quo

The above prevalent wisdom has not always been as popular as it is today.

One of the main reasons for the status quo is the forgotten part of our history, due to the emergence during the last 15-20 years of two “new” popular areas, neural networks and machine learning. Both of them are dealing with the same subject matter as pattern recognition however starting, basically, all over again, and eventually rediscovering the importance of symbolic representations.

(In contrast to pattern recognition, the professional milieu is not any more engineering, but psychological and computational/statistical, respectively, although both of them attracted many young physicists.)


[ The forgotten history: syntactic pattern recognition

• In North America, one of the few early general texts on pattern recognition, Pattern Recognition Principles (1974), by Tou and Gonzalez, had the last chapter (chapter 8) titled “Syntactic Pattern Recognition” and considered “structural” pattern representations.

• Among English books that came out in the ’70s and ’80s and devoted entirely to this topic, we had those by Fu (Syntactic Pattern Recognition and Applications), Grenander (Lectures in Pattern Theory), Gonzalez and Thomason (Syntactic Pattern Recognition), Watanabe (Pattern Recognition) and several others.


In the resulting excitement and during the making of so many careers in the above two new sister areas, some of the important lessons learned in syntactic/structural pattern recognition were lost, i.e. the critical role of (non-vector) pattern representations and formalisms was overlooked.

The forgotten history: syntactic pattern recognition


Syntactic pattern recognition

• Pioneers: Eden, Narasimhan, and Ledley (published their initial work in the early 60s) and others

• King-Sun Fu (of Purdue university, also instrumental in the founding of IAPR

and was its first president) mounted a productive and influential applied scientific program to shift emphasis from the vector space based representation to other, “structural”, forms of representation, predominantly those associated with formal grammars and its various generalizations.

Fu began his career in statistical pattern recognition; later, in the ’70s and early ’80s, he was largely responsible for the creation of a burgeoning subfield of syntactic pattern recognition, and his untimely death in 1985 had a big impact on the vitality of

this subfield.



Narasimhan (1964):

The aim of any adequate recognition procedure should not be merely to arrive at a “yes”, “no”, “don’t know” decision but to produce a structural description of the input picture.



There are applications of syntactic pattern recognition to almost any field, from seismic oil exploration to speech recognition, from face recognition to fingerprint recognition.



The main overlooked lesson from syntactic pattern recognition (already noted by its pioneers) is this:

even in this incomplete form, “structural” pattern and class representations have substantial advantages over their vector space counterparts, from both applied and theoretical points of view.

(However, see slides 23-26).


Compared to the vector space representation of the digitized image, under symbolic representation, one moves immediately into a more meaningful, higher level representation, with a generative class description.

]

23

What is the problem, then? Why have not these advantages materialized yet in a more apparent manner?

______________________________

Fundamental inadequacy of the (conventional) string representation

Take a string: afdbaaccbdfaddbbcacbffacda

=> no temporal information is represented in it (i.e. how the string was “formed”)

=> exponentially many candidate operations to consider that could have been involved in the generation of the string

[ Syntactic pattern recognition: the unrealized hopes


=> given a training set of strings, the inductive learning process simply cannot recover reliably the set of “generative operations”, i.e. to recover the class description

=> basic inadequacy of the underlying formal structure of the conventional syntactic (similarly all computational) formalisms: the “link” between the class and object representations is too week.

Syntactic pattern recognition: the unrealized hopes


Syntactic pattern recognition: the unrealized hopes

Thus, the conventional string is not an adequate/reliable form of representation: there are just too many formative object histories that are “hidden” behind this representation.

(The related observation applies to graphs and various numeric representations.)

A somewhat obvious problem―which is a consequence of the above fundamental inadequacy―is the presence of the second (“spurious”) alphabet of the non-terminals.

]


We need to be wise

About 2500 years ago Democritus wrote:

“Fools can learn from their own experience; the wise learn from the experience of others.”

____________

So, let’s try to be wise and learn as much as we can from the experience of physicists, mathematicians, and biologists.


[ So how should we apply the wisdom of physicists?

Going back to slide 5, since an incremental wisdom has not really worked for our field, how should we interpret “a radical novelty” for our needs?

I suggest that we should interpret it in two (equally important) ways, both pointing towards radically new forms of representation.


How should we apply the wisdom of mathematicians?

First: why the representation?

If we interpret correctly, from the applied point of view, the wisdom of modern mathematics, we would immediately accept that form the representational point

• the data operations that are not derivatives/compositions of the basic operations (specified by the underlying axiomatic structure of the “data space”) cannot be inductively recovered/discovered.


How should we apply the wisdom of physicists? ]

• we should demand from the model a radical explanatory novelty

we should expect it to offer some basic insights into the nature of information processes in the Universe

• we should demand radical novelty in its formal structure

we should expect it to embody a radically new formal structure

Second:


ETS formalism: its inspiration

From the very beginning, the ETS framework has been inspired by the formal/esthetical beauty and power of a dynamic (and generalized) version of the generative grammar model:

to support an evolving concept of class, one needs an evolving set of transformations that captures the class description and also modifies the corresponding (evolving) mathematical structure on the representation “space”.

(In that sense, if besides ETS there is another formal realization of this vision, it should definitely be investigated.)


ETS formalism: its inspiration

In mathematics, so far, we have been dealing with various static (abstract) structures.

For example, in group theory, which does study the subgroup lattice of a given group, there are, quite naturally, no expectations that one subgroup is obtained by modifying another one.

Even in a more “continuous” setting of a topological space, there are again no expectations that a topological structure itself is evolving.

-----------------------

In contrast, in ETS formalism, some of the central building blocks of the formal structure, the set of transformations, are being modified on the basis of the inductive experience.


[ ETS formalism: temporal information

Thus, it should not come as a surprise that, when we came to the formalization stage about 5 years ago, we had to begin literally from scratch.

The main difficulties have been (and will continue to be) associated with the need to introduce temporal information into a structural representation, i.e. with the concept of object’s formative/generative structural history.

And it is precisely this feature that characterizes the radical departure from all known mathematical paradigms.

ETS formalism: temporal information

Event environment versus object environment:In State 1, three unbonded oxygen atoms are shown. After the first “real” event has occurred, OA and OB become bonded, and the corresponding “ideal” event (primitive 1) is depicted on the right.

ETS formalism: temporal information ]

Note how one event (particle on the left or string on the right) is immediately followed by two events (two particles/strings).

From Edward Witten, Universe on a String, Astronomy, June 2002:

[ ETS formalism: (class) primitive transformations

initial sights

terminal sights

• Think of a primitive as an “elementary” process that transforms the initial “objects” into terminal ones: it is a symbolic “notation” of a typically nontrivial process (structured event).

• The circle and the square denote two site types: letters {a, b} and {x, y} are names of the variables that are allowed to vary over non-overlapping sets of numeric labels.

• Brackets [ ] signify that we are, in fact, dealing with a class of (original) primitives, where each original primitive carries concrete numeric labels.

time

ETS formalism: structs (segments of formative history)

representations of a more general structural object

number 3

Each i denotes an ETS primitive transformation (the order in which the primitive transformations are applied is captured in the representation).

ETS formalism: extructs (contexts)

• Examples of extructs: heavy lines identify the interface sites and crosses identify detached sites.• Contexts should be thought of as parts of the formative history that are necessary for the presence of the (immediately following) “important” segments of history.

ETS formalism: transformation

bodycontext

The “assembled” transform

context

body

Formal definition

40

ETS formalism: a supertransformation

A supertransform, (tau bold), is a generalization of the concept of transformation, and it can be thought of as an abstraction of the set of several “closely related” and inductively acquired transforms.

Here, all contexts have the same interface sites and all bodies have the same initial and terminal sites.

41

ETS formalism: class supertransform (structural class representation)

The class supertransform, [ ] , is obtained on the basis of the supertransform, by abstracting away the supertransform’s site labels.

42

ETS formalism: (single level) class representation

Class representation (associated with a class supertransform []) is defined as a pair

CLASS [] = ( [ ] , CB ) ,

where CB is the context-body association strength scheme, or simply class weight scheme:

CB: {| from } → R+ .

(Obviously, [] is the main, “structural”, part of the representation.)


ETS formalism: (structural) description of a single representational level

Transformation system TS is simply a finite set of class supertransforms:

TS = { [1], [2], . . . , [m] } .

ETS formalism: transition to the next level (a tentative form)

For each class supertransform in a transformation system, we choose a canonical supertransform (shown on the left) and construct the corresponding next-level primitive (shown on the right).

ETS formalism: transition to the next level

Simplified multi-level ETS representation with different time scales for each level.

Two consecutive levels are shown. The time scale for the higher level is measured in coarser units: t’0 corresponds to t0 , t’1 corresponds to t2 ,

t’2 corresponds to t5 .

The contexts of the transformations are not identified.

46

ETS formalism: multi-level view

A multi-level representational tower with a single-level sensor at level 0.

47

ETS formalism: multi-level view of class representation

Pyramid view (partial) of a k-th level class supertransform:

the pyramid is formed by all subordinate class supertransforms.


ETS model basics: the evolution of a class ]

Since any class is specified by a finite set of weighted k-th level (for some k) transformations, the class evolution is readily understood via modification of the set of transformations (structural change) and/or their weights (quantitative change). And this is exactly what you will observe in the functioning of the ETS intelligent process, discussed in the next talk.


The proceedings cover page

This is “Metamorphosis III” by Escher, which was chosen for the cover page of the proceedings as intimating an evolution of a class.


ETS formalism: representational completeness

A most distinguishing feature of this formalism is unprecedented representational completeness and explicitness.

This representational completeness radically changes the formal side of the modeling (the corresponding “future mathematics”).


ETS formalism: the intelligent process

In particular, in pattern recognition, the nature of basic algorithms changes radically, as you will see from the next talk: the processing is now basically concerned with a careful (algorithmic) “examination” of the input data and “recording” of the results of such examination in the ETS language.

This is the job of the intelligent process (which includes the learning and recognition stages):

• the modification of the structural memory (at various levels), i.e. of the class supertransforms, and occasionally, introduction of new levels

• the modification of the numeric memory (at various levels), which is needed, at present, to record the statistics related to various observed recurring associations (between various primitives as well as between the contexts and the bodies).


[ Learning without representation?

(See also Section 6 of my introductory paper in the proceedings).

• Within the vector space (VS) formalism, because of millennia old tradition, it appears that representation is “easy” but learning is “difficult”.

• Within the ETS framework, the present expectation is that representation is difficult but learning is easy. (Not quite true.)


Learning without representation? ]

However, since at the end of the VS leaning process we have

• no reliable/meaningful object or class representations => no “transferable knowledge” about the class objects is gained

and as a consequence

• the results of (carefully crafted) VS learning algorithms can hardly be used for any information processing needs other than “classification”,

it is very misleading to call such process “inductive learning”.After all, induction is the only candidate we have for the central intelligent process.


About our resources

Currently, the (financial) resources of our ETS group are absolutely negligible, which is why it is not surprising, given the scale of the undertaking, that the more “substantial” applications of the ETS formalism are still ahead.


Inductive informatics

Nevertheless, we are an optimistic lot and have big plans:

we even introduced the term inductive informatics for the new science emerging around the development and various applications of the ETS formalism (this is to honor the ideas of the true prophet of modern science, Francis Bacon)

moreover, on the philosophical side, as many scientists and non-scientists have noted, numeric formalisms have failed “to reveal the unity of nature”; so we believe that, as a general “symbolic” formalism, ETS, when developed, promises to fit the bill.


[ Conclusion

Going back to slide 6, I would like you to keep in mind the question raised there.

How mature are we as a science: are we capable of estimating the quality of the match between the (inductive) information processes as they exist in nature and our models of them?

Let’s come back to this question periodically and during the workshop’s concluding open discussion.

Conclusion

• Radical explanatory novelty of the ETS model

• Radical formal/structural novelty of the ETS model

the relationship between an object and its class

the nature of class and class description

the temporal nature of class and object descriptions

the nature of multilevel class description

the nature of class evolution

the first temporal form of structural representation (generaliz. Peano axioms)

its unprecedented representational completeness/transparency

still ahead: the mathematics of such structural representations

(the concept of class, rather than that of a set, will be the pivotal point)


Conclusion ]

• Undoubtedly, the golden age of our field is still ahead, and it appears that its arrival depends on the choice of the “right” representational formalism.

• The very hart of any scientific enterprise is the construction of a fundamentally new model, and much more rarely, of a new formalism.

• I do hope that some of you will participate in this most exciting development.

Pattern representation & the future of pattern recognition Lev Goldfarb ETS group Faculty of...

Documents

Transcript of Pattern representation & the future of pattern recognition Lev Goldfarb ETS group Faculty of...