Nature of External Representations in Problem Solving

8/3/2019 Nature of External Representations in Problem Solving

1/25

To appear in Cognitive Science, 00, 000-000.

The Nature of External Representations in Problem Solving1

Jiajie ZhangDepartment of PsychologyThe Ohio State University

1This research was in part supported by a Seed Grant from the Office of Research at The Ohio State University. I am especiallygrateful to Todd Johnson for many valuable discussions and suggestions. I would also like to thank Bill Clancey, Mari Jones,Don Norman, Keith Stenning, Krishna Tateneni, Hongbin Wang, and an anonymous reviewer for comments on an early draft,and Dwen Hall, Ben Tupaz, Hongbin Wang, and Kellie White for their assistance in the experiments. Correspondence and re-quests for reprints should be sent to Jiajie Zhang, Department of Psychology, The Ohio State University, 1827 Neil Avenue, Co-lumbus, OH 43210. Email: [email protected].

ABSTRACT

This article proposes a theoretical framework for externalrepresentation based problem solving. The Tic-Tac-Toeand its isomorphs are used to illustrate the procedures ofthe framework as a methodology and test the predictions ofthe framework as a functional model. Experimental resultsshow that the behavior in the Tic-Tac-Toe is determined bythe directly available information in external and internalrepresentations in terms of perceptual and cognitive biases,regardless of whether the biases are consistent with, incon-sistent with, or irrelevant to the task. It is shown that ex-ternal representations are not merely inputs and stimuli tothe internal mind and that they have much more important

functions than mere memory aids. A representational de-

terminism is suggested--the form of a representation de-termines what information can be perceived, what proc-esses can be activated, and what structures can be discov-ered from the specific representation.

External representations are involved in many cogni-tive tasks, such as multiplication with paper and pen-cil, grocery shopping with a written list, geometricalproblem solving, graph understanding, diagrammaticreasoning, chess playing, and so on. Few would denythat external representations play certain roles inthese tasks. However, in comparison with internalrepresentations, relatively little research has beendirected towards the nature of external representa-

tions in cognition. This might be due to the beliefthat very little knowledge about the internal mind canbe gained by studying external representations, ordue to the view that external representations arenothing but inputs and stimuli to the internal mind, orsimply due to the lack of a suitable methodology forstudying external representations.

This article explores the functions of externalrepresentations, using problem solving as the taskdomain and test bed. It takes the position that muchcan be learned about the internal mind by studyingexternal representations because much of the struc-ture of the internal mind is a reflection of the struc-ture of the external environment (e.g., Anderson,

1993; Shepard, 1984; Simon, 1981). It argues thatexternal representations are not simply inputs andstimuli to the internal mind; rather, they are so intrin-

sic to many cognitive tasks that they guide, constrain,and even determine cognitive behavior. By focusingon what information in external representations canbe perceived and how the information in externalrepresentations affects problem solving behavior, thisarticle develops a theoretical framework for externalrepresentation based (henceforth, ER-based) problemsolving. This framework is not only a functionalmodel that can make specific empirical predictionsbut also a methodology that can be used to systemati-cally analyze ER-based problem solving tasks.

This article is divided into five parts. The firstpart introduces the theoretical background, including

a definition of external representations, a discussionon the relationship between internal and external rep-resentations, and a brief review of the important rolesof external representations in cognition. The secondpart proposes the theoretical framework for ER-basedproblem solving. The third part uses the frameworkas a methodology to analyze the structure of the Tic-Tac-Toe and as a functional model to make specificpredictions about the behavior in the Tic-Tac-Toe.The fourth part reports three experiments designed totest the predictions of the framework and examine thegeneral properties of external representations. Thelast part summarizes the experimental results, evalu-ates the theoretical framework, and suggests a repre-sentational determinism.

THEORETICAL BACKGROUND

A Definition of External RepresentationsIn the present study, external representations are de-fined as the knowledge and structure in the environ-ment, as physical symbols, objects, or dimensions(e.g., written symbols, beads of abacuses, dimen-sions of a graph, etc.), and as external rules, con-straints, or relations embedded in physical configura-tions (e.g., spatial relations of written digits, visualand spatial layouts of diagrams, physical constraints

in abacuses, etc.). The information in external repre-sentations can be picked up, analyzed, and processedby perceptual systems alone, although the top-downparticipation of conceptual knowledge from internal


2/25

External Representations 2

representations can sometimes facilitate or inhibit theperceptual processes. In contrast, internal representa-tions are the knowledge and structure in memory, aspropositions, productions, schemas, neural networks,or other forms. The information in internal repre-sentations has to be retrieved from memory by cog-nitive processes, although the cues in external repre-

sentations can sometimes trigger the retrieval proc-esses. Let us consider multiplying 735 by 278 usingpaper and pencil. The internal representations are themeanings of individual symbols (e.g., the numericalvalue of the arbitrary symbol "7" is seven), the addi-tion and multiplication tables, arithmetic procedures,etc., which have to be retrieved from memory; theexternal representations are the shapes and positionsof the symbols, the spatial relations of partial prod-ucts, etc., which can be perceptually inspected fromthe environment (see Zhang & Norman, 1995). Toperform this task, people need to process the infor-mation perceived from external representations andthe information retrieved from internal rep-

resentations in an interwoven, integrative, and dy-namic manner.

External representations can be transformed intointernal representations by memorization. But thisinternalization is not necessary if external representa-tions are always available, and not possible if exter-nal representations are too complex. Internal repre-sentations can also be transformed into external rep-resentations by externalization. Externalization canbe beneficial if the benefit of using external repre-sentations can offset the cost associated with the ex-ternalization process.

The Relationship betweenInternal and External Representations

The importance of explicitly distinguishing externalrepresentations from internal ones has not been seri-ously considered until recently. In traditional cogni-tive science, most studies either exclusively focusedon internal representations or, when taking externalrepresentations into account, often failed to separatethem from internal ones. Thus, these studies oftenmistakenly equate external representations to internalrepresentations, or equate representations having bothinternal and external components to internal repre-sentations. As noted by Kirlik, Plamondon, Lytton,and Jagacinski (1993a, 1993b) and Suchman (1987),this confusion often leads one to postulate unneces-sary complex internal mechanisms to explain thecomplex structure of the wrongly identified internalrepresentation, much of which is merely a reflectionof the structure of the external representation.

When people do acknowledge the differencebetween internal and external representations, theyusually have different views on their relations. Oneview is that external representations are merely inputsand stimuli to the internal mind. In this view, even if

it is the case that many cognitive tasks involve inter-actions with the environment, all cognitive process-ing only occurs in the internal model of the externalenvironment. Thus, when an agent is faced with atask that requires interactions with the environment,the agent first has to create an internal model of theenvironment through some encoding processes, then

performs mental computations on the contents(symbols, subsymbols, or other forms) in this con-structed internal model, and then externalizes theproducts of the internal processing to the environ-ment through some decoding processes. This is acommon view in traditional AI and other fields ofcognitive science (e.g., see Newell, 1990, pp. 57).

A radically different view, offered by Gibson(1966, 1979), is that the environment is highly struc-turedfull of invariant information in the extendedspatial and temporal patterns of optic arrays. Theinvariant information in the environment can be di-rectly picked up without the mediation of memory,inference, deliberation, or any other mental processes

that involve internal representations. To Gibson, theinformation in the environment is sufficient to spec-ify all objects and events in the environment, andthus it is sufficient for perception and action. In ad-dition, the end product of perception is not an internalrepresentation of the environment; rather, it is theinvariant directly picked up from the environment.

There are a few recent approaches that empha-size the structures of the environment and people'sinteractions with them without denying the importantroles of internal representations. The situated cogni-tion approach, for example, argues that people's ac-tivities in concrete situations are guided, constrained,and to some extent, determined by the physical andsocial context in which they are situated (e.g., Bar-wise & Perry, 1983; Clancey 1993; Greeno, 1989;Greeno & Moore 1993; Lave, 1988; Lewis, 1991;Suchman, 1987). In this view, it is not necessary toconstruct an internal model of the environment tomediate actions: people can directly access the situa-tional information in their environment and act uponit in an adaptive manner. As another example, thedistributed cognition approach explores how cogni-tive activity is distributed across internal humanminds, external cognitive artifacts, and groups ofpeople, and across space and time (e.g., Hutchins,1990, 1995a, 1995b; Hutchins & Norman, 1988;Norman, 1988, 1991, 1993b; Zhang & Norman,1994). In this view, much of a person's intelligentbehavior results from interactions with external objects and with other people. For example, Hutchins(1990, 1995a, 1995b) has shown that the cognitiveproperties of a distributed cognitive system consist-ing of a group of people interacting with complexcognitive artifacts (e.g., the cockpit of a commercialairplane or the control room of a military ship) candiffer radically from the cognitive properties of the


3/25


individuals, and they cannot be inferred from theproperties of the individuals alone, no matter howdetailed the knowledge of the properties of those in-dividuals may be. Zhang & Norman (1994), focusingon distributed cognitive tasks that involve interac-tions between internal and external representations,also argue that the representation of a distributed

cognitive task is neither solely internal nor solelyexternal, but distributed as a system of distributedrepresentations with internal and external representa-tions as two indispensable parts.

The Important Roles ofExternal Representations

External representations are not simply inputs andstimuli to the internal mind. They have many im-portant properties. The most obvious one is that theycan serve as memory aids: extend working memory,form permanent archives, allow memory to beshared, etc. However, the properties that truly makeexternal representations crucial are not memory aids.

For many tasks, external representations are intrinsiccomponents, without which the tasks either cease toexist or completely change in nature. The followingdiscussion briefly reviews some of the properties ofexternal representations that cannot be simply con-sidered as memory aids.

Diagrams, graphs, and pictures are a few typicaltypes of external representations. They are used inmany cognitive tasks such as problem solving, rea-soning, and decision making. In the studies of therelationship between mental images and externalpictures, Chambers & Reisberg (1985; Reisberg,1987) showed that external pictures can give peopleaccess to knowledge and skills that are unavailablefrom internal representations. In the studies of dia-grammatic problem solving, Larkin & Simon (1987;Larkin, 1989), for example, argue that diagrammaticrepresentations support operators that can recognizefeatures easily and make inferences directly. In thestudies of logical reasoning with diagrams, Stenning& Oberlander (1995) argue that diagrammatic repre-sentations such as Euler circles limit abstraction andthereby aid processibility, that is, graphical repre-sentations can make some information interpretableand transparent in a specialized form at the expenseof limiting abstraction in general forms. The repre-sentation, perception, and comprehension of graphshave been extensively studied since last century (fora few integrative studies, see Bertin, 1983; Cleve-land, 1985; Schmid, 1983; Tufte, 1990). It is well-known that different forms of graphic displays havedifferent representational efficiencies for differenttasks and can cause different cognitive behaviors.For example, Kleinmuntz & Schkade (1993) showedthat different representations (graphs, tables, andlists) of the same information can dramaticallychange decision making strategies. Zhang (1996)

suggested that all graphs can be systematically stud-ied under a representational taxonomy based on theproperties of external representations.

The studies on literacy also show the importantfunctions of external representations. The classicalview on writing, originally developed by Aristotle(1938) and restated in our own time by Bloomfield

(1993) and Saussure (1959), is that writing merelytranscribes or re-represents speech from one externalrepresentation in auditory form to another externalrepresentation in visual form. For some people,however, it is not a simple transcription becausewriting supports reflective thought (Norman, 1993b)without which the logical, analytic, rational, and sci-entific modes of modern thought are impossible (e.g.,Goody, 1977; Ong, 1982). For example, Goody ar-gues that the shifts from the so-called prelogical tomore and more rational mode of thought resultedfrom the shifts from orality to various stages of liter-acy, that is, writing systems are not only the productsof the mind but also part of the determining features

of the mind. Without writing, the human mind wasso occupied by the participation in dynamic utteranceof speech that it could not organize and elaboratelogical relations in the analytic form of linear se-quences. Rational mode of thought was possible onlybecause certain procedures were made available bythe technology of wri ting. Ong also argues thatwriting has reconstructed cognition: writing systemsare not mere external aids but also internal transfor-mations of cognition. In a recent paper, Olson (1996)has made a convincing argument that writing doesnot merely transcribe but rather brings structuralproperties of speech into consciousness, that is, thedevelopment of writing was also the discovery of therepresentable structures of speech. From an evolu-tionary perspective, Donald (1991) also illustrated theimportant roles of external representations in theemergence of the modern mind. According to Don-ald, the changes in cognitive architecture mediated byexternal representations were no less fundamentalthan those mediated by biological changes in thebrain: the external symbolic system, especially writ-ing, is the most important representational systemresponsible for much of the virtually unlimited cog-nitive capacity of the modern mind.

In the study of the representational properties ofdistributed cognitive tasks, Zhang & Norman (1994)also identified several properties of external repre-sentations. First, they provide information that canbe directly perceived and used without being inter-preted and formulated explicitly. Second, they cananchor cognitive behavior. That is, the physicalstructures in external representations constrain therange of possible cognitive actions in the sense thatsome actions are allowed and others prohibited.Third, they change the nature of tasks: tasks with andwithout external representations are completely dif-


4/25


ferent tasks from a task performer' point of view,even if the abstract structures of the tasks are thesame (see also Norman, 1991).

The above brief review clearly demonstrates thatexternal representations are not simply inputs andstimuli to the internal mind, and they are much morethan memory aids. For many tasks, external repre-

sentations are so intrinsic to the tasks that they guide,constrain, and even determine the pattern of cognitivebehavior and the way the mind functions. Given thatexternal representations are so important, they needto be considered seriously, not as something trivial;and they need to be studied on their own right, not assomething peripheral to internal representations.The present study is a serious attempt to study exter-nal representations in a systematic manner. A par-ticular area, ER-based problem solving, is selected asthe task domain and test bed.

A THEORETICAL FRAMEWORK

The behavior in ER-based problem solving is con-strained both by the complexity of the environmentand by the limitations of the mind. On one hand, theenvironment is complex because of too much infor-mation, real time requirement, unpredictable out-comes, etc. On the other hand, the mind is limitedbecause of the limited bandwidth of informationprocessing, the limited capacity of working memoryand attention, the limited speed of mental operationsand learning, etc. (e.g., Norman, 1993a). The com-plexity of the environment and the limitations of themind, taken together, suggest that the determiningfactors of the behavior in ER-based problem solvingare not just the structures of the mind but also thestructures of the environment.

This section develops a theoretical frameworkfor ER-based problem solving. This framework is notonly a functional model that can make specific em-pirical predictions but also a methodology that canspecify the important components that need be ana-lyzed for any ER-based problem solving task. Themajor components of the framework are shown inFigure 1 and their details are described as follows.

Components of The FrameworkAbstract Structures

Each ER-based problem solving task has an ab-stract structure. It specifies the properties of the task

that are independent of specific representations andimplementations. Abstract structures are usually onlyconceivable to theorists because task performers usu-ally only deal with the specific representational andimplementational contents in which the abstractstructures are only implicitly embedded.

RepresentationsThe abstract structure of a ER-based problem

solving task can be implemented by different isomor-phic representations (not shown in Figure 1 but inFigure 4). Each isomorphic representation is a dis-tributed representation decomposed into an internaland an external representation, which are the knowl-edge and structure of the task in memory and in theenvironment, respectively. The present framework

demands such a decomposition because the repre-sentation of a ER-based problem solving task is nei-ther solely internal nor solely external but distributed(see Zhang & Norman, 1994).

Operations and InformationDifferent representations activate different op-

erations, not vice versa. It follows that operations arerepresentation-specific. External representations ac-tivate perceptual operations, such as searching forobjects that have a common shape and inspectingwhether three objects lie on a straight line. In addi-tion, external representations may have invariant in-formation that can be directly perceived without the

mediation of deliberate inferences or computations,such as whether several objects are spatially symmet-rical to each other and whether one group has thesame number of objects as another group. Internalrepresentations activate cognitive operations, such asadding several numbers to get the sum. In addition,internal representations may have information thatcan be directly retrieved, such as the relative magni-tudes of single-digit numbers.

Lookahead and BiasesThe basic unit in problem solving is an action. If

there are more than one alternative actions under acertain problem state, one needs to make a decisionon which action to take. The decision on actions canbe based on lookahead, or biases, or learned knowl-edge, or any combination of them.

Lookahead is the activity of mentally imaginingand evaluating alternative sequences of actions beforeactually selecting an action. For some tasks, it ispossible to lookahead all alternatives of completesequences and then take the best sequence of actionsthat leads to the goal. For many other tasks, how-ever, it is impossible to do complete lookahead, dueto the complexity of the tasks and the limited re-sources of attention and working memory. Perceptualand cognitive operations demand different amount ofattentional and working memory resources, thereforethey affect the amount of lookahead that can be per-formed. Generally speaking, perceptual operationsrequire less attentional and working memory re-sources than cognitive operations. Different percep-tual operations (e.g., sequential vs. parallel search)may also have different effects on lookahead, so dodifferent cognitive operations.


5/25


DirectlyPerceived

Information

Central Control

ExternalRepresentation

InternalRepresentation

Abstract TaskStructure

PerceptualOperations

CognitiveOperations

DirectlyRetrieved

Information

Actions

Lookahead BiasesLearned

Knowledge

Distributed Representation

Figure 1. The theoretical framework of ER-based problem solving. See text for details.

The directly perceived information from externalrepresentations and the directly retrieved informationfrom internal representations may elicit perceptualand cognitive biases, respectively, on the selection ofactions. If the biases are consistent with the task,they can guide actions towards the goal. If the biasesare inconsistent with the task, however, they can alsomisguide actions away from the goal. Learning ef-fect can occur if a task is performed more than once.Thus, the decision on actions can also be affected bylearned knowledge.

Central ControlThe central control is the most complex but least

specified component of the framework. It consists ofthe mechanisms of working memory and attention,interpreting and understanding, learning, deliberationand decision making, memory retrieval, and so on.Although the central control is crucial for problemsolving, the current framework does not attempt tospecify its details because it is not the current focus.The most important function of the central control

that is unique to ER-based problem solving is thecoordination of the interplay between perception andcognition: allocating and switching attention betweeninternal and external representations, integrating in-ternal and external information, and coordinatingperceptual and cognitive operations (for a few studieson these issues, see Carlson, Wenger, & Sullivan,1993; Dark, 1990; Weber, Byrd, & Noll, 1986).

The Key Assumption of the FrameworkThe key assumption of the framework is that externalrepresentations need not be re-represented as an in-ternal model in order to be involved in problemsolving activities: they can directly activate percep-tual operations and directly provide perceptual in-formation that, in conjunction with the memorial in-formation and cognitive operations provided by in-ternal representations, determine problem solvingbehavior. There are three points that need to beelaborated about this key assumption.

First, the key assumption denies the necessity ofthe internal model of external representations. Per-


6/25


ception is not a peripheral device to ER-based prob-lem solving. Rather, it is a central component, for thefollowing reasons. First, perception is not the simpleencoding processes that re-represent external repre-sentations as an internal model. The end product ofperception no longer models and mirrors the externalrepresentations because the end product is the infor-

mation and structure either directly picked up or al-ready highly analyzed, processed, and transformed byperceptual systems. Second, the end product of per-ception is not necessarily the intermediate data pre-pared for high-level cognitive mechanisms. Rather,the end product of perception is often the end productof the whole problem solving process. Third, theperceptual operations activated by external represen-tations are central mechanisms of ER-based problemsolving: they are no less fundamental than the cogni-tive operations activated by internal representations.For these three reasons, ER-based problem solvingcan also be called perceptual problem solving. Be-cause perceptual processes are central components of

ER-based problem solving and because they directlyoperate on external representations, external repre-sentations are also central components of ER-basedproblem solving. Thus, there is nothing to lose whenthe internal model is not available. The behavior inER-based problem solving is simply the interwoven,integrative, and dynamic processing of the informa-tion perceived from external representations and thatretrieved from internal representations.

Second, the key assumption does not mean thatexternal representations can function independentlywithout the support of anything internal or mental.External representations have to be processed by per-ceptual mechanisms, which are of course internal.The end product of these perceptual mechanisms isalso internal. However, this end product is notequivalent to the internal representation of the task asdefined in the present study. It is not an internalmodel of the external representation of the task, ei-ther. As defined earlier, the internal representation ofan ER-based task is the knowledge and structure ofthe task in memory; and the external representation isthe knowledge and structure of the task in the envi-ronment. The end product of perception is merelythe situational information in working memory thatusually only reflects a fraction (usually crucial) of theexternal representation.

Third, the key assumption does not mean thatexternal representations can always be directly andautomatically used without being interpreted andcontrolled by high-level cognitive mechanisms. Ifthe end product of perception is not the end productof problem solving, it has to be interpreted by high-level cognitive mechanisms for further processing.Because not everything in external representations isalways relevant to a task, high-level cognitivemechanisms need to use internal task knowledge

(usually supplied by task instructions) to direct atten-tion and perceptual processes to the relevant featuresof external representations.

Test and Justification of the FrameworkThe theoretical framework is not only a functionalmodel that can be tested empirically but also a meth-

odology that can be used to analyze ER-based prob-lem solving tasks. The Tic-Tac-Toe (henceforth,TTT) and its isomorphs are selected to demonstratethe procedures of the framework as a methodologyand test the predictions of the framework as a func-tional model. TTT is selected for these dual purposesfor the following reasons. First, it is simple enough toallow well-controlled laboratory studies. Second, itis complex enough to mimic real world complexproblems. For example, complete mental lookaheadis impossible for TTT, just like for most real worldcomplex tasks. Third, since the focus of the presentstudy is on external representations, the task shouldhave several isomorphs that have rich structures in

external representations such that the properties ofexternal representations and their effects on problemsolving behavior can be examined. TTT is a task thathas just this property. In next section, the theoreticalframework is first used as a methodology to analyzethe TTT isomorphs, then it is used to make specificpredictions on the problem solving behavior in theTTT.

TIC-TAC-TOE

The TTT is a well-known two-player game. A minorvariation of the original TTT is shown in Figure 2A(Line version). The task for the two players is to se-lect the circles in turn by coloring the circles withdifferent colors, one at a time. The one who first getsthree circles on a straight line (horizontal, vertical, ordiagonal) wins the game. The TTT is a draw game,i.e., when both players use optimum strategies, nei-ther can win.

Figures 2B, 2C, and 2D show three more iso-morphs of the TTT. In theNumberversion (Figure2B), the task is to select the numbers in turn by col-oring the numbers, one at a time. The one who firstgets three numbers that exactly add to 15 wins thegame. In the Shape version (Figure 2C), the task isto select the big circles in turn by coloring the objectsinside a big circle, one big circle at a time. The one

who first gets three big circles that contain a commonshape wins the game. In the Color version (Figure2D), the task is to select the big circles in turn bydrawing different background textures. The one whofirst gets three big circles that contain the same col-ored small circle wins the game.

The equivalence of the four isomorphs in Figure2 is shown in Figure 3. To make the mappings easyto understand, the nine elements in Number, Shape,and Colorwere arranged in the same spatial relations


7/25


as inLine. The center, corners, and sides in Line(Figure 3A) correspond to the number five, evennumbers, and odd numbers in Number(Figure 3B),the 4-object, 3-object, and 2-object big circles inShape (Figure 3C), and the 4-object, 3-object, and 2-object big circles in Color(Figure 3D), respectively.

Theoretical AnalysesIn this section, the four TTT isomorphs are analyzedin terms of the components of the theoretical frame-work in Figure 1.

Abstract StructuresThe TTT has four abstract properties. First, it

has nine elements. Second, it has eight winning trip-lets, each of which is a group of three elements thatconstitute a win. Third, the nine elements are dividedinto three symmetry categories. The elements in a

symmetry category are identical to each other, i.e., tomake a move, selecting one element is not differentfrom selecting another element in the same category.Fourth, the elements within a symmetry categoryshare a common invariant propertywinning invari-ant, which is the number of winning triplets in whichan element is part of. The winning invariants of the

three symmetry categories are 2, 3, and 4.For example, for Line in Figure 2A, the nine

elements are the nine circles. The eight winningtriplets are the 3-circle groups that lie on the 3 hori-zontal, 3 vertical, and 2 diagonal lines. The threesymmetry categories are the center, 4 corners, and 4sides. The winning invariants of the center, corners,and sides are 4, 3, and 2, respectively. For example,the center is an element of 4 winning triplets: 1 hori-zontal, 1 vertical, and 2 diagonal lines.

3

1

7

4

9

2

(A) Line (B) Number (C) Shape (D) Color

6

5

B

R

R

RGG

G

Y

Y

Y

O

O

O

B

B

P

P P

W

W

W

L

L

L

8

Figure 2. Four TTT isomorphs. (A)Line. Getting three circles on a straight line is a win. (B) Number. Getting threenumbers that exactly add to 15 is a win. (C) Shape. Getting three big circles that contain a common shape is a win. (D)Color. Getting three big circles that contain the same colored small circle is a win. The letters inside the circles indicate thecolors used in the experiments: B = Blue, G = Green, L = Light Blue, O = Orange, P = Pink, R = Red, Y = Yellow, W =

Brown.

G

O

R Y

OR

B

G

Y

B

P

B

W O

P W

Y

W

L

G

L

RP

L

2

5

9 4

37

6 1 8

(A) Line (B) Number (C) Shape (D) Color

center 5 4 objects 4 circles

corners even numbers 3 objects 3 circles

sides odd numbers 2 objects 2 circles

Figure 3. The mappings among the four TTT isomorphs. The center, corners, and sides in (A) correspond to five, evennumbers, and odd numbers in (B), 4-object, 3-object, and 2-object big circles in (C), and 4-object, 3-object, and 2-object bigcircles in (D), respectively.


8/25


Representations, Operations, and InformationEach of the four formal properties of the TTT is

represented differently in the four TTT isomorphs inFigure 2. Different representations activate differentoperations and provide different types of information.

Elements. The nine elements in Line, Shape,and Colorare represented externally. They all corre-

spond to nine distinct spatially-spread physical enti-ties, which can be perceptually separated and identi-fied. In Number, the nine elements are representedboth externally and internally: externally becausethey can be perceptually separated, and internallybecause the meanings (numerical values) of the ninedigits have to be retrieved from memory.

Winning Triplets. InLine, the eight winningtriplets are represented externally by the 3 horizontal,3 vertical, and 2 diagonal straight lines. To identify awining triplet is to search for three circles lying on astraight line. Whether three circles lie on a straightline can be perceptually inspected. In Number, theeight winning triplets are represented internally by

eight number triplets, each of which has three num-bers that add to 15. To identify a winning triplet is tosearch for three numbers that add to 15. Whetherthree numbers add to 15 has to be mentally com-puted. In both Shape and Color, the eight winningtriplets are represented externally, the former by theeight different shapes of the small objects and thelatter by eight different colors of the small circles.To identify a wining triplet is to search for three bigcircles that contain a common shape or a commoncolor. Both shapes and colors can be identified andsearched perceptually. However, the search forshapes in Shape is sequential whereas the search forcolors in Color can be parallel due to the pop-outeffect of the specific colors used in the isomorph.

Symmetry Categories. In Line, the three sym-metry categories are represented externally by spatialsymmetry: the center, 4 corners, and 4 sides. Forexample, the 4 corners are spatially symmetrical toeach other and therefore equivalent to each other.Spatial symmetry can be directly perceived. InNum-ber, the three symmetry categories are representedinternally by parity: the number five, 4 even num-bers, and 4 odd numbers. (Though 5 is an odd num-ber, for convenience, it is considered to form a sym-metry category by itself.) The symmetry informationin this version is not directly available because twonumbers having the same parity does not necessarilymean that they are equivalent to each other: they havemany other numerical properties such as dividable bythree, prime numbers, magnitudes, etc. In bothShape and Color, the three symmetry categories arerepresented externally by the quantity of objects in abig circle: 4-object, 3-object, and 2-object big circles.However, the symmetry information in this case cannot be directly perceived because two big circleshaving the same number of objects does not necessi-

tate that they are equivalent to each other.Winning Invariants. In Line, the winning in-

variant of a symmetry category is represented exter-nally by the number of straight lines connecting acircle: 4, 3, and 2 for the center, the corners, and thesides, respectively. The number of straight linesconnecting a circle can be directly perceived. In

Number, the winning invariants are represented inter-nally: 4 for the number five, 3 for even numbers, and2 for odd numbers. They are not directly representedand thus are not directly available. To get the win-ning invariant of a number, it must be grouped withall possible pairs of other numbers to form numbertriplets and the sums of the three numbers in all num-ber triplets have to be mentally computed to seewhether each sum is 15. Even with extensive mentalcomputations, this task is very difficult if not impos-sible. In both Shape and Color, the winning invariantin a symmetry category is represented externally bythe quantity of objects in a big circle: 4, 3, and 2,which can be perceptually identified.

Lookahead and BiasesDue to the complexity of the TTT problem space

and the limited capacity of working memory, com-plete mental lookahead is nearly impossible for theTTT isomorphs. For example, the task in Experimenthas forty-seven 8-step lookahead sequences for firstmoves even with symmetry considered. Thus, it isimpossible to do complete lookahead for first movesfor any TTT isomorphs. For second moves, there arethree and eight 6-step lookahead sequences, respec-tively, with and without symmetry being considered.Thus, complete lookahead is still very difficult, if notimpossible, for second moves for the TTT isomorphs.

When complete lookahead is difficult or impos-sible, people may use perceptual or cognitive biasesto make decisions. The winning invariants of theTTT isomorphs, if they can be perceived, can elicit amore-is-betterbias: if an element is involved in morewinning triplets, that is, its winning invariant islarger, it should be preferred, because it may blockmore pieces of the opponent and create more oppor-tunities for oneself. The more-is-better bias can beelicited in Line, Shape, and Colorbecause the win-ning invariants in these three versions can be directlyperceived from their external representations. Thus,for Line, Shape, and Color, the center and 4-objectcircles should be most preferred, corners and 3-objectcircles should be the next, and sides and 2-object cir-cles should be least preferred. The more-is-betterbias can not be elicited in Numberbecause the win-ning invariants are not directly available. Alterna-tively, the internal representation of numerical factsin Numbermay elicit a larger-is-better bias: thelarger a number is, the more quickly it may contrib-ute to the sum of 15. If this bias is elicited, thenlarger numbers should be preferred.


9/25


Perceptual Operations

quantity of objects

search shapes

identify circles

Directly Perceived Information

External Representation Internal Representation

Abstract TaskStructure

Perceptual Operations Cognitive Operations Directly Retrieved Information

Lookahead Biases

elements

winning triplets

symmetry categories

winning invariants

numerical values

sum-15 triplets

parity

# in sum-15 triplets

circles

numerical values

add 3 numbers

identify circles

(larger-is-better)

Cognitive Operations Directly Retrieved Information

common shape circlesquantity of objects

quantity of objects

circles

Lookahead Biases

(more-is-better)

Directly Perceived Information

S

H

A

P

E

N

U

M

B

E

R

Learned Knowledge

Learned Knowledge


External Representation Internal Representation


Figure 4. The components of the theoretical framework in Figure 1 are mapped onto the corresponding components in theNumberand Shape isomorphs. See text for details.


10/25


TheNumber and Shape Isomorphsunder the Framework

Based on the above analyses, the Numberand Shapeisomorphs are selected to show how the componentsof the theoretical framework in Figure 1 are mappedonto the corresponding components in these two iso-morphs. The mappings are shown in Figure 4. The

abstract task structure contains the four formal prop-erties of the TTT, which are represented differently inNumberand Shape. For Number, the four formalproperties are represented internally. In addition, theelements are also partially represented externally.The external representation of the elements activatesa perceptual operation (identify circles) and the inter-nal representation of the elements activates directlyretrievable information (numerical values) that elicitsthe larger-is-better bias. In addition, the internal rep-resentation of the winning triplets activates a cogni-tive operation (add 3 numbers). However, the inter-nal representations of symmetry categories and win-ning invariants do not activate any operations or di-

rectly retrievable information.For Shape, the four formal properties are all rep-

resented externally. The external representations ofthe elements and winning triplets activate perceptualoperations (identify circles and search for shapes),and the external representation of the winning invari-ants provides directly perceivable information(winning invariants) that elicits the more-is-betterbias. However, the external representation of thesymmetry categories does not provide directly per-ceivable symmetry information.

Predictions of the Framework and Overview ofthe Experiments

In the theoretical framework, the decision on actionsis based on lookahead, biases, and learned knowl-edge. Because complete mental lookahead is practi-cally impossible for the TTT tasks, the behavior inthe TTT tasks should be only determined by biasesand learned knowledge. Therefore, before learning,the behavior in the TTT tasks should be solely deter-mined by biases; during learning, it should be jointlydetermined by biases and learned knowledge; andafter learning, it should be solely determined bylearned knowledge in the form of a routine task withmemorized and compiled procedures. Specifically,we have the following predictions.

First, if the biases are inconsistent with the task,they can make the task more difficult by misguidingactions away from the goal. This prediction wastested in Experiment 1. Second, if the biases are con-sistent with the task, they can make the task easier byguiding actions towards the goal. This predictionwas tested in Experiment 2. Third, if the biases areirrelevant to the task, they should have no effects onthe decision of actions. This prediction was alsotested in Experiment 2. Because these three predic-

tions are based on the assumption of biases, the na-ture of the biases were examined in both experiments.The more-is-better bias was assumed to be the per-ceptual bias elicited by the external representations inLine, Shape, and Color, and the larger-is-better biaswas assumed to be the cognitive bias elicited by theinternal representation inNumber.

In addition to the above two experiments de-signed to test the specific predictions about biases,Experiment 3 was designed to examine the percep-tion of symmetry information and its effects onproblem solving behavior. In all three experiments,several other properties of the theoretical frameworkwere also examined, such as the difference betweenperceptual and cognitive operations and the differ-ence between sequential and parallel perceptual op-erations.

EXPERIMENT 1:

INCONSISTENT MAPPING

Experiment 1 focuses on the effect of inconsistentbiases on problem solving behavior. In addition, italso examines how the different operations activatedby different representations affect problem solvingbehavior. The four TTT isomorphs in Figure 2 werethe four conditions of this experiment. Because allTTT isomorphs have the same formal structures, forconvenience, we use the three symmetry categoriesofNumber, i.e., five, even numbers, and odd num-bers, to refer to the three symmetry categories of allTTT isomorphs for the rest of this article. For ex-ample, when we talk about even numbers in Shape,we actually refer to the 3-object big circles (see Fig-ure 3).

In this experiment, subjects played games againsta perfect computer opponent. In all conditions, thecomputer opponent always played first by randomlyselecting an even number as its first move. The com-puter's strategy (see the Appendix for details) wascarefully designed such that the computer couldnever lose and in order for the subjects to get draws,they had to strictly follow the 5-Oddstrategy:

Always select five as the first move. Always select any odd number as the second

move. For all other moves, simply block the piece that

can lead to an immediate win for the computer.

The 5-Odd strategy is necessary and sufficientfor subjects to get draws. The first and the secondmoves are crucial: if either or both are made incor-rectly, then subjects always lose, regardless of howother moves are made. Under this strategy, subjectsonly have to make decisions for the first and secondmoves because for all other situations, the subjectsonly have one choiceblocking the piece that canlead to an immediate win for the computer. In addi-


11/25


tion, subjects' first and second moves only depend onthe symmetry categories (i.e., five, even numbers,and odd numbers), not on specific elements.

As analyzed earlier, the winning invariants in theexternal versions of the TTT (Line, Shape, andColor) were assumed to elicit the more-is-better bias.Therefore, the preference order is, from most to least

preferred: center > corners > sides for Line and 4-object > 3-object > 2-object circles for Shape andColor. Since the 5-Odd strategy requires that the firstmove be always five and the second move alwaysany odd number, the more-is-better bias should leadto more correct first moves but meanwhile also moreincorrect second moves in Line, Shape, and Color.Thus, the more-is-better bias is inconsistent with the5-Odd strategy. In the internal version of the TTT(Number), the numerical facts of the nine numberswere assumed to elicit the larger-is-better bias.Therefore, larger numbers should be preferred inNumber. The larger-is-better bias is also inconsistentwith the 5-Odd strategy: it can make the correct se-

lection of the first move (five) more difficult becausethe bias towards larger numbers reduces the chanceof selecting five. However, the larger-is-better bias isirrelevant to the selection of second moves becausethere is no correlation between larger and smallernumbers and even and odd numbers.

Another issue this experiment examines is howthe different operations activated by different repre-sentations affect problem solving behavior. InLinethe winning triplets are represented externally bystraight lines, whereas in Number they are repre-sented internally by sums of numbers. Whether threecircles lie on a straight line can be perceptually in-spected, whereas whether three numbers add to 15has to be mentally computed. These two differentidentification processes for winning triplets may havedifferent effects on problem solving behavior. Dif-ferent external representations can also activate dif-ferent perceptual operations. The winning triplets inShape and Colorare both represented externally, theformer by shapes and the latter by colors. However,the shapes in Shape have to be searched sequentiallywhereas the colors in Colorcan be searched in par-allel. Thus, the difference between these two identi-fication process for winning triplets may also affectproblem solving behavior.

MethodSubjects

80 undergraduate students enrolled in introduc-tory psychology courses at The Ohio State Universityparticipated in the experiment to earn course credit.

StimuliThe four TTT isomorphs in Figure 2 were the

four conditions of this experiment. They were pro-grammed in SuperCard on Macintosh computers.

The four TTT isomorphs were controlled by the sameprogram because they have the same formal structure.The computer always made the first move in allgames. Its strategy was designed such that the sub-jects had to discover the 5-Odd strategy to get draws(see the Appendix). Subjects made moves by click-ing the pieces with a mouse. The pieces selected by

the computer and subjects were in different colors orbackground patterns such that they could be distin-guished.

Design and ProcedureEach subject was randomly assigned to one of

the four conditions. There were 20 subjects for eachcondition. The instructions were given to the subjects verbally. The first part of the instructions wasdifferent for different conditions but the second partwas the same for all conditions. The first and secondparts of the instructions forNumberare as follows:

Part 1. There are nine numbers on the screen. You

and the computer select numbers in turn byclicking the numbers, one at a time. Who-ever first gets any three numbers that exactlyadd to 15 wins the game.

Part 2. Due to the specific design of this game, youcan not beat the computer. So your task isto prevent the computer from winning, thatis, to get draws. The computer always startsfirst. There is a strategy. If you can figure itout, you can always get a draw. You need toplay this game over and over again until youget 10 draws in a row.

If a subject could get 10 draws in a row within50 games, the experiment was over. Otherwise theexperiment was over at the fiftieth game. Completemove sequences and time stamps for all games wererecorded by the computer.

ResultsOverall Performance

Figure 5 shows the percentage of subjects whogot 10 draws in a row within 50 games, the numberof games needed to get 10 draws in a row within 50games, and the number of games needed to get thefirst draw for the four conditions. If a subject couldnot solve a problem within 50 games, the value wasconsidered to be 50 games. In terms of the percent-age of successful subjects, the difficulty order was:Line < Color< ShapeNumber. Except of the dif-

ference between Numberand Shape (2 = 0.00,p =1.0), all other differences were significant (smallest

2 = 4.44, p = 0.04). In terms of the number ofgames to 10 draws, the difficulty order was: Line

Nature of External Representations in Problem Solving

Documents

Transcript of Nature of External Representations in Problem Solving