ANA SUSAC, ANDREJA BUBIC, JURICA KAPONJA, MAJA PLANINICand MARIJAN PALMOVIC

EYE MOVEMENTS REVEAL STUDENTS’ STRATEGIESIN SIMPLE EQUATION SOLVING

ABSTRACT. Equation rearrangement is an important skill required for problem solvingin mathematics and science. Eye movements of 40 university students were recordedwhile they were rearranging simple algebraic equations. The participants also reported ontheir strategies during equation solving in a separate questionnaire. The analysis of thebehavioral and eye tracking data, namely the accuracy, reaction times, and the number offixations, revealed that the participants improved their performance during the time courseof the measurement. The type of equation also had a significant effect on the score. Theresults indicated that the number of fixations represents a reliable and sensitive measurethat can give valuable insights into participants’ flow of attention during equation solving.A correlation between the number of fixations and participants’ efficiency in equationsolving was found, suggesting that the more efficient participants developed adequatestrategies, i.e. “knew where to look.” The comparison of eye movement data andquestionnaire reports was used for assessing the validity of participants’ metacognitiveinsights. The measures derived from eye movement data were found to be more objectiveand reliable than the participants’ reports. These results indicate that the measurement ofeye movements provides insights into otherwise unavailable cognitive processes and maybe used for exploring problem difficulty, student expertise, and metacognitive processes.

KEY WORDS: algebra, equations, expertise, eye tracking, inverse efficiency,metacognition, number of fixations, problem difficulty, strategy

INTRODUCTION

Mathematics is a fundamental school subject that students often perceiveas very difficult and abstract. However, regardless of potential difficultiesthey may face while dealing with this subject, students nevertheless haveto learn many of its basic concepts and acquire mathematical skills that,among others, include equation solving skills. Although students learnequations throughout their education, they nevertheless often havedifficulties with rearranging even the simplest constructions. Studyingtheir strategies and approaches to equation solving is therefore highlyrelevant, as it may allow professionals to better understand students’reasoning logic, strategies, and difficulties encountered while solvingequations and aid teachers in designing programs aimed at facilitatingequation learning.

International Journal of Science and Mathematics Education 2014# National Science Council, Taiwan 2014

Many different approaches have been used for studying equationsolving in students of different ages and proficiency statuses. In recentyears, in addition to behavioral quantitative and qualitative methodolo-gies, neuroimaging and eye tracking have also been used in this field(e.g., Cohors-Fresenborg, Kramer, Pundsack, Sjuts & Sommer, 2010;Jansen, Marriott & Yelland, 2007; Landy, Jones & Goldstone, 2008; Qin,Carter, Silk, Stenger, Fissell, Goode & Anderson, 2004; Sohn, Goode,Koedinger, Stenger, Fissell, Carter & Anderson, 2004). Among these, eyetracking represents a compelling methodological solution as it providesvaluable insights related to visual attention, especially when they are usedto complement tasks that involve a simple motor response (usually buttonpress) or participants’ descriptions of what they are attending to.Specifically, eye tracking, namely the measurement of eye movementsrelative to the head and the visual stimulus, provides a fast and reliablemeasure of some important aspects of visual attention. Eye trackersmeasure spatial and temporal features of eye movements and provideinformation on eye fixations and saccades. Among these, fixationsrepresent the maintaining of the visual gaze on a certain location in thevisual field, while fast eye movement from one location to another arenamed saccades. The resulting sequence of fixations and saccades iscalled a scan path.

Eye tracking has thus far been applied in various fields and for diversepurposes. First of all, in the widest sense, measurements of eyemovements have been used for providing a general insight into cognitiveprocesses involved in performing different tasks. For example, differentstages of problem solving have been identified using eye movements,suggesting a sequence of cognitive processes that starts with theformation of an internal problem representation and is followed byplanning processes (Epelboim & Suppes, 2001; Nitschke, Ruh, Kappler,Stahl & Kaller, 2012). Some studies have shown that perceptualproperties can guide attention and eye movements in ways that assist indeveloping the problem-solving insights which lead to improvedreasoning (Grant & Spivey, 2003; Thomas & Lleras, 2007). With respectto the more applied scientific fields, a very wide area of eye trackingapplication includes website design and advertising, in which themeasures of the first fixation, or the number and the duration offixations and saccades, provide information about the efficiency ofinformation transfer through the website or the advertisement.

Although eye tracking has been utilized for more than a century, in thepast it was only rarely used in educational studies (e.g. Hegarty, 1992;Hegarty, Mayer & Green, 1992). Only recently, it was applied more

widely to various problems in different disciplines, such as mathematics(Chesney, McNeil, Brockmole & Kelley, 2013; Merkley & Ansari, 2010;Moeller, Klein, Nuerk & Willmes, 2011; Schneider, Maruyama, Dehaene& Sigman, 2012), physics (Madsen, Larson, Loschky & Rebello, 2012;Madsen, Rouinfar, Larson, Loschky & Rebello, 2013; Smith, Mestre &Ross, 2010), chemistry (Tang & Pienta, 2012; Williamson, Hegarty,Deslongchamps, Williamson & Shultz, 2013), biology (Cook, Wiebe &Carter, 2008; Patrick, Carter & Wiebe, 2005), and geology (Maltese,Balliet & Riggs, 2013). A number of studies have used eye tracking toinvestigate student perception and interpretation of visual representationsof different scientific concepts like DNA replication (Patrick et al., 2005).Eye tracking is also used for exploring problem solving (Liu & Shen,2011) and problem difficulty (Tang & Pienta, 2012). Measurement of eyemovements provides much richer information than the conventionalassessment methods (e.g. written examination) and can be very useful forestimating the test item difficulty. It also gives new insights into students’strategies used for solving multiple-choice science problems, showingthat visual attention plays an important role in successful problem solving(Tai, Loehr & Brigham, 2006; Tsai, Hou, Lai, Liu & Yang, 2012).Furthermore, in educational research, eye tracking is used to studymultimedia learning to determine the effects of verbal explanationscombined with the standard text information (Liu, Lai & Chuang, 2011),to explore the use of animations and simulations in science learning (She& Chen, 2009), etc. These initial experiences indicate that eye trackingrepresents a very promising technique that offers various possibilities forstudying multimedia learning and instruction (van Gog & Scheiter, 2010).However, since there are still many technical, practical, ethical andmethodological challenges related to the use of eye tracking for studyinglearner–computer interactions (San Diego, Aczel, Hodgson & Scanlon,2012), the need to complement the eye tracking data with otherbehavioral measures of learning has been emphasized (Hyönä, 2010).Nevertheless, its features continue to be constantly refined and developed,and its benefits more recognized, as this method has the potential to gobeyond questions regarding what and when works in instruction, andaddress the question of how a particular instructional method causeslearning (Mayer, 2010).

Reading skills are important for all school subjects including scienceand mathematics. Despite the relevance of science texts that serve ascrucial tools in learning science concepts, most research traditionallyfocused only on the outcomes of text reading, and not on the textprocessing itself. An insight into the text processing aspect may be gained

EYE MOVEMENTS AND EQUATION SOLVING

using the eye tracking method that provides a quantitative and objectivemeasure of the flow of visual attention while reading a science text.Importantly, the effects of different aspects of reading texts from sciencetextbooks can be evaluated using eye tracking. For example, a recentstudy has shown the effect of the text structure in learning from a sciencetext (Ariasi & Mason, 2011). In addition, it was found that students whointegrated text and picture more often during learning from an illustratedscience text had higher learning performance (Mason, Tornatora &Pluchino, 2013b). Another study revealed a significant effect of picturelabeling on science text processing, suggesting that the integration of textand picture can be facilitated by the appropriate visual signaling (Mason,Pluchino & Tornatora, 2013a). The role of graphical overviews intextbook comprehension was also examined (Salmerón, Baccino, Cañas,Madrid & Fajardo, 2009). These results showed that the effect ofgraphical overview depended on the text difficulty, and the order ofreading the graph and the associated overview. This line of research mayhave practical implications and provide concrete guidelines regardingways of improving text and graphics comprehension.

First studies which used the eye tracking method in a more specificmathematical education context investigated reading while solving wordproblems in arithmetic (de Corte, Verschaffel & Pauwels, 1990; Terry,1992). Later, researchers utilized eye movement data in a variety ofcontexts, including number processing (Merkley & Ansari, 2010;Moeller, Fischer, Nuerk & Willmes, 2009), performing arithmeticoperation (e.g. Moeller et al., 2011; Schneider et al, 2012), andunderstanding math equivalence (Chesney et al. 2013). However, eyetracking was rarely used to study more complex, algebraic concepts(Anderson & Gluck, 2001). The present study is grounded in this field, asit aims to track how students solve simple algebraic equations. Studentsencounter equations throughout their mathematics education and use theirskills related to equation rearrangement not only in mathematics but alsoin, e.g., physics and chemistry. Nevertheless, despite these frequentencounters with equations problems, many students still have difficultieswith the rearrangement of even the simplest equations. Our previousstudy has shown that 15-year-old students often use concrete strategiesto rearrange an equation, such as inserting numbers into equations,whereas experts recognize patterns and show flexibility in choosing theappropriate strategy. The present study used eye tracking to furtherinvestigate students’ strategies in simple equation solving. We wantedto explore what kind of information eye tracking can provide inaddition to accuracy and reaction times (RTs). Our aim was to relate the

eye movement measures with participants’ efficiency and the use ofdifferent strategies during equation solving. Furthermore, we wanted tocombine the insights gained using eye movements with the participants’verbal responses based on introspective judgments. We attempted todetermine the equation difficulty and the repetition effects from the eyemovement parameters. Experts were expected to show a distinct pattern ofeye movements.

THEORETICAL BACKGROUND

Most educational studies rely on written or oral examinations, neither ofwhich gives direct information about the complexity of cognitiveprocesses involved in the task performance. In written exams, itemdifficulty that is based solely on accuracy of the answers on a givenproblem represents an indirect measure of cognitive demand. Sometimes,the time required to solve the problem is also measured, thus giving anadditional insight into the problem difficulty. On the other side,qualitative interviews provide more focused in-depth information, butmay suffer because of the subjective nature of the reported data. Anothermethod that is frequently used represents the so-called think-aloudmethod that involves participants thinking aloud as they are solving aproblem. Although it offers a very valuable window into the cognitiveprocesses during problem solving, it also affects the processing itself. Onthe contrary, eye tracking gives a somewhat more objective, measurableinsight into cognitive and attentional processes involved in problemsolving without influencing it.

An important area in which eye tracking methodology has thus farprovided numerous valuable insights is the field of expert performance.Many studies reported that experts have a smaller number of fixationsthan non-experts in various field of expertise. For example, experts inchess make fewer fixations per trial than less-skilled players (Reingold,Charness, Pomplun & Stampe, 2001). Smaller number of fixations is alsotypical for proficient readers when compared with beginners, which isusually expressed as a higher probability of a word being skipped(Brysbaert & Vitu, 1998; Drieghe, Brysbaert, Desmet & De Baecke,2004; Inhoff & Radach, 1998). However, experts sometimes have morefixations in the areas relevant to the task. The meta-analysis of eyetracking research in professional domains revealed that experts, whencompared with non-experts, had shorter fixation durations, more fixationson task-relevant areas, and fewer fixations on task-redundant areas

EYE MOVEMENTS AND EQUATION SOLVING

(Gegenfurtner, Lehtinen & Säljö, 2011). Similarly, Madsen et al. (2012)reported that the participants who correctly answered introductory physicsproblems spent a higher percentage of time looking at the relevant areasof the diagram. These results indicate that the experts direct their attentionto the problem-relevant areas, i.e. they “know where to look.”

Eye tracking data can also provide insights into the problemsolving steps that experts go through during problem solving.Knowing how experts solve problems is valuable for improving theteaching methods. Knoblich, Ohlsson & Raney (2001) suggested thata change of an initially misleading representation in insight problemsolving leads to the allocation of visual attention. Grant & Spivey(2003) argued that attention is not just an outcome, but a process thatcan facilitate problem solving itself. They found that perceptuallyhighlighting the critical diagram component significantly increased thefrequency of correct solutions. Furthermore, Cohors-Fresenborg et al.(2010) emphasize the important role of metacognitive monitoring,namely individuals’ awareness of their own strategies and efficiency,for succeeding in school algebra. This metacognitive monitoring isprobably related to the experts’ guiding their top-down attention tothe problem-relevant areas.

In addition to the insights eye tracking may provide in the field of theexpert problem solving and metacognition, this method is also very usefulfor exploring the dynamics of problem solving, problem difficulty, andmetacognition among non-experts. Although eye tracking measures, suchas the number or duration of fixations, reflect more than pure attentionalprocesses, they nevertheless may be regarded as indirect measures ofattention (Duchowski, 2007). Attention is closely related to othercognitive processes that take place during problem solving. Consequently,eye tracking could be used as an indirect measure of cognitive load, andthus, problem difficulty. Of course, detailed research is needed toestablish a connection between eye tracking measures and problemdifficulty. Furthermore, other measures of problem difficulty should beused complementary. Similarly, given that eye tracking provides anobjective measure of behavior, it may be a useful technique for assessingindividuals’ metacognitive activities. As already mentioned,metacognitive monitoring is crucial for mathematics achievement(Cohors-Fresenborg et al., 2010). Metacognitive skills have beenshown to improve the quality of student learning, but it has also beenshown that many students lack them. Furthermore, it has also beenused as a tool for enhancing metacognitive processes in computer-based learning environments (van Gog & Jarodzka, 2013).

METHODS

Participants

Forty paid volunteers (24 females, age 23±3 years), with normal orcorrected-to-normal vision, participated in the study. All participants wereundergraduates at the University of Zagreb. Twenty-two participants werestudying mathematics, science, and engineering, while the remaining 18students were majoring in fields not connected to mathematics, such aslanguages, law, and kinesiology. Each participant gave an informedwritten consent before taking part in the experiment.

Apparatus

Eye movement data were recorded using a stationary eye tracking systemwith a temporal resolution of 500 Hz and a spatial resolution of 0.25–0.50° (SMI iView Hi-Speed system, Senso Motoric InstrumentsG.m.b.H.). The distance between the eyes and the monitor was 50 cm.Prior to every recording, the gaze of each participant was calibrated with a13-point calibration algorithm. The gaze direction was calculated as avector between corneal reflection (which is stable, i.e. it depends only onhead movements) and pupil position (i.e. the calculated center of thepupil). Microsaccades were automatically grouped in a fixation. Thefixations were detected automatically using the “Event Detected Method,”which is built into the eye tracking device. Blinks were correctedautomatically.

Materials

In each trial, simple equations consisting of three elements (numbers orletters) were presented in the center of the visual field. The presentednumbers and letters were black, displayed in 24 pt size Ariel font on thewhite background. Simultaneously, a potentially correct or incorrectanswer was presented below the equation. Participants’ task was to makex subject of the equation. They indicated their response by looking to theleft of the screen if the suggested answer was true (DA; Croatian word for“yes”) or to the right if the answer was false (NE; Croatian word for“no”). An example of an equation with the offered answer used in thestudy overlaid with the scan path of one participant is presented in Fig. 1.

During the experiment, three types of equations were used:

A equations: x⋅a= b,B equations: x

a ¼ b ,

EYE MOVEMENTS AND EQUATION SOLVING

C equations: ax ¼ b .

The offered answers were of the following types: x=a⋅b, x ¼ ab , and x ¼ b

a .

Within all presented equations, a and b stand for different letters and numberswhich all appeared with the same probability during the experiment.

Procedure

In the beginning of the study, participants were familiarized with the taskand procedure. After the preparation for eye tracking recording andcalibration, they were presented with a training block that consisted of sixequations equivalent to those used in subsequent experimental trials.

During the experiment, participants were presented with the threepreviously described types of equations, which were randomized acrossfour blocks. Each block consisted of 15 equations of each equation type,amounting to an overall of 45 presented equations per block. Two blockscontained equations with numbers, while the other two blocks containedequations with letters (symbols). The order of presentation of blockscontaining number and letter equations was counterbalanced acrosssubjects. In the present study, only equations with letters were analyzed.During all experimental trials, no feedback was given to the participants.

After the eye tracking recording, the participants were asked tocomplete a questionnaire designed for assessing their strategies duringequation solving. They were asked to recall whether they looked at theoffered answer before or during equation solving, or if they firstindependently solved the equation, and then checked the accuracy ofthe offered answer. Participants were also asked to rank different equationtypes by difficulty. Questionnaire data are missing for two participants.

Figure 1. Example of an equation used in the study. The offered answer is presentedbelow the equation, and the two potential answers (true or false) are shown on the left andright side of the display. Scan path of a participant is overlaid. Centers of circles representpositions of fixations and lines represent saccades. Radius of the circle illustrates theduration of the fixation. The numbers in circles mark the numeric order of fixations.

The whole experiment, including preparation, eye movement calibration,recording, and completion of the questionnaire, lasted around 30 min.

Data Analysis

Recorded eye movements were analyzed using BeGaze software. First,we visually inspected scan paths which show participants’ eye fixations,and the order in which they occurred (Fig. 1). The motivation for thevisual inspection of scan paths was threefold. First, we were interested invisually checking the quality of the recorded data. Second, given that theparticipants indicated their responses using eye movements, responseaccuracy was also determined by visual inspection. In this process, thefinal location of participants’ eye gaze was recorded as their response(true or false). Finally, this procedure allowed us to inspect whetherparticipants were looking at the offered answer while solving equations.

In order to determine the time needed to solve the task (RT) which wasdefined as the duration of the gaze to equation and answer, an area ofinterest (AOI) was defined as a rectangle including both the equation andthe offered answer (dashed rectangle in Fig. 1). Dwell time (time spentlooking within the area of interest) was evaluated and recorded as areaction time measure for each trial. In addition, the number of fixationson the defined AOI containing the equation and the offered answer wasrecorded, and evaluated for each trial. Finally, inverse efficiency, namelya composite measure of participants’ efficacy in solving different equationtypes that takes into account both their response accuracy and speed, wascalculated as a ratio between their reaction times and accuracy (Townsend& Ashby, 1978). Lower values on this measure indicate better efficiencyon a particular task. Inverse efficiency is used to account for the speed–accuracy tradeoffs. Within the analysis of the eye movement data,reaction times, and number of fixations, outliers defined as results outsidethe M±3z range were not included in the analysis. Furthermore, onlycorrect responses were included in the analysis of reaction times and thenumber of fixations.

To determine the effects of equation type and repetitions, a two-wayrepeated-measures analysis of variance (ANOVA) on accuracy, reactiontimes, and the number of fixations was conducted. Repeated-measurespost hoc tests using a least significant difference adjustment were appliedto further assess differences between equation types and blocks. Inaddition, the Pearson correlation coefficient, one-way ANOVA, andMcNemar’s chi-square test were used for investigating participants’metacognitive insights and the relationship between eye movements and

EYE MOVEMENTS AND EQUATION SOLVING

expertise. A threshold of pG0.05 was used for determining the level ofeffect significance within all conducted tests.

RESULTS AND DISCUSSION

Repetition and Equation Type Effects

In order to test the differences between participants’ accuracy in solvingdifferent equation types during two presentation blocks, a two-wayrepeated-measures ANOVA was conducted. The obtained results indicat-ed a statistically significant main effect of equation type (F(1,39)=10.10,pG0.01) and block (F(2,78)=4.59, pG0.05), while the interaction effectwas not significant (F(2,78)=1.90, p90.05). Participants were moreaccurate on the A and B types of equation than on the C equations (pG0.05). Accuracy increased in block 2 compared to block 1 (pG0.05).Figure 2 shows the percentage of correct responses for different equationtypes, separated for block 1 and block 2.

A two-way repeated-measures ANOVA used for testing the differencesbetween participants’ RTs across different equation types and presenta-tion blocks revealed a statistically significant main effect of both factors:equation type (F(1,38)=37.85, pG0.001) and block (F(2,76)=28.33, pG0.001), as well as the interaction effect (F(2,76)=7.79, pG0.01).Generally, participants had longer RTs for the equations C than for theA and B equations (pG0.05), and their RTs were shorter in block 2 thanin block 1 (pG0.05). From block 1 to block 2, the biggest reduction in

Figure 2. Accuracy (percentage of correct responses) for the equation types A, B, andC, separated for block 1 and block 2. Error bars represent 1 SEM

RTs was observed for the equations C. RTs needed to solve differenttypes of equations in block 1 and block 2 are presented in Fig. 3.

The results obtained while comparing the number of fixations duringsolving different equation types within the two presentation blocksshowed a significant main effect of equation type (F(1,38)=52.31, pG0.001) and block (F(2,76)=21.04, pG0.001), as well as the interactioneffect (F(2,76)=5.46, pG0.01). Generally, the number of fixations waslargest for the equations C, followed by the equations A and then theequations B (pG0.05). In addition, participants’ number of fixations waslarger in block 1 compared to block 2 (pG0.05). From block 1 to block 2,the biggest reduction in the number of fixations was observed for theequations C. Figure 4 shows the number of fixations during solving of thethree different types of equations in block 1 and block 2.

Overall, the obtained data show that our participants were successful inequation rearrangement. On average, their accuracy amounted to 95 %which probably does not reflect the abilities of the general studentpopulation because our participants volunteered for the study knowingthat they would be required to solve equations. Such high level ofaccuracy could have posed a problem if our goal was to determine theability of the general student population to rearrange equations. As wewanted to explore successful students’ strategies in equation rearrange-ments, high accuracy was desirable in our study. In addition, highaccuracy was expected because we explored how students solved a taskoften encountered during their schooling that they should haveoverlearned before college. However, it is still interesting that ourparticipants were majoring in very different fields, some of which were

Figure 3. Reaction times for the equation types A, B, and C, separated for block 1 andblock 2. Error bars represent 1 SEM

EYE MOVEMENTS AND EQUATION SOLVING

not in any way connected to mathematics (e.g. law). Therefore, it isencouraging that they were still able to rearrange the equations whichsuggests that they acquired that skill earlier, during their primary andsecondary school. However, the C equations (a/x=b) already posed aproblem for some participants. This was expected because the Cequations are usually solved in two steps while only one step is neededfor the A and B equations. In our earlier study, we found that the mainwrong strategy used for solving the C equations was “if there is divisionand ‘a’ is brought on the other side of the equation, the result has to be aproduct.”

Nevertheless, the lowest accuracy and the longest reaction times for theC equations suggest that this was the most difficult type, whereas the Aand B equations were not statistically different in terms of accuracy andRTs. The number of fixations was largest for the C equations indicating apositive correlation between the equation difficulty and the number offixations. Taking that into account, a larger number of fixations for theequations A than the equations B suggests that the equations A were moredifficult. This result is corroborated by our previous study with anothergroup of participants who had lower accuracy and longer RTs for theequations A compared to the equations B. This indicates that the numberof fixations represents a reliable measure of task difficulty that can beeven more sensitive than RTs.

As expected, the accuracy increased and the RTs decreased during thetime course of the measurement. This indicates that practice improvesperformance even if the participants know how to solve the equations.

Figure 4. Number of fixations for the equation types A, B, and C, separated for block 1and block 2. Error bars represent 1 SEM

The number of fixations was also reduced during the measurement,suggesting that participants developed more efficient strategies, i.e. theyknew where to look for each equation type. The biggest reduction in thenumber of fixations was observed for the C equations, once againimplying their larger difficulty as participants needed more time todevelop efficient strategies for solving them.

Assessing Students’ Strategies During Equation Solving

First, student equation solving strategies were investigated by analyzingthe eye movement data. Although eye gaze does not represent a directmeasure of strategic processes nor can it be equated to attention, scan pathanalysis nevertheless gave us valuable insights into participants’ flow ofattention during the equation solving. Figure 5a shows a scan path of oneparticipant for one equation. The scan path reveals that the participant waslooking first at the equation, then to the offered solution and then back tothe equation again. At the end of the trial, she made a decision regardingthe correctness of the offered response and gave the response accordingly.Figure 5b shows a scan path of the same participant for another equation.This scan path suggests that the participant first solved the equation, andonly then looked at the offered solution, decided if it was correct and gavethe response accordingly. Scan paths revealed that most of theparticipants most of the times checked the offered answer before orduring equation solving.

In addition, student strategies for solving equations were assessed witha separate questionnaire in which participants reported whether theylooked at the offered answer before or during the equation solving, or ifthey first independently solved the equation, and then checked theaccuracy of the answer. While answering this question, 15 participantsanswered “I always independently solved the equation, and then checkedthe offered answer.”; 9 participants answered “I always checked theoffered answer before or during the equation solving.”; and 14participants answered “Sometimes I checked the offered answer, and

Figure 5. Scan paths of one participant while solving two equations. The circles markthe fixations in numeric order. Duration of the fixation is represented by the size of circle.In the questionnaire completed after the experiment, the participant stated that she did notuse the offered answer during equation solving

EYE MOVEMENTS AND EQUATION SOLVING

sometimes I independently solved the equation.” Interviews conductedduring our previous study suggested that experts usually took intoaccount the offered answer which helped them to solve equations fasterby recognizing the relevant patterns.

Although participants’ verbal report may seem as the potentially mostrelevant and valid source of information regarding their equationrearrangement strategies, a more careful analysis reveals that eyemovements provided valuable information on some aspects of strategyuse that otherwise would not have been available. For example, using eyemovements, we were able to directly monitor whether the participantswere using the offered answer during the equation solving instead ofrelying on their recalled reports. Therefore, only eye tracking gave us anobjective and quantitative measure of how often the participants werelooking at the offered answer. Having these two independent sources ofinformation allowed us to later compare the objective eye trackingmeasure with participants’ reports from questionnaires. As will be seen inthe next section, this comparison of two information sources providedvery informative, useful, and somewhat surprising findings.

Assessing the Validity of Students’ Metacognitive Insights

The nature and validity of some aspects of participants’ metacognitiveinsights was first explored by associating and comparing the obtainedquestionnaire reports with the eye movement data gathered duringequation solving. We focused on this issue because previous studieshave emphasized the importance of different metacognitive activities forsuccess in mathematical problem solving (Cohors-Fresenborg et al., 2010;Kramarski & Mevarech, 2003). Specifically, we investigated the scanpaths of participants who stated that they did not check the offeredanswers during equation solving in the questionnaire. In contrast to theirverbal reports, scan paths revealed that participants often checked theoffered answers. Specifically, the data from 15 participants who reportedsolving equations independently was analyzed in order to check howmany times they returned from the offered answers to the equations(Table 1). Although the expected frequency of returns associated withindependent equation solving would have been 0, the analysis revealedthat overall these participants returned from the offered answers toequations in 51.5 % of all trials. This indicates that their metacognitiveability, or the ability to introspectively evaluate and report the utilizedstrategies of equation solving, was far from ideal. However, it has to beemphasized that there was substantial variability among participants

which indicates that they differed with respect to the validity of theirmetacognitive insight. It may be suggested that the participants whochecked the offered answers more frequently were making a morepronounced error in monitoring their behavior than those who only rarelyturned their eyes towards the answers. Thus, differences in the frequencyof answer checking may provide a rough indicator of participants’metacognitive accuracy. Consequently, it may be speculated that theparticipants who looked at the offered answers more frequently had lowermetacognitive accuracy when they claimed they were not looking at them.To quantify the relationship between metacognitive accuracy andefficiency in equation rearrangement, we calculated the correlationbetween frequency of checking the answer (Table 1) and the inverseefficiency using the Pearson correlation coefficient. The obtained positivecorrelation between these two measures indicates that the participantswith higher metacognitive accuracy were more efficient in equationsolving (r(13)=0.52, pG0.05).

The validity of participants’ metacognitive insight was further exploredby bringing together the questionnaire reports, participants’ accuracy, andthe scan path data gathered during the equation solving. Specifically,

TABLE 1

Frequency of checking the answer before solving the equation among participants whoreported independently solving the equations without consulting the solutions

ParticipantA typex·a=b

B typex/a=b

C typea/x=b All types (%)

P1 26 23 29 87P2 8 3 6 19P3 5 2 7 16P4 29 28 27 93P5 9 5 6 22P6 30 26 30 96P7 14 3 8 28P8 12 3 8 26P9 30 30 30 100P10 25 18 17 67P11 29 29 28 96P12 20 12 11 48P13 10 6 7 26P14 12 2 12 29P15 9 3 5 19

EYE MOVEMENTS AND EQUATION SOLVING

since each participant was asked to rate and rank the difficulty of eachequation type, we investigated whether these reports served as reliablepredictors of the calculated inverse efficiency, a composite measure thatbest approximates equation difficulty. For each participant, we comparedthe equation ranks determined by their inverse efficiency, questionnairereports, as well as the number of fixations. The obtained results indicatedthat the number of fixations, i.e. the eye movement data, significantlybetter (χ2 (1)=3.86, pG0.05) predicted the objective measure of equationdifficulty derived from inverse efficiency (correct for 25 out of 38participants) when compared to participants’ answers (correct for 16 outof 38 participants).

The order of equations ranked by difficulty obtained from participants’inverse efficiency, questionnaire reports, and the number of fixations isshown in Fig. 6. The leading rank order (from the least difficult to themost difficult) derived from inverse efficiency was B A C suggesting thatB equations were easier than A equations. The same ranking combinationwas most frequent for the number of fixation, thus indicating that thismeasure is a good predictor of equation difficulty. However, leading rankorder obtained from participants’ questionnaire responses was A B C,indicating their assurance that A equations were easier than B equations.We obtained the same ranking in the previous study for non-experts.Many participants believe that A equations are easier because theycontain multiplication, while B equations include a division that isconsidered more difficult than multiplication. However, they overlook thefact that the result in A equations is a quotient, and that a dividend and adivisor cannot be exchanged, whereas in B equation the result is a product

Figure 6. Frequency of different ranking combinations (from the least difficult to themost difficult question) determined from inverse efficiency, participants’ responses, andthe number of fixations

and the order of factors is not important. Given this, in cases in whichparticipants used the offered response during equations solving, it waseasier (and faster) to recognize the correct response for B equations.Nevertheless, most participants responded that C equations were the mostdifficult, which is in agreement with the inverse efficiency and thenumber of fixations.

Overall, some of our participants had rather poor metacognitive insightinto their strategies and the equation difficulty. Eye movement data havetherefore proven to be a more objective and a more reliable measure thanparticipants’ reports. This finding is in accordance with previous studiesthat have indicated low correspondence between participants’ question-naire reports regarding own metacognitive skills and their behavioralperformance (Veenman, Prins & Verheij, 2003; Veenman, Van Hout-Wolters & Afflerbach, 2006).

Eye Movements and Expertise

In order to explore the efficiency of equation solving in more depth,a composite measure that combines participants’ accuracy andresponse times, i.e. inverse efficiency, was used. As mentionedbefore, while investigating student equation solving strategy, thismeasure showed an association with eye movement data, namely thenumber of fixations. This relationship was further explored bycalculating the correlation between inverse efficiency and the numberof fixations using the Pearson correlation coefficient (Fig. 7). Theobtained results indicate a significant positive correlation betweenthese two measures (r(38)=0.83, pG0.01).

Figure 7. The correlation between inverse efficiency and the number of fixations duringequation solving

EYE MOVEMENTS AND EQUATION SOLVING

A somewhat different insight into the effects of expertise in solvingdifferent types of equations was gained by dividing participants intogroups based on their overall efficiency in solving different equationtypes. Specifically, the participants were divided into three equal groupsthat were then compared with respect to their equation solving strategiesidentified using scan paths. In a sense, it may be suggested that thesegroups differ with respect to their equation rearrangement expertise,although this division is only relative given the overall high accuracy ofall participants. A one-way ANOVA was used in order to test thedifferences in the frequency of returns from the offered answer to theequation among participants from three groups. The obtained resultsindicate statistically significant differences between the groups (F(2,37)=9.14, pG0.01), such that the group with the lowest efficiency showed ahigher number of returns from the answers to the equations duringequation solving when compared to the group with the highest efficiency(pG0.05).

These results indicate that experts have fewer fixations than non-experts, in agreement with some previous eye tracking studies onexpertise (e.g. Reingold et al., 2001). However, for some tasks expertshave been reported to have more fixations than novices in the task-relevant areas (Madsen et al., 2012). This pattern of the reported results isnot as contradictory as it may seem at first sight. A more careful analysisreveals that the strategy used for the equation solving and the need for itsoptimization depends on the task difficulty. When presented with adifficult task, the experts have an enhanced information uptake, but theirsearch is optimally directed to relevant areas because of their previousexperience. The situation is the opposite for easy tasks. The task used inthe present study was easy and repetitive, so the experts found an efficientstrategy rather quickly and later did not have the need to explore anyfurther.

CONCLUSION

Student ability to rearrange equations is an important skill required forproblem solving in many disciplines. In this study, we have used eyetracking to investigate the strategies of university students who were ableto solve simple algebraic equations. In particular, we wanted to explorethe possibilities offered by this methodology and combine them with thestandard behavioral measures. The data on accuracy and RTs have shownthat the participants improved their performance during the time course of

the measurement. Accuracy and RTs have also shown that the C type ofequations used in this study was the most difficult, which can beexplained by its composition and the operations required for its successfulrearrangement. Corresponding effects of training and equation type werefound with respect to the number of fixations, indicating the reliability ofthis measure. Moreover, it seems that the number of fixations can be aneven more sensitive measure than accuracy and RTs, given that this indexseparated the A and B equation types by difficulty which the lattermeasures could not do. These results demonstrate that eye tracking offersa very valuable way for measuring some aspects of students’ attentionalprocessing during equation solving.

Measurements of eye movements also provided means forassessing students’ strategies during the equation solving. Inparticular, scan path analysis gave us an objective measure of thefrequency of participants’ checking the offered solution during theequation rearrangement. The comparison of these data with partic-ipants’ questionnaire reports provided us with an opportunity tovalidate students’ metacognitive insight. Our results suggest thatsome participants were not really aware whether they were lookingat the offered answer during equation solving. Similarly, someparticipants had rather poor insight into the difficulty of differentequation types. The eye movement data (the number of fixations)better predicted the equation difficulty derived from the inverseefficiency than participants’ reports. The obtained results indicatethat the participants’ reports do not represent a very objective andreliable measure, and that the eye movement data might be a veryuseful complementary source of information on students’ strategiesand problem difficulty.

Finally, we attempted to relate eye tracking parameters with thelevel of expertise since a different pattern of eye movements forexperts and non-experts was expected. Indeed, we found a positivecorrelation between the number of fixations and the inverse efficiency,a composite measure that combines participants’ accuracy and responsetimes. The results suggest that the non-experts (with larger inverseefficiency) have more fixations than the experts. This probably reflectsthe fact that the experts developed an efficient strategy and that they“knew where to look.” Our findings indicate the adequacy of usingeye tracking in expertise research. Overall, our study has shown thatthe measurement of eye movements provides valuable insights intocognitive processes that otherwise would not be available, thusindicating novel possibilities of its use in educational studies.

EYE MOVEMENTS AND EQUATION SOLVING

ACKNOWLEDGMENTS

This research is a part of the project Investigation of students' strategies insolving simple algebraic equations funded by the University of Zagrebdevelopment fund.

REFERENCES

Anderson, J. R. & Gluck, K. (2001). What role do cognitive architectures play inintelligent tutoring systems? In V. Klahr & S. M. Carver (Eds.), Cognition &instruction: Twenty-five years of progress (pp. 227–262). Mahwah, NJ: LawrenceErlbaum Associates.

Ariasi, N. & Mason, L. (2011). Uncovering the effect of text structure in learning from ascience text: An eye-tracking study. Instructional Science, 39, 581–601.

Brysbaert, M. & Vitu, F. (1998). Word skipping: Implications for theories of eyemovement control in reading. In G. Underwood (Ed.), Eye guidance in reading andsceen perception (pp. 125–148). Oxford: Elsevier.

Chesney, D. L., McNeil, N. M., Brockmole, J. R. & Kelley, K. (2013). An eye forrelations: eye-tracking indicates long-term negative effects of operational thinking onunderstanding of math equivalence. Memory & Cognition, 41, 1079–1095.

Cohors-Fresenborg, E., Kramer, S., Pundsack, F., Sjuts, J. & Sommer, N. (2010). The roleof metacognitive monitoring in explaining differences in mathematics achievement.ZDM Mathematics Education, 42, 231–244.

Cook, M., Wiebe, E. N. & Carter, G. (2008). The influence of prior knowledge onviewing and interpreting graphics with macroscopic and molecular representations.Science Education, 92, 848–867.

de Corte, E., Verschaffel, L. & Pauwels, A. (1990). Influence of the semantic structure ofword problems on second graders’ eye movements. Journal of Educational Psychology,82, 359–365.

Drieghe, D., Brysbaert, M., Desmet, T. & De Baecke, C. (2004). Word skipping inreading: On the interplay of linguistic and visual factors. European Journal of CognitivePsychology, 16, 79–103.

Duchowski, A. (2007). Eye tracking methodology: theory and practice. London: Springer.Epelboim, J. & Suppes, P. (2001). A model of eye movements and visual workingmemory during problem solving in geometry. Vision Research, 41, 1561–1574.

Gegenfurtner, A., Lehtinen, E. & Säljö, R. (2011). Expertise differences in thecomprehension of visualizations: A meta-analysis of eye-tracking research in profes-sional domains. Educational Psychology Review, 23, 523–552.

Grant, E. R. & Spivey, M. J. (2003). Eye movements and problem solving: Guidingattention guides thought. Psychological Science, 14, 462–466.

Hegarty, M. (1992). Mental animation: Inferring motion from static diagrams ofmechanical systems. Journal of Experimental Psychology: Learning, Memory, andCognition, 18, 1084–1102.

Hegarty, M., Mayer, R. E. & Green, C. E. (1992). Comprehension of arithmetic wordproblems: Evidence from students’ eye fixations. Journal of Educational Psychology,84, 76–84.

Hyönä, J. (2010). The use of eye movements in the study of multimedia learning.Learning and Instruction, 20, 172–176.

Inhoff, A. W. & Radach, R. (1998). Definition and computation of oculomotor measuresin the study of cognitive processes. In G. Underwood (Ed.), Eye guidance in readingand sceen perception (pp. 29–54). Oxford: Elsevier.

Jansen, A. R., Marriott, K. & Yelland, G. W. (2007). Parsing of algebraic expressions byexperienced users of mathematics. European Journal of Cognitive Psychology, 19, 286–320.

Knoblich, G., Ohlsson, S. & Raney, E. G. (2001). An eye movement study of insightproblem solving. Memory & Cognition, 29, 1000–1009.

Kramarski, B. & Mevarech, Z. R. (2003). Enhancing mathematical reasoning in theclassroom: The effects of cooperative learning and metacognitive training. AmericanEducational Research Journal, 40, 281–310.

Landy, D., Jones, M. N., & Goldstone, R. L. (2008). How the appearance of an operatoraffects its mathematical precedence. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.),Proceedings of the 30th Annual Conference of the Cognitive Science Society.Washington DC, USA (pp. 2109–2114). Austin: Cognitive Science Society.

Liu, C. J. & Shen, M. H. (2011). The influence of different representations on solvingconcentration problems at elementary school. Journal of Science Education andTechnology, 20, 621–629.

Liu, H. C., Lai, M. L. & Chuang, H. H. (2011). Using eye-tracking technology toinvestigate the redundant effect of multimedia web pages on viewers’ cognitiveprocesses. Computers in Human Behavior, 27, 2410–2417.

Madsen, A. M., Larson, A. M., Loschky, L. C. & Rebello, N. S. (2012). Differences invisual attention between those who correctly and incorrectly answer physics problems.Physical Review Special Topics-Physics Education Research, 8, 010122.

Madsen, A. M., Rouinfar, A., Larson, A.M., Loschky, L. C. & Rebello, N.S. (2013). Canshort duration visual cues influence students’ reasoning and eye movements in physicsproblems? Physical Review Special Topics - Physics Education Research, 9, 020104-1–020104-16

Maltese, A. V., Balliet, R. N. & Riggs, E. M. (2013). Through their eyes: Tracking thegaze of students in a geology field course. Journal of Geoscience Education, 61, 81–88.

Mason, L., Pluchino, P. & Tornatora, M. C. (2013a). Effects of picture labeling on sciencetext processing and learning: Evidence from eye movements. Reading ResearchQuarterly, 48, 199–214.

Mason, L., Tornatora, M. C. & Pluchino, P. (2013b). Do fourth graders integrate text andpicture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers & Education, 60, 95–109.

Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study oflearning with graphics. Learning and Instruction, 20, 167–171.

Merkley, R. & Ansari, D. (2010). Using eye tracking to study numerical cognition: Thecase of the ratio effect. Experimental Brain Research, 206, 455–460.

Moeller, K., Fischer, M. H., Nuerk, H. C. & Willmes, K. (2009). Sequential or paralleldecomposed processing of two-digit numbers? Evidence from eye-tracking. QuarterlyJournal of Experimental Psychology, 62, 323–334.

Moeller, K., Klein, E., Nuerk, H. C. & Willmes, K. (2011). Three processes underlyingthe carry effect in addition—Evidence from eye tracking. British Journal of Psychology,102, 623–645.

EYE MOVEMENTS AND EQUATION SOLVING

Nitschke, K., Ruh, N., Kappler, S., Stahl, C. & Kaller, C. P. (2012). Dissociable stages ofproblem solving (I): Temporal characteristics revealed by eye-movement analyses.Brain and Cognition, 80, 160–169.

Patrick, M., Carter, G. & Wiebe, E. (2005). Visual representations of DNA replication:Middle grades students’ perceptions and interpretations. Journal of Science Education& Technology, 14, 353–365.

Qin, Y., Carter, C. S., Silk, E., Stenger, V. A., Fissell, K., Goode, A. & Anderson, J. R.(2004). The change of the brain activation patterns as children learn algebra equationsolving. Proceedings of National Academy of Science, 101, 5686–5691.

Reingold, E. M., Charness, N., Pomplun, M. & Stampe, D. M. (2001). Visual span inexpert chess players: Evidence from eye movements. Psychological Science, 12, 48–55.

Salmerón, L., Baccino, T., Cañas, J. J., Madrid, R. I. & Fajardo, I. (2009). Do graphicaloverviews facilitate or hinder comprehension in hypertext? Computers & Education, 53,1308–1319.

San Diego, J. P., Aczel, J. C., Hodgson, B. K. & Scanlon, E. (2012). Digital approaches toresearching learners’ computer interactions using gazes, actions, utterances andsketches. Educational Technology Research and Development, 60, 859–881.

Schneider, E., Maruyama, M., Dehaene, S. & Sigman, M. (2012). Eye gaze reveals a fast,parallel extraction of the syntax of arithmetic formulas. Cognition, 125, 475–490.

She, H. S. & Chen, Y. Z. (2009). The impact of multimedia effect on science learning:Evidence from eye movements. Computers & Education, 53, 1297–1307.

Smith, A. D., Mestre, J. P. & Ross, B. H. (2010). Eye-gaze patterns as students studyworked-out examples in mechanics. Physical Review Special Topics: Physics EducationResearch, 6, 020118.

Sohn, M. H., Goode, A., Koedinger, K. R., Stenger, V. A., Fissell, K., Carter, C. S. &Anderson, J. R. (2004). Behavioral equivalence but not neural equivalence: Neuralevidence of alternative strategies in mathematical thinking. Nature Neuroscience, 7,1193–1194.

Tai, R. H., Loehr, F. J. & Brigham, F. J. (2006). An exploration of the use of eye-gazetracking to study problem-solving on standardized science assessments. InternationalJournal of Research & Method in Education, 29, 185–208.

Tang, H. & Pienta, N. (2012). Eye-tracking study of complexity in gas law problems.Journal of Chemical Education, 89, 988–994.

Terry, P. W. (1992). The reading problem in arithmetic. Journal of EducationalPsychology, 84, 70–75.

Thomas, L. E. & Lleras, A. (2007). Moving eyes and moving thought: On the spatialcompatibility between eye movements and cognition. Psychonomic Bulletin & Review,14, 663–668.

Townsend, J. T. & Ashby, F. G. (1978). Methods of modeling capacity in simpleprocessing systems. In N. J. Castellan Jr. & F. Restle (Eds.), Cognitive theory (VolumeIIIth ed., pp. 199–239). Hillsdale, NJ: Erlbaum.

Tsai, M. J., Hou, H. T., Lai, M. L., Liu, W. Y. & Yang, F. Y. (2012). Visual attention forsolving multiple-choice science problem: An eye-tracking analysis. Computers &Education, 58, 375–385.

van Gog, T. & Scheiter, K. (2010). Eye tracking as a tool to study and enhancemultimedia learning. Learning and Instruction, 20, 95–99.

van Gog, T. & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitiveand metacognitive processes in computer-based learning environments. In R. Azevedo

& V. Aleven (Eds.), International handbook of metacognition and learning technol-ogies (pp. 143–156). New York: Springer.

Veenman, M. V. J., Prins, F. J. & Verheij, J. (2003). Learning styles: Self-reports versusthinking aloud measures. British Journal of Educational Psychology, 73, 357–372.

Veenman, M. V. J., Van Hout-Wolters, B. H. A. M. & Afflerbach, P. (2006).Metacognition and learning: Conceptual and methodological considerations. Metacog-nition Learning, 1, 3–14.

Williamson, V. M., Hegarty, M., Deslongchamps, G., Williamson, K. C. & Shultz, M. J.(2013). Identifying student use of ball-and-stick images versus electrostatic potentialmap images via eye tracking. Journal of Chemical Education, 90, 159–164.

Ana Susac, Jurica Kaponja and Maja Planinic

Department of Physics, Faculty of ScienceUniversity of ZagrebZagreb, CroatiaE-mail: ana@phy.hr

Andreja Bubic

Chair for Psychology, Faculty of PhilosophyUniversity of SplitSplit, Croatia

Marijan PalmovicLaboratory for Psycholinguistic Research, Department of Speech and LanguagePathologyUniversity of ZagrebZagreb, Croatia

EYE MOVEMENTS AND EQUATION SOLVING