Metacognition in ELEKTRA

23
Metacognition in ELEKTRA * Jean-Loup Castaigne LabSET — IFRES — University of Liege January 24, 2007 1 Why certitude degrees in learning? In an evaluation based only on identifying correct and incorrect answers there is little information available for both teacher and learner other than right or wrong. Adding certitude degree to evaluation brings a new dimension, allowing more precise conclusions. For instance what conclusion should raise a teacher when 95 % of his students succeed answering a question? What other conclusion if those 95 % successful students only have a mean confidence of 10% in their correct answer? Maybe the teacher will consider this task as not achieved by his students despite 95% of correct answer. The way a test will be corrected will also be different if a majority of students are failing a question with a mean certitude degree of 10% for that question or with a mean certitude degree of 90%. In the first case doubt can be expressed regarding the question itself. In the second case it appears students are confident in their believe which is an incorrect information. This last situation may be considered as dangerous as students will trust what they think they know. Learning does not move someone from total ignorance to perfect knowledge. Often people will already have some knowledge or representation about what is teached, even if these representation or knowledge might be erroneous. Evalua- tion should not be limited to either knowledge or correct answer and ignorance or incorrect answer. Information is what reduces doubt. “Partial information exists. To detect it is necessary and feasible” (De Finetti, 1965) 1 . certitude degrees give a teacher a more detailed image of the knowledge of a student: they reveal partial information. * ELEKTRA : “Enhanced Learning Experience and Knowledge TRAnsfer” is an European research project that aims at revolutionising technology-enhanced learning. 1 De Finetti, B. (1965). Methods of discriminating levels of partial knowledge concerning a test item. Bristish Journal of Math. & Statist. Psychol., 18, 87-123. 1

description

How metacognition was trained in ELEKTRA project of game based learning of physics

Transcript of Metacognition in ELEKTRA

Page 1: Metacognition in ELEKTRA

Metacognition in ELEKTRA∗

Jean-Loup CastaigneLabSET — IFRES — University of Liege

January 24, 2007

1 Why certitude degrees in learning?

In an evaluation based only on identifying correct and incorrect answers thereis little information available for both teacher and learner other than right orwrong. Adding certitude degree to evaluation brings a new dimension, allowingmore precise conclusions.

For instance what conclusion should raise a teacher when 95 % of his studentssucceed answering a question? What other conclusion if those 95 % successfulstudents only have a mean confidence of 10% in their correct answer? Maybethe teacher will consider this task as not achieved by his students despite 95%of correct answer.

The way a test will be corrected will also be different if a majority of studentsare failing a question with a mean certitude degree of 10% for that question orwith a mean certitude degree of 90%. In the first case doubt can be expressedregarding the question itself. In the second case it appears students are confidentin their believe which is an incorrect information. This last situation may beconsidered as dangerous as students will trust what they think they know.

Learning does not move someone from total ignorance to perfect knowledge.Often people will already have some knowledge or representation about what isteached, even if these representation or knowledge might be erroneous. Evalua-tion should not be limited to either knowledge or correct answer and ignoranceor incorrect answer. Information is what reduces doubt. “Partial informationexists. To detect it is necessary and feasible” (De Finetti, 1965)1. certitudedegrees give a teacher a more detailed image of the knowledge of a student:they reveal partial information.

∗ELEKTRA : “Enhanced Learning Experience and Knowledge TRAnsfer” is an Europeanresearch project that aims at revolutionising technology-enhanced learning.

1De Finetti, B. (1965). Methods of discriminating levels of partial knowledge concerninga test item. Bristish Journal of Math. & Statist. Psychol., 18, 87-123.

1

Page 2: Metacognition in ELEKTRA

2 What is metacognition?

2.1 Metacognition in general

Metacognition has been defined by Flavell (1976)2: “Metacognition refers toone’s knowledge concerning one’s own cognitive processes or anything relatedto them, e.g. , the learning-relevant properties of information or data” . Manyinterpretations were made from the original definition of metacognition butmetacognition is agreed to involve both

• knowledge about one’s own knowledge and

• knowledge about one’s own cognitive processes.

Metacognition has to do with the active monitoring and regulation of cognitiveprocesses. Metacognition is also relevant to work on cognitive styles and learningstrategies in so far as the individual has some awareness of their thinking orlearning processes.

Within metacognition “knowledge about memory” is defined as metamem-ory (Flavell & Wellman, 1977)3. Metamemory itself is declined into declar-ative metamemory (e.g. knowledge about strategy variables) and proceduralmetamemory with its two components: the monitoring component and the con-trol and self-regulated component.

Metacognition

• Metamemory

1. Declarative metamemory

2. Procedural metamemory

(a) monitoring component(b) control and self-regulated component

Schraw and Dennison (1994)4 separate the knowledge and the regulation ofcognition, each one being defined into different parts.

1. The knowledge of cognition is composed of:

(a) Declarative knowledge: knowledge about one’s skills, intellectual re-sources, and abilities as a learner.

(b) Procedural knowledge: knowledge about how to implement learningprocedures (e.g., strategies).

(c) Conditional knowledge: knowledge about when and why to use learn-ing procedures.

2. The regulation of cognition refers to:2Flavell, J.H. (1976). Metacognitive aspects of problem solving. In Resnick (Ed.), The

nature of intelligence. (pp. 231-235). New Jersey : Lawrence Erlbaum Associates.3Flavell, J.H., & Wellman, H. (1977). Metamemory. In R. V. Kail Jr. & J. Hagen (Eds.),

Perspectives on the development of memory and cognition (pp. 3–33). Hillsdale, NJ: Erlbaum4Schraw, Gregory, and Rayne Sperling Dennison, 1994. Assessing Metacognitive Aware-

ness? In Contemporary Educational Psychology 19, 460-475.

2

Page 3: Metacognition in ELEKTRA

(a) Planning: planning, goal setting, and allocating resources prior tolearning.

(b) Information management: skills and strategy sequences used on-lineto process information more efficiently (e.g., organizing, elaborating,summarizing, selective focusing).

(c) Monitoring: assessment of one’s learning or strategy use.(d) Debugging: strategies used to correct comprehension and perfor-

mance errors.(e) Evaluation: analysis of performance and strategy effectiveness after

a learning episode.

The EARLI 5 metacognition special interest group often refers to the maincomponents of metacognitive monitoring and metacognitive control that wasdefined from the works of Nelson (1996) 6 and Nelson & Narens (1994) 7 – seefigure 1.

Figure 1: Main components of metacognitive monitoring and metacognitive control definedby Nelson (1996) and Nelson & Narens (1994)

Leclercq & Poumay (2004) 8 propose an “operational definition” of metacog-nition: “Observable (or not) judgments, analysis and /or regulations effectuated

5EARLI: European Association for Research on Learning and Instruction6Nelson, T. O. (1996). Gamma is a measure of the accuracy of predicting performance

on one item relative to another item, not of the absolute performance on an individual item.Applied Cognitive Psychology, 10, 257–260.

7Nelson, T. O., & Narens, L. (1994). Why investigate metacognition? In J. Metcalfe &A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 1–25). Cambridge, MA:Bradford Books.

8personal translation of “les jugements, analyses et/ou regulations observables effectues parl’apprenant sur ses propres performances (processus ou produits d’apprentissages), ceci dansdes situations de PRE, PER ou POST performance” from Leclercq, D. & Poumay, M. (2004).Une definition operationnelle de la metacognition et ses mises en oeuvre. 21e Conferenceinternationale de l’AIPU, Morocco: Marrakech.

3

Page 4: Metacognition in ELEKTRA

by a learner on his/her own performances (learning processes or products), insituations of pre, per or post performance (mainly testing or learning).”

2.2 Certitude degrees and metacognition

Ebel (1965)9 defined confidence weighting as “a special mode of responding toobjective test items, and special mode of scoring those responses. In generalterms, the examinee is asked to indicate not only what he believes to be thecorrect answer to a question, but also how certain he is of the correctness of hisanswer”. certitude degrees are confidence weighting. In Nelson & Narens’s maincomponents of metacognitive monitoring and metacognitive control certitudedegrees with testing take place in the monitoring side, retrieval of knowledgecategory, in the Confidence in retrieved answers box. In Leclercq & Poumay’soperational definition certitude degree with testing will be observable judgmentseffectuated by a learner on his/her own performances (products), in situationsof per performance (testing).

2.3 Metacognition in the demonstrator

One can ask if metacognition is usable as such for teenagers 14 years old. Inbrief, publications that were founded consider tested components of metacogni-tive monitoring and control not significantly different between adults and as lowas 10 years old children. Below the age of 10, components significance evolveswith age. Flavell, Friedrichs & Hoyt (1970)10 have showed significant correla-tion between predicted and actual memory span in children from the 4th gradeand no significant correlation was founded below that age, including at nurseryand kindergarden. Young children show unrealistic performances prediction.Schneider (2006)11 offers three theoretical reasons: 1) insufficient metacognitiveknowledge (young children not monitoring their memory activities or lacking inunderstanding about the interplay among relevant factors), 2) predominance ofwishful thinking over expectations thus predictions reflecting their desires) and3) belief in the power of effort. Duell (1986)12 confirm that as children get olderthey demonstrate more awareness of their thinking processes.

In the Elektra demonstrator the use of certitude degrees so far is neitherrelevant of a “judgment of learning” nor a confidence in retrieved answer. Judg-ments of learning (jols), are defined as judgments that “occur during or afteracquisition and are predictions about future test performance on recently stud-ied items” (Nelson & Narens, 1994).

9Ebel, R. (1965). Confidence weighting and test reliability. Journal of Educational Mea-surement, pages 49–57.

10Flavell, J. H., Friedrichs, A., & Hoyt, J. (1970). Developmental changes in memorizationprocesses. Cognitive Psychology, 1, 324–340.

11Schneider 2006, The development of metacognition in childhood and adolescence, keyn-note of the 2nd international biennial conference of the Metacognition Special Interest Group(SIG 16) of the European Association of Research in Learning and Instruction (EARLI), July2006, Cambridge

12Duell, O.K. (1986). Metacognitive skills. In G. Phye & T. Andre (Eds.), CognitiveClassroom Learning. Orlando, FL: Academic Press.

4

Page 5: Metacognition in ELEKTRA

2.3.1 What is a JOL?

Imagine a learner studying pairs of linked words13 such as game–elektra.The learner receives the instructions that each pair should be learned so thatsubsequently when prompted with the cue (e.g., “game–?”), the learner wouldrecall the target (e.g., “elektra”). An interval of time elapses between thetermination of studying a given item and the onset of the jol for that item.This interval could be

• extremely brief (e.g., as in an immediate jol that occurs as close in timeas possible to the offset of the studied item) or

• could be delayed for a more lengthy amount of time (as in a delayed jol)filled with other activity and/or other to-be-learned items.

Then the first jol occurs, prompted by a cue that usually consisted of onlythe cue from the studied item (e.g., “game–?”). The learner generates a jolby choosing the predicted likelihood (e.g., on a Likert-type rating scale or on ascale ranging from 0% to 100% in steps of 20%) of remembering the item on aneventual criterion test that might occur later (e.g., 10–20 minutes later). Otherpairs of words are studied, and jols are made for each of them, until everypair had been studied and had received a jol. Finally, following an intervalof perhaps 10–20 minutes (filled with other items) from the time of studying agiven item, the person received the eventual memory test on that item. Thememory test is self-paced, usually asking the person to recall the target whenprompted by the cue (e.g., when prompted by “game–?”, the person attemptto recall “elektra”).

With this example we demonstrate that what is asked in the demonstratoris neither true jols nor confidence in retrieved answers.

2.3.2 JOL or confidence in retrieved answer

We suggest that metacognition in elektra is in a first time limited to the useof certitude degrees with the objective of improving the learner metacognitiveself assessment. Kruger & Dunning (1999) 14 showed that “learners whose skillsor knowledge bases are weak in a particular area tend to overestimate theirability in that area”. In other words, they don’t know enough to recognizethat they lack sufficient knowledge for accurate self-assessment. In contrast,learners whose knowledge or skills are strong may underestimate their ability.These high-ability learners don’t recognize the extent of their knowledge or skills.Kruger and Dunning’s research also shows that “it is possible to teach learners atall ability levels to assess their own performance more accurately”. It is in thatperspective that certitude degrees will be implemented in elektra. With theuse of certitude degree in elektra we can at least give the learner/player somefeedback about their cognitive strategies with the confidence in their retrievedanswers.

13adapted form Nelson, Narens & Dunlosky (2004). A Revised Methodology for Research onMetamemory: Pre-judgment Recall And Monitoring (PRAM). Psychological Methods, Vol.9, No. 1, 53–69.

14Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties inrecognizing one’s own incompetence lead to inflated self-assessment. Journal of Personalityand Social Psychology, 77, 1121-1134.

5

Page 6: Metacognition in ELEKTRA

Consolidated Research Report M1-6 specifies that metacognition within elek-tra should:

• support and facilitate the development of metacognitive abilities and strate-gies and

• support the assessment of knowledge.

We recommend to be very cautious in identifying what is assessment of knowl-edge and what is support to metacognition. Feedbacks should always be veryclear in regard to what they refer to, either cognition–knowledge or metacognition–knowledge about knowledge. With elektra we can provide teachers a tool tobegin a reflection with their students about metacognition. The distinctionbetween cognitive and metacognitive strategies is important, partly because itgives some indication of which strategies are the most crucial in determiningthe effectiveness of learning. It seems that “metacognitive strategies, that allowstudents to plan, control, and evaluate their learning, have the most central roleto play in this respect, rather than those that merely maximize interaction andinput . . . Thus the ability to choose and evaluate one’s strategies is of centralimportance” (Graham, 1997) 15 . Learners who are metacognitively aware knowwhat to do when they don’t know what to do; that is, they have strategies forfinding out or figuring out what they need to do. “The use of metacognitivestrategies ignites one’s thinking and can lead to more profound learning and im-proved performance, especially among learners who are struggling” (Anderson,2002) 16.

3 Metacognition in games

Prakash (1999) 17 reports that he heard on the radio a sixth grader explainingwhat she was learning from playing the Stock Market Game. This game was anactivity designed to help children become familiar with how the stock marketfunctions. She said, “This game makes me think how to think”. What thisstatement reveals is that this young learner was beginning to understand thereal key to learning; she was engaged in metacognition using a game.

Saldana (2004) 18 reports using a Master Mind c© modified (order was notimportant) to assess metacognitive processes use to solve the enigma. Thesolution could be found independently or with three levels of assistance

1. focusing on the metacognitive processes present in the task such as

• planning aims / strategies,

• supervision and control of aims / strategies,

• revision of aims / strategies,

15Graham, S. (1997). Effective language learning. Clevedon, England: Multilingual Mat-ters.

16Neil J. Anderson, The Role of Metacognition in Second Language Teaching and Learning,Brigham Young University

17Prakash, S. (Reporter). (1999, March 19). Market games [Radio series episode]. Allthings considered. Washington: National Public Radio.

18Saldana, D. (2004) Dynamic Master Mind: interactive use of a game for testing metacog-nition. School Psychology International, Vol 25(4): 422-438.

6

Page 7: Metacognition in ELEKTRA

• meta knowledge of possibilities and limitations

• meta knowledge of tasks / strategies;

2. scaffolding each and every one of the selfregulatory steps included in thetask;

3. modelling of the task solution process

On top of working metacognitive process this experiment also showed diversifi-cation in methods for adaptivity.

Additional literature research is still under process. Please keep this docu-ment for private use within elektra.

4 About the demonstrator, 2006 version

These comments only cover some metacognitive aspects of the demonstrator.

4.1 A surprise

When Galileo says that the most important thing is to first define what youwant to do, he is asking George to identify the objective. This is a metacognitiveactivity and looks like what is described by Saldana (2004) as “planning aims”in his Master Mind c© experiment. It is a pleasant surprise to see that theformulation of that question turned it into a metacognitive activity.

Figure 2: Cut scene from the elektra demonstrator, 2006

7

Page 8: Metacognition in ELEKTRA

4.2 General comments

1. The asked question should not have been ‘Are you sure?” but somethinglike “With what you know and what you have learned, how confident areyou that you will succeed opening the door in this try?” Asked the way itwas, it was confusing as we mentioned as soon as we had the opportunityto look at the demonstrator. But it was too late to change as the validationin ORT Marseilles had already begun. UG document about metacognition19 mentioned that “As shown for example in the first validation results,the current formulation of the confidence rating is not understood by mostindividuals of the target group”. We argue that

(a) the formulation of the question, especially that lack of “sure of what?”

(b) the fact that the question was neither a jol nor a confidence inretrieved answer,

(c) the lack of explanations related to the metacognitive aims and theabsence of tutorial explaining how to use certitude degree,

(d) the lack of training of students in general regarding expression oftheir confidence,

(e) the use of verbal expression e.g., “sure” is confusing by itself andshould not be used when retrieving certitude (see Shuford, Albert &Massengill (1966)20 recommendations for the use of subjective prob-abilities instead of verbal expressions)

are all explanations of the confusion. As mentioned here above, in collect-ing jol Nelson & Narens themselves are using a probabilistic scale from0% to 100% by steps of 20%. In the validation report21 ORT mentionsthat: “It seemed that most pupils didn’t really understand the meaningbehind the confidence degree, or the questions that was asked.” We to-tally agree with their recommendation that suggest: “It seems that thequestion related to the confidence degree wasn’t very clear, it is thereforerecommended to formulate the question in a clearer way (adapted to 13-14years old pupils)”. To avoid confusion we propose section 7 some sugges-tion about what to explain in the tutorial. Shuford (1966) and Leclercq(198322, 198623, 199324) defined 4 rules when using certitude degrees:

(a) the use of probabilistic value instead of words;

(b) if points are awarded tariff should respect the decision theory;

(c) feedbacks concerning the good or not good use of certitude degreeprovided to students;

19Kickmeier-Rust, Albert & Linek (2007) A psychological perspective on metacognition andthe usage of confidence ratings. Draft for internal use of elektra consortium

20Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-surement procedures. Psychometrika, 31(2):125–45.

21Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report22Leclercq, D. (1983). Confidence marking, its use in testing. Postlethwaite, Choppin (eds)

Evaluation in Education, Oxford : Pergamon, 1982, vol. 6, 2, pp. 161-287.23Leclercq, D. (1986). La conception des questions a choix multiple, Bruxelles, Ed. Labor.24Leclercq, D. (1993). Validity, reliability and acuity of self-assessment in educational test-

ing. In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-Assessment, number F 112 in NATO ASI Series, pages 114–131. Berlin: Springer Verlag.

8

Page 9: Metacognition in ELEKTRA

(d) students must be trained to use certitude degrees.

The respect of these rules is mandatory for anyone willing to use certitudedegrees and avoid confusion or deception.

2. Chosen certitude and chosen answer are not recorded into the game. Bothteacher and learner should be able to view the results, success and failure,during tries and tasks. These data should influence the feedbacks. Assuch they can be part of micro-adaptivity.

3. The feedback should be more precise. Galileo might say something like:

When I asked you how confident you were about succeeding this task, youtold me you were XX% confident succeeding it.If (FAILURE) then

If (chosen certitude degree > 50%) thenYou did not succeed. But you were very confident you wouldsucceed. So you’ve made a bad assumption. You should be morecareful in your estimations. Try again and be more modest withyour confidence.

elseYou did not succeed. But you confidence in your ability tosucceed was low. So your assumption was good: you have failedbut you knew it was risky so you associated doubt with yourtry. I congratulate you for that. Give it another try.

end Ifelse

If (chosen certitude degree ≤ 50%) thenYou’ve succeeded. Congratulations. But your confidence inyour ability was too low. You should be more confident in yourestimations.

elseYou’ve succeeded. Congratulations. And your estimation was agood assumption: you were sure and you’ve succeeded. That’sgreat.

end Ifend If

Notice how feedbacks about cognition and metacognition are in different sen-tences as suggested in section 2.3.2, page 5. Knowledge feedbacks are clearlydifferent than support of metacognition. When a few tries have been made, thefeedback can even turn to be more precise. This will be described in the lastpart of this document.

4.3 Experiment in world 2 and metacognition

We also suggest that, in world 2, the player should be able to turn on and offthe flashlight. He should only be allowed to move the blinds and the screen

9

Page 10: Metacognition in ELEKTRA

when the torch is turned off. After he has placed all the objects the way hewants, he will have to turn the flashlight on to see the results.

Even if the player does not find the correct solution he will see the result ofwhat he built when turning the flashlight on. He will see it in 3D and will also seethe associated top view. When the player turns on the flashlight Galileo mightalso ask a question like “How sure are you that this construction will produce anarrow beam of light?” When turning it off Galileo will give a feedback (see theprevious algorithm). Instead of Galileo always asking the same question aboutconfidence, we suggest that the switch of the light might be something like

off on with certitude degree of0% 20% 40% 60% 80% 100%

Table 1: Flashlight on–off switch enhanced with certitude degree

4.4 The metal-wooden door and the blinds

Imagine that the player succeed placing the blinds and screen in LeS1.2a inworld 2 with a very good confidence. Then he/she fails in the real world withthe metal-wooden door. What kind of feedback can we give about what he/shehas learned in world 2? What kind of feedback can we give about his/herconfidence in succeeding the task based on what he/she has experimented inworld 2? We have no answer for these questions because the problem, from ourpoint of view, is that the metal-wooden door does not evaluate what might havebeen learned with the blinds and screen in world2. This seems impossible to bechanged. So we recommend not to use certitude degrees at the moment withthe opening of the metal–wooden door situation.

5 Deeper with certitude degree

The demonstrator is working on one single expression of the player’s jol. It isdifficult for him to gain any metacognitive knowledge with a single measurement.The only feedback we can actually give with the actual state of development ofthe demonstrator is a good\bad performance associated with a confident\nonconfident jol as showed in table 2.

good badperformance performance

confident Best Worstgood knowledge wrong knowledgegood confidence bad confidence

non confident good knowledge lack of knowledgebut but

lack of confidence good confidence

Table 2: Feedback class associated with the combination of true-false and confidence

10

Page 11: Metacognition in ELEKTRA

5.1 Theoretical background

The following is based on the works of Hunt (1993), Jans & Leclercq (1999) andLeclercq (2003) and gives a much more detailed measure of knowledge with theassociation of a confidence chosen out of a scale of 6 degrees of certitude (seefigures 3 to 5, pages 11 et 12). All figures are spectral distributions of knowledge.On the left hand side are the incorrect answers. They are distributed by thelearner chosen confidence, ranking from left to right from 100% down to 0%. Inthe middle is the grey area of not answered questions. On the right hand sideare the correct answer, also distributed by confidence but ranking from left toright from 0% to 100%. Figure 3 showed Darwin Hunt (1993) 25 suggestion todistinguish between three types of knowledge situations in which a person canbe in relation to a piece of content : “misinformed, uninformed, informed”.

Figure 3: Spectral distributions of Hunt’s three situations of knowledge (1993)

These situations were redefined –see figure 4– by Jans & Leclercq (1999) 26

as: “dangerous knowledge, unusable knowledge, usable knowledge”.Unhappy with the “unusable knowledge” unable to make the difference be-

tween a student making a mistake with a low confidence and a student hav-ing a low confidence in a correct answer, Leclercq (2003) 27 divided it into“unawareness” and “mid knowledge” as showed in figure 5. This is an impor-tant distinction as mid knowledge needs working on metacognitive judgment, a

25Hunt, D. (1993). Human self-assessment : Theory and application to learning and testing.In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-Assessment,volume F 112 of NATO ASI Series, pages 177–189. Berlin: Springer Verlag

26Jans, V. and Leclercq, D. (1999) Mesurer l’effet de l’apprentissage a l’aide de l’analysespectrale des performances. In Depover, C. and Noel, B., editors, L’evaluation descompetences et des processus cognitifs. Modeles pratiques et contextes, pages 303-317. Brux-elles : De Boeck Universite

27Leclercq, D. (2003). Un diagnostic cognitif et metacognitif au seuil de l’universite. Leprojet MOHICAN mene par les 9 universites de la Communaute Francaise Wallonie Bruxelles.Liege : Editions de l’Universite de Liege.

11

Page 12: Metacognition in ELEKTRA

Figure 4: Spectral distribution of knowledge kinds by Jans & Leclercq (1999) based on Hunt’sthree situation of knowledge

metacognitive part, and unawareness needs to first work on knowledge, a cogni-tive part. Again, as mentioned in section2.3.2 page 5, do not confuse assessmentof knowledge and support of metacognition with certitude degrees. It is already

Figure 5: Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999) unusable knowledge

much better than a simple correct or not correct feedback. But certitude degreesallows to give much more refined feedbacks.

12

Page 13: Metacognition in ELEKTRA

5.2 Application in the game

Certitude degrees will add a dimension in storytelling as we define a new “socio-affective” goal to the game: George is to gain the trust of Galileo Galilei tobecome the new trustfully leader of the Galileans. Galileo’s trust in George willbe based on his good use on certitude degrees, mainly based on confidence inretrieved answers. The point is to give a general feedback on how the player isusing certitude degree. Not only immediatly after after a single judgment as itis in the actual demonstrator but with enough measure to pin tendencies likeover– and under–estimation, see section 2.3, page 4.

5.3 Technical implementation

So on top of what was suggested here above, the program should computetwo values and communicate with the player about confidence (mcdca) andimprudence (mcdia) as defined by Leclercq & Poumay (2004):

1. mcdca is the mean certitude degree when the tries are successful or

Confidencce = CDcA=

∑n(n · pn)NCDcA

(1)

where

• CDcAstands for mean certitude degree for correct answers;

• n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%)

• pn is the number of use of the n certitude degree associated withcorrect answers;

• NCDcAis the number of use of the certitude degrees in correct an-

swers;

2. mcdia is the mean certitude degree when the tries are unsuccessful or

Imprudence = CDiA=

∑n(n · qn)NCDiA

(2)

• CDiAstands for mean certitude degree for incorrect answers;

• n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%)

• qn is the number of use of the n certitude degree associated withincorrect answers;

• NCDiAis the number of use of the certitude degrees in incorrect

answers.

Exemple : Out of a test of 7 questions a student has 5 correct answers and2 incorrect answers. All 7 answers were associated with a certitude degree asshowed in table 3 page 14.

To compute confidence we focus on the correct answers only: first we countthe number of use of certitude degree 0% (only for correct answer!). So n is 0%.In our exemple, there is 1 use of certitude degree 0% with a correct answer. Sop0 = 1. We multiply those two numbers by each other: n · pn giving 0 · 1 = 0.

13

Page 14: Metacognition in ELEKTRA

Question Answer Certitude degree1 C 802 C 203 C 1004 C 05 C 806 I 607 I 0

Table 3: Exemple for the computation of confidence and imprudence (C for correct answerand I for Incorrect answer).

Then we do the same for n = 20% which means that we count the number ofuse of certitude degree 20%, only for correct answer! In our exemple there is 1use of certitude 20% within all correct answers: p20 = 1. We multiply those twonumbers by each other: n · pn giving 20 · 1 = 20. Same thing for n = 40% whichmeans that we count the number of use of certitude degree 40%, remember onlyfor correct answer! In our exemple there is no use of certitude 40% within allcorrect answers: p40 = 0. We multiply those two numbers by each other: n · pn

giving 40·0 = 0. Same thing for each certitude degree (60%, 80%) until we reachcertitude of 100% (n = 100%) with one use of certitude 100% making p100 = 1.We multiply those two numbers by each other: n · pn giving 100 · 1 = 100. Wesum all n · pn into [(0 · 1) + (20 · 1) + (40 · 0) + (60 · 0) + (80 · 2) + (100 · 1)]and divide by NCDcA

or the number of use of certitude degree associated withcorrect answers, in our case 5. Final equation being [0+20+0+0+160+100]

5 = 56To compute Imprudence, it is the same process but focusing only on incorrect

answers. So we finish with [(0 ·1)+(20 ·0)+(40 ·0)+(60 ·1)+(80 ·0)+(100 ·0)]and divide by NCDcA

or the number of use of certitude degree associated withincorrect answers, in our case 2. Final equation being [0+0+0+60+0+0]

2 = 30. Soin our exemple confidence is 56% and imprudence 30%.

It is in the computation of those mean values that the use of a scale with6 levels makes sense. First Miller’s (1956) 28 works demonstrated that peopleare able to work with 7 plus or minus two certitude degrees to choose from.And the 6 levels are very easy to remember as they are from 0% to 100% stepof 20%. Shuford et al.(1966)29 demonstrated the necessity of using subjectiveprobabilities instead of verbal expressions. Finally it is after a number of answersthat mean values are computed taking into account every choice made by thestudent. In the case of elektra, after collecting certitude degrees during thestudent’s experimentations it is expected that the mean certitude of correctanswers (mcdca) –that is to say for the successful tries, should above 50%.Mixing well the use of all 6 levels of the certitude degree scale, students shouldlearn to weight their certitude in a correct way when they are about to succeed.A student will be considered as confident when his confidence is above 50%.On the other side the mean certitude of incorrect answers (mcdia) –that is to

28Miller, G. (1956). The magical number seven, plus or minus two: Some limits on ourcapacity for processing information. Psychological Review, (63):81–97

29Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-surement procedures. Psychometrika, 31(2):125–45

14

Page 15: Metacognition in ELEKTRA

say for the unsuccessful tries, should be below 50%. Students should learn tobe modest in confidence using low value on the 6 level scale. An accident mayalways occur like using a high certitude degree and failing but the tendencyshould be that when about to fail student learn to be prudent. An imprudentstudent will be a student with an imprudence above 50%. The best performanceis

• a confidence of 100% or a mean certitude degree for the successful answersof 100% and

• an imprudence of 0% or a mean certitude degree for the unsuccessfulanswers of 0%

For gaming reasons, LMR asked that instead of trying to reach 100% on onescale and 0% on another scale the student should always try to maximize hisscore (like gaining points) or to minimize his level (like note losing live pointswhen fighting). To comply we suggest to use prudence instead of imprudence

prudence = 100 − imprudence (3)

Imprudence or mean certitude degree of incorrect answer should ideally be aslow as possible. Over 50% is the limit when the student becomes imprudent.With the inversion presented in equation (3) goals are exactly the opposite.Prudence should ideally be as close to 100% as possible. A student will bequalified as prudent if his prudence is over 50%.

5.4 Graphic interface

Prudence and confidence will be the indicators of Galileo’s trust gained by theplayer. We suggest to represent prudence and confidence to use the certimeterthat was invented by Galileo and suggested by Frederic Pourbaix. There willbe 2 certimeters, one for the confidence (mean certitude of correct answers) andone for prudence (100 − mean certitude of incorrect answers), see figure 6. Bothcertimeter will be graduated from 0% to 100%. Suggested colors might be fromred to light red between 0% and 50% and from light green to green from 50%to 100%.

Including prudence and confidence we offer a possible answer to ORT rec-ommendation30: “The idea of confidence degree [. . . ] and may have the formof points in the game, which will bring the pupils to think further about theiranswer and will bring additional anticipation.” The aim of the player will be toreach green level score on both certimeter meaning that Galileo can trust himwhen he says that he is confident or when he says that he doubt. Of course tomake the “eclipse machine” work the player will have to gain knowledge andcomprehension of optic physics and astronomy. But the player should also payattention to some metacognitive aspects: confidence and prudence. If the playerfails to gain Galileo’s trust but manage to get the “eclipse machine” workinghe will gain access to the Galilean’s secret headquarters and to the book. Butmaybe Galileo will be a bit reluctant to help the player using the book. . . andgive him a feedback in that way, unless the player starts the game all over againand tries to improve his score on both certimeter. Success in the game could

30Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report

15

Page 16: Metacognition in ELEKTRA

Figure 6: Rough design of a possible way to communicate to the player the level of Galileo’strust gained: the cursor on each certimeter takes the color of the position in the color grada-tion. Both cursor must be at least light green to gain Galileo’s trust.

then be achieved by reaching the Galilean’s HQ. But another level of successwill be to reach HQ and to gain Galileo’s trust.

5.5 Realistic use of certitude degrees

The feedbacks could take the form described in this algorithm here after. Thelimit values of 50% used in this alogrithm should be variables. Four limits areto be created and defined at a default level of 50

conf-lim-ccd the limit to judge a single judgement with a correct answer or the limitto be declared confident in a single judgment

prud-lim-ccd the limit to judge a single judgment with an incorrect answer or the limitto be declared prudent in a single judgment

mcdca-lim the limit to judge the mean certitude degree for correct answer as good ornot or when is the player confident or not in the game so far

mcdia-lim the limit to judge the mean certitude degree for incorrect answer as goodor not or when is the player prudent or not in the game so far

Every situation is symbolized by a group in between brackets (e.g., F + -).

• First with a letter

F for failure in case of an unsuccessful try

S for success in case of a successful try

16

Page 17: Metacognition in ELEKTRA

• a plus (+) or (-) sign

+ if the chosen confidence degree is above 50%

- if the chosen confidence degree is below 50%

• a plus (+) or (-) sign

+ if the mean confidence degree is above 50%

- if the mean confidence degree is below 50%

ccd (chosen certitude degree) : matrix;mcdca (mean certitude degree of correct answers) : integer;mcdia (mean certitude degree of incorrect answers) : integer;

print When I asked you how confident you were about succeedingthis task, you told me you were {ccd}% confident succeeding it.If (FAILURE) then

If (ccd > 50) thenIf (mcdia > 50) then

print Situation A (F + +)else

print Situation B (F + -)end If

elseIf (mcdia ≤ 50) then

print Situation C (F - -)else

print Situation D (F - +)end If

end Ifelse

If (ccd ≤ 50) thenIf (mcdca > 50) then

print Situation E (S - +)else

print Situation F (S - -)end If

elseIf (mcdca ≤ 50) then

print Situation G (S + -)else

print Situation H (S + +)end If

end Ifend If

17

Page 18: Metacognition in ELEKTRA

5.5.1 Situation A (F + +)

Situation: Test unsuccessful, confident in answer, tend to be overconfidentwhen wrong.Feedback: You did not succeed. And you were very confident you wouldsucceed. So you’ve made a bad assumption. You should be more careful in yourestimations. When you failed you tend to choose a too high certitude degree.So when you fail your average prudence is {100-mcdia}%. I consider thatit should be over {mcdia-lim}%. You should try to be less confident whenyou choose a certitude degree. You should learn to be a bit more modest andhumble. You tend to believe you know when you’re still learning. So give itanother try. A try to be more modest if you doubt, even a little.

5.5.2 Situation B (F + -)

Situation: Test unsuccessful, confident in answer, tend to be prudent whenwrong.Feedback: You did not succeed. You were very confident you would succeed.So you made a bad assumption but I consider this assumption as an accident.Usually you seem prudent in your own estimations. Every time you’ve failedyou’ve chosen before a low certitude degree. So when you fail your prudenceis {100-mcdia}%. I consider that it should be over {mcdia-lim}% so I trustyour judgement. Keep going like that when you choose a certitude degree andmaybe go back to the table for some more experimentation.

5.5.3 Situation C (F - -)

Situation: Test unsuccessful, prudent in answer, tend to be prudent whenwrong.Feedback: You did not succeed. But you certitude in your ability to succeedwas low: you were prudent. So your assumption was good: you have failed butyou knew it was risky so you associated doubt with your try. I congratulateyou for that. It is important for me to know if I can trust you. In case case ofcourse you’ve failed but before failing you told me that you were not sure. Andas usual when you say that you’re not sure you do not succeed. So this failureis not important at all. It seems you’re quite good at being prudent: usuallywhen you’ve failed you’ve chosen before a low certitude degree. So when youfail your overall prudence is {100-mcdia}%. I consider that it should be over{mcdia-lim}%. That is a good performance. Continue to do so and I will keepmy trust in your judgments. But you still need to succeed this task: give itanother try.

5.5.4 Situation D (F - +)

Situation: Test unsuccessful, prudent in answer, tend to be overconfident whenwrong.Feedback: You did not succeed. But this time your assumption was good: youhave failed but you knew it was risky so you associated doubt with your try. Icongratulate you for that. You should continue to behave like that because itdoes not appear you’re very skilled at this on the long run. For instance, whenyou fail your average prudence is {100-mcdia}%. I consider that it should be

18

Page 19: Metacognition in ELEKTRA

over {mcdia-lim}%. That is why I said you’re not very skilled. With such avalue I will wait for you to improve before fully trusting your judgement. Maybeyou should go back to the table for some more experimentation before giving itanother try. And do not forget to keep going like you just did in giving yourconfidence.

5.5.5 Situation E (S - +)

Situation: Test successful, prudent in answer, tend to be confident when right.Feedback: To be completed

5.5.6 Situation F (S - -)

Situation: Test successful, prudent in answer, tend to be under confident whenright.Feedback: To be completed

5.5.7 Situation G (S + -)

Situation: Test successful, confident in answer, tend to be under confidentwhen right.Feedback: To be completed

5.5.8 Situation H (S + +)

Situation: Test successful, confident in answer, tend to be confident whenright.Feedback: To be completed

5.6 Even more complex feedbacks

It will really be enjoying for the player to hear the same long feedback. Wesuggest that all these information should be delivered not at once but over anumber of tries. Counter of try and counters of times being in the same situation(e.g., F + - or S + +) should be implemented into the if–then–else loops. So thefirst time the player is in situation A he will received a more simple feedbackbut the second time he will receive a different feedback. It will be only afternumber of correct or incorrect answers that the computed mean values will haveany meaning and so can be given to the player. Prudence and confidence canalready have significant values computed on 55 value of certitude chosen by theplayer if we had the suggested game situation into the tutorial.

6 Certitude degrees in LU1.2b

The Labset proposition for Elektra learning situation 1.2b (the slope) is similarto what we propose for learning situation 1.2a (the blinds) : every time Georgewants to release a marble he is asked his confidence in reaching the target on ascale from 0% to 100% by step of 20%. We suggest that beneath to the cursorsfor the magnet and the fan a third cursor with “confidence to reach the target”record the confidence associated with each try. Feedbacks will be less systematic

19

Page 20: Metacognition in ELEKTRA

as the player/learner should be more aware of his confidence score showed in thecertimeter. When the player successfully dropped all 3 balls in the basket Galileowill ask him a transfer question. The player will have to chose out of a numbera drawings the one that will represent the trajectory of light when turned on.He will also be asked about his confidence. Imagine they are only 4 drawing tochoose from and the player knows that if he fails he can immediately try again.Without confidence degrees the player may try any of the 4 propositions untilhe finds the right one. But with confidence degrees he knows that if he choosesa wrong one and is confident in his guessing he will loose some of Galileo’s trust.So he might know the right solution, be confident and then will answer witha high confidence degree. He also may guess the right solution but then thechosen certitude should be lower. If guessing all proposition until he finds thecorrect one, all confidence will be low31. So teacher will have information onhow did the student reached transfer.

7 The tutorial for certitude degrees

ORT recommended32 that “The idea of confidence degree should be explained inthe beginning of the game (tutorial)”. The player will receive some explicationsabout the Galileans. It is at that moment that certitude degrees should be ex-plained. We propose a theoretical information based on transmission–receptionand a gaming situation similar to experimentation–reactivity.

The tutorial should include

1. the player should gain the trust of Galileo Galilei

2. Galileo’s trust is based on the fact the when someone says “I know” hecan trust that knowledge. And when someone doubt that person is rightto doubt.

3. To gain Galileo trust everytime the player will experiment to solve anenigma he will be asked how confident he his that he will succeed resolvingthe enigma at this exact moment, on this precise try.

4. To indicate his self-confidence the player will chose one certitude degreebetween 0%, 20%, 40%, 60%, 80% and 100%. Points will be gained andlost based on

• if what the player just tried is successful and

– if he has chosen a high certitude degree he will gain confidencepoints

– if he has chosen a low confidence degree he will loose confidencepoints

• else if what the player just tried is unsuccessful and

– if he has chosen a high certitude degree he will loose prudencepoints

– if he has chosen a low confidence degree he will gain prudencepoints

31with the limitation that certitude will go up as the number of possible answers goes down.32Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report

20

Page 21: Metacognition in ELEKTRA

5. In other word if the player bet 100% on a try and if that try is

• successful then he will gain a lot of confidence points

• unsuccessful then he will loose a lot of prudence points

6. On the other hand if the player bet 0% on a try and if that try is

• successful then he will loose a lot of confidence points

• unsuccessful then he will gain a lot of prudence points

We suggest that this “theoretical” explanation is to be followed with a GamingSituation based on betting certitude degrees. This might be something like theShannon Guessing Game (1951)33 or another gaming situation that UL willprovide.

7.1 Shannon’s guessing game

“The new method of estimating entropy exploits the fact that anyone speakinga language possesses, implicitly, an enormous knowledge of the statistics of thelanguage. Familiarity with the words, idioms, cliches and grammar enableshim to fill in missing or incorrect letters in proof-reading, or to complete anunfinished phrase in conversation. An experimental demonstration of the extentto which English is predictable can be given as follows; Select a short passageunfamiliar to the person who is to do the predicting, He is then asked to guessthe first letter in the passage. If the guess is correct he is so informed, andproceeds to guess the second letter. If not, he is told the correct first letterand proceeds to his next guess. This is continued through the text. As theexperiment progresses, the subject writes down the correct text up to the currentpoint for use in predicting future letters.” Gilles & Leclercq (1994) 34 haveadapted this game to train student to use certitude degrees: each time a letteris chosen a certitude for it being the right next letter in the sentence is asked.Metacognitive indices like realism are computed and a graphical representationof the confidence in retrieved answer is drawn. The student tries to find thetext and to score well in metacognitive judgment.

In elektra the passage might be a simple phrase like The secret book ofthe Galileans is in the third drawer of the desk. The player will face an emptyparchment that will be filled with letters written by a 3D feather when theplayer types a letter and a certitude degree. If the letter is incorrect it willburn, disappear and be replaced by the correct letter. Prudence and confidencewill be computed and a metacognitive feedback will be generated. Then theplayer is to guess the next letter and so on.

7.2 Tutorial for teacher

We also suggest that informations about metacognition and certitude degreesshould be available for teachers. These informations should cover a wide domain,from objectives in the elektra game to proposition of reflection path about

33Shannon, C. E. (1951) Prediction and entropy in printed English, Bell Syst. Tech. J., 30,1951, 50-64

34Leclercq, D. & Gilles, J.-L. (1994). GUESS, un logiciel pour entraıner a l’auto-estimationde sa competence cognitive. Actes du colloque QCM et questionnaires fermes, Paris: ISIEE

21

Page 22: Metacognition in ELEKTRA

metacognition in classroom and support to further metacognitive activities forthe students.

8 Conclusion

Knowing how to learn, and knowing which strategies work best, are valuableskills that differentiate expert learners from novice learners. Metacognition, orawareness of the process of learning, is a critical ingredient to successful learning.“Metacognition is an important concept in cognitive theory. It consists of twobasic processes occurring simultaneously: monitoring your progress as you learn,and making changes and adapting your strategies if you perceive you are notdoing so well” (Winn, W. & Snyder, D., 1998).35

Contents

1 Why certitude degrees in learning? 1

2 What is metacognition? 22.1 Metacognition in general . . . . . . . . . . . . . . . . . . . . . . . 22.2 Certitude degrees and metacognition . . . . . . . . . . . . . . . . 42.3 Metacognition in the demonstrator . . . . . . . . . . . . . . . . . 4

2.3.1 What is a JOL? . . . . . . . . . . . . . . . . . . . . . . . 52.3.2 JOL or confidence in retrieved answer . . . . . . . . . . . 5

3 Metacognition in games 6

4 About the demonstrator, 2006 version 74.1 A surprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.2 General comments . . . . . . . . . . . . . . . . . . . . . . . . . . 84.3 Experiment in world 2 and metacognition . . . . . . . . . . . . . 94.4 The metal-wooden door and the blinds . . . . . . . . . . . . . . . 10

5 Deeper with certitude degree 105.1 Theoretical background . . . . . . . . . . . . . . . . . . . . . . . 115.2 Application in the game . . . . . . . . . . . . . . . . . . . . . . . 135.3 Technical implementation . . . . . . . . . . . . . . . . . . . . . . 135.4 Graphic interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.5 Realistic use of certitude degrees . . . . . . . . . . . . . . . . . . 16

5.5.1 Situation A (F + +) . . . . . . . . . . . . . . . . . . . . . 18

35Winn, W. & Snyder D. (1996). Cognitive perspectives in pyschology. In D.H. Jonassen,ed. Handbook of research for educational communications and technology, 112-142. NewYork: Simon & Schuster Macmillan

22

Page 23: Metacognition in ELEKTRA

5.5.2 Situation B (F + -) . . . . . . . . . . . . . . . . . . . . . 185.5.3 Situation C (F - -) . . . . . . . . . . . . . . . . . . . . . . 185.5.4 Situation D (F - +) . . . . . . . . . . . . . . . . . . . . . 185.5.5 Situation E (S - +) . . . . . . . . . . . . . . . . . . . . . . 195.5.6 Situation F (S - -) . . . . . . . . . . . . . . . . . . . . . . 195.5.7 Situation G (S + -) . . . . . . . . . . . . . . . . . . . . . 195.5.8 Situation H (S + +) . . . . . . . . . . . . . . . . . . . . . 19

5.6 Even more complex feedbacks . . . . . . . . . . . . . . . . . . . . 19

6 Certitude degrees in LU1.2b 19

7 The tutorial for certitude degrees 207.1 Shannon’s guessing game . . . . . . . . . . . . . . . . . . . . . . 217.2 Tutorial for teacher . . . . . . . . . . . . . . . . . . . . . . . . . . 21

8 Conclusion 22

List of Tables

1 Flashlight on–off switch enhanced with certitude degree . . . . 102 Feedback class associated with the combination of true-false and

confidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Exemple for the computation of confidence and imprudence (C

for correct answer and I for Incorrect answer). . . . . . . . . . . . 14

List of Figures

1 Main components of metacognitive monitoring and metacognitivecontrol defined by Nelson (1996) and Nelson & Narens (1994) . . 3

2 Cut scene from the elektra demonstrator, 2006 . . . . . . . . . 73 Spectral distributions of Hunt’s three situations of knowledge (1993) 114 Spectral distribution of knowledge kinds by Jans & Leclercq (1999)

based on Hunt’s three situation of knowledge . . . . . . . . . . . 125 Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999)

unusable knowledge . . . . . . . . . . . . . . . . . . . . . . . . . 126 Rough design of a possible way to communicate to the player the

level of Galileo’s trust gained: the cursor on each certimeter takesthe color of the position in the color gradation. Both cursor mustbe at least light green to gain Galileo’s trust. . . . . . . . . . . . 16

23