Cognitive Measurements to Design Effective Learning Environments

1 Open University of the Netherlands2 New York University, USA

Fred Paas1 & Slava Kalyuga2

I C L E P S WORKSHOP 2005 August 30 3

Overview

COGNITIVE MEASUREMENT: GROUP-BASED INSTRUCTIONMeasurement of cognitive load Interpretation of performance, mental effort, and combined scores

COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTIONDiagnostic assessment of organized knowledge structures Applying combined efficiency measures in adaptive training

Basic Assumption

If individuals are to learn effectively in a learning environment, the architecture of their cognitive system, the learning environment, and interactions between both must be

understood, accommodated, and aligned

Cognitive Measurements

Working Memory

Sensory Memory

Long Term Memory

Cognitive architecture

• Perceive incoming information

• Attend to information • Limited capacity• Limited duration• Separate processors for visual and auditory information

• Permanently store all knowledge and skills in a hierarchical network (schemas)

• Unlimited capacity

Schema: • categorizes elements of information according to the manner in which they will be used

• consists of a multi-dimensional web of interconnected nodes of information• can be treated by WM as a single entity, and if the learning process has occurred over a long period of time, it may incorporate a huge amount of information

• can be processed consciously or automatically

Intrinsic

represents the load that performing a particular task imposes on our cognitive system

The Concept Cognitive Load

Extraneous not relevant for learning

Germane relevant for learning

Determined by the number of information elementsand their interactivity

Determined by the manner in which the informationis presented to learners

Intrinsic

Extraneous

Germane

Capacitymax.

Capacitymin.

Cognitive Load TheoryInstructional Techniques

DecreasingExtraneous

Load Goal-free effect Worked-example effect Completion effect Split-attention effect Modality effect Redundancy effect

MinimizingExtraneous

LoadIncreasing

Germane Load Variability effect Self-explanation effect Imagination effect Interactivity effect

MaximizingGermane Load

Sequencing effect Fading support effect

ManagingIntrinsic Load

Measurement of Cognitive Load

Objective measures Task and performance Secondary task Psychophysiological

Subjective measures Rating scales

Goal-Specific Problem Solving

Start

Equationswith goalas only

unknown

Equation withsubgoal(s) and

unknown(s)

Equationwith subgoal

as onlyunknown

Equationwith goal andunknown(s)

Problem solved

Problem statement and

equations in

working memory

Subgoal(s)added to

working memory

yes

yes

yes

yes

no

no

no

no

no

noSolve equations

and add newknown to

working memory

Given: A car that starts from rest and accelerates uniformly at 2 meters/s2 in a straight line has an average velocity of 17 meters/s.

Goal: How far has it traveled?Operators: s = v * t , v = .5V and V = a * t

(V=final velocity, v=average velocity, a=accelaration, t=time, s=distance)

Goal-Free Problem Solving

Start

Stop searchEquation

with only oneunknown

Problem statement and

equations in

working memory

yes

no

Solve equationsand add new

known toworking memory

Given: A car that starts from rest and accelerates uniformly at 2 meters/s2 in a straight line has an average velocity of 17 meters/s.

Calculate the value of as many variables as you can.

Measurement of Cognitive Load

Objective measures Task and performance Secondary task Psychophysiological

Subjective measures Rating scales

Rapid RT Slow RT

Cognitive resources to simple primary task

Cognitive resources to complex primary task

Fixed cognitive capacity Fixed cognitive capacity

Resources to secondary task

Resources to secondary task

Objective measures Task and performance Secondary task Psychophysiological

Subjective measures Rating scales

Measurement of Cognitive Load

Heart rate variability

Eye movementsPupil dilation

very, very low mental

effort

very, very high mental

effort

neither low nor high

mental effort

In solving or studying the preceding problem I invested:

Measurement of Cognitive Load Objective measures

Task and performance Secondary task Psychophysiological

Subjective measures Rating scales

Subjective measures: Rating scales (NASA-TLX)

How to interprete performance and mental effort scores?

Instructional Condition Performance (1-10) Mental Effort (1-10)

A

B

C

D

2

2

8

8

9

2

3

9

Mental Effort

PerformanceM = P

Efficiency = 0high efficiency

low efficiency

Efficiency of Instructional Conditions

Motivation in Instructional Conditions

Mental Effort

Performance

Motivation = 0

high motivation

low motivation

Instructional Efficiency/Motivation

Mental Effort

Performancehigh motivation

low motivation

high efficiency

low efficiency

Find usable objective technique

Intrinsic items1- how easy or difficult do you consider probability theory at this moment? 

Extraneous items2- how easy or difficult is it for you to work with learning environment? 3- how easy or difficult is it for you to distinguish important and unimportant information in the learning environment?4- how easy or difficult is it for you to collect all the information that you need in the learning environment? 

Germane items5- how easy or difficult was it to understand the solution in the last animation?

Use measures to personalize instruction

Distinguish between different types of load

Measurement of Cognitive Load: Challenges

COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTION

Expertise reversal effect:

Cognitive load effects depend on levels of learner expertise:instructional designs or procedures that are effective for novices may be ineffective for more proficient (expert) learners.

Instructional implications:

- instructional techniques need to change with alterations in expertise;

- it is critical to have simple rapid measures of learner proficiency (performance and mental effort).

Real time (rapid online) diagnostic assessment of organized knowledge structures.

Why organized knowledge structures (schemas)?

Cognitive studies of expertise:

organized knowledge base in LTM is central to cognitive processing (De Groot, 1946/1965, Chase & Simon, 1973); they affects the way we process information in WM and solve problems (Novices vs Experts).

Solve for x: 5x = - 4

x = - 4/5

Solve for x: 5x = - 4

5x/5 = - 4/5

Solve for x: 5x = - 4

Diagnostic cognitive assessment should be

• sensitive to different cognitive attributes

• sensitive to different levels of proficiency

• practically usable

Typical time scale of cognitive processes: up to several seconds.

Rapid diagnostic approach:

general: What is the highest level of organised knowledge structures (if any) a person is capable of retrieving and applying to the briefly presented material?

first-step method: Presenting learners with a task for a limited time and asking them to indicate their first step towards solution.

Rapid verification diagnostic technique: Presenting learners with a series of possible task

solutions for a limited time and asking them to rapidly verify the suggested solution steps.

Physics (kinematics) vector addition motion problems

A ship is traveling at 7 m/s. A dog runs across the deck at the same speed in a direction of 60° relative to the direction of motion of the ship. What is the velocity of the dog relative to the sea?

R(mental effort rating)

P(performance score)

Pmax

Rmax

Ecr =Pmax

Rmax

E > E cr

E < E cr

P2

P1

R2R1

P1

R1E1=

P2

R2E2=

A

B

E = P/R

Combined efficiency measures

Using rapid diagnostic techniques in adaptive training for dynamic learning task selection (tailoring levels of task complexity and learner support).

Initial Diagnostic Test

STAGE 1

E 29

Diagnostic Test -4x=3

Yes

No

Yes

No

STAGE 2Diagnostic Test 3x+7=3

Yes

No

Yes

No

STAGE 3 Diagnostic Test (3x+2)/2=3

Yes

No

Yes

No

STAGE 4FinalDiagnostic Test

2 fully worked out examples each followed by a problem solving exercise

4 shortened worked examples

2 completion tasks (with 1 step to complete) each followed by a problem solving exercise

END

-3x=7

4x+3=2

(2x+1)/3=2

-7x=2

6x+4=3

(5x+4)/3=2

2 completion tasks (with 2 steps to complete) each followed by a problem solving exercise

4 problem solving exercises

2 worked examples

4 shortened worked examples

4 shortened worked examples

2 worked examples

2 worked examples

29

E 49

49

E 69

E 69

E 29

? E 19

?

E 29

?

E 29

?

E 19

?

E 19

?

Selection algorithm governing the selection of learning tasks with different levels of difficulty (stages 1-4) and support (worked examples, completion tasks, and conventional tasks/problem solving exercises). Adapted from Kalyuga and Sweller (2004)