A Value-Based Approach for A Value-Based Approach for Quantifying Scientific Quantifying Scientific

Problem Solving Problem Solving Effectiveness Within and Effectiveness Within and

Across Educational SystemsAcross Educational Systems

Ron Stevens, Ph.D. IMMEX Project UCLA School of Medicine

Vandana Thadani, Ph.D. Loyola Marymount University

The Challenging Question:The Challenging Question:

What is a suitable description of What is a suitable description of problem solving that can capture problem solving that can capture important cognitive and important cognitive and performance information about an performance information about an individual’s problem solving, yet individual’s problem solving, yet provide rapid and meaningful provide rapid and meaningful comparisons within and across comparisons within and across science domains and educational science domains and educational systems?systems?

Why Would Such a Why Would Such a Measure(s) Be Useful?Measure(s) Be Useful?

Could document the development of Could document the development of problem solving skills:problem solving skills: Throughout the year.Throughout the year. Across science domains.Across science domains.

Would allow comparisons:Would allow comparisons: Across students classrooms, teachers Across students classrooms, teachers

and help guide professional and help guide professional development.development.

Across schools and school systems.Across schools and school systems.

Important Properties for a Important Properties for a Generalizable Measure(s)Generalizable Measure(s)

Face, Construct, Concurrent and Face, Construct, Concurrent and Divergent validityDivergent validity

ReliabilityReliability ScalabilityScalability UnderstandabilityUnderstandability Adaptability / ExtensibilityAdaptability / Extensibility

Theoretical Groundings: Theoretical Groundings: Strategy and Skill Strategy and Skill

DevelopmentDevelopment

Each individual selects the best strategy Each individual selects the best strategy for them on a particular problem. for them on a particular problem.

People adapt strategies to changing rates People adapt strategies to changing rates of success. of success.

Paths of strategy development emerge as Paths of strategy development emerge as students gain experience; and,students gain experience; and,

Improvement in performance is Improvement in performance is accompanied by an increase in speed and accompanied by an increase in speed and reduction in the data processed.reduction in the data processed.

A Framework for Assessing A Framework for Assessing Problem Solving SkillsProblem Solving Skills How well / rapidly were the problems How well / rapidly were the problems

solved? (solved? (easy to assess, but little strategic informationeasy to assess, but little strategic information)) Can hard / easy problems be solved? (Can hard / easy problems be solved? (more more

difficult to assess, IRT estimates are usefuldifficult to assess, IRT estimates are useful)) What problem solving strategy was used? What problem solving strategy was used?

((more difficult to assessmore difficult to assess)) Are the problem solving strategies Are the problem solving strategies

improving with practice? (improving with practice? (more difficult to more difficult to assessassess))

What strategy will the student next use? What strategy will the student next use? ((hard to assesshard to assess))

……and the ability to generalize across and the ability to generalize across domains and educational systems.domains and educational systems.

Problem-Solving WithProblem-Solving With IMMEX IMMEX tmtm

IMMEXIMMEX is a web-based learning is a web-based learning system for promoting problem system for promoting problem solving and decision making skills.solving and decision making skills.

The The IMMEXIMMEX system includes real- system includes real-time student modeling capabilities time student modeling capabilities for assessing and reporting student for assessing and reporting student progress.progress.

Hazmat:Hazmat: A Hazardous A Hazardous Materials SimulationMaterials Simulation

Face Validity – The TasksFace Validity – The TasksFace Validity – The TasksFace Validity – The Tasks

The The HazmatHazmat task task is to is to analyze a toxic analyze a toxic spill by spill by using multiple using multiple chemicalchemicaland physical tests.and physical tests.

http://www.immex.ucla.edu/docs/publications/pdf/its_paper.pdfStevens, R., Soller, A., Cooper, M., Stevens, R., Soller, A., Cooper, M., and Sprang, M. (2004). and Sprang, M. (2004).   Intelligent Tutoring Systems.Intelligent Tutoring Systems. Lester, Lester, Vicari, & Paraguaca (Eds). Springer-Vicari, & Paraguaca (Eds). Springer-Verlag Berlin Heidelberg, Germany. Verlag Berlin Heidelberg, Germany. 7th International Conference 7th International Conference Proceedings (pp. 580-591).Proceedings (pp. 580-591).

Layers of Assessment Tools for Layers of Assessment Tools for Investigating Skill Investigating Skill

DevelopmentDevelopment

Student Ability EstimatesStudent Ability Estimates

First builds a model of the difficulties of each case based on student performance. Then each student is evaluated against this model.

Strategy ModelsStrategy ModelsSelf-organizing neural networks cluster similar

performances into strategic topology

maps.

Progress ModelsProgress Models

Probabilistic models of sequences of neural network

strategies.

Problem Sets Contain Problem Sets Contain Multiple Cases of Multiple Cases of Varying Difficulty Varying Difficulty

As expected, flame test negative compounds are more difficult than positive ones.

The student abilities follow a normal distribution.

This data is useful for comparing problem solving ability with other student assessments like standardized tests.

It does NOT indicate HOW the problem was solved.

Defining Strategies with Defining Strategies with Artificial Neural Networks – Artificial Neural Networks –

The IdeaThe Idea Postulate a number of major strategies that Postulate a number of major strategies that

can be applied to the problem….we often use can be applied to the problem….we often use 36.36.

Train the neural network with thousands of Train the neural network with thousands of performances from students of many abilities performances from students of many abilities using the tests they selected as input data.using the tests they selected as input data.

The performances compete with each other The performances compete with each other for each neural network node such that for each neural network node such that those most similar are clustered together on those most similar are clustered together on the map. the map.

Defining Strategies with Defining Strategies with Artificial Neural Networks – Artificial Neural Networks –

The DataThe Data

http://www.immex.ucla.edu/docs/publications/anndistinguish.htmJournal of the American Medical Informatics Association 3: 131-8, 1996. Stevens, R.; Lopo, A.; Wang, P.

Developing Progress Models Developing Progress Models From Sequences of Neural From Sequences of Neural

Network StrategiesNetwork Strategies –The Idea –The Idea

Postulate a number of states that represent Postulate a number of states that represent transitions students may pass through (often 3-5).transitions students may pass through (often 3-5).

Train Hidden Markov Models with many strategy Train Hidden Markov Models with many strategy sequences. sequences.

Use the transition and emission matrices from the Use the transition and emission matrices from the modeling to develop learning progress modelsmodeling to develop learning progress models..

Developing Progress Models Developing Progress Models From Sequences of Neural From Sequences of Neural

Network StrategiesNetwork Strategies –Examples –Examples

Developing Progress Models Developing Progress Models From Sequences of Neural From Sequences of Neural

Network StrategiesNetwork Strategies –The Data –The Data

State 1 55% SolvedState 2 60% SolvedState 3 45% SolvedState 4 54% SolvedState 5 70% Solved

Student Problem Solving Student Problem Solving Strategies Stabilize Strategies Stabilize

RapidlyRapidly

Collaborative Grouping Collaborative Grouping Accelerates Strategy AdoptionAccelerates Strategy Adoption

The The Big ProblemBig Problem for for DisseminationDissemination

ANN and HMM modeling are very ANN and HMM modeling are very useful research tools……butuseful research tools……but

Each problem set has its own ANN Each problem set has its own ANN topology and state transitions.topology and state transitions.

So a teacher implementing a dozen So a teacher implementing a dozen IMMEX problem sets would need to IMMEX problem sets would need to understand 24 models! understand 24 models!

One Solution: A Value-based One Solution: A Value-based ApproachApproach

Explore expressing problem solving Explore expressing problem solving as a as a valuevalue relating the efficiency of relating the efficiency of the process to the effectiveness of the process to the effectiveness of the outcomes.the outcomes.

Strategic Efficiency Strategic Efficiency

Students demonstrating high Students demonstrating high strategic efficiency will make the strategic efficiency will make the most effective problem solving most effective problem solving decisions using the least number of decisions using the least number of resources available. Resources can resources available. Resources can be costs, risks, time, etc.be costs, risks, time, etc.

A quality measure is also needed as A quality measure is also needed as not all resources will be equally not all resources will be equally applicable to the problem at hand.applicable to the problem at hand.

Start with ANN Defined Start with ANN Defined Strategies That Differ in Strategies That Differ in

the Data Selected and Solve the Data Selected and Solve RatesRates

Nodal Solve Rates (%) 61 65 58 47 37 36 64 64 53 48 47 42 64 70 75 57 53 51 74 83 76 59 58 53 24 80 83 72 61 59 72 81 80 58 55 56

Efficiency Index: Outcomes Efficiency Index: Outcomes Obtained vs. Resources Obtained vs. Resources

UsedUsed For example, for one strategy 9 of the 22 items For example, for one strategy 9 of the 22 items

were selected by the majority of the students and were selected by the majority of the students and with a solve rate of 1.33 the EI = 3.25with a solve rate of 1.33 the EI = 3.25

EI= * max outcome

obtainedoutcome

maxoutcome

resourcesused

resourcesavailable

Relating the Relating the EIEI to Strategies to Strategies and Problem Difficultyand Problem Difficulty

Steps for Calculating a Steps for Calculating a Strategic Efficiency Index Strategic Efficiency Index

((EIEI))1.1. Train ANN and calculate the Train ANN and calculate the EIEI for for

each node.each node.2.2. For each new performance, determine For each new performance, determine

the matching strategy from ANN, and the matching strategy from ANN, and assign the associated nodal assign the associated nodal EIEI. .

3.3. Determine if the case was solved or Determine if the case was solved or not.not.

4.4. Repeat for additional cases and Repeat for additional cases and average average EIEI and solved rate and solved rate

5.5. Calculate Quadrant Value Calculate Quadrant Value (QV)(QV)

Calculate Calculate QVQV from Nodal from Nodal EIEI and Outcomeand Outcome

A performance at a particular node can either under perform or over perform the nodal average

Final Wrinkle--Problem Sets Final Wrinkle--Problem Sets Vary in Difficulty and EIVary in Difficulty and EI

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

Elements

Math

Reactions

DensityForensics

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.90.9

A Low DataLow Outcomes

High DataLow Outcomes

Low DataHigh

Outcomes

Low DataHigh

Outcomes

The Solution: Normalize The Solution: Normalize Performances to Quadrants Performances to Quadrants Defined by the Mean EI and Defined by the Mean EI and

Solved RatesSolved Rates

Quadrant 1QIR(1)

High EfficiencyLow Outcomes

Quadrant 2QIR(2)

Low EfficiencyLow Outcomes

Quadrant 4QIR(4)

High EfficiencyHigh Outcomes

Quadrant 3QIR(3)

Low Ef

The Classes of Individual The Classes of Individual Teachers Often Show Similar Teachers Often Show Similar

QV AveragesQV Averages

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

CASE 1 CASE 2 CASE 3 CASE 4 CASE 5

0 1 2 0 1 2 0 1 2 0 1 2 0 1 20

1

2

3

4

5

0

1

2

3

4

5

6

7

8

0 1.0 1.5 2.0 0 1.0 1.5 2.0 0 1.0 1.5 2.0 0 1.0 1.5 2.0 0 1.0 1.5 2.0

CASE 1 CASE 2 CASE 3 CASE 4 CASE 5

0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

6.5 Elements Reactions

EI R

Average Solved

Individual student problem solving progress from classes of four teachers.

Reliability – Across Classroom PerformancesReliability – Across Classroom Performances

QV Changes With Student QV Changes With Student ExperienceExperience

A

B

0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.00

1

2

3

4

5

6

Case 5Case 4Case 3Case 2Case 1

0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0

0

1

2

3

4

5

6

7

8

9

Average Solved

EI R

Quadrant 1QIR(1)

High EfficiencyLow Outcomes

Quadrant 2QIR(2)

Low EfficiencyLow Outcomes

Quadrant 4QIR(4)

High EfficiencyHigh Outcomes

Quadrant 3QIR(3)

Low Ef

Construct Validity – Strategic Changes With ExperienceConstruct Validity – Strategic Changes With Experience

Concurrent Validity: Concurrent Validity: Correlations of IRT Correlations of IRT

Problem Solving Scores and Problem Solving Scores and QV Indices with QV Indices with

Achievement Scores Achievement Scores IRT

Elements Density Forensics Math Reactions CAT

Reading CAT

Language CAT Math

Elements 1.000 Density 0.159 1.000 Forensics 0.223 0.148 1.000 Math 0.228 0.052 0.156 1.000 Reactions 0.171 0.268 0.239 0.180 1.000 CAT Reading 0.335 0.100 0.479 0.331 0.308 1.000 CAT Language 0.341 0.132 0.398 0.328 0.346 0.751 1.000 CAT Math 0.430 0.234 0.389 0.332 0.332 0.675 0.665 1.000

QV

Elements Density Forensics Math Reactions CAT

Reading CAT

Language CAT Math

Elements 1.000 Density 0.157 1.000 Forensics 0.002 0.170 1.000 Math 0.063 0.081 -0.075 1.000 Reactions 0.168 0.042 0.294 0.146 1.000 CAT Reading 0.293 0.109 0.447 0.079 0.195 1.000 CAT Language 0.279 0.275 0.296 0.138 0.227 0.751 1.000 CAT Math 0.266 0.327 0.301 0.061 0.190 0.675 0.665 1.000

Are Students Problem Solving to Are Students Problem Solving to Their Abilities? … it may depend Their Abilities? … it may depend

on their teacher.on their teacher.

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

AVG

(M-S

S)

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

AVG

(M-S

S)

Correlations Between QV Correlations Between QV and Achievement Test and Achievement Test

Scores is Independent of Scores is Independent of Content DomainContent Domain

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

600

AVG

(M-S

S)

problem

Phyto Phiasco Version 1.1

Road Trip

Roots Quest Version 2

Who Messed With Roger Rabbit? Version 2

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

600

AVG

(M-S

S)

problem

Phyto Phiasco Version 1.1

Road Trip

Roots Quest Version 2

Who Messed With Roger Rabbit? Version 2

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

600

AVG

(M-S

S)

problem

Phyto Phiasco Version 1.1

Road Trip

Roots Quest Version 2

Who Messed With Roger Rabbit? Version 2

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

AVG(quadrant)

0

100

200

300

400

500

600

AVG

(M-S

S)

problem

Phyto Phiasco Version 1.1

Road Trip

Roots Quest Version 2

Who Messed With Roger Rabbit? Version 2

SummarySummary

Detailed machine learning models of Detailed machine learning models of problem solving progress can be problem solving progress can be developed.developed.

By focusing on the value of the By focusing on the value of the problem solving process these problem solving process these models can be generalized and models can be generalized and aggregated across content domains.aggregated across content domains.