+

Seeing the elephant: Using mixed methods

to understand complex learning

Cindy E. Hmelo-Silver

Rutgers University

+Indian Parable:

The Blind Men and The ElephantThe First approached the Elephant,

And happening to fall

Against his broad and sturdy side,

At once began to bawl:

"God bless me! but the Elephant

Is very like a wall!"

The Second, feeling of the tusk

Cried, "Ho! what have we here,

So very round and smooth and sharp?

To me `tis mighty clear

This wonder of an Elephant

Is very like a spear!"

The Third approached the animal,

And happening to take

The squirming trunk within his hands,

Thus boldly up he spake:

"I see," quoth he, "the Elephant

Is very like a snake!” (Saxe, n.d.)

+Overview

Complex Learning

Design-based Research

Example 1: STELLAR

Example 2: VMC

Discussion

+What is Complex Learning?

“True knowledge −− understanding −− develops through

exploration, rumination, interpretation, judgment, and the

application of information.” (Hawkins, 1997)

“Complex learning aims at the integration of knowledge,

skills, and attitudes; the coordination of qualitatively different

constituent skills; and the transfer of what is learned to daily

life or work settings.” (van Merrienboer, Kirschner, & Kester,

2003)

+Complexity of Learning Environments

The strengths of design studies lie in testing theories in the crucible of

practice; in working collegially with practitioners, co-constructing

knowledge; in confronting everyday classroom, school, and community

problems that influence teaching and learning and adapting instruction

to these conditions; in recognizing the limits of theory; and in capturing

the specifics of practice and the potential advantages from iteratively

adapting and sharpening theory in its context. (Shavelson et al, 2003)

Its not just the technology….

Pedagogy

Curriculum

Participant structures

AND

Technology

+Answering the question “under what

circumstances”

“What works” is underpinned by a concern for “how, when, and

why” it works, and by a detailed specification of what, exactly, “it”

is. This intimate relationship between the development of theory

and the improvement of instructional design for bringing about

new forms of learning is a hallmark of the design experiment

methodology.

(Cobb et al, 2003, p. 13)

+Design Experiments (Brown, 1992)

+ Supporting new forms of learning

Conducted in a limited number of settings

Example of broader class of phenomena

Embodied conjecture (Sandoval, 2004)

+DBR Cycle

•Comparison of enactments

•Micro cycles within enactments

•Theory

•Embodied conjectures

•Aspects of design

•Documenting context

•Attend to unexpected

•Documenting Learning

• Embodied Conjecture

• “t”theory

What? How?

Why?Revise

?

+Mixing Methods

Drowning in data, e.g.,

Video

Artifacts

Pre and post tests,

Qualitative

Ethnography

Interaction Analysis

Grounded theory

Quantitative

Descriptive

Inferential

Statistical modeling

+

Example 1: STELLARWith Sharon Derry, Anandi Nagarajan, Ellina Chernobilsky

+Adapting PBL to Teacher Education

Initial implementation (Hmelo-Silver, 2000)

Paper cases

One wandering facilitator for 6-7 groups

Limitations

Cases were oversimplification

One wandering facilitator for 6-7 groups

Difficulty identifying fruitful learning issues

because of limited and variable prior knowledge

+STELLAR Learning Environment Goals:

Provide perceptually rich video cases of practice

Affordances for meshing conceptual knowledge with terrain of practice

Support students in identifying learning issues and beginning self-directed learning

Extend skilled facilitation resources

Implementation

Knowledge Web (KW): Learning sciences hypermedia designed according to Cognitive Flexibility Theory (Spiro et al, 1992)

Video cases indexed to KW

PBL online activity structure

+ Knowledge Web

+Videocase Library

+Example Problem

+Group

whiteboard

+Findings (Derry et al, 2006)

Pre-post across institutions using video analysis task

Demonstrated significant gains from pre- to post test for both sites, despite differences in implementations

Quasi-experimental design over 3 semesters

Participants: All pre-service teachers taking Educational Psychology

STELLAR PBL students

126 students from Educational Psych subject pool

Tracer concepts “Understanding”

Moderate to large effects over the three years on quantitative measures

Between group variability striking

+Visual Representations for

Contrasting Case Analysis (Hmelo-

Silver et al, 2008)Group 1

0

5

10

15

20

25

30

35

40

45

50

300 400 500 600 700 800 900 1000 1100

Lines

Codes

Facilitator

CHS

Ann

Fran

Cathy

Fauna

Luke

Other monitoring

SDL

Group monitoring

Individual monitoring

Grounded beliefs

Personal beliefs

Elaborations

Explanations

Transforming

Elaborated telling

Telling

Acknowledgement

Summary

Disagreements

Agreements

Modifications

New Ideas

Metacognitive questions

Explanation questions

Information questions

Personal talk

Concept talk

Tools as help

Tools as a problem

Task talk

View Other Proposals

Research Library

White Board

Discussion Board

Notebook

KW

Video

+Contrasting Cases (cont’d)

Group 2

0

5

10

15

20

25

30

35

40

45

50

450 550 650 750 850 950 1050 1150 1250 1350 1450

Lines

Codes

Facilitator

CHS

Matt

Bob

Carla

Caitlin

Liz

Helen

Other monitoring

SDL

Group monitoring

Individual monitoring

Grounded beliefs

Personal beliefs

Elaborations

Explanations

Transforming

Elaborated telling

Telling

Acknowledgement

Summary

Disagreement

Agreement

Modifications

New Ideas

Metacognitive questions

Explanation questions

Information questions

Personal talk

Concept talk

Tools as help

Tools as a problem

Task Talk

View Other Proposals

Research Library

White Board

Discussion Board

Notebook

KW

Video

+But what about other concepts?

Similar type of rubric developed to measure “transfer”

(Hmelo-Silver et al, 2009)

Components of transfer rubric:

1. requires understanding,

2. involves activating appropriate prior knowledge and

applying something learned in a new situation,

3. involves abstraction and cognitive flexibility,

4. can be near or far transfer, and

5. can be preparation for future learning.

+Scoring Scale

0 “Knows nothing.” No evidence that any aspect of the concept is understood or attended to, or

evidence that concept is rejected or not understood. Concept very unlikely to be used

correctly in planning or implementation unless student teacher receives and is open to

intensive assistance.

1 “Needs substantial scaffolding.” Indicate that there is some limited understanding and

acceptance of idea and that limited range of acceptable implementation of idea is occurring.

However, there are major omissions, weaknesses, or misunderstandings in relation to the idea,

and the student teacher will probably need substantial assistance to help design and

implement the idea successfully.

2 "Demonstrates early expertise.” Indicate idea is likely understood with some range and depth

and is being implemented with at least moderate success as conceptualized. However, there

are some weaknesses or omissions that should be addressed, and this part of the student

teacher’s work could be improved in important ways with some assistance.

3 “Expert.” Evidence that idea is well conceptualized in depth, detail, over a range of uses

(given limits of current assessment context—type of assignment, word limits, etc.) and is

being implemented successfully and reflectively with sophisticated understanding, even

though improvements might still be possible. Encouragement and positive feedback but little

assistance would be appropriate.

+Quantitative Results

Pretest and Posttest Scores by Class Type

Class N Pretest Posttest

STELLAR 33 0.71 (0.31) 2.02 (0.69)

Traditional 37 0.61 (0.36) 0.68 (0.34)

F (1,67) = 114.323, p < .001, d=2.55

+Qualitative results: Understanding why

Large variability in groups

Examined STELLAR whiteboards for contrasting cases

analyses

Engagement with “Transfer”

Group A

6 female students who had some difficulty

Mean gain= 1.40, SD=0.89

Group B

6 female students, rarely needed any assistance

Mean gain= 1.33, SD= 0.61

+

Group A

Discussed transfer in 3 of 4 problems

In Problem 1, Jenny proposed explanation for enduring

understanding that child in video developed:

“In the case of Brandon, he needed to have an understanding

of how and why he was able to solve the block problem in

order to transfer his ideas onto the pizza problem. "The first

factor that influences successful transfer is degree of mastery

of the original subject" (How People Learn, 53). Brandon was

able to continue to solve such a problem because of his

complete understanding of how he was able to arrive at the

solution for the block problem.”

+Group A: Encapsulating

Knowledge

In problem 2, Rina used the concept of transfer in thinking about assessment as she offered this proposal:

…The portfolio should have a final summary of the students' work and questions regarding the students' learning, so that the students can explain and evaluate their own thinking. (knowlege web [sic]) The students should be able to transfer their prior knowledge of concepts such as force and motion in order to create their vehicle, while also allowing the activity to expand on that knowledge. …another important facet of understanding is application (sic). Ms. Baker will know whether the students acquired enduring understanding by how much they can apply this knowledge to real world problems. One way of doing is to have Ms. Baker create another problem that will use the same concepts in a real world setting, and evaluating whether the students were able to apply the concepts they had learned.

+Group B: Getting started

All students mentioned transfer in P1; 2-4 students in subsequent problems

Kathy wrote:

…Second, Brandon was able to recognize a connection between the pizza problem and the tower problem that he did weeks earlier. Moreover, he made this connection relatively quickly and without much effort. He was able to show us, by using his chart and the manipulatives (blocks), exactly how the pizza problem mapped into the tower problem. His understanding of the pizza problem therefore facilitated a new, and deeper, understanding of the block problem; this process is called transfer. Brandon’s seemingly effortless use of transfer provides evidence that he understood the problem, because “transfer and wide application of learning are most likely to occur when learners achieve an organized and coherent understanding of the material.” (How People Learn, p. 238-239)…

+Group B: Moving Along

On later problems, fluid application of concept part of shared understanding

Micki writes:

…another way to look for enduring understanding would be the students' transfer and application of principles of force and motion especially to real world situations. This would show the student's understandings of information previously and transfer it to the problem at hand, which is a real world problem that allows students to work with hands-on material.

Mimi followed this up by incorporating Micki’s comment and a previous proposal from another group member:

To put these two ideas together, [t]he teacher could bring together individual explanation and transfer as evidence of enduring understanding. An activity could be created at the end of each project that would ask the individual members of the group to use the principles gained to explain a real world scenario. Likewise, an activity could be designed to facilitate transfer of the instructional objectives. For instance, one of the objectives was learning the scientific inquiry process. The teacher could present a real world problem that would require the students to use the same scientific process to solve. (This would also facilitate transfer)

+Conclusion

Quantitative methods can show what students learn Pre-post tests

Experimental, quasi-experimental designs

Quantifying verbal data can show how students learn CORDTRA representations can support interpretation

Other kinds of qualitative data analysis can explain how learner engagement affects what they learn

All of these are part of a program of design-based research

+

Example 2with Carolyn Maher, Marjory Palius, Grace Agnew, Robert Sigley,

Chad Mills

www.videomosaic.org

+Video Mosaic Collaborative (VMC)

Preserves a major video data collection on student reasoning From diverse schools settings

To be available as open source

From 40 doctoral dissertations

Makes available new tools for Teachers

Educators

Researchers

From longitudinal/cross sectional studies spanning 25 years

Videos following the same student cohort from elementary school through high school and beyond

Over 4500 hours of video

+Video Mosaic Collaborative

(VMC)

+Design-based Research for Teacher

Professional Development

In-service and pre-service teachers

Primary, middle and secondary mathematics

Counting and fractions strand

At all sites pre and post: Video assessment recognizing forms of reasoning

Content assessment

Belief assessment

At selected sites In-depth qualitative analysis

+Theoretical Perspective

Importance of making sense of students’

conversations and how tools mediate learning

(Hmelo-Silver, 2003)

Being aware of the contextual resources

(media, other Ss, prior experience) that Ss

use influence collaborative knowledge

construction (Arvaja et al., 2006)

Attending to social interactions in collaborative

knowledge construction (Palincsar, 1998)

35

+Results for Video Assessment:

Counting Strand

+An Online Enactment: Using

Resources for Reasoning

Graduate mathematics education hybrid course

Four online groups (three, 6 Ss; one, 7 Ss)

Two + week unit In class problem solving

Individual study of videosand related readings

Group discussion questions

Online Design eCollege CMS

Streaming video / linked papers

Minimal online instructor intervention

Data Postings from online

threaded discussions

Pre and post tests (math, Ss reasoning)

+Research Questions

To what extent do videos and readings promote online

discussion within and across groups?

How, if at all, do learners relate practice to online

discussion?

What evidence, if any, is there of Ss enjoyment in

studying readings and videos?

To what extent do Ss relate videos to readings in their

online discussions?

38

+Coding

All posts coded for comments related to

Video (V)

Readings (R)

Additional sub categories of comments relating

videos/readings to:

Own problem solving (PV/PR)

Others’ problem solving (OV/OR)

Earlier interventions (EV/ER)

Affect (AV/AR)

Practice (TV/TR)

39

+Video: Ankur’s Challenge

Shows two groups of 10th graders (2 in one & 3 in other) working on the problem:

How many different block towers can be built, four tall, selecting from three colors of blocks such that the towers have at least one block of each color?

Approximately 8 minutes

http://hdl.rutgers.edu/1782.1/rucore00000001201.Video.000062055

40

+

Romina’s Proof

+ Discussion Questions

(1) Describe Romina’s strategy for solving the Ankur’s

Challenge problem.

(2) In your opinion, is this solution a convincing one?

Why or why not?

(3) According to the Yackel & Hanna chapter, both von

Glaserfeld and Thompson equate reasoning with

learning (p. 227). From this perspective, in what

ways do explaining and justifying contribute to

learning mathematics?

+

+Summary of Quantitative Analysis

Across all groups: Studying videos generated reflections about own and

classmates’ problem solving

Studying videos of students’ reasoning was enjoyable

But still left us with question of ways in

which which Ss related resources to their

practice

+Connections

+

Bringing New Tools on

Board: The

VMCAnalytic

Allows users to create multimedia

artifacts using the VMC repository

Create narrative with video for purpose

(e.g., research, professional

development)

Used in a variety of ways by instructors,

researchers

Needed to be able to classify and

identify what differentiates high quality

and low quality “analytics”

Data sources: 27 VMCanalytics from

several different classes and

researchers

VMC Analytic by Hmelo-Silver (2011)

VMC Analytic by Horwitz (2011)

+What do different methods tell us?

Class Math Depth

LS Depth Clarity Coherence

Design-based Research 0.78 1.00 1.22 1.33

Intro to

Math Ed 1.96 1.80 2.36 2.20 Critical

Thinking 2.00 2.40 2.30 2.00

Practicum 3.00 2.25 2.63 2.63

No Class 1.17 1.67 3.00 3.00

Quantitative methods tells us about

broad strokes

+Qualitative methods

Looking deeper with contrasting case analysis

Example 1: Analytic illustrating how students can move from particular to general

Concepts from both learning sciences and mathematics clearly articulate

Indicative of designer's understanding of students’ learning trajectory

Example 2: Analytic illustrates teacher questioning during early algebra exploration

Students claims not supported by video segments selected

Textual descriptions of events were vague

Video not well chosen for intended purpose of relationship of teacher questioning and student engagement

+Word Clouds

Analytics coded for emergent

themes

Explored use of word clouds

as a learning analytic

Mathematics Education

ideas

Learning Sciences ideas

Allow us to see dominant

themes within the two areas of

interest

+Synergy

Quantitative and qualitative methods answer different questions

Help move closer to seeing the whole elephant

As part of design research, answer different questions for example,

What students learn

How enactments differ

Ways resources are used in different enactments

How tools are used

In all these questions, both have value

Providing the broad brush

Providing rich detail

Guidance for redesign of theory and learning environement

+Challenges in DBR in Complex

Learning Environments

By their very nature, design studies are complex, multivariate, multilevel, and interventionist, making warrants particularly difficult to establish (Shavelson et al, 2003)

Data overload and responding to emergent questions while trying to stay systematic and focused

Resource challenges

Need to expertise in range of research methods

Documenting design decisions and rationales in the heat of the moments

Staying accountable to both theory and practice

+Questions?

Email: cindy.hmelo-silver@gse.rutgers.edu