WCSE 2015

Faculty & Students as collaborators, co-creators and makers

Simon.Bates@ubc.ca@simonpbatesbit.ly/batestalks

Photo: Wikipedia

Edwin Thompson Jaynes

1922-1998

Photo: Wikipedia

Jaynes, E. T., 1993, `A Backward Look to the Future, ' in Physics and Probability, W. T. Grandy, Jr. and P. W. Milonni, Cambridge Univ. Press,

“But it required a few years before I perceived what a science teacher's job really is.

The goal should be, not to implant in the student's mind every fact that the teacher knows now; but rather to implant a way of thinking that will enable the student, in the future, to learn in one year what the teacher learned in two years.

Only in that way can we continue to advance from one generation to the next.”

Jaynes, E. T., 1993, `A Backward Look to the Future, ' in Physics and Probability, W. T. Grandy, Jr. and P. W. Milonni, Cambridge Univ. Press,

Overview

How things are changing; technology as an example driver

Implications for instructors; the anatomy for future success

The case for students as co-creators, and four examples of how to enact that

CC BY-NC-SA 2.0 https://flic.kr/p/2ZdABF

Technology - scale and pace

Slide credit: Eric Grimson (MIT)

Technology - scale and pace

Slide credit: Eric Grimson (MIT)

Technology - reach and unbundling

Technology - reach and unbundling

Technology - disruption

Graph extracted from http://vikparuchuri.com/blog/on-the-automated-scoring-of-essays/

Technology - implications

Changing the

what, where, when, how, from whom and with whom

of many aspects of life, …and learning is included

Technology - implications

So what are the

we need to embrace, develop and refine?

skills, values and habits

Students as collaborators, co-creators

Four examples (of increasing complexity to implement)

1. “Things I wish I’d known”

2. The most important course you can take

3. Undergraduate learning assistants in the classroom

4. Student-generated assessment and content

“Things I wish I’d known”

“Things I wish I’d known”

The most important course you will take

The most important course you will take

The most important course you will take

Undergraduate learning assistants in the classroom

Undergraduate learning assistants in the classroom

S. Pollock, 2007 PERC Proc. 951, p.172

Undergraduate learning assistants in the classroom

S. Pollock, 2007 PERC Proc. 951, p.172

Undergraduate learning assistants in the classroom

S. Pollock, 2007 PERC Proc. 951, p.172

Students as producers of assessment content for learning

bit.ly/batestalks

Students as producers of assessment content for learning

bit.ly/batestalks

Selected results and analysisEngagement - how do students use the system?

Benefits - what is the impact on learning?

Question quality - how good is what students produce?

Relevant publications:

Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002 (2014)

Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education, 1-15 (2014).

Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) 10, 020105

Student-generated assessment - Education in Chemistry (2013) 13 1

Quality of student authored content

Bloom’s Taxonomy of levels in the cognitive domain

Score Level Description

1 Remember Factual knowledge, trivial plugging in of numbers

2 Understand Basic understanding of content

3 Apply Implement, calculate / determine. Typically one-stage problem

4 Analyze Typical multi-step problem; requires identification of strategy

Evaluate Compare &assess various option possibilities; often conceptual

Synthesize Ideas and topics from disparate course sections combined. Significantly challenging problem.

Text

Question quality

0%

5%

10%

15%

20%25%

30%

35%

40%

45%

50%

1 2 3 4 5 6

Taxonomic Category

Per

cent

age

of S

ubm

itte

d Q

uest

ions

First semester N = 350

Second semester N = 252

Explanation quality

0 Missing No explanation provided or explanation incoherent/irrelevant

1 Inadequate Wrong reasoning and/or answer; trivial or flippant

2 MinimalCorrect answer but with insufficient explanation/justification/ Some aspects may be unclear/incorrect/confused.

3 Good Clear and detailed exposition of correct method & answer.

4 ExcellentThorough description of relevant physics and solution strategy. Plausibility of all answers considered. Beyond normal expectation for a correct solution

0 1 2 3 4 0 1 2 3 40

20

40

60

Num

ber o

f que

stio

ns

Assessment 1 Assessment 2

Explanation Quality

Question quality summary (UoE 2011)

2 successive years of the same course (N=150, 350)

‘High quality’ questions: 78%, 79%

Over 90% (most likely) correct, and majority of those wrong were identified by students.

69% (2010) and 55% (2011) rated 3 or 4 for explanations

Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.

That’s not commonBottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-361 (2011)

107 Year 2 biochem students 56 / 35 / 9 % of questions in lowest 3 levels.

Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)

“9,713 assessment items submitted by 50 instructors in the United States reported that 93% of the questions asked on examinations in introductory biology courses were at the lowest two levels of the revised Bloom’s taxonomy”

Students as producers of assessment content for learning

Why not short answer Qs?

Why not …. anything?

Students as producers of assessment content for learning

Why not short answer Qs?

Why not …. anything?

Students as producers of assessment content for learning

Why not short answer Qs?

Why not …. anything?

Test Kitchen: Adaptive Comparative

Judgement

Students as producers of assessment content for learning

Why not short answer Qs?

Why not …. anything? LEARNING OBJECTS

Test Kitchen: Adaptive Comparative

Judgement

PHYS101: Energy and Waves

PHYS101: Energy and Waves

PHYS101: Energy and Waves

PHYS101: Energy and Waves

PHYS101: Energy and Waves

Implementation logistics Cohort split into 4 groups

Each week one group tasked with creating LOs

Each submission counts for 2.5% of final grade

Repeat cycle twice per Semester

Students can submit >2 LOs & receive grade for best 2

Short survey on submission

Students encouraged to apply CC licenses

Results: engagement0 100 200 300

LO 1

LO 2

LO 3

LO 4

LO 5

LO 6

LO 7

LO 8

Number of students

AssignedOptional

Results: time on task

0 100 200 300 400

Less than 0.5h

0.5 to 1 h

1 to 2h

2 to 3h

3 to 4h

4 to 5h

More than 5h

Number of students

Results: self-reported change in understanding

0 200 400 600 800

None

Little

Moderate

Good

Excellent

Number of students

0200400600800

Number of students

before creating it after creating it How much did you understand the topic your LO was based on

PHYS101: Energy and Waves

3. Successes

http://youtu.be/BObyt_NJYrE

Sample 2 - Standing Wave in a bowl

Sample 2 - Standing Wave in a bowl

Sample 3 - Colour Loss Underwater

Student generated exam content

before creating it after creating it How much did you understand the topic your LO was based on

Resources for use in class

http://blogs.ubc.ca/phys101

Not quite the whole story• Despite these outstanding examples, many students

didn’t like the assessment

• difficulty level vs other assessed components of the course

• credit weighting

• Students dropped these assessments more than other coursework

• Strange ‘phase transition’ for LO vs exam grades

One final thought….why this all works