The Use of Screen Recorders to Achieve More Informative Assessment of Students’ Understanding.

Deborah Abdel Nabi & Paul RogersPsychology School of Health & Social SciencesUniversity of Bolton

Aims of Presentation

• Introduce a new computer-assisted assessment method recently developed in Psychology at the University of Bolton– Project funded by University TQEF sabbatical (2007)

• Highlight potential uses across the psychology curriculum.

Research design and data analysis (RDA) for BSc (Hons)

Psychology courses.

University of Bolton ‘Year 1 - Introduction to Research Methods In Psychology’

Year 2 - Research Methods II and Further Methods

Year 3 – Experimental Report for Honours’ Project

Conventional methods of teaching and assessing RDA

Lecture on Research DesignLecturer explains a particular research design

Lecture on Statistics

Overview of appropriate tests to analyse the data

Statistical Analysis Workshop

Traditionally at UoB Year 1stats analysis conducted using

calculator and formula

3 years ago restructured RDA in Year 1 (re-focus

emphasis on research design rather than calculation of

stats)

Students taught Statistical Package for the Social

Sciences (SPSS) Software from Year 1

Highly structured and detailed handouts for workshops

Module Assignment (s)

• Submit output file from an analysis (e.g. descriptives)• Answer questions on the results (interpreting output)

Problems

• How good are they at navigating around SPSS?

• How much do they really understand in the output ?(Tend to answer questions with ‘text-book’ phraseology)

• Did they get a friend’s copy?

Problem (ctd): Negative skew in grades

30 40 50 60 70 80 90 100

MarkAssignment1 2006-7 (Hard Copy)

0

10

20

30

40

50

60

Fre

qu

ency

Mean = 87.26Std. Dev. = 12.023N = 164

Problems (ctd): Lack of Relationship with other Measures of RDA Knowledge

Grades didn’t correlate with their score on a 30-item MCQ

test on RDA

Correlation 2006/7 (Hard Copy Submission of Assignment 1)

1 .095

. .112

164 164

.095 1

.112 .

164 164

Pearson Correlation

Sig. (1-tailed)

N

Pearson Correlation

Sig. (1-tailed)

N

MarkAssignment1

MCQs

MarkAssignment1 MCQs

Conclusions

New assessments of introductory RDA using SPSS were

better than previous but had room for improvement

(lacked informational value re. the depth of their

understanding)

Solution?

Could have incorporated more (and more specific

questions) on the analysis in written format BUT

• Can’t get people to write answers to everything we wanted to evaluate eg:– competency and fluency at data input,– variable specification – navigation around software

• Suspected there would be a tendency to stick to taught academic linguistic style when writing answers

We wanted to provide a means of promoting answers in

the students own words

i.e. wanted to recruit the truly generative processes

required for oral explanation

• Better comprehension (e.g. Muller, Sharma & Reimann, 2006)

• Better recall (e.g. Bekerian & Dennett, 1988; Goolkasian & Foos, 2002)

• Authenticity of understanding (e.g.Zhou Qi-jia, 2006)

Need means of getting students to demonstrate

and articulate their understanding

One-on-one observation / examination?• Impractical• Intimidating

The Idea

Screen Recorder Software

Record narrated movies of the content of your PC screen

Windows Media Encoder

The Project (Semester 1 and 2 2007/8)

First assessment in ‘Introduction to Research

Methods in Psychology’ students were instructed how to

create a small (approx 5 minute) WME movie of them:

• inputting data into SPSS to a specified format and• explaining why they are taking particular steps in order

to perform a given analysis and;• providing a description of (i.e. interpreting) the output

file

• Upload the file to WebCT for marking

• Observe a ‘model answer movie’

Example of a Movie File Created by A Student

Proposed Potential Impact of the Project

Proposed Impact on Student Learning

Practice

(To produce good movie)

Increased engagement

Deeper encoding

Easier retrieval

Deeper comprehension

Multi-Modal

(Visual, auditory and kinaesthetic)

Encoded on multiple levels

(Brunye, Taylor and Rapp,

2007).

Learning advantages Rich memories

Easier retrieval

Suits all learning styles

Observation of Performance

Watch and hear own knowledge repeatedly in

own words(Muller, Sharman and

Reimann, 2006)

Provides opportunity for reflective learning and possibly / enhanced

metacognitive awareness

Feedback

(Model answer)

Detect differences between their submission

and model answer

Increases metacognitive awareness and

updates concepts

Results

Effect on The Assessment Technique on Mark Achieved for Assignment 1

Frequency Distribution of Marks Across Cohorts

30 40 50 60 70 80 90 100

MarkAssignment1 2006-7 (Hard Copy)

0

10

20

30

40

50

60

Fre

qu

ency

Mean = 87.26Std. Dev. = 12.023N = 164

20 40 60 80 100

MarkAssignment1 2007-8 (Movie)

0

5

10

15

20

25

30

Fre

qu

en

cy

Mean = 72.06Std. Dev. = 16.989N = 145

Correlation 2006/7 (Hard Copy Submission of Assignment 1)

1 .095

. .112

164 164

.095 1

.112 .

164 164

Pearson Correlation

Sig. (1-tailed)

N

Pearson Correlation

Sig. (1-tailed)

N

MarkAssignment1

MCQs

MarkAssignment1 MCQs

Correlations 2007/8 (Movie Submission of Assignment 1)

1 .231**

. .003

145 136

.231** 1

.003 .

136 143

Pearson Correlation

Sig. (1-tailed)

N

Pearson Correlation

Sig. (1-tailed)

N

MarkAssignment1

MCQs

MarkAssignment1 MCQs

Correlation is significant at the 0.01 level (1-tailed).**.

Conclusions

When students are required to:• Demonstrate their knowledge / competency of research

design and SPSS and • Articulate their understanding of the output of stats

tests in their own words

• Mean mark drops and distribution not so negatively skewed.

• Not ‘perceived’ as competent as when just a hard-copy submission of an analysis is required.

Conclusions

Movie is more representative of their actual RDA

knowledge and understanding than traditional hard copy

submissions.

This is more informative for both tutor and student and

allows for better feedback and early intervention.

Feedback from Students

Feedback Questionnaire Results

Did they rehearse more?

1 = 1-2 times

2 = 3-6 times

3 = 7-10 times

4 = 11-15 times

5 = 16-25 times

1 2 3 4 5

No of Times Practiced Prior to Recording

0

2

4

6

8

10

12

14

1

13

9

2 2

Mean = 2.67Std. Dev. = 0.961N = 27

Was the multimedia format useful for learning?

2 3 4 5 6 7

Opportunity for Verbal Rehearsal

0

2

4

6

8

10

12

1 1 1

2

11 11

Mean = 6Std. Dev. = 1.271N = 27

2 3 4 5 6 7

Audio Playback is Useful

0

2

4

6

8

2

3

5

8 8

Mean = 5.58Std. Dev. = 1.447N = 26

4 5 6 7

'Doing' Something was Beneficial

0.0

2.5

5.0

7.5

10.0

12.5

2

3

12

10

Mean = 6.11Std. Dev. = 0.892N = 27

1 2 3 4 5 6 7

Visual / Graphical Feedback Is Beneficial.

0

2

4

6

8

1 1

2

8 8

4

2

Mean = 4.58Std. Dev. = 1.391N = 26

Was Model Answer Movie Useful Form of Feedback?

3 4 5 6 7

How useful was the model answer movie?

0

1

2

3

4

5

6

7

2 2

4

7

4

Mean = 5.47Std. Dev. = 1.264N = 19

1 = Not useful at all

7 = Very useful

Movies watched: 421 times 173 hours

Overall Conclusions and Future Work

• Performance on this assignment is a good (better)

reflection of research design and stats understanding

• It persuades them to engage more with the assignment

• Students generally found it useful and enjoyable.

• The multimedia format is perceived to be beneficial– Dual (triple) representation– Integrative working memory– Generative processes aid comprehension

Future Work

The usefulness of method has been consolidated by

increasing the amount of movie assignments in the

RDA module to two (and more advanced methods

modules)

Real possibility for extending this assessment technique

to e.g.

Honours Dissertation (RDA presentation movies)

and other modules such as Biopsychology

Enhance private study and revision

Own narrated recording of lecturer’s PP or create movies of mind maps with their own narration

Foundation component of a embryonic PLE

The End

Any questions?

Help to improve confidence?

1 = Didn’t improve it at all

7 = Improved it a lot

1 2 3 4 5 6 7

How much has using WME improved your confidence in using SPSS?

0

2

4

6

8

10

12

1 1 1

3

12

5

4Mean = 5.04Std. Dev. = 1.427N = 27

Did they enjoy using WME?

1 2 3 4 5 6 7

How much did you enjoy using WME?

0

1

2

3

4

5

6

7

1

2 2

4

7 7

4

Mean = 4.89Std. Dev. = 1.625N = 27

1 = Not at all

7 = I enjoyed it a lot