Post on 28-Mar-2015
1
Evaluating Student Learning Experiences
Michael ProsserHigher Education Academy
1
2
Academy Mission
The Academy’s mission is to help institutions, discipline groups and all staff to provide the best possible learning experience for their students.
3
Presentation based upon:
• Over 20 years research into the student learning experience in higher education in the United Kingdom, Australia, Sweden and Hong Kong
• Experience in Australia with the Course Experience Questionnaire
Interpretations of the results of evaluations of student learning experiences are not value or theory free:
• Interpretations in terms of student satisfaction
• Interpretations in terms of student learning experiences
4
Overview of the student learning perspective
Figure 1: Model of Student Learning
CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS'
PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals)
COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment)
STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep)
STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality)
5
STUDENT APPROACHES TO LEARNING
Surface Approach
Intention to reproduce
- rote memorise information needed for assessment
- failure to distinguish principles from examples
- treat tasks as external impositions
- focus on discrete elements without integration
Deep Approach
Intention to understand
- meaningfully memorise information for later use
- relate new ideas to previous knowledge
- relate concepts to everyday experiences
- relate evidence to conclusions
Approaches vary between tasks and modules
6
STUDENTS’ EPERIENCES OF THE LEARNING CONTEXT
Early research by Entwistle and Ramsden (1983) using both interviews and questionnaires identified a number of student experiences relating to the way they approached their studies
Student experiences of:
Quality of teaching – including quality of feedback
(NSS: Teaching, Assessment and Feedback))
Clearness of goals of course and standards of assessment
(NSS: Assessment and feedback)
Workload so high that it was not possible to understand everything
Assessment measuring reproduction and not understanding
were found to relate to how they approach their studies and to learning outcomes (exam results and other indicators)
7
RELATIONSHIP BETWEEN COURSE EXPERIENCES AND APPROACHES
Study of over 8000 students in first year subjects around Australia
Amongst the data collected were students responses to:
contextualised Ramsden's Course Experience Questionnaire and a
contextualised Biggs Study Process Questionnaire
1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS
8
Approaches to Study
Surface Approach
32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture
35. The best way for me to understand what technical terms mean is to remember the textbook definitions
Deep Approach
28. I try to relate ideas in this subject to those in other subjects, wherever possible
34. In trying to understand new ideas, I often try to relate them to real life situations to which they might apply.
Biggs Study Process Questionnaire
9
Student Experiences Of Learning Environment
Good Teaching
15. The staff made a real effort to understand difficulties students might be having with their work.
Clear Goals and Standards
1. It was always easy to know the standard of work expected
6. I usually had a clear idea of where I was going and what was expected of me in this subject.
Appropriate Workload
25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-).
Appropriate Assessment
8. To do well in this subject, all you really need is a good memory (-).
Ramsden’s Course Experience Questionnaire
10
Factor Analysis of Experiences of T & L Context and Approach to Study
_________________________________________________________________
Scale Factors
____________________________________
1 2
_________________________________________________________________
Experiences of Context
Good teaching .80
Clear Goals and Standards .67
Appropriate Workload -.69
Appropriate Assessment -.65
Approach to Study
Surface Approach .81
Deep Approach .73
_________________________________________________________________
Principal Components, Variamax Rotation, n=8837
11
In each subject:
1. A deep approach is associated with experiences that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback)
2. A surface approach is associated with experiences that the workload is too high and assessment tests reproduction
That is, variation in students’ experiences of the learning environment within subjects is associated with the approaches to study within subjects – within subject variation in perception is not measurement error.
1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS
1221
Comments on NSS
• Substantial amount of research linking results on surveys such as the NSS with student approaches to study and learning outcomes
• Student experiences are a function of both their prior experiences and understandings and the course design and teaching
• Do not necessarily improve student satisfaction by focusing on satisfaction - the individual items or scales – need to better understand why they responded the way they
• Spread of results – proportions responding in certain ways – better than mean of responses
• Changes is scores over time of surveys such as the NSS represent broad cultural change – departmental & institutional experiences
• Can only expect small effect sizes in changes in scores over time - .2 of a SD – if over 3 - 5 years changes of the order of .1 to .2 points (3.5 to 3.6 or 3.7).
• Follow up with more qualitative studies – interviews, open-ended written statements, focus groups to better understand experiences
13
Comments on NSS
• Substantial amount of research linking results on surveys such as the NSS with student approaches to study and learning outcomes
• Student experiences are a function of both their prior experiences and understandings and the course design and teaching
• Do not necessarily improve student satisfaction by focusing on satisfaction - the individual items or scales – need to better understand why they responded the way they
• Spread of results – proportions responding in certain ways – better than mean of responses
• Changes is scores over time of surveys such as the NSS represent broad cultural change – departmental & institutional experiences
• Can only expect small effect sizes in changes in scores over time - .2 of a SD – if over 3 - 5 years changes of the order of .1 to .2 points (3.5 to 3.6 or 3.7).
• Follow up with more qualitative studies – interviews, open-ended written statements, focus groups to better understand experiences
14
Overview of the student learning perspective
Figure 1: Model of Student Learning
CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS'
PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals)
COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment)
STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep)
STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality)
15
Recent Review of Research and Evaluation on Student Learning Experience in
E-learning
Rhona Sharpe, Greg Benfield, Ellen Lessner and Eta DeCicco,:
• Much research on design of learning experiences – teacher, course or program focused
• Much research on student observable behaviours
• Little or no research on the student learning experiences of those designs (intentions, perceptions, beliefs, outcomes etc)
• Little research on the student experience of the integration of online with face-to-face – how e-learning relates to their whole learning experience and to their lives and jobs
16
Learning Through Blended Discussion
Peter Goodyear, Rob Ellis, Mike Prosser
(acknowledge Australian Research Council Support)
Work in Progress
Student experience of learning through online and face-to face discussion
Using a phenomenographic approach to analysing student learning experiences
• Key aspects of differences in the way something is experienced – not rich, thick descriptions
• Second-order description and analysis – not describing what it is but how it is experienced.
• Categories of description
17
Investigate associations amongst:
• how the students approached their discussion within blended learning context,
• how this variation was related to the quality of what they thought they were learning through discussion, and
• their performance in the course as a whole
In-depth, semi structured interviews; short open-ended questionnaire, assessment results
Interviews: 30 minutes, transcribed, main data for construction of categories and outcome space
Short open-ended questionnaire – same questions and leading questions for interview
18
Questions:
1. What did you learn through discussion in your course? This includes all of the discussions that you were involved in in the course
2. How did you approach engaging in face-to-face discussion in your course? What sorts of things did you do to engage in the discussions? Why did you use these strategies to engage
3. How did you approach engaging in online discussion in your course? What sorts of things did you do to engage in the discussions? Why did you use these strategies to engage
Sample:
51 second your university undergraduate students studying social work in a large, research intensive metropolitan Australian University – 19 completed semi-structured interviews
19
Category Quote
A: Checking Ideas
Fragmented
Getting the teacher’s point of view … it’s good being able to talk and make sure you are really learning what you are supposed to be learning
B: Acquiring Ideas
Fragmented
It elaborates the readings even more like it sort of expands the readings a bit … Like you sort of remember it a bit more
C. Developing Ideas
Cohesive
It sort of gives you different views of what people are getting out of the readings .. I guess it gives me an appreciation that people do see things differently, that its not clear cut
D. Challenging Ideas
Cohesive
It challenges my belief, which is always good … because a belief is something that is based on knowledge and experience and you understanding of the world, and if it is being challenged you are testing it … If my beliefs are challemde I believe that my understanding of concepts is more complete
Categories of Description of Conceptions of Learning through Discussions
20
Category Online Discussion Quote Online Discussion
A
Surface
Engaging in online discussions to read all postings to avoid repetition
I tend to read all of them first. Because I tend to want to write something a bit different … and sort of stand out a little bit because I thought I would get good marks for that … But then if I read all of them as well I can reply to some of them …
B
Surface
Engaging in online discussion to use postings to add to my ideas
C
Deep
Engaging in online discussions to evaluate posting to challenge ideas
D
deep
Engaging in online discussions to evaluate postings to reflect key ideas
It just makes me think … It wasn’t really that original but it was something which I hadn’t thought before. So I mean, I didn’t respond to it because I didn’t have much to say .. It is something for me to think about and reply later.
Categories of Description of Approaches to Learning through Online Discussion
21
Conceptions of Learning through discussion
Approaches to Discussion Online
_________________________
Totals
Surface (A&B) Deep (C&D)
Fragmented (A&B) 26 0 26
Cohesive (C&D) 9 16 25
Totals 35 (69%) 16 (31%) 51
Chi-square=24.2, phi=.69, p<.001 Statistically significant and large effect size
No cases of Fragmented conception of learning with Deep approach to discussion on-pile
Student understanding of why they are engaging in discussion fundamental to adopting deep approaches to discussion online.
22
Aspects of the experience of learning through discussion
Final Mark
___________________________
Effect Size
Mean Standard Deviation
Conceptions
Fragmented 63.5 8.4
Cohesive 70.0 8.4
T test: t=2.8, p<.05 d=.77 (large)
Approaches online
Surface 64.9 8.6
Deep 70.6 8.7
T test: t=2.2, p<.05 d=.66 (medium to large)
Relationship between Conceptions, Approaches and Performance
23
• Focus on the student experience rather than satisfaction – better understand their experiences
• Comprehensive and aligned set of student evaluation instrument designed and interpreted in terms of student experiences
• Conceptual model of teaching and learning based upon the student experience underlying all developments
• Follow up with more in depth qualitative studies
• Focus on the student experience