The evaluation life-cycle Tristram Hooley – Postgraduate Training Co-ordinator Rob Shaw –...
-
Upload
luis-burton -
Category
Documents
-
view
233 -
download
2
Transcript of The evaluation life-cycle Tristram Hooley – Postgraduate Training Co-ordinator Rob Shaw –...
The evaluation life-cycle Tristram Hooley – Postgraduate Training Co-ordinator
Rob Shaw – Learning Technologist
Project aims
1. To produce and evaluate a high-quality online portal to provide training in online research methods;
2. To act as a self-supporting online resource to enhance understanding of both the theoretical and practical aspects of online research methods including web-based questionnaires, virtual synchronous and asynchronous interviews;
3. To draw on a wide range of successful good practice case studies, cover associated ethical issues of online research, provide important resource links and technical guidance.
The Team
Project initiated by • Clare Madge (Lecturer)• Henrietta O Connor
(Lecturer)• Jane Wellens
(Educational Developer)
Key work undertaken by• Rob Shaw (Learning
Technologist)
Project supported by
• Tristram Hooley (PG Training Coordinator)
• Lisa Barber (Cartographer)
• Bill Hickin (Senior Computer Officer)
• Julia Meek (Evaluation Consultant)
Problems with on-line resources
• Expensive;• Rarely used;• Difficult to use;• Inflexible – rarely meet our exact needs;• Frequently out of date;• Anxiety about the reliability of the information;• Frequently poorly referenced.
Conventional classroom
two way communication
On-line learning environment
Brilliant!
Fantastic!
Academically rigorous!
Interesting!
Publish on-line
one-way communication
Addressing These Problems Through Evaluation
Academic content • Academic peer review
Graphic design/look • User feedback
Usability • Heuristic (expert) evaluation
• Watching sample users (cognitive walkthrough)
Flexibility • Classroom evaluation
Designing Evaluation
• Ask the experts – speak to an evaluation consultant;
• Evaluate early and often;
• Use a range of evaluation techniques;
• Try to evaluate with a range of different users who are drawn from your target audiences;
• Act on your evaluation – redraft, redesign and re-evaluate.
Development and Evaluation
Projectidea
Project group
evaluation
Expert evaluation
Usability studies
User feedback
Design
Classroomtesting
AcademicPeer
Review
CompleteProject
Stages of Evaluation
Project group
evaluation
Expert evaluation
Usability studies
User feedback
Ongoing Regular written feedback and discussion in team meetings.
Evaluation activities completed so far:
Initial usability studies (October - November)
Heuristic evaluation Focus on the design - prioritised report produced.
Pilot with group of potential users – feedback anecdotal and via evaluation questionnaire.
Cognitive walkthrough. Observation of use of site by potential users and eliciting of immediate feedback.
Project aims
1. To produce and evaluate a high-quality online portal to provide training in online research methods
2. To act as a self-supporting online resource to enhance understanding of both the theoretical and practical aspects of online research methods including web-based questionnaires, virtual synchronous and asynchronous interviews
3. To draw on a wide range of successful good practice case studies, cover associated ethical issues of online research, provide important resource links and technical guidance.
First draft
Most recent draft
‘Informal’ team evaluationPositives :• Swift response so glaring problems are likely to
be picked up early and not embedded;• A consensus is usually ‘right’;• Forum for debate and discussion of possibilities;• Helpful for whole team to have an insight into the
process and the ongoing development.Negatives• Can be a tendency to ‘go round in circles’;• Ability to see the site with a ‘fresh eye’ diminishes over
time – Over-familiarity with the ‘rules’ of the site;• Can be difficult to be objective.
Developments inspired by the team: Before
Developments inspired by the team: After
then
Heuristic evaluation
• Heuristic - “process or method for attempting the solution of a problem”
• Heuristic evaluation carried out by experts in web design and human-computer interaction.
• Focus on the design, navigation and accessibility of the site rather than content.
• Report produced with any problems given a severity rating from 0-4.
Extract from report
Heuristic evaluation
Positives:• Navigation and design flaws highlighted;• External input is more objective;• Severity rating allows easy setting of priorities.• Provides reassurance about things that are going well
Negatives:• Need to strike a balance between carrying out formal
evaluation early enough to inform the design process, and maximising cost-effectiveness;
• Some of the points made referred to ‘known issues’;
Heuristic evaluation: Before
Heuristic evaluation: After
Cognitive walkthrough
• A structured way in which user behaviour can be observed.
• Aims to discover aspects of the site that the user finds difficult to understand.
• The user uses the site. The development team watch them and ask them to describe what they see.
Cognitive walkthrough
Positives :• Real insights gained into potential issues with new
users;• Extremely practical and easy to implement;• Encourages team members to view with a fresh eyes;• Throws up surprises.
Negatives• Can be subject to personal ‘quirks’;• Not always easy to draw generalisable conclusions;• Danger of over-reaction.
Screen shots
Before
After
User-group pilot• Introduced to a group of students on a research
methods training programme.• Students carry out a mini-package of material from
one section of the training package in their own time.• Feedback primarily via questionnaire.• E.g.
Pilot with group of potential users• Can provide a more measured response which
is useful in tandem with the rapid feedback from cognitive walkthrough.
• Allows compilation of records and more accurate means of analysis and comparison.
• Depends on people giving their time to help. Only the more motivated are likely to respond given the time required.
• Can identify individuals who are interested in getting involved or finding out more.
Stages of Evaluation
Future evaluation activities:
• Case studies with users (February – April 2005) • Small groups of target users will be followed using the site,
keeping diaries on ease of use/content. Focus groups.
• Content evaluation (February – April 2005) • Subject experts to evaluate content of completed modules
(e.g. Chris Mann).
• Formal user study (Autumn 2005) • Users will be observed using the package and asked to
provide feedback through questionnaires.
Conclusions
To avoid unpleasant surprises:
• Use a range of methods and target groups
• Evaluate as often as you can
• Act quickly upon the results