A Lifecycle Approach to Evaluating MyArtSpace Giasemi Vavoula with Julia Meek Mike Sharples Peter...

A Lifecycle Approach to Evaluating MyArtSpace

Giasemi Vavoula

withJulia Meek

Mike SharplesPeter LonsdalePaul Rudman

[email protected], [email protected] 2 / 16WMUTE 06

Outline

What is MyArtSpace What is Lifecycle Using Lifecycle to evaluate

MyArtSpace Results: Mobile learning for schools

in museums Conclusions


MyArtSpace – Situation of concern

School visits to museums Visits often isolated from classroom work

“Learning is a cumulative process involving connections and reinforcement among the variety of learning experiences people encounter in their lives: at home, during schooling, and out in the community and workplace” (Dierking et al. 2003)

“making the links between school and museum learning explicit, genuine, and continuous affords real opportunities for school students to have enjoyable learning experiences in both settings.” (Griffin 2004)

Making the visit personal and relevant Static displays, generic labels

Structuring enquiry learning. Children need specific support in:

planning appropriate investigations managing investigations interpreting results (de Jong et al., 2006)

Design linked pre, on, and post-museum visit learning experiences

Personalise interaction with exhibits

Design enquiry-led museum learning


MyArtSpace – What it is Service on mobile phones for enquiry-

led museum learning Aim to make school museum visits more

engaging and educational Learning through structured enquiry,

exploration, connection Combines

physical space (museum, classroom) virtual space (online store and gallery) personal space (mobile phones)

Museum test sites Urbis (Manchester) The D-Day Museum (Portsmouth) The Study Gallery of Modern Art (Poole)

About 3000 children during 2006


2. At the museum: Students are given Nokia 6680

multimedia mobile phones Use phones to ‘collect’

exhibits by typing a two-letter code

Prompted to type their reason for collecting

Document their experience (take pictures, record sounds, write comments)

All exhibits and recordings are sent automatically to a personal web space

MyArtSpace – How it works1. In the classroom:

Discuss ‘big question’ to explore, by collecting evidence from the museum visit

3. Back at school: Personal website shows their

notes, recordings, pictures, exhibits

They can view others’ collections, and items provided by the museum

They can create personal online galleries to show teachers and classmates, and to friends and family outside school


Lifecycle – What it is A methodology for the evaluation of

Educational Technologies Basic premises:

Start evaluation very early in the design process – and keep evaluating throughout

Undertake evaluation activities at key points addressing questions:

Who are the stakeholders and the users of this evaluation activity?

Why is this activity being undertaken (purpose/goals)? What evaluation methods/resources will be used When should the evaluation be undertaken? How will the results be communicated?

Feed evaluation outcomes into following stages, or re-iterate previous stages

Lifecycle Evaluation Toolkit


Lifecycle – What it is (2) Shares concepts and methods with software

engineering, educational and learning technology evaluation (User-Centred Systems Design, lifecycle approach to software engineering, etc.)

Emphasises the central role of Evaluation throughout learning system design, from system conception to final assessment.

Provides a framework for planning evaluation activities and using their outcomes.

Meek (2006), Meek & Sharples (2001)Contact: [email protected]


Lifecycle for MyArtSpace - Stakeholders

The Sea: MyArtSpace design and development team

Culture Online (UK Dept. for Culture, Media and Sport): funding body

Teachers: users Students: users Local Education Authority reps: expert

users / promoters Museum curators / education experts:

users / clients / test sites Education/accessibility consultants: experts


Lifecycle for MyArtSpace – Evaluation activities

Project Phase Activity Methods

Consultation Consultation workshop Round-table discussion

Consultation Requirements analysis Workshop

Design - Build Usability of designs Heuristic evaluation

Build First user trial Field study

Build Second user trial Field study

Deploy Usability of designs Heuristic evaluation

Deploy Third user trial Field study

Deploy Stakeholder consultation

Structured interviews

Deploy Teacher evaluation Questionnaire/interviews


Lifecycle for MyArtSpace – Evaluation activities (2)

Consultation workshop Development team, funding body, evaluation team/mobile learning

experts Improvement of initial concept

Requirements workshop Development team, funding body, teachers, LEA reps, evaluation

team, museum reps, accessibility expert Educational, user and accessibility requirements

Usability evaluations Heuristic evaluations of mobile phone, student/teacher web interface,

museum curators web interface; at critical points in interface design Field studies

Full scale trial with KS2 students at Urbis; main focus: system performance, usability

Full scale trial with KS3 students at D-Day; main focus: issues from previous trial, educational experience, usability

Final full scale trial with KS3 students at D-Day; main focus: educational and user experience, system impact

Stakeholder interviews After-pilot deployment interviews; focus: great expectations –

fulfilled? Teacher surveys

Questionnaires and semi-structured interviews with teachers; main focus: what works and what doesn’t, why, perceptions and attitudes, space for improvement

Jan 05

Dec 06

Jan 06

Aug 05

Jun 05

May 05

Conception

Consultation

Design

Build

Deploy

Market

Pro

ject T

imelin

e


Lifecycle for MyArtSpace – Use of evaluation output

MyArtSpace concept

MyArtSpace

designs

MyArtSpace system

Formulate

concept

Requirements

analysis

Usability evaluatio

nsField

Studies

Evaluation activity

System development stage

Teacher evaluati

ons

MyArtSpace

marketing

Stakeholder

Consultation


Lifecycle for MyArtSpace – A focused framework

Emergent evaluation framework for mobile learning systems and services

Expectations Reality Gaps

MicroIndividual activities performed during the learning experience – focus on usability/usefulness

How do stakeholders expect the system to work technically?

How does the system work technically for stakeholders?

What were the gaps between technical expectations and reality?

How does the Micro level affect the Meso level and vice versa?

MesoEducational aspect of the experience – focus on learning breakthroughs and breakdowns

How do stakeholders expect the system to work educationally?

How does the system work educationally for stakeholders?

What were the gaps between the educational expectations and reality?

How does the Meso level affect the Macro level and vice versa?

MacroEffect on educational practice – focus on educational transformations

What was the stakeholders’ vision for the system?

What impact / transformations does the system bring about?

What were the gaps between the expected system impact and the actual impact the system appears to have?


Lifecycle for MyArtSpace – Preliminary results

Mobile learning for schools in museums: Micro level (individual learning activities)

Students enthusiastic about creating their own content: taking dozens of pictures and recording lots of audio

Text comments not very popular – in recent trial, they override text comments and record audio instead

Meso level (the learning experience) “Made me look at artwork more” Students stop before exhibits and think – should I collect? Is

this relevant? Prompted to abstract main ideas of their galleries by e.g.

asking to provide a good title Durable visit output to take away

Macro level (the educational experience) Connect museum visit to school learning (“didn’t really use

museum visits for specific purposes (before)”, “give us an easier follow up lesson to make the value of the trip more worth while”)

Increase engagement


Lifecycle for MyArtSpace – Preliminary results (2)

Example inter-level influences (micro-meso): ‘Aggressive’ collecting: the more open and exploratory the

learning task is at the meso level, the more overuse is made of collection activities at the micro level…

… which makes harder the post-visit manipulation of collections at the micro level, which in turn hinders the learning experience at the meso level

learning task needs to be well designed and appropriately constrained

pre, on and post museum experiences need to be allocated enough time

No explicit link between pictures and comments at micro level results in difficulty in interpreting pictures post-museum at the meso level

Functionality needed: annotate picture with audio/text comment


Conclusions MyArtSpace:

Encourages students to stop and think about each exhibit Engaging, fun Consolidates formal and semi-formal learning experiences MyArtSpace is being re-branded as OOKL for commercial

release Lifecycle:

Focus on both usability and pedagogy early in the design Helped draw a full picture of the use of new technology to

be understood, assessed and improved Involved stakeholders in evaluation activities early on,

making it everyone’s business to provide input to evaluation activities and make use of the outcomes!


Partners - contacts Culture Online (Department of Culture, Media and

Sport) http://www.cultureonline.gov.uk/

The SEA – software development http://the-sea.com

University of Birmingham (CETADL) University of Nottingham (LSRI) Urbis, Manchester Study Gallery, Poole D-Day Museum, Portsmouth Lifecycle Ltd

Contact: Julia Meek ([email protected]) Partners: Julia Meek, Peter Lonsdale, Paul Rudman, Mike

Sharples, Giasemi Vavoula (MAS evaluation team leader), <other two>

A Lifecycle Approach to Evaluating MyArtSpace Giasemi Vavoula with Julia Meek Mike Sharples Peter...

Documents

Transcript of A Lifecycle Approach to Evaluating MyArtSpace Giasemi Vavoula with Julia Meek Mike Sharples Peter...