Experience-driven evaluation methods for mobile cultural applications

Areti Galani Media, Culture, Heritage (MCH)Newcastle University areti.galani@ncl.a

c.uk

Motivation:

Our experience of heritage/art/culture is progressively mediated by digital applications delivered on mobile platforms.

Embedded in our everyday life Every use of heritage/art apps (and

indeed any kind of technology) subtly changes/shapes our own sense of heritage/art/culture and so forth.

Evaluating how we experience heritage/art/culture with or through technology is fundamental in understanding heritage experience(s) at large.

A ‘wicked problem’:

How do we experience heritage / art / culture with or through mobile applications? How do we evaluate the ‘felt experience’ of heritage/art apps?

Wright, Mccarthy, Meekison (2004)

‘Making sense of experience’, in Blythe et al. (eds) Funology (p. 48)

“People do not simply engage in experiences as ready-made, they actively construct them through a process of sense making. This process of sense making is reflexive and recursive.”

Designing an experience vs. designing for experience

Experience:

Situated Personal / private Dynamically constructed – unpredicted

Rock Art on Mobile Phones(RAMP) web app

I’ve found it!

...to engender speculation and sense of place

Second Moon art app (iOS, Android)

...to intrigue and delight

Rock Art on Mobile Phones

Research project at Newcastle University Funder: Arts and Humanities Research

Council 3 web apps for 3 rock art sites in rural

Northumberland + a desktop site Launched in Lordenshaw (near Rothbury)

in July 2011 Participatory design approach 7,011 sessions (71.89% new sessions);

5,059 users; at least 290 new users in-situ

Objective: To evaluate whether the app supported the discovery of rock art, speculation about its meaning and sense of place.

Evaluation approach:

10 self-selected study participants who had not used the apps before (2 solo and 4 groups of 2 visitors)

Shadowing of users on site using the app Debriefing qualitative interviews with

image elicitation Mind maps based on prompts

The carver Neolithic people Sounds of the land Change The views

Using the app to discover rock art panels

I liked the sense of discovery. When you're looking at-you've got the little map and you actually find the things. [i.e. rock art panels]And some of those things are quite hard to see, when you're actually in front of the rock. (P1)

In some weird way it almost becomes like a little adventure to find it, you know? We became, like, really determined to find it, kind of like, “Where is it? I gotta find it”, so I think people will look for it if they know that one's there. (P3)

Integrating the app in the exploration of cup and ring marks

Yeah, it wasn’t immediately obvious to me where the ring marks were. You could see one clearly and that was because of the first picture, you knew what you were looking for. They are quite good representational of the rock you’re looking at – you tell by the shape of the rock where you are. Even though it’s not on the rock before you go to the map, but when you look at the map it’s much more obvious where the ring patterns are. And you find quite a lot of them. (P1)

Recognising own internal responses in the dialogic content of the app

P4: When she was saying, ‘That could be, I think that was more ritual or, you know, in the woods,’ but he was like, ‘Ah, but if you think about it, the wood wouldn't have been here-‘ rather than, ‘definitely wasn't’.  P3: Absolutely. The fact that, like, she was making comments about things that I had already made comments on, you know, just like what you said – “I can't believe it, people were here, making some of these things six thousand years ago”, that literally, 15 minutes beforehand, so when I heard her say it I was just like, ‘There you go’. So, it makes it feel like people are up there with you, even if someone’s up there by themselves, you get the sense that there are two other people there, you know?

Speculating about the meaning of rock art

I didn’t think they were A, B, C & D. The only other thing was maps, perhaps astronomical indicators. (P1)

Connection with the landscape

The heather – was out, purple, gorgeous. So much about the individual elements of the land. So, like the gorse. Kept stepping around it, and the ferns as well. All these elements have their own look and personality. They stuck out to me clearly, and the rock art was just another part of the landscape. (P2)

Points of dissatisfaction and the ‘evaluation effect’

The informal style of content challenged the participants who often mentioned in the debriefing interviews that they needed:

More content More detailed content More assertive/authoritative content

BUT Not too much content so it is easily

accessible in the field Not too authoritative content so they

could still feel that they could question and speculate about it

Some participants also felt that, surely, there was more information than the app was letting 0n

Second Moon

Art project by British artist Katie Paterson Commissioned by: Locus+, TWAM,

Newcastle University’s Institute for Creative Arts Practice

Funders: Arts Council England, Catherine Cookson Foundation

Part of 2013 British Science Festival Launched in Newcastle, GNM: Hancock

(August 2013)

It involved:

A fragment of moon rock ‘orbiting’ around the earth for one year using commercial airfreight

An app to visualise the travels of the fragment in relation to one’s own position and the orbit of the moon (iOS and Android, c. 1,880 downloads)

Exhibitions of the fragment along its route Educational activities and engagement

programme delivered by GNM: Hancock

Objective:To capture user response to the mobile app component of the commission

Evaluation approach:

7 Second Moon self-selected app users

Final four weeks of the project (last orbit)

Online diary using Google docs Open ended ‘blank’ pages + optional textual and visual prompts

Participants were asked to: ‘live with’ the app record a minimum of 6 entries record a final/wrap up entry if they

wished to

Meet Mary and Tom(pseudonyms)

2 out of 7 diarists in the Second Moon evaluation Mary

50 years old Darlington Art student with interest in art +

science (25 entries; 2,242 words)

Tom 40 years old Majorca Art critic and curator (12 entries; 1,516 words)

Mary

Monday 25 AugustHow was your first visit to Second Moon?

At first sight it feels quite simplistic, but I looked at it several times on the first day, and then I started noticing how far the (real) moon moves in just a few hours.  It gave me a new appreciation for the distance that separates the earth and the moon for it to be travelling so fast.  I was a little disappointed that the second moon seemed to be stuck and not moving, but it made me curious about where it was.  Zooming in told me something about how the place looked (where it was stuck), but I couldn't identify it with a name - was it Canada or USA.  So then I found myself googling that latitude to try to discover which town it was in.  I think it turned out to be in Oregon somewhere - presumably waiting to clear customs.

At first I thought the app would be better if it named the places where second moon has stopped or was travelling over.  But then I decided I quite enjoyed trying to figure that out for myself.

Mary

Sunday 31 August

I’ve been checking the app, but keep forgetting to log into the diary (would be good if I could get to the diary from the app and just do it immediately).so some of the following entries as remembered days later - and maybe a bit mixed up in terms of where the moon and second moon were.  But my feelings and thoughts are accurate.

Mary

Thursday 4 September

Second Moon is flying over China - just north of Nanjing.  Makes me think of all the lovely people I met when I lived in China.[...]

Tom

Thursday 4 September

I’ve placed my iPad on the desk and left it there with the app running, as a second screen or maybe a window. It sits to the right of my computer. […]

On the iPad, the Second Moon flies over some Chinese region. I’ve zoomed in so I can see the land, the cities, rivers and lakes. The aerial view scrolls slowly while the pulsating white circle stays firmly in the center of the screen. It feels like looking at a landscape, and for some reason, it is quite relaxing, like looking out the window of an airplane.

Mary

Wednesday 10 September

How long did you spend with the Second Moon App today?Not long - It has started to repeat itself.  I’m looking for some new insight.

Mary

Saturday 13 September

I travelled to the north of Scotland and it was good to see my position moving on the map!(I think this second moon experience has reminded me that I’m a traveller by nature, and sitting still feels like missing out on all the things there are to see and experience)

Mary

Second Moon has now landed.

How would you sum up your experience of the Second Moon App?

I think I’d sum up my experience as one of gaining perspective.It’s been interesting to stand so far back from the earth and think about the movement of the moon around our planet and our planet in the solar system, and our solar system in the vastness of space.  I can begin to see how some people obsess about the idea of finding other life out there ...its impossibly sad, and scary to think that we are alone.It’s also given me perspective on the busy way we lead our earthbound lives, and the odd ways in which we carve up this planet into territories and languages and tribes.  In the grand scheme of the universe whatever we do on a daily basis is pretty meaningless!Thanks for giving me the opportunity to take part...and best of luck with the project.

Tom

Second Moon has now landed.

How would you sum up your experience of the Second Moon App?

When I first installed the app, after watching it for a while I closed it and never opened it again because I thought “I had seen it”, just as when you watch a painting or another type of artwork and, although you may be fascinated by it, there comes a moment when you tell yourself that’s it, ok, I’ve seen it, and then you move onto the next thing.

I was wrong when I came to the same conclusion watching Second Moon for the first time. After writing this diary (the best I could, given that my travels turned out to be more distracting than inspiring for this experience) I have understood that I was not looking at an artwork, in the sense of something finished (as most artworks are: the result of a process), but that I was witnessing an event.

The ‘diary effect’

Stimulated participants to do repeat use of the app over 2-4 weeks

Diary format engender a special kind of attention – self-reflection and imagination

Diary prompts catalysed specific direction in people's interest (deep interest vs. peripheral use)

HOWEVER The diaries allowed us to see the ‘felt

experience’ of the app in action

5 lessons we learnt about apps:

Heritage/art apps cannot be all things to all people

Good usability ≠ meaningful experiences BUT meaningful experiences require good usability

People make sense and appropriate technology throughout their experience – scaffolding people’s experience with technology enhances their heritage/art experience

Mobile apps do not fragment/distract from the heritage/art experience, unless they are ‘designed’ to do so, or they are bad design.

Experience of heritage/art apps is made of ‘micro-moments’ – experience of heritage/art with or through apps extends beyond these micro-moments.

Tensions and opportunities in evaluating the ‘felt experience’ of mobile heritage/art apps

Opportunities The evaluation process is an extension

of the meaning making process through the recounting of the experience – this is not a weakness of experience-driven methods.

Capture change as a result of sense making

The boundaries between evaluating and studying are blurred – evaluation is not the end of a process

Tensions Resist the ‘value reduction’ approaches

to evaluation of art/culture/heritage Capturing the sensory and embodied

aspects of use requires commitment The research /evaluation methods

themselves are reflexive – ambiguity

With Thanks to:

RAMP project: Dr Aron Mazel, Dr Debbie Maxwell AHRC (AH/H037608/1)http://rockartmob.ncl.ac.uk

Second Moon project: Ms Rebecca FarleyACE, NICAP, Locus+, TWAM