'Stories & Numbers' - A Framework for Measuring Engagement
-
Upload
jason-ryan -
Category
Business
-
view
8 -
download
1
description
Transcript of 'Stories & Numbers' - A Framework for Measuring Engagement
1
Learnings from the live web:
‘Stories & Numbers’
A Framework for Measuring Engagement
MCN 2009
Dr. Jason Ryan, iCrossing
2
Introduction
This presentation
1. Background/context
2. What is engagement?
3. Framework for measurement
4. Case study
5. The ROI of engagement
6. Opportunities/challenges for Museums
About me
Working in digital for 14 years
Joined Cogapp in 1998 - many projects with Museums & Galleries
UX and consultancy work for cultural, government and commercial clients
Left Cogapp in 2007 to join iCrossing, a global digital marketing agency
3
Context – online has changed, is changing…
We started by adapting existing models – •marketing•catalogs•structures•collections
Somewhere along the way realised it was about doing things differently…
Explosion of content creation and distribution tools = disruptive
From passive online consumers to active participants
Users increasingly building their own interfaces to and contexts for our content
The web IS social
Everyone is having to adapt…
44
Many organizations see their website as the sum total of their online existence
– but every organization will always exist in a far broader network
- the question is how we choose to listen to and engage with those networks
The network view
5
Designing a distributed web experience
6
Web analytics is about individual “eyeballs on page” metrics such as visits and time spenton site
Analytics is often ignored – it needs to work with qualitativeobservations to tell a better story and understand what success online looks like
But visitors complete actions such as commenting and watching videos that can not be tracked by simply calculating time on site
And people interact with a site’s content by sharing and discussing it in the broader network - on blogs, forums, social networks and content streams and feeds.
How do we evaluate visitor engagement that takes place off our owned website?
What can we learn from that offsite engagement?
Measurement models need to change
7
Measurement models need to change
Recognising the user as active participant – and considering the total online experience – not just the onsite experience
There is a paradigm shift underway from mass media measures (traffic) to engagement measures
Active attention
Deliberate actions
Brand and content advocacy
Development of a user-centric Engagement Framework which recognises:
Awareness, Actions and Advocacy
8
Advocacy(Creators)
Actions(Contributors)
Awareness(Consumers)
A user-centric framework
Volume &Brand Control
9
Foundations
10
11
A
Method – Engagement Scorecard
12
Channel 4: Bow Street Runner
Channel 4 is one of 4 terrestrial broadcasters in the UK
It has a public service broadcasting remit – including education programming
It was felt that education programming was not reaching the intended audience
Channel 4 took the bold step of taking all education programming online - Including taking content to the audience
iCrossing did a pilot project with Channel 4 to measure engagement with the educational flash game 'Bow Street Runner'
Bow Street Runner is historical role playing game set in Georgian London – and explores the foundation of the first police force
13
Channel 4: Bow Street Runner
14
Bow Street Runner The Game
An episodic 5-part RPG built by Littleloud based on the 5 part TV series City Of Vice
Features video shot in high-definition
channel4.com/cityofvice
The game was not marketed – awareness and traffic was driven solely by social media outreach – seeding content on relevant blogs
Players are Bow Street Runners
Level up from Runner to Chief Inspector
Investigate crime >
Collect evidence >
Present case >
Solve crime
15
To understand the landscape of the networks in which our potential audience exists
Objectives
To understand how users engaged with the game outside of the Channel 4 networks
To measure the effectiveness of the social media seeding plan
To measure how awareness spreads within the network
To evaluate the Measuring Engagement framework
16
1
Discovering Engagement Online
• Engagement with online content is distributed across networks
• Multiple measurements at multiple points are required to capture all interactions
Influential sites Social networking Forums Social bookmarking
17
Involvement: Are people coming to the site & playing the game?
Interaction: How far do they get in the game, do they come back, do they interact with other users and Channel 4?
Intimacy: What do they feel about the game, do they bookmark it?
Influence: How far do users spread knowledge of the game through associated networks?
A
Method – Engagement Scorecard
18
19
20
Web Analytics Engagement Metrics
Visits Visits
Time on Page Time on Page
Bounce Rate Bounce Rate
Comments on c4 forum
Rank & Esteem data
Return Visits
Email Sign-up
Comments on non-C4 sites
Positive Sentiment
Negative Sentiment
# Tags/ Bookmarks
Links
# Influential Blogs
Influential Blogs: Positive Sentiment
Influential Blogs: Negative Sentiment
Page Rank
Average Search Engine Ranking
City of Vice Viewing Figures
Going beyond Web Analytics
21
Results
22
Intimacy
23
I finished the 3rd game but missed the esteem points at the
end. I wanted to play 3 again but it had already saved my password for game 4 which
doesn't exist yet. The password for 3 is gone. If I want to play it again, I have to start over… I’m
not going to do that 1
Based on what I read, I thought the episodes were supposed to come out
every Monday, but Ep. 2 took an entire month to show up. Hopefully… we'll be seeing a weekly release schedule from
here on 2
I was hoping episode four would be up today,
following the general weekly pattern lately,
but seems not 4
Oh brother... on episode 2 with the nails in the hatch... whenever I try to pull them upwards I get my mouse outside the
screen and I have to start all over!!! This happened twice now … this is sooooo
frustrating 3
Makes me wish for programs of similar quality to be developed here
for education. Imagine doing problem-based learning in this
format… today’s kids would want something a bit edgier and more interactive. Who’d be up to the task? Discovery Learning? The
History Channel? 1
Flash loading
problemsRelease DatesPasscodes Loading Time
Gameplayfrustrations
Negative Sentiment
24
The game is really well done and uses a pretty decent cast of
actors… the actors are caught in full-motion and talk in
real time 1
Historically accurate detective game where you have to
investigate the locations in order to find clues and
evidence 3
The game was created as a companion to the UK television
series City of Vice and is not only well-acted, but also takes great pains
to be historically accurate 2
Bow Street Runner… is a murder mystery that takes place in the seedier
neighborhoods of Georgian London… you poke around at things, talk to people, and
little by little the facts are uncovered 4
GameplayActors Plot Atmosphere
Positive Sentiment
25
Seeding Networks
Network map snapshot for Bow Street Runner
26
Traffic Source – first 6 weeks
• Over 300,000 visits in the first 6 weeks
•Traffic from referring sites accounts for most visits > 75%
27
Performance of Seeded Sites• 35% of all traffic came from 17 seeded sites; over 75% of all traffic from extended network
• Top 4 seeded sites each sent more than 10,000 visitors
• Of the 17 seeded sites, 8 sent more than 1,000 visits to the game
•Sites from Turkey, Brazil, Japan and Taiwan also sent more than 10,000 visitors
Seeded sites
Non-seeded, UK
Non-seeded, non-UK
28
Total Playing Time
47,826 hours47,826 hours
29
Seeding Networks - Insights
The blog seeding of BSR was very successful in attracting gamers
Seeded sites also attracted links from other gaming websites
Seeding drove majority of traffic to game (>75%)
Seeded sites attracted nearly 1000 links between them
Traffic for the game never reached the same level after the initial seeding
The sites identified through direct response info, web stats, and desk research
Seeding was done too early
Seeding could also have targeted ‘gatekeeper’ and 14-19 year old demographic
30
Outcomes
31
The importance of social tools to encourage Discussion and Advocacy
Insight into the effect of decisions – (beta, delays emails)
Sentiment fed into the design and allowed fixes to be prioritised
Understanding how quickly the community take ownership of the product
Seeding worked driving 75% of the traffic
Outcomes
Provided a benchmark for similar initiatives
32
Moved from a small Education pilot to contract to measure online engagement for the whole ‘Disarming Britain’ TV series and related online activity
Highlighted the real possibility of an AGILE PLANNING FRAMEWORK – from seeding to creative execution
Validation of the approach and framework –
led to a Forrester Case Study – the first on Measuring Engagement
Outcomes
What if marketing and creative just got better over the course of a campaign?
33
Street Crime Season
34
Disarming Britain blog: Network growth
Week Two Week Four
Week TenWeek EightWeek Six
External incoming link
Blog Post/Page
35
Top Referring Sites to C4 MicroSite (you don’t always get what you want)
Airgun EnthusiastsAirgun Enthusiasts
US Rifle Enthusiasts US Rifle Enthusiasts
Anti-Knife CommunityAnti-Knife Community
Grime Fans CommunityGrime Fans Community
Football CommunityFootball Community
Q&A SiteQ&A Site
Pro Gun Ex- SoldierPro Gun Ex- Soldier
Dutch BlogDutch Blog
Gun AdvertisingGun Advertising
Community Action SiteCommunity Action Site
36
Disarming Britain Season – themes of discussion
Themes of wider discussion
Glamorisation of violent crime
Issues relating to child discipline in school and in the home
Mentioned in the news alongside the Home Office campaign and reports of fatal stabbings
Media scaremongering
37
Channel 4 & Home Office Bebo friend networks
Friends with one profile
Contributed content to both profiles (Comment or Whiteboard post)
Friends with both profiles
38
Which content drives traffic
“Spy shots of McLaren P11 supercar”
“McLaren P11 pictures surface”
“Nissan's Nuvu: electric, cartoonish”
“Aston Martin and Mercedes 'to strike £235m (€300m) deal‘”
“Test Drives Audi's new S4!”
39
The value of content
Visit Value Link Value
Total Value this month:
The value of content…
40
Summary – Outcomes & Insights
The user-centric framework is a useful tool for evaluating user engagement
Ability to measure ROI for government and non-profit organisations in terms of social impact
Potential to benchmark and index engagement scores by web traffic
Always design-in measurement
Don’t be limited by analytics tools – aggregate everything!
“Numbers & stories” – is a practical approach that allows analytics insights to ‘travel’ within organisations
41
Thoughts about Museums
Web visits is a metric that is ‘understandable’ to Museums – it is rooted in the real world
How many Museums go beyond visits and page views as the primary measure and attempt to evaluate how engaged visitors are ?
With more and more free quality content and media online, Museums, as well as commercial brands, face competition for attention
Unlike many commerical ‘brands’ Museums have rich and potentially very engaging online content
By extending their measurement models Museums can:
• drive more traffic• provide better experiences, and • even find better ways to monetise their content
42
JASON RYANVP, Head of User Experience, iCrossing UK+44 (0) 1237 828 146 [email protected]
thank you.
43
What is engagement?
Awareness:
Have people heard about you?
How do you know?
Conversation:
What is depth and quality of the conversation?
Advocacy:
Who are the influencers?
What are they saying about you?
Actions:
What actions are being taken?
Sales, downloads, subscriptions, ratings?
A relationship