Measuring the Value and Effect of Learning 2.0 Programs in Libraries
Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value
-
Upload
the-20-adoption-council -
Category
Education
-
view
109 -
download
3
description
Transcript of Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value
![Page 1: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/1.jpg)
© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
Ted Hopton, United Business MediaDonna Cuomo, The MITRE Corporation
Measuring Success and Business
Value: Metrics and Analysis
![Page 2: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/2.jpg)
2© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Who We Are
Ted Hopton
Wiki Community Manager
United Business Media
www.adventuresinsocialmedia.org
@Ted_Hopton
Donna Cuomo, PhD
Chief Information Architect
The MITRE Corporation
@Donnalc300
![Page 3: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/3.jpg)
3© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
United Business Media
Who is UBM?– ~ 6000 employees world-wide across hundreds of offices plus
telecommuters
– B2B media:
“We serve specialist business communities with tradeshows and other live ‘in person’ events; data, marketing and information products; print products; and targeting, distribution and monitoring services.”
– Organizationally stove-piped
More than a dozen divisions serving distinct markets
Each has a CEO and operates largely independently
– Commonalities across divisions include
Exhibitions/conferences (BTW, we produce this event, E2.0!)
Print publications
Digital/Online Services
– Launched internal Jive SBS community Sept. 2008
![Page 4: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/4.jpg)
4© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Objectives
Our Community’s Objectives Are Simply Stated
![Page 5: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/5.jpg)
5© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Minimal Engagement Level: Logins and Contributors
Contributors = Created content of some kind
![Page 6: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/6.jpg)
6© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Consuming Content: Page Views
Per Member Views: Comparable over time as community grows
![Page 7: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/7.jpg)
7© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Active Members, Consumers and Contributors
![Page 8: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/8.jpg)
8© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Page Views by Area
![Page 9: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/9.jpg)
9© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Page Views by Area, cont.
![Page 10: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/10.jpg)
10© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Page Views Per Employee
Comparable view across divisions of different size
![Page 11: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/11.jpg)
11© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
![Page 12: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/12.jpg)
12© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
![Page 13: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/13.jpg)
13© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Where In the Community Is Activity Taking Place?
![Page 14: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/14.jpg)
14© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Who Is Creating and Viewing Content?
Learn what the active members are doing so you
can share success stories and ideas
![Page 15: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/15.jpg)
15© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Minimal = <6 activities Light = 6 - 20 activities
Moderate = 21 - 50 activities Heavy = 51- 200 activities
Super = >200 activities
Active Members by Level of Activity
![Page 16: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/16.jpg)
16© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Quantitative Metrics
Minimal = <6 activities Light = 6 - 20 activities
Moderate = 21 - 50 activities Heavy = 51- 200 activities
Super = >200 activities
Active Members by Level of Activity
![Page 17: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/17.jpg)
17© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Objectives
But, Back to Our Objectives…
![Page 18: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/18.jpg)
18© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Qualitative Metrics: Survey
Annual User Survey– How often they used it in
specific ways Several times/week
Weekly
Monthly
Occasionally
Never
![Page 19: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/19.jpg)
19© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Qualitative Metrics: Survey
List of positive outcomes– Strongly Agree
– Somewhat Agree
– Somewhat Disagree
– Strongly Disagree
![Page 20: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/20.jpg)
20© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Qualitative Metrics: Survey
Why Don’t You Use It More?– Strongly Agree
– Somewhat Agree
– Somewhat Disagree
– Strongly Disagree
Blunt, negative statements– Ask for it!
– Let them tell you how they feel
OK, this is going to hurt a little– It’s a benchmark
– Listen, learn, then improve
![Page 21: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/21.jpg)
21© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Qualitative Metrics: Survey
The Bottom Line Question– How likely are you to recommend using it to a colleague?
– Scale of 0-10, with zero least likely and 10 most likely
Net Promoter Score© (NPS)– Scores of 9-10 = Promoters (7-8 = Passive Positives)
– Scores of 0-6 = Detractors
– NPS = % Promoters - % Detractors
Positive NPS means more people promoting than detracting
Negative NPS means the opposite
A Truly Comparable Metric– Even very different communities can compare NPS
![Page 22: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/22.jpg)
22© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Qualitative Metrics: Wins
Success Stories: “Wins”– Invite people to write up and submit wins
Small, medium and large
All kinds:
•cost-saving, revenue
•“Soft” wins: collaboration, communication, efficiency, innovation…
– Celebrate Wins
– Share them as examples for others
![Page 23: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/23.jpg)
23© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Lessons Learned
Metrics will (and should) evolve– Technology evolves new measurement capabilities
– Your skill with the technology will evolve better at pulling out data you need
– Your understanding of your community & objectives will evolve with experience
– Your community will evolve it’s alive, it will change, it will surprise you!
Numbers + Surveys + Stories– Collect all three
– Look at the whole picture
![Page 24: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/24.jpg)
24© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Lessons Learned
There are no generally accepted benchmarks for Enterprise 2.0 communities
– No one knows what results you *should* see for your community
– Your Culture + Your Objectives = A Unique Community
Benchmark Against Yourself– Aim for progress and improvement over time
– Compare different parts of your community with each other
![Page 25: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/25.jpg)
25© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
The MITRE Corporation MITRE is a private, independent, not-for-profit organization,
chartered to work in the public interest
Founded in 1958 to provide engineering and technical services to the U.S. Air Force
Currently manages 4 Federally Funded Research and Development Centers
– Department of Defense
– Federal Aviation Administration
– Internal Revenue Service/Department of Veterans Affairs
– Department of Homeland Security
Supports a broad and diverse set of sponsors within the U.S. government, as well as internationally
25
Bedford, Mass.
McLean, Va.
7,000 employees worldwide
![Page 26: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/26.jpg)
26© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Use Case 1: Improve MITRE’s Research Program Selection Process
Deploy an innovation management tool as the host environment for the research competition
Enable codification of the research competition process
Collect all ideas in one place and record all participation (eliminate early “weeding out” of ideas)
Better support “teaming” of proposers from across the corporation with similar ideas (improve collaboration)
Encourage much broader participation in both proposing ideas and commenting on ideas
Improve user experience and satisfaction of participation (visibility, feedback, standard process, idea targeting)
Use an externally-hosted application (cloud service)
![Page 27: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/27.jpg)
27© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Idea Market
27
Idea Market, powered by Spigitas a cloud service
FY10 Research Strategic Plan, powered by Sharepoint Wiki
Face-to-face “elevator proposals”
![Page 28: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/28.jpg)
28© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
First Year Numbers (as of 20 May 2009)
2842 of 7278 (39%) MITRE employees clicked into Idea Market
– 1445 Read-only Visitors
– 597 Idea Owners (non-stakeholders, submitted ideas, commented, voted)
– 575 Voters (only voted on ideas)
– 192 Commenting Users (non-stakeholders, non-idea owners who provided written content)
– 33 Members are Active Stakeholders (MIP Leadership who submitted an idea, review, comment, reply or vote)
840 Ideas submitted
750 Comments (threads) initiated
969 Replies to Comments
463 Reviews submitted
643 MIP Tech Support inquiries handle
5564 Votes cast
Achieved transparency, breadth of participation
![Page 29: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/29.jpg)
29© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
MIP User Satisfaction Summary: 2008 vs. 2009 Respondents reported considerable engagement with the
process
Clear improvement in perceptions that the process is understandable and predictable
Clear improvement in perceptions of fairness/consistency in proposal evaluation
Still only a (sizable) minority of respondents who feel that they understand or have received adequate rationale for funding decisions
More participants (41%) agreed that the process improved their competition experience than disagreed (27%)
Used survey technique to assess user experience
![Page 30: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/30.jpg)
30© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Use Case 2: Social Bookmarking (with Laurie Damianos)
Hypothesized social bookmarking would improve resource sharing, leveraging others research across teams and corporation
– Some use of external tools like delicious already occurring
Could feed expertise finding
Could replace current “knowledge zones” – subject-based websites maintained by corporate stewards (consider corporate goals, not just end user goals)
Increase participation in knowledge sharing
Subscription to topic areas
More granular information management technique via tagging
Ease of ‘re-finding’ information or highlighting resources (recommended, lesson learned, etc)
![Page 31: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/31.jpg)
31© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Social Bookmarking
Corporately stewarded collections
Tips
Popular topics
Bookmarked resources
RSS
urc
![Page 32: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/32.jpg)
32© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
onomi Pilot Usage(from 2007/2008)
Average information provider has 56bookmarks tagged with 5.4 terms.12000 page visits/mos
0% 5% 10% 15% 20% 25% 30%
other tag collectionsedit my tag
quick lookup (user, tag)delicious top 15
tag filtersadd bookmark via bookmarklet
onomi tag collectionpoponomi
other bookmarks by taghelp
searchbookmark page (URL)
delete bookmarkmy tag collection
browse usersadd bookmark by pop-up
my bookmarks by tagedit bookmarkadd bookmark
other bookmarksmy bookmarks by tag
onomi bookmarks by tag
most popular activity: viewing
other people’s bookmarks
14% of onomi’s visitors are
contributors
tags
154K tags18K unique
27K bookmarks
17% internal bookmarks
We have no other way to share external
resources
Supports need for internal social
bookmarking service
83% external bookmarks
![Page 33: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/33.jpg)
33© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
![Page 34: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/34.jpg)
34© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Center # Color
1 Lavender
2 Blue
5 Red
19 LightMagenta
20 Sepia
48 Melon
Size: tenure
MITRE-Babson SNA Study
(Bill Donaldson, Donna Cuomo, Sal Parise, Bala Iyer)
Study Findings• Its not just frequency of use that is important with these tools, but rather, who you are connecting to (uniqueness)
• Brokerage, in both social networks and technology-mediated networks, has a positive impact on personal innovativeness
• Both of the technology networks (ListServ and social bookmarking/ tagging) provide unique, significant value
• The social network & the technology networks complement each other
![Page 35: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/35.jpg)
35© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Measuring Business Value
Understand and explicitly document your goals for deploying a particular social tool. What benefits do you expect to achieve?
Think about what evaluation methodologies and objective metrics would help you understand if these desired behaviors are occurring
End users are not your only user group– Research analysts, commenters/reviewers, expertise finding, recommending, ….
Increasing the number of unique connections in a person’s social network has value
Cutter study notes it isn’t always possible to measure business value of a social tool from a prototype
– Need critical mass, long tail effect of many one-to-one benefits, takes 2-3 years for behavior changes to occur, etc
Use realistic benchmarks of participation to assess the adoption success of your tool
– Social tool use and patterns not always comparable to business tool use
![Page 36: Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value](https://reader034.fdocuments.us/reader034/viewer/2022051515/54c7bea64a7959dc418b45cf/html5/thumbnails/36.jpg)
36© Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.
#e2conf @Ted_Hopton @@Donnalc300
Engage. Evangelize. Empower.
www.20adoptioncouncil.com