14 Oct 2010, TELECENTRE-EUROPE Summit TELECENTRE-EUROPE Summit: Third Sector contribution to...
-
Upload
aden-izard -
Category
Documents
-
view
218 -
download
2
Transcript of 14 Oct 2010, TELECENTRE-EUROPE Summit TELECENTRE-EUROPE Summit: Third Sector contribution to...
14 Oct 2010, TELECENTRE-EUROPE Summit
TELECENTRE-EUROPE Summit:
Third Sector contribution to e-Inclusion in Europe: Impact Measurement and Analysis framework
Budapest, 14th October 2010
Gabriel Rissola, JRC-IPTS
Maria Garrido, Technology & Social Change Group | University of Washington
Joint Research Centre (JRC)Institute for Prospective Technological Studies The European Commission’s Research-Based Policy Support Organisation
14 Oct 2010, TELECENTRE-EUROPE Summit
Institute for Prospective Technological Studies
Part of Joint Research Centre of the EC
IPTS: Research Institute supporting EU policy-making on socio-economic, scientific and/or technological issues
14 Oct 2010, TELECENTRE-EUROPE Summit
Last TE Summit in Istanbul: IA policy relevance and constraints
Demonstrate Impact is useful for:
Get a recognition of telecentres role in EU policy making:- Evidence is essential to get policies adopted (including funding programs)- Evidence helps to set priorities, make the right choices
Smarter action by combining social innovation and economic innovation, stimulated by technological innovation
…but few IA good practices were identified in the field
In Vienna Study on Inclusive Innovation for Growth and Cohesion, 1000 cases were analysed but only 52 had measured outputs and outcomes (mostly from UK and IRL), for example:
– UK Onlines Centres trained 76,000 people; active job seekers increased from 66% to 100%
– EOL UK digital literacy training helped 76 people to get a job, saving £670,000 in state subsidies
– FIT Ireland trained 6500 young unemployed people in ICT; 3500 found a job, saving $13 million.
– Also examples of improved welfare benefits distribution, savings through ICT-enabled home care.
14 Oct 2010, TELECENTRE-EUROPE Summit
Last TE Summit in Istanbul:Measuring Impact (outputs, outcomes)
eIN
CLU
SIO
N
Acc
ele
rati
ng
Enab
ling
De
ma
nd
Su
pp
ly
MEASURE CATEGORY DOMAINS BY MAIN
FOCUS OF INITIATIVESTYPICAL OUTPUT
Su
pp
lypublic access points availability # of regular customers
basic digital literacy # individuals completing courses
tax relief & incentives for affordability % of eligible beneficiariesusing the incentives
broadband % Δ broadband coverage
eAccessibility # of accessible public services
Community/ territorial initiatives Holistic not possible to indicate 1
learning through ICT
skill-building for employability # individuals completing courses
ageing-well at home services
healthcare services Holistic not possible to indicate 1
access to welfare entitlement Holistic not possible to indicate 1
De
ma
nd
Contribution to Regional growth
# individuals completing courses
# of homes covered
TYPICAL POTENTIAL OUTCOME
beneficiaries becoming regular Internet users
Various measures of community social capital
Various health outcomes
trainees finding a job
fewer days spent in institutional care/hospital
Various learning outcomes
Beneficiaries increase in disposable income
Cristiano Codagnone et al, EC study March 2009 (Vienna Study on Inclusive Innovation for Growth and Cohesion)
14 Oct 2010, TELECENTRE-EUROPE Summit
Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge
• Increasing political awareness on the critical role of Telecentres and NGOs in promoting e-Inclusion goals through different social and digital programs
• Wide range of target groups addressed - from immigrant communities, women, impoverished youth to prisoners and isolated people in rural areas to people with physical disabilities and the elderly – which goes even beyond Riga’s targets
• However, 3rd Sector organisations working in the eInclusion field don’t balance their efforts in critical areas (like IA)
14 Oct 2010, TELECENTRE-EUROPE Summit
Last TE Summit in Istanbul:Telecentre eInclusion critical role and challenge
• NGOs often struggle to systematically:– collect information from their beneficiaries without
overburdening staff capacity– articulate how and the extent to which their programs
impact the lives of the people they serve – use this evidence as a tool to gain more visibility and
recognition to their work especially among policy makers and donors
• Measuring Outputs (quantity of trainees, quantity of Internet users) and End user satisfaction surveys are important, but insufficient; need to go a step forward and find a way to systematically measure Outcomes
14 Oct 2010, TELECENTRE-EUROPE Summit
Practitioner perspectiveTelecottage impact measurement system
• Adapted UK’s IMProving Accountability, Clarity and Transparency (IMPACT) to Hungary
• Prepare impact report of the real social effect of their work.
• Primary data (outputs), with Secondary (outcomes).
• Created intended effects and possible indicators, on whether life is better or not.
• Most important effects of telecottages: – strengthened community spirit,
– improved human conditions,
– increase public trust
• Challenge: – to adapt intended effects and possible indicators to other local
contexts
– to measure them in an aggregated way at regional/national/EU level
14 Oct 2010, TELECENTRE-EUROPE Summit
Funder perspectiveMS UP impact measurement system
• Driven by need for systematic and credible evidence of how UP helps unemployed & others excluded to get essential skills to find a job.
• Most important thing is that evaluation scales globally. Is it improving lives? Are they moving along a trajectory?
• UP goes beyond self-reported quarterly reports, doing evaluations and case studies.
14 Oct 2010, TELECENTRE-EUROPE Summit
Funder perspectiveMS UP impact measurement system
• Lessons learnt: – track results on ongoing basis and make adjustments as needed
– metric that decided to avoid is how many people they put into jobs;
– commissioned research can be expensive;
– strongest grantees & partners are data driven and report back consistently, which builds a solid picture on impact over time
– Need for common indicators, plural of anecdote is not evidence.
• Opportunity: telecentres are multipliers
14 Oct 2010, TELECENTRE-EUROPE Summit
1. Consensus building on the objectives of measuring impact– to provide funders or investors with data on impact; – to provide a tool for organisations to manage their own choices
internally– to better understand long-term processes of social change and impact.
2. Targeted methodological framework for IA based on the state-of-the-art (there is no “perfect” social metric framework)
3. Targeted set of indicators– Relevant enough as to measure different e-Inclusion dimensions– Simple and easy to implement by organizations of different sizes– Appropriate enough as to provide consolidated results related to the e-Inclusion
goals of the DAE– Compatible with existing indicators but going deeper into the specificity of the
measured field.
Building an IA frameworkPossible steps
14 Oct 2010, TELECENTRE-EUROPE Summit
4. Need to facilitate social organizations self-analysis and self-interpretation of their own local impact. Indicators should be:– Available online (save costs and facilitates its distribution)– Translated and contextualized in each country (localisation)– Complemented by adequate guidelines for a proper data collection, self-analysis
of data and production of reports for different organisational scopesGuidelines for implementation and interpretation would surely be necessary
5. Need a software for the implementation of an IA system based on those indicators
• To facilitates the process of user data collection and impact analysis for NGOs
• To create with these data a repository that can be aggregated and analyzed globally
• To protect anonymity of NGO users and of the organizations themselves
• Easily implementable by the interested organizations in each local context
• Centrally maintained for the provision of constantly-updated consolidated information
Building an IA frameworkPossible steps
14 Oct 2010, TELECENTRE-EUROPE Summit
Measuring indicators is a complex endeavor due to the complexity of isolating the effects of one single variable (e.g. effect of e-Skills training in people’s lives).
This process is further complicated by the diversity of NGO programs, target groups, local factors, individual factors, etc.
Different types of impact can be measured (i.e. Access, frequency and purpose of ICT use, skills development, size and diversity of social networks, user perception of usefulness of ICT for improving their lives)
However, how to connect them with desired Outcomes remains an issue (e.g. “Qty of DL trainees that become regular Internet users”, “DL trainees that improve their employment status”)
Building an IA frameworkIndicators to be measured
14 Oct 2010, TELECENTRE-EUROPE Summit
FEEDBACK FROM THE AUDIENCE
Constraints- Voluntary organisations- Data protection laws- Local advocacy skills
Org needs/Selling points- Carefully select and agree the objectives of IA- Start simple, small- Highlight useful bits of data- Simple and quick implementation- Pay attention to incentives
14 Oct 2010, TELECENTRE-EUROPE Summit 16
Thank you
[email protected] http://is.jrc.ec.europa.eu/pages/EAP/eInclusion.html http://is.jrc.ec.europa.eu/pages/EAP/eLearning.html
[email protected] http://tascha.washington.edu/
14 Oct 2010, TELECENTRE-EUROPE Summit
Last TE Summit in Istanbul One example (female migrant users)
Perceived usefulness of ICT Appreciated features of TCs
Satisfaction with Social services
END USER SATISFACTION IS IMPORTANT, BUT INSUFFICIENT
+ Education and Employment services
14 Oct 2010, TELECENTRE-EUROPE Summit
Building an IA frameworkObjectives of the Metrics
Metrics are useful to show what works and what deserves to be grown (from The Open Book of Social Innovation):
• There are many metrics for judging whether innovations are working – at various stages of development.
• Metrics can play a decisive role in determining whether innovations are scaled up, or deserve to be.
• A recent survey found 150 different metrics in use in the non-profit sector. However, relatively few of these are actually used to make decisions.
• This field has failed to make progress due to the confusion and conflicts of interest between three different tasks performed by metrics: – to provide funders or investors with data on impact; – to provide a tool for organisations to manage their own choices internally– to better understand long-term processes of social change and impact.
1. NEED TO BUILD A CONSENSUS ON THE OBJECTIVES OF MEASURING IMPACT
14 Oct 2010, TELECENTRE-EUROPE Summit
There is no “perfect” social metric framework | Pros and Cons
Example 1: Sustainable Livelihoods Framework• Increasingly popular among development agencies
• It recognizes that people have a wide range of resources at their disposal and their choice for deploying those resources are critical to assess impact
• Very closely align to Amartya Sen’s capabilities approachLIMITATION: DIFFICULT TO IMPLEMENT AND MEASURE HOW PEOPLE’S CHOICES
AND MOTIVATION AFFECTS THE OUTCOME OF A PROJECT
Example 2: Cost-benefit analysis• Used primarily by public authorities and government agencies
• Its goal is to assess a particular project taking into account costs and benefits
• Benefits and costs are quantified in money termsLIMITATION: ASCRIBING MONETARY VALUES TO CERTAIN SOCIAL IMPACTS THAT
ARE VERY DIFFICULT TO MONETIZE
Building an IA frameworkSocial metrics system
2. NEED TO BUILD A TARGETED METHODOLOGICAL FRAMEWORK FOR I.A.
BASED ON THE STATE-OF-THE-ART
14 Oct 2010, TELECENTRE-EUROPE Summit
• Relevant enough as to measure some dimensions of the impact of e-Inclusion initiatives driven by NGOs in the communities they serve;
• Simple and easy to implement by organizations of different sizes which are active in the field of e-Inclusion at regional/national level in Europe (extendible to smaller NGOs and other actors in a second stage)
• Appropriate enough as to provide consolidated results related to the e-Inclusion goals of the Digital Agenda of Europe.
• Compatible with existing indicators but going deeper into the specificity of the measured field.
Building an IA frameworkSelection of indicators
3. NEED A TARGETED SET OF INDICATORS
14 Oct 2010, TELECENTRE-EUROPE Summit
Aggregated level
• E-Inclusion activities
• Other social services
• NGO mission
• Reach of the work (local, regional, national)
• Target groups
• Partnerships
Building an IA framework Some examples of indicators
Organisational level
• User demographics
• NGO services and activities where users participate
• Digital skills level
• Frequency of ICT use
• Types of ICT use (email, chat, job searching, homework, etc)
• Employability situation
• Social inclusion situation
• Users perception of value of ICT to improve their lives
14 Oct 2010, TELECENTRE-EUROPE Summit
5. NEED A SOFTWARE FOR THE IMPLEMENTATION OF AN I.A. SYSTEM BASED ON THOSE INDICATORS
Building an IA frameworkSoftware for processing data
• To facilitates the process of user data collection and program impact analysis for NGOs
• To create with these data a repository that can be aggregated and analyzed by country and at European level
• In doing so, to protect anonymity of NGO users and of the organizations themselves
• Easily implementable by the interested organizations in each local context
• Centrally maintained for the provision of constantly-updated consolidated information at European level
14 Oct 2010, TELECENTRE-EUROPE Summit
Stakeholders interested in the metrics to be produced
• Organisation’s staff (managers, fundraisers, etc)
• Local policy officers in charge of social services and community development
• Donors/funders
• Policy makers operating at different levels, including the EC
Users who will collect the data
• eFacilitators in contact with the end users
• Volunteers
• Staff managing the telecentres
• Telecentre network coordinators
Building an IA frameworkStakeholders and Users
14 Oct 2010, TELECENTRE-EUROPE Summit
• Review of indicators currently used by EUROSTATS, OECD, ITU, and academic articles
• Needs assessment to determine the kind of information that social organizations are currently gathering about the beneficiaries of their programs and additional data they are interested in collecting.
• Piloting the Impact Measurement System in different contexts
• Wide implementation of the system across Europe promoted by a multistakeholder partnership (MSP) or an umbrella organization who represents several stakeholders (like TE)
Building an IA frameworkOther needs