Communities Connect Network Study of Community Technology in Washington State
description
Transcript of Communities Connect Network Study of Community Technology in Washington State
Measuring and reporting outcomes for BTOP grants:the UW iSchool approach
Samantha BeckerResearch Project ManagerU.S. IMPACT Study
1UW iSchool evaluation framework
Communities Connect NetworkStudy of Community Technology in Washington State
In 2007, UW conducted a phone survey across 211 agencies in the state identified as community technology providers. 47of these agencies completed the survey, and 7 sites were visited for more in-depth study.
This was combined with earlier data to provide a snapshot of CT in Washington State
2UW iSchool evaluation framework
Findings from the CCN survey indicated that CT was having an impact on communities.
• Three levels of benefits were identified– Individual– Family – Community
• Six domains were found to be important– Employment/economic– Academic skills and literacy– Social inclusion and personal growth– Independence– Access to information and resources– Communication
From 2007 CCN study
3UW iSchool evaluation framework
This lead to the development of the community technology impact analysis framework
Context Analysis
Situated Logic Model
Outcome Measuremen
t
Validation and
Reflective Practice
• Used to frame the Washington Community Technology Opportunity Program goals
• Six domains were found to be important– Employment/economic– Academic skills and literacy– Social inclusion and personal growth– Independence– Access to information and resources– Communication
– A similar process was used to frame the current WA BTOP evaluation framework
4UW iSchool evaluation framework
Identify major policy issues confronting the PCC’s community, such as:• Workforce development• Education• Poverty• Civic engagementAnd link them to activities used to address these issues, such as:• Access to technology and information• Skills building• Development of local content
Step 1: Policy issue mappingContext Analysis
5UW iSchool evaluation framework
Identify other stakeholders concerned with the policy goals
– individuals– groups– organizations– institutions
Collect data about what they do and how they experience PCC servicesInclude stakeholders who are working to achieve similar policy goals or who are affected by the PCC
Step 2: Stakeholder analysisContext Analysis
6UW iSchool evaluation framework
Step 3: Develop a policy or stakeholder logic model
Situated Logic Model
Inputs
• Facilities• Technology• Knowledge• Relationship
s
Activities
• Technology access
• Training and support
• Awareness building
Outcomes
• Effective and efficient service delivery
• Improved decision making
• Gain employment
Impacts
• Improved workforce
• Reduction of poverty
• Improved community health
Policy Issue: Workforce Development
7UW iSchool evaluation framework
Step 4: Develop a PCC Logic Model to link goals with measurable indicators
Situated Logic Model
Inputs
• Facilities• Technology• Knowledge• Relationships
Activities
• Technology access
• Training and support
• Awareness building
Outputs
• Hours of access to technology
• Number of clients participate in training
Impacts
• Clients get jobs
• Clients earn GEDs
• Clients are able to use technology independently
• Clients engage in their communities
8UW iSchool evaluation framework
Sidebar: what’s a measurable indicator?Indicators need to meet certain utilitarian standards. Beyond the actual content of the indicator, they should also be:• specific, unique and unambiguous;• observable, practical, cost effective to collect,
and measurable;• understandable and comprehensible;• relevant (measures important dimensions,
appropriate and related to the program, that are of significance, predictive and timely);
• time bound; and • valid, providing reliable, accurate, unbiased,
consistent, and verifiable data(Hatry, 2006)
UW iSchool evaluation framework 9
Step 4: Bridge the logic models to show how the CTC supports larger policy goals
Situated Logic Model
Inputs
• Facilities•Computers• Internet
connection• Software• Staff
Activities
•Open technology access
•Computer classes
•Tutoring
Outputs
•Hours of access to technology
•Number of clients participate in training
•Number of clients looking for jobs
Impacts
•Clients get jobs•Clients earn
GEDs•Clients are able
to use technology independently
•Clients engage in their communities
Inputs
• Facilities• Technology• Knowledge• Relationships
Activities
• Technology access• Training and
support• Awareness building
Outputs
• Citizen technology access
• Increased citizen knowledge
• Use of technology to support social services
Impacts
• Improved workforce
• Reduction of poverty
• Improved community health
Example: Workforce Development Community technology
1. Workforce development clients use community technology to look for and apply for jobs
2. Clients get jobs
3. Workforce is improved
1 2 3
Workforce Development
Community Technology
10UW iSchool evaluation framework
Identify the most important outcomes to measure:• Tie activities to immediate policy goals
in the PCC community• Link to the larger community policy
context to evaluate overall impact on stakeholders and the community
Step 6: Measure outcomesOutcome Measurement
Step 7: Report outcomesReport outcome measures in the context of the situated logic model to show contribution to community policy goals.
11UW iSchool evaluation framework
Re-examine relationship between outcomes and policy issues:• Validate measures• Challenge assumptions• Interview stakeholders
Step 8: Validate Outcome MeasuresValidation and Reflective Practice
Step 9: Reflect on performanceUse outcome/impact measures to inform your work:• Establish the value of your work• Improve effectiveness• Understand your organization’s contribution to
the community you service
12UW iSchool evaluation framework
Measuring and reporting outcomes for WA BTOP
1UW iSchool evaluation framework
WA BTOP’s reporting system relates to a community technology logic model
Inputs Activities Outputs Impacts
SRs are asked to keep track of and report inputs, activities, outputs, and impacts on a quarterly basis for the duration of the grant in order to:
• Show the value of the services offered• Show the changes in use as a result of the grant• Evaluate the impact of the grant on the outcomes of clients
2Measuring and Reporting OutcomesUW iSchool evaluation framework
Impact types for WA BTOP were informed by the CCN study, BTOP policy goals, and PCC self-identified anticipated outcomes
Computer skills enhancement• Help clients gain computer and
Internet skills to enable them to independently use technology
• Can be at any skills level and includes multimedia training
Employment skills and opportunities• Help clients learn how to use
computers and the Internet to strengthen their ability to become employed
• Provide access to employment opportunities
Education enhancement• Provide supplemental learning
support for students challenged by limited learning resources
Access to information and services• Enable clients to use digitally accessible
information for personal needs like managing health problems
• Enable clients to access government information, benefits, or services
Life skills and social inclusion• Includes special services for people with disabilities• Help clients learn to use technology for managing personal relations• Help clients learn about and use technology for money management,
finding support, interacting with the community, and daily living skills (e.g. getting bus schedules)
• Teach and empower clients to use digital technology to express themselves and participate in their communities
4UW iSchool evaluation framework
• SRs are not expected to have outcomes to report for every category or indicator.
• SRs were asked to review the list of possible outcomes and choose those they wish to track.
• No single client survey can accommodate all PCCs, but samples and advice are provided for SRs for designing instruments and tracking sheets.
Tracking outcomes
4UW iSchool evaluation framework