Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor...
-
Upload
jake-fraser -
Category
Documents
-
view
221 -
download
4
Transcript of Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor...
![Page 1: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/1.jpg)
Presentation at The Conference for Family Literacy
Louisville, KentuckyBy Apter & O’Connor Associates
April 2013
Evaluating Our Coalition: Are We Making an Impact?
![Page 2: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/2.jpg)
Critical stakeholder mass
Greater efficienci
es
Broader goals
accomplished
Community benefits
Why a Coalition?
![Page 3: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/3.jpg)
![Page 4: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/4.jpg)
Evaluation can guide the change process
Evaluation can monitor the health of your coalition
Evaluation can “tell your story”
Evaluation can “dig deeper”
Evaluation can measure your success and answer the “so what” questions
Why Evaluate?
![Page 5: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/5.jpg)
Effectiveness
Why Evaluate? Provide Accountability to
Community, Funders & Stakeholders
EffectivenessEffectiveness
Quality Efficiency
Effectiveness
![Page 6: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/6.jpg)
Step 1: Defining YOUR Coalition
Cooperative Group
• Short term goals/tasks
• Linkages may be formalized but still advisory
Network• Dialogue• Informal
linkages• Information
exchange
![Page 7: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/7.jpg)
Collaborative • Long-term
commitment• Formal agreements• Shared leadership
and decision making
• Shared community-wide indicators of success
Partnership• Share/merge
resources• Joint planning but
individualized authority
• Frequent communication
• Common issue
![Page 8: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/8.jpg)
SO WHAT!! The Impact of Your Coalition
![Page 9: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/9.jpg)
Step 2: Meeting Evaluation Challenges
1. Lack of clarity about the expected outcomes from the coalition: What to evaluate
2. Coalitions take time to build3. Coalitions are not static4. Measurement: Attribution vs.
Contribution (who gets “credit”)
![Page 10: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/10.jpg)
Challenge #1: Lack of Clarity about Expected Outcomes
![Page 11: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/11.jpg)
Some Solutions• Reflect back to your logic model:
reasonable, feasible, focused outcomes
• Clarity about what the coalition will bring to the community that individual programs cannot.
• The evaluation must match the level of anticipated outcome
![Page 12: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/12.jpg)
Communities
Systems
Agencies/Organizations
Individuals/clients
Level of Impact
![Page 13: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/13.jpg)
Individuals
Change in behaviors,
attitudes, skills
![Page 14: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/14.jpg)
Agencies/Organizations
Change in behaviors, attitudes, knowledge and skills
Change number and type services provided
Change in policies, proceduresLeadership development
![Page 15: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/15.jpg)
Systems
Change in behaviors, attitudes, knowledge and skills
Increase in coordination/collaboration/cohes
ion within or across systemsChange in service delivery
Joint decision making
![Page 16: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/16.jpg)
Community
More resources/services for the community
Change in legislation, regulation, policy
Heightened community awareness/civic action
Demonstrated change in social, economic, environmental conditions
![Page 17: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/17.jpg)
Challenge #2: Coalitions Take Time to Build
![Page 18: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/18.jpg)
• Clarify the “development phase” of your coalition and the key tasks that need to be accomplished
• That will lead you to appropriate questions and important data to collect
Some Solutions
![Page 19: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/19.jpg)
Evaluating the right thing at the right time!
• Planning and forming• Why do we need this
collaboration? • Who should be involved? • Do we have a common vision?
Formative Evaluation:--Readiness and capacity
“Developmental Evaluation”:Evaluating the Right Thing at the Right Time…
![Page 20: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/20.jpg)
Developing • Are we on track? • Is our coalition meeting the needs of
members? • Are we achieving early outcomes?
Process Evaluation:-- Beginning to implement
Evaluating the Right Thing at the Right Time
![Page 21: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/21.jpg)
Evaluating the right thing at the right time!
Maturing and Sustaining • What have we
accomplished?• What are the benefits for
our coalition and for whom?• What is the value of our
effort?Outcome/summative evaluation-- Accountability
Evaluating the Right Thing at the Right Time
![Page 22: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/22.jpg)
Challenge #3: Service Systems and Communities Are Not Static
![Page 23: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/23.jpg)
Solutions to enhance evaluation
• Revisit your logic model:– Are the new strategies are still
in line with your vision– Does the coalition membership
still include the “right” stakeholders
Some Solutions
![Page 24: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/24.jpg)
Challenge #4: Attribution vs. Contribution
![Page 25: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/25.jpg)
• How do we insure “self interest” of member agencies/organizations are met?
• How do we distinguish outcomes of the coalition vs. outcomes by programs funded by the coalition vs. other contextual factors?
![Page 26: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/26.jpg)
Solutions to enhance evaluation
• Do a good job of collecting the right data– Reflects the coalition expected level of
impact– Reflects the coalition developmental
stage
• Critical importance of process to outcome: continue to measure perception of members
• Shared measurement systems
Some Solutions
![Page 27: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/27.jpg)
Not everything that counts can be counted!
![Page 28: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/28.jpg)
Phases of Shared Measurement System
STEP I Design
STEP 2Develop
STEP 3Deploy
•Agree on system and relation to logic model
•Review current state of knowledge/data
•Agree on governance structure
•Identify approach, metrics, confidentiality, etc.
•Develop platform and tools
• Test and refine platform/tools
•Staff for data management and synthesis
•Learning forums
•Ongoing support
•Review, refine, improve, ongoing evaluation of usability and impact
Source: FSG Analysis, www.fsg.org 2011
![Page 29: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/29.jpg)
The COMET® System
![Page 30: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/30.jpg)
• Clarity among organizations about WHY you are collecting this data and how it will be used
• Getting organizations to agree on a set of shared indicators that reflect their work
• Silo nature and reporting requirements of funders
• And………….
Cautions….
![Page 31: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/31.jpg)
Outcome
Process
Formative
Measure infrastructure,
functioning and procedures of the coalition
Measure indicators of
extent of implementation
Measure change and realization of
vision
Evaluation Types are All Related….
![Page 32: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/32.jpg)
SURVEYS: member benefits, perceptions of stakeholders, informants, community
INTERVIEWS/FOCUS GROUPS stakeholders, beneficiaries, community members
OBSERVATIONS: site visits, meetings
RECORDS, DOCUMENT REVIEW
CASE STUDIES: policy analysis, beneficiaries, build a story of the coalition’s work and impact
INDICATORS: statistical analysis for significance, trend data, change in targets, cost-benefits
EV
ALU
ATIO
N
Tools of the Trade
![Page 33: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/33.jpg)
EV
ALU
ATIO
N
Collect information from SEVERAL SOURCES (community residents at large, targeted clientele, elected officials)
Collect information using SEVERAL METHODS (survey, focus group and site observations for perceptions)
Consider OUTSIDE EVALUATORS (to add credibility, expertise, objectivity)
Collect both QUANTITATIVE (the numbers) and QUALITATIVE(the story) data
USE your findings
Tools of the Trade
![Page 34: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/34.jpg)
Pre-Conditions• Urgency for change
• Adequate financial resources
• Influential champions
Conditions for having impact• A common agenda (vision)• Continuous
communication• A shared measurement
system with a “short list” of indicators
• Mutually reinforcing activities (relating to vision
• A centralized backbone structure
Successful Coalitions
Kania and Kramer, Collective Impact, 2011 www.ssireview.org
![Page 35: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/35.jpg)
1. Selection of a core set of community outcome indicators by the literacy coalition
2. Provide training and technical assistance in performance measurement
3. Assist community literacy programs to track and use their own outcomes
4. Provide support for community-wide outcome data collection
5. Provide support for analysis of the outcome information.
Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008
Contributions of a Literacy Coalition
![Page 36: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/36.jpg)
6. Use the findings to help attract funding
7. Use the findings to identify and report community literacy program needs and literacy condition
8. Use the findings to help identify and disseminate successful (“best”) practices in your community and elsewhere
9. USE THE FINDINGS TO CELEBRATE GOOD LITERACY OUTCOMES
Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management for Community Literacy Coalitions, Washington, DC 20008
Contributions of Evaluation
![Page 37: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/37.jpg)
For more information…
Literacy Powerline www.literacypowerline.com
Apter & O’Connor Associates
![Page 38: Presentation at The Conference for Family Literacy Louisville, Kentucky By Apter & O’Connor Associates April 2013 Evaluating Our Coalition: Are We Making.](https://reader036.fdocuments.us/reader036/viewer/2022062511/551c16d25503469e4f8b568c/html5/thumbnails/38.jpg)
For more information…The Literacy Powerline Site
www.literacypowerline.com or
The Apter & O’Connor Associates Site
www.apteroconnor.comor
[email protected]@apteroconnor.com