Evaluating community Evaluating collaborationEvaluating capacityTom Kelly Annie E. Casey Foundation

Tanja Kubas-Meyer Making Connections Providence

www.aecf.org public.me.com/tomkellyjrtkelly@aecf.org slideshare.net/tomkaecf

(410) 223-2932

Established in 1948 by Jim Casey, founder of UPS

Casey “cousins” with Casey Family Program, Marguerite Casey Foundation, Jim Casey Youth Opportunity Initiative

Mission: To improve the lives of America’s most vulnerable children

Operates foster care as Casey Family Services

Annie E. Casey Foundation

Annie E. Casey Foundation

Began grantmaking in 1980’s– State child welfare reform– Juvenile detention reform– New Futures – community/schools

Grantmaking in the 1990’s– Urban workforce – Jobs Initiative– Urban children’s mental health– Teen pregnancy prevention – Plain Talk– Rebuilding Communities Initiative

Grantmaking in the 2000’s– Community change initiative – Making

Connections– PRI’s and responsible redevelopment –

East Baltimore

Overview of Comprehensive Community Initiatives (CCIs) Goals focus on multiple levels,

including individual/family, neighborhood, organizations, community, region

Focus on building capacity of neighborhood stakeholders and local institutions

Attend to interrelated influences Multiple players, layers, systems

Challenges in Evaluating Comprehensive Community Initiatives: Outcomes are either too distal

(reduced poverty rates) or too abstract (improved community capacity)

Difficulty developing a counterfactual (e.g., Community wide intervention, other programs tackling same issue, lack of control group or comparable community)

Outcomes and theories of change shift, as community needs fluctuate and learning occurs

Unexpected outside influences

Range of Tools for a Continuum of Data Needs/Questions

Random Control

ComparisonT1 vs T2

SurveysOutcome Evaluation

Sampling

Anecdote

Focus Groups

Observations

Measuring Success

Change in PLACE Change in PEOPLE Change in COMMUNITY

CAPACITIES Change in SYSTEMS OF SUPPORT

& SERVICES Change in ENVIRONMENT &

CONTEXT Contribution of Weed & Seed -

Explain WHAT, HOW and WHERE the initiative is making a difference

RBA Model (in one slide)1. Who are we concerned about? Population

2. What conditions or change do we want for this population? Result

3. How can we measure these results? Indicators

4. How would we know if we are making progress? Performance measures

5. Who has a role to play? Partners6. What works? Best practices7. What do we propose to do? Our theory

and strategies

How Much We Did OUR

EFFORTS

How Well We Did

Quantity# of Customers # of Activities

Quality% Timely activities% Attending% Customers Satisfied $ Costs per unit% Standards met

What Difference We Made for Customers

By the #s

Changes in …# Behavior# Attitude# Circumstance# Knowledge# Skill

OUR

EFFECT

By % of AllChanges in Changes in …% Behavior% Attitude% Circumstance% Knowledge% Skill

Results…what difference did we make in terms of: IMPACT on children and

families INFLUENCE on policies,

practice, systems, and services

LEVERAGE of resources

Impact, Influence, LeverageWe seek to make a positive difference in

the lives of children and families through three types of RESULTS:

 Impact: Changes in a condition of well being for the children, adults, families or communities directly served by grants, programs, agencies, or service systems.  Influence: Changes in capacity (including cultural competence), policies, regulations, systems, practice or public opinion. Leverage: Changes in investments by other public or private funders in our strategies to improve outcomes for children and families.

Place-based community accountability Program-level AND population-

level (whole neighborhood) accountability– Beyond traditional contractual

obligations to a funder– Strengthening shared and

collaborative accountability to the whole population across multiple partners and organizations

– Increased partnership of all partners, players, and funders working in a place

Accountability requires…

Funders and decisionmakers (i.e., those with power and resources) are held accountable for the decisions they make and they must publicly affirm and report their targets and results

All stakeholders must have publicly available data with which to hold all accountable

Community residents must be prepared to be organized and participate and act in decisionmaking and accountability processes

Transparent and open decisionmaking processes and structures need to exist to hold all stakeholders accountable

Incentives for positive progress and sanctions for negative change

Results Accountability

www.resultsaccountability.com

http://raguide.org/

www.aecf.org “Casey ResultsNet”

Weed & Seed

CollaborationCoordinationCommunity participationLeverage

Weed & Seed

CollaborationCoordinationCommunity participation

AUTHENTIC DEMANDLeverage

CORE CAPACITYASSESSMENT

Measuring Leverage

A Practical Guide to Documenting Influence and Leverage In Making Connections Communities

Reisman, Jane; Langley, Kasey; Stachowiak, Sarah; Gienapp, Anne

www.organizationalresearch.comwww.aecf.org

Making Connections

Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based strategy

10+ year investment in 10 communities (2000 to 2010)

Hartford IndianapolisLouisville MilwaukeeOakland ProvidenceDenver Des MoinesSan Antonio Seattle/White Center

Results/Outcomes

Adults have work and earnings Families save, build and protect

assets Families access quality services Families are connected to social

supports Children are healthy and prepared

to succeed in school (grade-level reading)

Local Learning Partnerships (LLPs) Established to support local Making

Connections sites (for 10-years and beyond) with data and information to help strategy development, and self-evaluation

Collaborative group of data holders, researchers, evaluators, and data users (including nonprofits and residents)

Multiple roles:– Building/consolidating local data warehouses– Performance and results measurement and feedback to

work teams– Quantitative and qualitative data collection on families and

neighborhoods, including evaluation of process and outcomes

– Promoting local capacity (by residents and systems) to use data strategically

Community capacities

Program implementation and management

Data, self-evaluation, and accountability

Communication, public policy, and will-building

Resident leadership and engagement (Authentic Demand)

Financial sustainability and infrastructure

Community capacities

SCOPE – coordinated and integrated services and service systems

SCALE – policy and system changes that reach large numbers of families

SUSTAINABILITY – financing and infrastructure

Authentic demand/resident engagement

Data, learning, and accountability

Measuring Community Capacities Began with stages of change

framework (INSITES, Bev Parson)

Maintaining Business as

Usual

Building Awareness of And Demand for Change

Exploring New

Approaches and Small

Scale Changes

Refining, Coordinating,

and Increasing

the Impact of Strategies

Effective Approaches are Taking Hold and

Transforming Business as

Usual

Maintaining Business as

Usual

Building Awareness

of And

Demand for Change

Exploring New

Approaches and Small

Scale Changes

Refining, Coordinating, Increasing the Impact

of Strategies

Effective Approaches are Taking

Hold

SCOPE

SCALE

SUSTAIN

RESIDENTS

DATA

Building a scale

Developed common (general) indicators across sites

Worked with sites to revise and better define concepts and stages

Tested in individual sites Added a “score”

Maintaining Business as

Usual

Building Awareness of And Demand for Change

Exploring New

Approaches and Small

Scale Changes

Refining, Coordinating,

and Increasing

the Impact of Strategies

Effective Approaches are Taking Hold and

Transforming Business as

Usual

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Using the scale

Implemented collaborative and participatory process of measuring capacities in each site

Evaluators facilitated a review and discussion of specific areas of work (eg, workforce, assets, school readiness)

What is the evidence? Site specific score

Contributing to evaluation Annual assessments produce a

site specific T1T2T3 Use additional data to answer

questions like:– Did technical assistance help sites

advance? – What patterns exist across sites?– Does higher capacity lead to

more/bigger/better results for families?

Lessons

Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of concepts and theory of change

(decision) More important to focus on change within sites than comparing sites given different environments/context

Lessons/Challenges Important to anchor discussion of

capacities in terms of results/work– Capacity for WHAT?

Iterative process Still difficult to summarize across

sites Need additional verification of

evidence and perspectives Tool/scale less important than the

process

Measuring…

Resident engagement Community engagement Participation Social networks/social capital Civic engagement Community organizing

Authentic Demand “Honeycomb”

Residents advocate for results in strategy work groups and play key decision making roles

Membership networks continue to grow in size and effectiveness

Resident leadership training institutionalized in community colleges and other local partners

Organizing leads to policy changes at local and state level

Resident leaders provide impetus for forming service provider networks

What Does Authentic Demand Look Like?(The Forms)

Leadership strategies that offer learning opportunities and training to residents and other partners focused on various skills and capacities including strategic planning, data analysis, policy advocacy and others;

  Strong social networks that build

relationships and reciprocal exchange among and between members of the community, and link residents to new resources and opportunities;

What Does Authentic Demand Look Like?(The Forms)

Community organizing efforts that mobilize community members in the collective and in strategic alliances with other stakeholders seeking better outcomes for children, families and neighborhoods

  The kind of civic participation that enables

residents to establish community priorities and then hold elected officials and service providers accountable through voting, local forums, community research, and consumer and policy advocacy.

How do we know that it’s meaningful?(Interim Outcomes)

1. Voice. There are ample opportunities for community members to not only make their needs, desires and opinions known, but also to participate with dignity in environments where decisions affecting the community are being made.

2. Accountability. Those who represent community interests at decision-making tables are representative of and accountable to well-defined and legitimate constituencies, including voters, members of a community or faith-based organization, or agency leadership. As important, there is evidence of a “demand environment” services, resources and opportunities are determined more by the needs and desires of those in the community rather than what organizations have available or what service providers “decide” families and the neighborhood need.

3. Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, mentoring, or other skill building opportunities that enhance the competence and confidence of community members at decision-making tables and in civic life.

4. Identity. People believe they hold a stake in the larger community. In addition, community identity is inclusive and represents its diversity -- different cultural groups within the community are genuinely represented and embraced as members of the community.

5. Reciprocity. Community members are actively encouraged to contribute their time and talent, because there is a clear understanding that everyone has something to offer the change agenda. Service to the community is rewarded through public recognition, ceremonies, and events. A social “norm” of participation and of action is nurtured.  

6. Choice. Not only are there many options to engage in and contribute to change efforts, but those options connect to the needs and desires of the community – what has value for them – not the agendas of someone else.

SKILLS AND COMMUNITY CAPACITYWhat/How Much Was Done How Well Was It Done

# of residents trained and prepared to take up engagement or leadership (completing training in community engagement, leadership, budget analysis, race matters, RBA, etc.)

# of partners completing training in authentically engaging the community (social networks, race matters/Undoing Racism, etc.)

% of training graduates getting involved in issue organizing or taking on new leadership roles

Deepened knowledge and ability in various types of skills and capacities (identified site-by-site)

Improved rates of program participation across Making Connections strategies

# of residents running and securing public office

CONSTITUENCY, SEATS AT THE TABLE AND COMMUNITY ACCOUNTABILITY

What/How Much Was Done How Well Was It Done # of residents attending town hall

or community meetings

# of parents engaged

# of public forums held

# of new participants attending forums to express perspectives and needs with elected officials

# of residents (i.e. constituency) that leaders can mobilize for an event, action or campaign

Increase in the number of tables where residents are represented at the community and system levels

Increased diversity of participants in town hall/community meetings

% of the new residents getting involved that remain involved after one year

Increase in the number of examples of collaboration across individuals and organizations to solve complex problem

Increase in the stability and capacity of resident-led organizations or associations (increase in budget; increase in staff; increase in membership; increase in collaborations)

What Difference Did It MakeImpact Influence Leverage

Increase in access to jobs via resident/network referrals

Increased access by residents and families to community resources and job supports

Increase in SCHIP or health care coverage for vulnerable children and families

# of positive community solutions achieved by residents and partners working together (key wins are identified site-by-site based on existing strategies). For example:

1) Increase in bi-lingual resources, staff and literacy options in the schools

#/% of targeted partners adopting or embedding new practices that incorporate authentic demand principles

#/% of targeted partners engaging residents in boards, committees and as advisors

Evidence of a positive change in practice due to partners responding to feedback from community members

Funders begin to require their grantees to demonstrate evidence of community capacities on their board, staff and practice

New funds are invested in fostering authentic demand skill and capacity building for residents and public/private agencies

New funds invested in engaging and strengthening community constituencies

New funds invested in mentoring opportunities

New funds invested in and towards engaging and strengthening resident and partner capacity and effectiveness in diversifying seats at the table

Multi-year funding secured for authentic demand

Tools & ways of measuring Software for tracking program,

process, and contextual data in a central place (Backpack- Backpackit.com)

Tracking resident engagement (Constant Contact and YS.net)

Mapping participation (GIS)

Tools & ways of measuring pre/post...or often better...post-

then assessments of learning charting timelines learning meetings core capacity assessment individual and group interviews social network analysis

Core

Link Node

Cluster Periphery

Hub

Social Network Analysis: A Few Helpful Definitions

Source: Monitor Institute

Lessons learned

Other resources Race Matters Toolkit

http://www.aecf.org/racematters.aspx

Guide to Evaluating Advocacy & Policy Change http://www.organizationalresearch.com/publications/a_guide_to_measuring_advocacy_and_policy.pdf

When and How to Use External Evaluators

http://www.aecf.org/publications/data/using_external_evaluators.pdf

Tools and Resources for Assessing Social Impact (TRASI) (beta)

foundationcenter.org/trasi