Dr. Judith Samuels Presentation to MJB Seminar 2015-10-28

Managing Director

Impact and Performance Assessment Department (IPAD)

UJA-Federation New York

October 28, 2015

Judith Samuels, PhD

Measuring our Impact:

State-of-the-Art Approaches to Evaluation

Seminar at

Meyers-JDC-Brookdale

HEADLINE HERE: PROSE CONTENTAgenda

• Our work

• Philosophy

• Methods

• Tools

• Models

IPAD Team: Jennifer Rosenberg, Jen Simbol, Alex Bebeshko

HEADLINE HERE: PROSE CONTENT

1. Develop and implement our impact measurement strategy

- Top-down approach starting with UJA-Federation NY mission

- Unified approach across all grants (over 1,000) to all grantees (over 400)

- Impact as related to MISSION

- Organization’s mission

- Commission/Department’s mission

- Initiative's mission

- Specific grant’s mission

IPAD works in 5 areas



2. Ensure every grant has proper, appropriate evaluation

- Level of sophistication of evaluation follows program/grant size

- Rethinking “chasing” the counterfactual

- Use of logic model for planning and to guide evaluations



Our Evaluation Framework:

Mission Impact

Goals Outcomes

Objectives Outputs

HEADLINE HERE: PROSE CONTENTLogic Model Template to Inform Planning:

Key (Color Code for Outcomes and Impact/Goals Boxes): Color – Impact on individual, Color – Impact on institution (agency, synagogue, etc.), Color – Impact on community

:

How to use this tool to inform your planning As you begin thinking about designing a new program/initiative, this logic model template will help you to frame your thinking on what you want to accomplish and what is the most effective and efficient strategy to do so. The logic model displays how investments link to results, and can inform planning, implementation, communication, and evaluation. When using this tool, start at the end and identify your goals and the impact you wish to have. To help you think this through, begin by asking yourself the following questions:

What is the impact we wish to have and on whom (i.e. individual, institution, community).

Do we want to see changes in an individual or a group/community’s behavior, action, attitude, perception or feeling?

Do we want to raise awareness on an important issue or for an individual or group/community to learn something new?

Do we want to see a change in an institution (i.e. the way it operates or makes decisions)? Once you have clearly defined goals/impact statements and outcomes, continue working backwards through the chart, asking yourself:

What is the best strategy or intervention (if you do not already have one) and how does it lead us to our desired goals?

What resources (i.e. financial, human, etc.) will we need to achieve it?

What are the outputs that will indicate whether or not we are on track to achieve our outcomes? Please note: when determining the appropriate output number, use your best judgement based on research from similar models in the field, or your previous experience funding similar types of projects/initiatives. Make sure that the output number is realistic in magnitude based on your goals (i.e. if you want to change the Jewish community of New York’s behavior, you likely need to reach far more than 100 people).




3. Increase the rigor & contribution of funded research studies

• ~ 40 Research/evaluation studies funded

• Expanding the pool of possible consultants, vetting along the way

• Explicit procedure to developing and reviewing RFPs

• Use of rubrics and explicit criterial to choose the best proposals






4. Move to state-of-the-art grants management & data usability

• Online grantee portal

• Grant reporting feeds directly into data warehouse

• Improved grantee relationships, dashboard capabilities






4. Move to state-of-the-art grants management & data usability

5. Provide analytic framework to assist in determining Core

(unrestricted) support to Network agencies


Well-managed,

excellent agenciesThat

Advance UJA-

Federation’s mission

Governance

Performance

Jewish Community

Building

Cooperation/Joint

Investment

Community Reach

History

JCC Pilot20% of the allocations to the JCCs

History80 % of allocations to the JCCs

Governance

25%

Performance

25%

Jewish Community Building

25%

Cooperation/Joint Investment

10%

Community Reach

15%

Weighting(Approved by the Impact

Committee)

Breakdown of Core

Allocations

Criteria Development

HEADLINE HERE: PROSE CONTENTExample of how we choose measures:Sources that Informed the Excellence Measures

Governance and Performance Measures

• Leadership Theory and Practice by Peter Northhouse

• Robinhood Leadership Assessment Tool

• Nonprofits Assistance Fund Financial Management Self-Assessment

• Assessment of Agency Executives from Nonprofit Leadership Services

• McKinsey & Co. Nonprofit Board Self Assessment Tool

• Hillel International’s Evaluation

• USAID: Leading and Managing Framework

• Community Resource Exchange Board Assessment Questionnaire

• Harvard Business Review

• NYC Mayor's Office of Contract Services Capacity Building and Oversight Unit

• Blackbaud Online Marketing Benchmarks for Nonprofits

• Independent Sector Principles for Good Governance and Ethical Practice

• Annual Hillel Assessment

HEADLINE HERE: PROSE CONTENTData Collection Methods:

• Data we have on hand

• Ie., Audited financial statements, targeted grant reports

• Surveys to Executive Directors and Board Presidents before

site visits (and additional informants as appropriate)

• Basic agency information

• Site visits – Trained Lay Leaders and Staff

• Interview guides with open-ended queries

• Interview with Executive Director, Board President, “consumers”

• Post site visit surveys

• Use of psychometrics to test reliability and validity

Governance Is the agency well managed and fiscally sound?

PerformanceDoes the agency track and evaluate program quality, outcomes, and impact? What is the agency’s

reputation for excellence overall and performance on targeted grants in particular?

Jewish Community BuildingDoes the agency inspire Jewish life and foster bonds and relationships across diverse groups of Jews

locally and globally?

Cooperation/Joint InvestmentDoes the agency meet the rules and responsibilities of being an affiliate of UJA-Federation and

contribute to the value of the network?

Community ReachHow large is the total community being served? Is the agency addressing needs in the broader

community?

History

Hillels Pilot Assessment Criteria

Criteria for Future Core Grant

Allocations

Governance Is the agency well managed and fiscally sound?

Performance



Community Reach

History

Governance

Leadership and Accountability

Management Practices

Financial Stability

Governance


Allocations

Governance

PerformanceDoes the agency track and evaluate program quality, outcomes, and

impact? What is the agency’s reputation for excellence overall and

performance on targeted grants in particular?



Community Reach

History

Performance

User Satisfaction

& Enagement

Performance on UJA-Fed

Targeted Grants

Accreditations/

External Ratings

Program Outputs,

Outcomes and Impact(i.e. Technology

Infrastructure)

Performance


Allocations

Governance

Performance

Jewish Community BuildingDoes the agency help Jews in need, inspire Jewish life, and foster bonds

and relationships across diverse groups of Jews locally and globally?


Community Reach

History


Jewish Impact

Quality/ Innovation of

Jewish Programming

Mission

Alignment(Jewish Focus)



Allocations

Governance

Performance


Cooperation/Joint InvestmentDoes the agency meet the rules and responsibilities of being an affiliate of

UJA-Federation and contribute to the value of the network?

Community Reach

History


Fundraising

Co-Branding

Exhibits Shared

Commitment to Communal Planning and

Action



Allocations

Governance

Performance



Community ReachHow large is the total community being served? Is the agency addressing

local community needs?

History

Community Reach

Use of Resources

Responsive to

Community Needs

Inclusive

Organizational

Practices(Community

Engagement)

Community Reach


Key: ▲▲ = Top Scores, ▲ = Positive/In top Quartile, ● = Neutral (Q1-Q3), ▼ = Negative/In Bottom Quartile (Below Q1) , ▼ ▼ = Below 70

Sample Individual Agency Score CardAgency: XXXX

Criteria Measure High Score

Low Score

Avg. Score

YourScore

Your Overall Criteria Score

E

X

C

E

L

L

E

N

C

E

Governance(25%)

Accountability Practices 93 61 80 62 ▼▼58Financial Stability 85 28 65 28

Leadership/Management 95 56 80 84

Performance(25%)

Accreditation/External Ratings 98 48 80 88

● 86User Satisfaction and Engagement 100 53 91 98

Program Outputs, Outcomes & Impact 90 47 72 71

Performance on UJA-Targeted Grants 92 56 83 88

M

I

S

S

I

O

N

A

L

I

G

N

M

E

N

T


(25%)

Mission Alignment 92 46 78 77

●88Jewish Impact 98 54 83 90

Quality/Innovation of Jewish Programming 100 65 88 97


(10%)

Exhibits Shared Commitment to Communal Planning and Action

97 55 83 96

▲ 95

Co-branding 98 45 82 97Fundraising 95 26 80 92

Community Reach(15%)

Size Within Sector 100 20 61 70

● 83Responsive to Community Needs 96 54 83 85

Inclusive Organizational Practices 100 77 85 93Your Overall Score: ● 80

FOR INTERNAL USE ONLY, NOT FOR CIRCULATION

HEADLINE HERE: PROSE CONTENTLocal Agency Assessments on a 2 year Cycle

• Hillel Pilot Assessment• In process, completion 3/2016

• To Core Allocation Committee 3/2016

• Impacting FY 2017 funding

• JCC Targeted Assessments• In process

• Overnight Camp Pilot Assessment• Model adaptations Spring 2016

• Assessment period summer 2016

• JCC Assessment (round 2)

Dr. Judith Samuels Presentation to MJB Seminar 2015-10-28

Government & Nonprofit

Transcript of Dr. Judith Samuels Presentation to MJB Seminar 2015-10-28