Welcome to the Allegany Franciscan Ministries2014 Major Grant Evaluation & Success Measures Webinar

We will begin PROMPTLY at 11:00 a.m.

~~ Please MUTE your phone ~~

Please do not use a headset as they cause feedback that is disruptive to others on the call.

April 29th, 2014

ReflectionBlessed are they who give without expecting even thanks in return, for they shall be

abundantly rewarded.Blessed are they who translate every good thing they know into action, for ever higher

truths shall be revealed unto them.Blessed are they who do God's will without asking to see results, for great shall be their

recompense.Blessed are they who love and trust their fellow beings, for they shall reach the good in

people and receive a loving response.Blessed are they who after dedicating their lives and thereby receiving a blessing, have

the courage and faith to surmount the difficulties of the path ahead, for they shall receive a second blessing.

Adapted fromfrom the book "Peace Pilgrim"

Agenda Welcome and Reflection The Application Evaluation – our philosophy Success Measures & Outcomes

Definitions Examples Resources

Q & APresentersJessica Gonzalez, Program Officer Erin Baird, Director, Grants and TechnologyShelley Robertson, Evaluation Consultant

Our goals for this webinar To help you and your staff team…

Understand our evaluation philosophy, process & expectations.

Better understand the difference between program outcomes & indicators; and the difference between those outcomes that are measureable & meaningful and those that are not.

Begin to conceptualize program success measures before your initial meetings with the Program Officer.

Have the tools and knowledge to submit a successful application.

2014 Major Grants

In each region, 4 to 6 organizations are likely to be funded.

Complete Major Grant Applications are due by May 29, 2014 at 12 pm (noon).

Site Visits will occur in late July and early August.Board will finalize decisions in September. Grant agreements will then be developed and signed;

initial payments will be made between. November and January, depending on the project.

Major Grant Timeline

Hillsborough, Pinellas, Palm Beach, Martin & St. Lucie

CountiesNotes

LOI Due March 20, 2014 Online application must be submitted by 12 pm (noon).

Notification of our decision; decline or invite to submit full application By April 25 We will e-mail and send a letter to you.

Major Grant Full Application Technical Assistance Webinar April 30 For those selected to submit a full application; please hold date

(date subject to change).

Deadline to submit full application May 29, 2014 Online application must be submitted by 12 pm (noon).

Site Visits & Meetings July & August

Decisions Announced late September We will work with applicants between September & December to finalize budgets & success measures.

Grant Period Begins November 1, 2014 or later

2014 Allegany Franciscan Ministries Strategies Community education and outreach designed to

increase knowledge and support behavioral changes.

Healthcare services and basic needs. Increased capacity and coordination of the

healthcare delivery system. Advocacy and community efforts that educate,

engage, raise awareness, or mobilize communities.

The Application – How to ApplyRead the entire 2014 Major Grant Application

Instructions document.Go to our web site, www.afmfl.org.Click on the “Current Partner” button.Log on to the online application system.Confirm or edit all contact information.Under “LOI” you will see “Application”.Click on “Edit Application”.

The Online Application – Some Basics Complete your responses in Word© and then

paste into the appropriate field. Pay attention to character limits. Document will

not save if you are over the character count. Don’t use acronyms or jargon. Save your work frequently. When uploading documents, click “save as draft”

to finalize each upload. Before submitting, “Print Packet” and read/have

someone else read.

The Application

The Application

Budget

Evaluation Philosophy

As a good steward, Allegany Franciscan Ministries is dedicated to assuring that our investments move us toward our mission. Our evaluation process is clear, simple, and focused on outcomes. The evaluation process is respectful of organizations with different size, sophistication, values, beliefs, and traditions. It balances qualitative and quantitative methods to inform and guide our decisions while improving services provided by our grant partners.

Getting to measures An organization should articulate how the

program leads to long-term change for people.

Success measures document progress being made towards that long-term change.

Allegany Franciscan MinistriesSUCCESS MEASURES FORM

Organization Name: Project Title: Year: From (month/year): To (month/year):

Outcomes Please state one outcome per row.

Indicators What is the specific statistic you will track to determine your success on this outcome? What data tool will you use to collect this data?

ActivitiesWhat are the key activities that will lead to each outcome?

Measurement planWhat is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

Participants increase knowledge about how to cook healthy meals.

85 percent of participants are able to identify four healthy cooking methods and up to 10 healthy food choices.

Provide 10 hours of training to 20 participants in 5 two-hour sessions.

We will administer a paper pre and post test to participants.

Coalition members will increase their advocacy activities

90 percent of members will take at least two advocacy actions during the six months after training.

Recruit and train a coalition of up to 20 advocates.

Members will report actions at monthly coalition meetings; staff will document efforts.

complete one page – and one page only - per year of grant

Definitions

Outcomes: improvement and/or change in knowledge, skills, attitudes, or behavior. Outcomes may also refer to desired changes in a community in response to community-level and system-wide activities.

Indicators: the specific items used to track a program’s success. Indicators describe observable, measureable characteristics or changes. An indicator should include the specific characteristic measured, what data tool is used, when it is measured, and who collects the data.

 

Definitions

Activities: what a program does with its resources to fulfill its mission.

Measurement plan: states any sampling strategies, how the data is entered, how you will analyze the characteristics to assess whether you were successful, and how data reported. For example, what counts as an increase?

Program outcomes

Program outcomes state the changesin clients’ knowledge, skill,

attitude, or behavior.

Example: Program outcomes Clients increase their knowledge about how to reduce risk

behaviors. An indicator could be that 80% of clients increase their

knowledge about what are risk behaviors, consequences of risk behaviors, and how to reduce risk behaviors as measured by comparing post test to pretest scores on the curriculum-embedded tool.

Clients increase their skills in preparing healthy food. Clients demonstrate improved confidence toward changing

behavior. Clients experience decreased health risk behaviors.

Organizational capacity outcomes

Organizational outcomes state changes in capacity areas such as revenue, fiscal

management, board development, operations, etc.

The organization should be able to explain – although probably not measure – how the change in capacity will impact those they serve, such as improved service quality or more clients served.

Example: Organizational capacity outcomes

The organization increases the capacity of its board of directors. An indicator would be that the organization meets current

needs on 10 benchmarks on the board development section of the Capacity Benchmarking tool (see resources).

The organization increases its fundraising capacity. An indicator could be that the organization achieves

benchmarks on a capacity tool or that the organization (a) develops at least two new revenue streams, (b) reduces expenses by 5 percent in comparing 2010 to 2009, and (c) that 90% of staff report that the new system saves them time.

Choosing outcomes: meaningful and measurable What we want to see:

Fewer youth exhibit at risk behaviors due to undiagnosed mental health problems.

What we don’t want to see:Fewer youth

abuse substances and show increased stability and life skill.

Choosing outcomes: meaningful and measurable What we want to

see:75% of recipients

increase their intake of healthy food daily.

What we don’t want to see: 75% of recipients

report eating more fruits, vegetables and protein after starting our food delivery program.

Choosing outcomes: meaningful and measurable What we want to see:

“Organization A” will provide primary medical services to individuals and families with no health insurance.

What we don’t want to see: Provide safety net

medical services for individual and families without health insurance or who cannot afford or are not eligible for conventional health care insurance.  

25

Creating Outcome Statements:

• Step one: describe your client (i.e., age, gender, status)

• Examples: at-risk middle school youth, adults with mental illness, frail seniors, Hispanic teens, adults with disabilities, low-income parents

The Outcome Formula

26

• Step two: identify the issue or topic you are addressing.

• Examples: healthy lifestyle choices, employment, substance abuse, grades, sports, self-esteem, coping skills, health status.

Creating Outcome Statements:

The Outcome Formula

27

Client Verb Adjective

Type Example/topic

  Will Increase Knowledge about

 

  Decrease

Skill(s) in

  Maintain Attitude about

    Behaviors such as

     

     

Examples:

1. At -risk middle school youth will decrease behaviors such as participating in gangs, getting in fights, and using drugs.  2. Hispanic teens will increase skills in job seeking.

Creating Outcome Statements:

The Outcome Formula

Demonstrate

Outputs vs. Outcomes Outputs answer the

question “how much” .

Outputs include number of clients, service units, hours, and appointments.

Outcomes are client-focused.

Outcomes relate to knowledge, skills, attitudes, or behaviors.

29

Characteristics:Observable and measurable.Client-focused.Unambiguous.State a time frame.Match.Understandable.

Indicators

30

The Indicator FormulaTells what you will measure (see table below), how, when, and who is measuring.

For this type of outcome

Tell us…. Examples:

Knowledge

What three main things the clients learn

 

Skill What, specifically, they can demonstrate

 

Attitude

What they will or will not feel  

Behavior

What they will or will not do  

31

Indicator ExamplesTool:

• Pick one: survey, pretest and posttest, posttest only, interview, case records, observation form, or official statistics. Tell us the name of the tool and who created it.

• Examples: curriculum pretest and posttest, online survey created by USF, case records kept by the agency, report cards from the school, ADL checklist provided by the State Department of Elder Affairs, survey provided by DJJ.

Target: • Put the percentage of your clients that you think will achieve the

outcome. The target is NOT the client population. • Examples: 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% .

32

The Indicator FormulaHow examples:

• how the tool is administered.

When examples: • at the end of the program, beginning, every 30

days, 90 days after completing the program, etc.

Who examples: • program staff, non-program staff, outside agency

personnel, other agencies.

33

Indicator Examples

Knowledge: [Clients] will learn three elements of sportsmanship as measured by an end of

the program posttest administered by staff. Skill:

[Clients] will demonstrate that they can prepare a resume, find an open job through the paper or online, and answer interview questions as measured by non-program staff observation during the job fair at the end of the program.

Attitude: [Clients] will state that they feel hopeful about a positive future as

demonstrated by exit interviews conducted by program staff prior to leaving the shelter.

Behavior: [Clients] will not be adjudicated for gang activity, fights, or drugs while they are

in the program as measured by DJJ statistics collected by the School Resource Officer.

Making It Measurable

Direction of change.

Area of change.Target population.Degree of change. Time frame.As measured byBaseline.

Example: To decrease the percentage of 18 to 20 year old college students reporting binge drinking from 13 percent to 12 percent by 6/30/2015 as measured by the CORE survey.

Hints Outcomes: State one outcome per row.

Indicators and measurement: What is the specific statistic will you track to determine your success on this outcome? What data tool will you use to collect this data?

Activities: What are the key activities that will lead to each outcome?

Measurement plan: What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

Evaluation Resources

In addition, Dr. Shelley Robertson is available at no charge to you for up to two hours.

Dr. Robertson can assist you in identifying which of these resources might be most helpful for your specific project.

She can also help as you develop or refine the outcomes to be tracked and reported, and determine appropriate indicators and measurement tools.

Next steps

Log in to the grant administration system and begin the application.

Contact Jessica with application questions earlier rather than later.

Complete the Full Application by May 29, 2014, 12 pm (noon).

Communicate with your team and discuss possible outcomes with your team.

Contact InformationFor all questions and technical support: contact

Jessica Gonzalez at 727-507-9668 or jgonzalez@afmfl.org

For help with outcomes and success measures, first contact Jessica Gonzalez and then our evaluation consultant, Shelley Robertson, 239-633-3241, info@snrobertson.com

Questions?