INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENT The Logic and Illogic of Logic Models Jay A. Mancini...

INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENT

The Logic and The Logic and Illogic of Logic Illogic of Logic

ModelsModelsJay A. Mancini

March 17, 2008ISCE SIG on Evaluation Research


Evaluation Research Evaluation Research Processes: Inquiry and Processes: Inquiry and

DiscourseDiscourseAsking questionsConsidering elementsLogic model serves as vehicle

for focusing on “connections”


The Search for Explanatory The Search for Explanatory ToolsTools

For clarifying ideas and procedures

A visual way to “Take them by the hand”

We have more experience with words, less so with symbols

A picture, if not worth a thousand words, is at least worth a couple of hundred


The Essence of Logic The Essence of Logic ModelsModels A framework for examining “fit”

Examples: Intervention goals and supportive activities Parts and sequences Cumulative thinking and procedures

Tools for determining whether our inquiry and discourse are: Plausible Doable Testable

Ultimately knowing how well our words and deeds are aligned


Two Very General Types of Two Very General Types of Logic ModelsLogic Models

Project activity modelSuch as service delivery process

Theory and research modelStructural model of concepts and

measuresBoth seek to avoid

Theory failure (bad ideas)Implementation failure (bad procedures)


Results Management Logic Results Management Logic ModelModelParticular spin on program and

evaluation researchManage results rather than activities

But one example of using logic modelsSource: Mancini, J.A., Huebner, A.J.,

McCollum, E., & Marek, L. (2005). Program evaluation science and family therapy. In D.H. Sprenkle & F.P. Piercy (Eds.), Research methods in family therapy (2nd Ed.) (pp. 272-293. NY: Guilford.


Logic Model Template

Note:Monitoring and evaluation activities permeate all aspects of the logic model.

Identified Needs and Assets

Indicators

Short Term:

Long-term:

Resources

ActivitiesResults

Short Term:

Long-term:


Step 1: Identified Needs and Step 1: Identified Needs and AssetsAssets

The first step is to identify needs and assets in the community which allow program personnel to clearly identify and understand the issues.

Because this is the basis for the entire program, it is important to use accurate and valid information rather than uninformed hunches.

Be sure to look at needs and assets at many levels, from the community systems such as health, education, legal, etc. to families and to individuals.


Needs and assets analysis is a systematic way of gathering information that helps set priorities for action while also recognizing the assets that already exist in a community.

In the logic model process the information that evaluators are gathering to understand severity of a problem also assists them in determining program priorities, and results that accrue from program activities.

Census data, observations, records and existing data bases, and literature reviews are sources of existing information to identify needs and assets.


Step 2: Desired ResultsStep 2: Desired ResultsThe second step in the logic

model process is to articulate the desired result or change you expect to see because of your program.

This is arguably the most important aspect of a logic model because it provides the focus that all successful programs need.

At the completion of the program, what will be different?


Results should be defined as short-term and long- term.

Specifying results is useful because it enables program professionals to have a more realistic sense of how parts of the program theory fit, and what can reasonably be expected to change over a specific time period.


Short-Term ResultsShort-Term ResultsChanges in knowledge, attitudes,

and/or behaviors that are anticipated as a result of participating in programs can be considered short-term results.

More proximal than distal, expected to occur sooner rather than later; may also be precursors to other expected results.


Long-Term ResultsLong-Term ResultsSystem change or changes in

organizations, community capacity, program support, and/or program sustainability may be considered long-term results.

Also longer-term changes in individuals, families, and communities: Behaviors, attitudes, knowledge.


Step 3: IndicatorsStep 3: IndicatorsIndicators flow logically from the

desired results. How will you know if the results have been achieved, with what evidence?

By determining program indicators, program activities and results can be measured and tracked. Consequently notions, hunches, hearsay, and anecdotes are not good indicator candidates.


Indicators can be far-ranging, including results from surveys and focus groups, agency administrative data, population data, in-depth conversations, observations, as well as other measures.

You must specify program indicators for each of your program goals.


Step 4: ActivitiesStep 4: ActivitiesActivities describe programs that are

implemented to achieve the desired results.

Program activities become important only when they are linked to desired results.

Attention to the choice of programs, research upon which it is based, population to be served and critical elements of successful programs must all be considered.


Step 5: ResourcesStep 5: ResourcesResources include the people,

stakeholders, curricula, spaces, approvals, funds and anything else needed to conduct the activities to achieve the desired results to meet the identified needs and capitalize on the available assets.


Identified Needs and Assets

Indicators

Short Term:

Long-term:

Resources

Activities

Logic Model Template

Note:Monitoring and evaluation activities permeate all aspects of the logic model.

Results

Short Term:

Long-term:


Identified Needs:

• Child problems post- divorce• Legal system clogged with custody and visitation disputes• Parental stress from post-divorce conflict

Desired Results:

Short Term:• Parents and children gain knowledge on improving relationships Mid-range:• Parental communication and conflict management will improveLong-term:• Child adjustment post-divorce will improve• Custody and visitation court disputes will decrease

Activities:

• Train referral sources• Provide educational sessions for parents and children

Resources:

• Mental Health Center• Judges & Court Service Workers• Volunteers

Indicators:

• N of referrals• Attendance/ completion rates• Pre- and post- test parental functioning scores• Youth functioning scores• Children’s school success• Rates of returning to court to settle disputes

Figure 1 Logic model of Children and Divorce Seminar

Note: Monitoring and evaluation activities permeate all aspects of the logic model.


Identified Needs:

• Child problems post- divorce• Legal system clogged with custody and visitation disputes• Parental stress from post-divorce conflict

Desired Results:

Short Term:• Parents and children gain knowledge on improving relationships

Mid-range:• Parental communication and conflict management will improve

Long-term:• Child adjustment post-divorce will improve• Custody and visitation court disputes will decrease




Activities:

• Train referral sources• Provide educational sessions for parents and children

Resources:

• Mental Health Center• Judges & Court Service Workers• Volunteers

Indicators:

• Number of referrals• Attendance/completion rates• Pre- and post-test parental functioning scores• Youth functioning scores• Children’s school success• Rates of returning to court to settle disputes




Identified Needs:

• Lack of information about signs & symptoms of depression• Lack of available mental health resources• Easy access to drugs and alcohol• Poor parental monitoring• Boredom and lack of alternative activities for teens

Desired Results:

Short Term:• Class participants will demonstrate knowledge gain• Increase in mental health resources for youthMid-range:• Increased number of teens in need will receive counseling or other treatmentLong-term:• Incidence of adolescent depression will decrease

Activities:

• Provide school- based mental health services• Provide classes for teens, teachers and parents

Resources:

• Community Services Board• Schools• Parents• Teens• City Officials

Indicators:

• Completion rate of class participants• Pre and post-test measures of knowledge gains• Changes in number of teens who receive referrals• Number of programs for youth• Depression screening results on instrument annually in school settings

Figure 2 Logic Model of a Youth Development Community Intervention



Identified Needs:

• Lack of information about signs & symptoms of depression• Lack of available mental health resources• Easy access to drugs and alcohol• Poor parental monitoring• Boredom and lack of alternative activities for teens

Desired Results:

Short Term:• Class participants will demonstrate knowledge gain• Increase in mental health resources for youth

Mid-range:• Increased number of teens in need will receive counseling or other treatment

Long-term:• Incidence of adolescent depression will decrease




Activities:

• Provide school- based mental health services• Provide classes for teens, teachers and parents

Resources:

• Community Services Board• Schools• Parents• Teens• City Officials

Indicators:

• Completion rate of class participants• Pre and post-test measures of knowledge gains• Changes in number of teens who receive referrals• Number of programs for youth• Depression screening results on instrument annually in school settings




Theories of Change and Logic Theories of Change and Logic ModelsModels

Articulation of connectionsExplicit rather than tacitElevation of assumptionsFollowing example shows

steps in developing a successful youth employment initiative

INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENT““Micro-Step” Assumptions that Micro-Step” Assumptions that

Make Up Make Up Change TheoryChange Theory Training for attractive occupations is

(or can be) provided in accessible locations

Information about its availability will reach the target audience.

When young people hear of the program’s availability, they will sign up for it.

They will attend regularly. Where necessary, stipends (and

perhaps child care) will be available to youth while they are in training.

Trainers will offer quality training and they will help youth learn marketable skills.

Trainers will attend regularly and provide helpful and supportive counsel.

Youth will learn the lessons being taught about work habits and work skills.

Youth will internalize the values and absorb the knowledge.

Having attained the knowledge and skills, the youth will seek jobs.

Jobs with adequate pay will be available in the areas in which training was provided.

Employers will hire the youth to fill the jobs.

The youth will perform well. Employers will be supportive. Youth will remain on the job and they will

become regular workers with good earnings.

Carol Weiss: “Nothing as Practical as Good Theory”


Research Logic ModelsResearch Logic ModelsTwo examples from social

gerontologyStudy one (R03 funding)Study two (R01 proposal)


Individual:AgeHealth levelRaceGenderPsychological well-beingEmployment aspirationsRelated work historyAttitudes toward aging

Family:Marital statusFamily sizeFamily supportiveness Family financial needsQuality of relationshipsFamily responsibilities

Community:Current position/ time in positionJob/position demandsDistance between home and workCloseness to residentsFeeling appreciated at workplaceFeeling rewarded/benefited at workConflict at workplaceAffinity/liking for coworkersSense of community/belongingTime spent with coworkers off-siteSense of shared responsibilitySense of collective competence

Retention intentionsJob satisfactionJob competence or “readiness”Job commitment

Retention

RETENTION-RELATED OUTCOMES

ECOLOGICAL LAYERS

COMMUNITY RESULTS

Conceptual Framework: Community Ecology and Retention of Long-Term Care Employees


Figure 1: Proposed Ecological Model of Retention-Related Outcomes, Retention Intentions, and Employment Outcomes

EMPLOYMENTSTATUSLength of

EmploymentEmployed vs.Not Employed

RetentionIntentions

Job CompetenceJob CommitmentJob Satisfaction

Job MatchInteractions with

Elderly

NOTE: Exogenous demographic variables are gender, race/ethnicity, socioeconomic status, and age; the validity of the model with be examined within these contexts. Though not represented by vectors in the model both direct and indirect effects will be examined across study categories.

INDIVIDUALPsychological Well-Being, Health Well-Being,

Locus of Control, Optimism

FAMILY AND RELATIONSHIPSFamily Responsibilities and Worries, Relationship Quality

and Support, Competing Family and Work Demands, Family Structure

COMMUNITY CONNECTIONSWork Community Capacity, Supervisor Support, Affinity

For Coworkers, Connections with Coworkers (time spent off site), Work Worries

Ecological Layers Employment-Related Outcomes


Program Procedure Logic Program Procedure Logic ModelModel

Oriented around service delivery elements

Actually a set of job descriptions

Simple at first glance but reflect numerous complex processes


Program and Research Logic Program and Research Logic ModelModel

Teen parent educationExpected positive outcomes

model


Multiple Trails: Mediation and Multiple Trails: Mediation and ModerationModeration

Variability in idea complexityDegrees of descriptiveness


Program Discharge Logic Program Discharge Logic ModelModel

Depicts paths taken according to service utilization

Shows ebb and flow between ongoing support and little support post-hospitalization


The Plus Side of Logic ModelsThe Plus Side of Logic ModelsShows how we are conceiving

a process or a projectForces an examination of “fit”

How logical are we?Is the line-up correct?

Helps crystallize thinking and procedures

Visual aid for explanation to others


Logic Model CautionsLogic Model Cautions Model is only as good as our thinking May be overly simplistic

Theory itself should be simple, but not too simple May be overly complex

On its own, does it tell a story? If not considered malleable may eventually not

reflect program and research realities Logic model is an approximation and probably

incomplete Boxes, circles, and arrows can be juxtaposed in

various ways, depending on who is drawing Be mindful of discerning association from

causation Model is only as good as our thinking

INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENTBoxes, Ovals, and Arrows: Boxes, Ovals, and Arrows:

Unleashing Would-be Artists on Unleashing Would-be Artists on Program Processes Program Processes

ABCDEF

Results


ReferencesReferences Bowen, G.L., Orthner, D.K., Martin, J.A., & Mancini, J.A. (2001).

Building community capacity: A manual for U.S. Air Force Family Support Centers. Chapel Hill, NC: A Better Image Printing.

Mancini, J.A., Huebner, A.J., McCollum, E., & Marek, L. (2005). Program evaluation science and family therapy. In D.H. Sprenkle & F.P. Piercy (Eds.), Research methods in family therapy (2nd Ed.) (pp. 272-293. NY: Guilford.

Orthner, D.K., & Bowen, G.L. (2004). Strengthening practice through results management. In A.R. Roberts & K. Yeager (Eds.), Handbook of practice based research (pp. 897-904). NY: Oxford.

Rossi, P.H., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

Weiss, C.H. (1995). Nothing practical as good theory. In J.P. Connell, A.C. Kubisch, L.B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Vol. 1, Concepts, methods, and contexts (pp. 65-92). Washington, DC: Aspen.


Jay A. ManciniJay A. ManciniJay A. Mancini is the Senior

Research Fellow at Virginia Tech’s Institute for Society, Culture, and Environment, and also a Professor in the Department of Human Development ([email protected])

INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENT The Logic and Illogic of Logic Models Jay A. Mancini...

Documents

Transcript of INSTITUTE FOR SOCIETY, CULTURE AND ENVIRONMENT The Logic and Illogic of Logic Models Jay A. Mancini...