Post on 15-Apr-2017
—Tools for Research Project, Program, and Evaluation Design—
Introduction to Logic Models
June 27, 2016
Discovery/EHS Professional Development
1
Presentation Overview
》Section 1, Explain:
• What is systematic inquiry?
• What is logic modeling?
• What are the purpose(s) of using a logic model?
》Section 2, Review:
• Different types of logic models (e.g., Basic, Theory,
Activities, Outcomes).
• Tips/Reasons for using each type of model.
》Section 3, Discuss:
• How can I create/design a logic model?
• Break-down and examples of logic model components.
Section 1: Systematic Inquiry & Logic Models
》Explain the idea behind “systematic-inquiry”
》Connect “big-picture” idea of systematic inquiry
to logic modeling.
》Discuss the benefit of thinking about the whole
process/cycle of research (*outcomes first)
before designing a research project or program
plan.
Systematic Inquiry Cycle
"Systematic implies [...] a thoughtful process
of asking critical questions [a.k.a, "inquiry"],
collecting appropriate information, and then
analyzing and interpreting the information
for a specific use and purpose." – Program Development and Evaluation
University of Wisconsin-Extension, Cooperative Extension, 2008.
http://www.uwex.edu/ces/4h/evaluation/documents/glossary.pdf
Planning and Design
Protecting Participant
Rights
Developing and
Modifying Instruments
Data Collection
and Processing
Data Analysis
and Reporting
What is a Logic Model?
“A program logic model is a picture of how your program works – the theory and assumptions underlying the program. ...This model provides a road map of your program, highlighting how it is expected to work, what activities need to come before others, and how desired outcomes are achieved.” - W.K. Kellogg Foundation Evaluation Handbook (1998, p. 35)
“…a valuable tool for discovering and displaying the links between activities and outcomes.” - New Directions for Evaluation (2008, p. 17)
Key Features of Logic Models
》Systematic – Visual – Dynamic - Flexible
》Sequential but not always linear
》Shows the “What” and “How” of your project/program
》May show the “Why” behind your project/program
》Shows area of strengths, weaknesses, gaps, and
limitations
》Promotes strategic/programmatic management
》Typically has 5 components (e.g., Resources/Inputs,
Activities, Outputs, Outcomes, Impacts)
Basic Logic Model Outline
W.K. Kellogg Foundation (January 2004)
Basic Logic Model Components
》PLANNED WORK
• Resources/Inputs: human, financial, organizational, or
community resources a program has available to do the work
• Activities: processes, tools, events, technology, and actions
that are an intentional part of program implementation
Basic Logic Model Components
》 INTENDED RESULTS
• Outputs: direct products of program activities
• Outcomes: specific changes in program participants’
behavior, knowledge, skills, status
• Short-term: 1-3 years; Long-term: 4-6 years
• Impact: fundamental intended or unintended change as a
result of program (outcomes 7-10 years)
Basic Logic Model Example
Put on Your Dancing Shoes
Go Dancing w/Friends
Your Dance Moves get Posted on YouTube
YouTube Video goes
Viral
You become Famous/
Overnight Sensation
Logic Model “Logic”
Certain resources are needed to operate your project/program:
If . . . Then . . .
you have access to them, you can use them to accomplish your
planned activities.
you accomplish your planned
activities,
you will deliver the amount of
product/service that you intended.
you accomplish your planned
activities to the extent intended,
your participants will benefit in specific
ways.
these benefits are achieved, certain changes might occur under
specified conditions.
Why Use Logic Modeling?
》Planning and Design
》 Implementation and Management
》Evaluation and Communication
How Logic Models Better Position Research/Programs for Success
Program ElementsCriteria for Program
Success
Benefits of Program
Logic Models
Planning and DesignProgram goals and objectives, and important side effects are well defined ahead of time.
Finds “gaps” in the theory orlogic of a program and work toresolve them.
Program goals and objectivesare both plausible and possible.
Builds a shared understandingof what the program is allabout and how the parts worktogether.
Program Implementation and Management
Relevant, credible, and usefulperformance data can beobtained.
Focuses attention ofmanagement on the mostimportant connections between action and results.
Evaluation, Communication, and Marketing
The intended users of theevaluation results have agreed onhow they will use theinformation.
Provides a way to involve andengage stakeholders in thedesign, processes, and use ofevaluation.
W.K. Kellogg Foundation (January 2004) p.6
Systematic Inquiry Cycle
Planning and Design
Protecting Participant
Rights
Developing and Modifying
Instruments
Data Collection
and Processing
Data Analysis and
Reporting
Determine purpose(s),
resources, activities, and
anticipated outcomes;
ensure participants’
safety/rights and
credibility for your work.
Planning and Design
》 Provides a road map for ideas/plan/design
》 Describes sequence of events
》 Connects the need for the program with desired results
》 Allows you to see missing links or limitations
“Most of the value in a logic model is in the process of creating, validating, and modifying the model …The clarity of thinking that occurs from building the model is critical to the overall success of the program.” - W.K. Kellogg Foundation (2004, p. 43)
Systematic Inquiry Cycle
Planning and Design
Protecting Participant
Rights
Developing and Modifying
Instruments
Data Collection
and Processing
Data Analysis and
Reporting
Manage, monitor, and
modify activities to
improve progress and
final outcomes.
Implementation and Management
》Helps visualize and understand how investments/inputs contribute to achieving intended projects or program goals
》Monitors incremental progress toward outcomes
》Facilitates project or program improvements
“Program logic modeling is a learning and management tool that can be used throughout a program’s life . . . result(ing) in effective programming and offering greater learning opportunities.” - W.K. Kellogg Foundation (2004, p. 4)
Systematic Inquiry Cycle
Planning and Design
Protecting Participant
Rights
Developing and Modifying
Instruments
Data Collection
and Processing
Data Analysis and
Reporting
Document outcomes and
share knowledge with
others.
Evaluation and Communication
》Documentation of outcomes
》Shared knowledge about what works and why
》 Improves likelihood of sustaining a program over time should you use your work for future academic endeavors (e.g., publication, funding)
“A program is a theory and an evaluation is its test. . . to organize the evaluation to provide a responsible test, the evaluator needs to understand the theoretical premises on which the program is based.” - Carol Weiss (1998, p. 55)
Section 2: Logic Model Types/Tips/Uses
》Review 3 different types of logic models:
• Theory-Approach Model
• Outcomes-Approach Model
• Activities-Approach Model
》Review reasons and tips for using each type of
logic model.
》Review additional examples of logic models.
How to create a “logic model”…
Chris Lysy (http://freshspectrum.com/)
Logic Model Tips/Types
》There is no “Best” Logic Model - Choose the model that fits your research project or program best and provides information you need in the format that is most helpful.
》Don’t worry if your model doesn’t look like others but DO worry if you cannot explain your model to others.
》Typically logic models use table and flow chart formats with text and arrows or a graphic representation of program ideas.
》A logic model does not have to be linear. Concept maps or other visuals may better describe complex program concepts.
More Logic Model Tips
Chris Lysy (freshspectrum.com)
》Models do not need to be drawn to scale.
Three Approaches to Logic Modeling
》Theory Approach Model (conceptual)
》Outcome Approach Model
》Activities Approach Model (applied)
Theory Approach Models
》Emphasize the theory of [logic] influencing design and
plan for the research project/program
》Explain how and why your project/program will work
》May have additional parts (e.g., describes problem or
issue addressed by project/program, connects
strategies to activities)
Theory Approach Model Example
W.K. Kellogg Foundation (January 2004)
W.K. Kellogg Foundation (January 2004)
Why use a Theory-based Approach?
》Demonstrates that project/program implementers have
deep understanding of project/program aspects/ideas
》Allows consideration of context (to see “Big Picture”)
》Theories of Change (ToC) and Theories of Action
(ToA) are simply extensions of Logic Modeling – and
help explain the rationale or logic behind why certain
entities are implementing a project/program.
》Facilitates project/program improvement and
evaluation
Outcomes Approach Models
》Still emphasizes the theory of change influencing
design and plan for the project/program…
》…But focuses during early stages of project/program
planning on connecting resources and/or activities with
desired results (a.k.a. outcomes)
》Subdivides outcomes and impact that may result from
a given/distinct set of activities over time (i.e., short-
term, long-term, impact)
Outcomes Approach Model Example
W.K. Kellogg Foundation (January 2004)
Why use an Outcomes-based Approach?
》Demonstrates that project/program implementers have
understanding of anticipated results
》Can use when project/program theory is
underdeveloped – but not UNdeveloped
》Facilitates project/program summative evaluation
Activities Approach Models
》Focus on project/program implementation process
emphasizing higher detail on “nuts-and-bolts” of
project/program plan.
》Clearly link activities to each other to map
implementation
》Provides more detail about steps and sequence of
project/program implementation
Activities Approach Model Example
W.K. Kellogg Foundation (January 2004)
Why use a Activities-based Approach?
》Demonstrates that project/program implementers have
understanding of relationships among project/program
activities
》Allows for studying interactions of activities
》Facilitates program monitoring/management
》Participants like them because they show what you
will actually do
》Facilitates program formative or process evaluation
Basic Logic Model (More Examples)
Outcomes Approach Model (More Examples)
Section 3: Logic Model Break-Down & Development
》Break-down of basic logic model components:
• *Remember to think “in order of design” – Outcomes, Impact,
Resources/Inputs, Activities, and Outputs.
》Review of logic model component examples.
• *If time allows – Review examples of student research/projects.
》Offer additional logic model resources.
Review of Basic Logic Model Components
Resources
InputsActivities Outputs Outcomes Impact
In order to
accomplish our
set of activities
we will need the
following…
In order to
address our
problem or
asset we will
conduct the
following
Activities…
We expect
that once
completed or
under way
these
activities
will produce
the following
evidence of
service
delivery…
We expect that
if completed or
ongoing these
activities will
lead to the
following
changes in 1–3
then 4–6
years…
We expect
that if
completed
these
activities will
lead
to the
following
changes in
7–10 years…
Break-down of Logic Model Components
Resources
InputsActivities Outputs Outcomes Impact
In order to
accomplish our
set of activities
we will need the
following…
In order to
address our
problem or
asset we will
conduct the
following
Activities…
We expect
that once
completed or
under way
these
activities
will produce
the following
evidence of
service
delivery…
We expect that
if completed or
ongoing these
activities will
lead to the
following
changes in 1–3
then 4–6
years…
We expect
that if
completed
these
activities will
lead
to the
following
changes in
7–10 years…
Outcomes
Outcomes Example(s)
Outcomes are specific changes in
attitudes, behaviors, knowledge,
skills, status, or level of functioning
expected to result from program
activities, most often at an
individual or group level (Short-
term: 1-3 years; Long-term: 4-6
years).
Earlier Example: Dancing YouTube
video goes viral.
Another Example: Teachers create
more inquiry-based STEM units for
their classrooms, and have
increased comfort levels creating
inquiry-based STEM units (medium-
term), and a deeper understanding
of how to teach inquiry-based
STEM (long-term).
》 Sometimes “reverse-thinking” is required – What results do I expect…(i.e., think about outcomes first)
》 SMART GOALS: Specific, Measurable, Action-oriented, Realistic, and Timed
Differences between Outcomes vs. Outputs
Outputs Outcomes
Direct results of program activities
(e.g., # of meetings, students
enrolled)
Results you expect to achieve over
time as a result of implementing
activities.
》 Remember the differences between components of a logic model - *Conducting an activity is not the same as achieving results from accomplishing that activity.
Think About Basic Logic Model Components
Resources
InputsActivities Outputs Outcomes Impact
In order to
accomplish our
set of activities
we will need the
following:
In order to
address our
problem or
asset we will
conduct the
following
activities:
We expect
that once
completed or
under way
these
activities
will produce
the following
evidence of
service
delivery:
We expect that
if completed or
ongoing these
activities will
lead to the
following
changes in 1–3
then 4–6 years:
We expect
that if
completed
these
activities will
lead
to the
following
changes in
7–10 years:
Impacts
Impacts Example(s)
Impacts are collective
organizational, community, and/or
system-level changes expected to
result from program activities.
Earlier Example: You become
famous via overnight sensation.
Another Example: Students have
better critical thinking skills and
choose STEM career pathways at a
higher rate.
》 Long-Term: 7 – 10 years, such as improved conditions, increased capacity, or policy changes.
Think About Basic Logic Model Components
Resources
InputsActivities Outputs Outcomes Impact
In order to
accomplish our
set of activities
we will need the
following:
In order to
address our
problem or
asset we will
conduct the
following
activities:
We expect
that once
completed or
under way
these
activities
will produce
the following
evidence of
service
delivery:
We expect that
if completed or
ongoing these
activities will
lead to the
following
changes in 1–3
then 4–6 years:
We expect
that if
completed
these
activities will
lead
to the
following
changes in
7–10 years:
Resources/Inputs: Available and Limiting
Enabling/Limiting Resources Example(s)
Enabling: Funding, existing
organizations, collaborating
partners, staff, time, equipment,
supplies.
Limiting: Attitudes, lack of
resources, policies, laws,
regulations, geography.
Earlier Example: Put on your
dancing shoes.
Another Example: New PD
curriculum developed; faculty
trained to teach workshop; space in
McGuffey Hall provided for PD
workshops.
》 Resources and/or barriers, which potentially enable or limit program effectiveness.
Activities
Activities Example(s)
Activities are processes, tools,
techniques, events, technology,
and actions used to bring about
desired results
• Products: promotional materials,
educational curricula
• Services: education, training
• Infrastructure: organizational
structure, relationships, capacity
Earlier Example: You go dancing
with your friend.
Another Example: Teachers
participate in a 2-week PD
workshop and learn to create
inquiry-based STEM units as part of
their curricula.
Break-down of Logic Model Components
Resources
InputsActivities Outputs Outcomes Impact
In order to
accomplish our
set of activities
we will need the
following…
In order to
address our
problem or
asset we will
conduct the
following
Activities…
We expect
that once
completed or
under way
these
activities
will produce
the following
evidence of
service
delivery…
We expect that
if completed or
ongoing these
activities will
lead to the
following
changes in 1–3
then 4–6
years…
We expect
that if
completed
these
activities will
lead
to the
following
changes in
7–10 years…
Outputs
Outputs Example(s)
Outputs are the direct results of
program activities. They indicate if
a program was delivered to the
intended audiences as planned.
• Size and/or scope of
services/products delivered
• Number of classes taught,
meetings held, or materials
products
• Participation rates and
demography
Earlier Example: Your dance moves
get posted on YouTube.
Another Example: 24 new inquiry-
based science units are created as
a result of teacher PD workshop.
Presentation References
》 W. K. Kellogg Foundation. (January 2004). Using Logic Models to
Bring Together Planning, Evaluation, and Action: Logic Model
Development Guide. Battle Creek, Michigan.
》 Weiss, C. H. (1998). Evaluation: Methods for studying programs
and policies. (2nd Ed). Upper Saddle River, NJ: Prentice Hall.
》 Hendricks, M., Plantz, M. C., & Pritchard, K. J. (2008). Measuring
outcomes of United Way–funded programs: Expectations and
reality. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and
evaluation. New Directions for Evaluation, 119, 13–35.
》 Chris Lysy graphics: http://freshspectrum.com
Helpful Resources
Logic Models Development
》The Evaluation Forum (1999). Outcomes for success.
》Targeting Outcomes of Programs.
http://deal.unl.edu/TOP/synopsis.htm
》 Innovation Network, Inc. electronic logic model
development tool http://www.inetwork.org
Planning and Design
》W.K. Kellogg Foundation, http://www.wkkf.org
》Taking Stock. http://www.horizon-research.org
For more information:
》 Emily Ryan
• ryanee2@miamioh.edu
Discovery Center for
Evaluation, Research, and
Professional Learning
210 E. Spring St.
408 McGuffey Hall
(513) 529-1686
discoverycenter@MiamiOH.edu
》 Yue Li
• liy@miamioh.edu
》 Kristen Morio
• moriokl@miamioh.edu