JARED C. SCHULTZ, PH.D., CRCUTAH STATE UNIVERSITY
Using Quantitative Data for
Effective Program Evaluation
Misleading Title?
The first critical question is not “How do we measure?”
The first critical question is “What do we measure?”
This presentation is not an overview of quantitative data collection procedures.
This presentation is intended to provide a foundation to address the question of “What do we measure?”
Transformational Learning
Adult learners bring the context of their experience to learning
Has a positive and negative impact on learning
“becoming critically aware of one’s own tacit assumptions and expectations and those of others and assessing their relevance for making an interpretation” (Mezirow, 2000,p.4)
Take a few minutes and write down your beliefs, attitudes, and emotions regarding program evaluation. Positive and Negative. Ambitions and Concerns.
What Program Evaluation is Not
Only a requirement for reporting to RSA and other oversight entities.
Program evaluation, quality assurance, and continuous improvement are not events, or things to implemented quickly. They represent shifts in organizational ideology and culture, and require a long-term, systemic view.
What is Program Evaluation?
“The use of social research methods to systematically investigate the effectiveness of social intervention programs…” (Rossi, Lipsey, & Freeman, 2004).
Data Collection and AnalysisIlluminates Effectiveness and Informs
Continuous ImprovementIs useful for all aspects of organizational
functioning.
An issue of organizational culture.A systemic method of addressing quality of
services, outcomes, and personnel.
Addresses processes and systems within the organization.
Role of Data
Informs Program DevelopmentInforms Program EvaluationInforms Program ImprovementInforms Professional HiringInforms Professional EvaluationInforms Professional Improvement /
Development
Bead Experiment
We need to hire: 2 Counselors Supervisor Data Analysis Specialist
Processes and Systems
Process The use of inputs into an action to achieve a desired
outcome. Inputs include people, materials, equipment, methods,
measurement, environment.
System – network of processes within an organization that work together for the aim of the organization.
(Neave, 1990; Deming, 2000)
An Example of Systems Thinking
A State agency implemented an incentive payment program that was directly linked to case closures.
Incentive Program Goals
Improve Rehabilitation Counselor Retention, Decrease Turnover
Increase Rehabilitation Counselor Compensation
Increase the quantity of Successful closures, Productivity
It was originally presented as a Trial Program
Since Implementation
Voluntary terminations have increased from an average of 15 per year to 21 per year
Voluntary resignations have not declined with the existence of the incentive program
Compensation
An average of 25 counselors receive the incentive in a given year ($894 before taxes)
On average 91 counselors receive no incentive
The average increase in compensation across the board is $180 before taxes per counselor for the 3 year period
Increase Productivity
Average 26's Per Counselor
23.5
24
24.5
25
25.5
26
26.5
27
27.5
28
28.5
2002 2003 2004 2005 2006 2007
Asking the Program Evaluation Question
Identify the ProblemDefine the problem in terms of processes and
systems Draw out the process Identify the way that processes influence the system
Models and Theories
Models and theories provide the context for making the questions concrete.
Provide the structure for understanding the variables.
Don’t assume that models and theories do not apply to your unique situation.
References
Deming, W.E. (2000). Out of the crisis. Cambrige, MA: The MIT Press.
Mezirow, J. (2000).Learning to think like an adult: Core concepts of transformation theory. In J.Mezirow and Associates (Eds.). Learning as transformation: Critical perspectives on a theory in progress. San Francisco: Jossey-Bass.
Neave, H.R. (1990). The Deming dimension. Knoxville, TN: SPC Press.
Rossi, P.H., Lipsey, M.W., & Freeman,H.E. (2004). Evaluation: A systematic approach (7th Ed.). Thousand Oaks, CA: Sage.
Top Related