JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program...
-
Upload
marshall-oneal -
Category
Documents
-
view
214 -
download
0
Transcript of JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program...
![Page 1: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/1.jpg)
JARED C. SCHULTZ, PH.D., CRCUTAH STATE UNIVERSITY
Using Quantitative Data for
Effective Program Evaluation
![Page 2: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/2.jpg)
Misleading Title?
The first critical question is not “How do we measure?”
The first critical question is “What do we measure?”
This presentation is not an overview of quantitative data collection procedures.
This presentation is intended to provide a foundation to address the question of “What do we measure?”
![Page 3: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/3.jpg)
Transformational Learning
Adult learners bring the context of their experience to learning
Has a positive and negative impact on learning
“becoming critically aware of one’s own tacit assumptions and expectations and those of others and assessing their relevance for making an interpretation” (Mezirow, 2000,p.4)
Take a few minutes and write down your beliefs, attitudes, and emotions regarding program evaluation. Positive and Negative. Ambitions and Concerns.
![Page 4: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/4.jpg)
What Program Evaluation is Not
Only a requirement for reporting to RSA and other oversight entities.
Program evaluation, quality assurance, and continuous improvement are not events, or things to implemented quickly. They represent shifts in organizational ideology and culture, and require a long-term, systemic view.
![Page 5: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/5.jpg)
What is Program Evaluation?
“The use of social research methods to systematically investigate the effectiveness of social intervention programs…” (Rossi, Lipsey, & Freeman, 2004).
Data Collection and AnalysisIlluminates Effectiveness and Informs
Continuous ImprovementIs useful for all aspects of organizational
functioning.
![Page 6: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/6.jpg)
An issue of organizational culture.A systemic method of addressing quality of
services, outcomes, and personnel.
Addresses processes and systems within the organization.
![Page 7: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/7.jpg)
Role of Data
Informs Program DevelopmentInforms Program EvaluationInforms Program ImprovementInforms Professional HiringInforms Professional EvaluationInforms Professional Improvement /
Development
![Page 8: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/8.jpg)
Bead Experiment
We need to hire: 2 Counselors Supervisor Data Analysis Specialist
![Page 9: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/9.jpg)
Processes and Systems
Process The use of inputs into an action to achieve a desired
outcome. Inputs include people, materials, equipment, methods,
measurement, environment.
System – network of processes within an organization that work together for the aim of the organization.
(Neave, 1990; Deming, 2000)
![Page 10: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/10.jpg)
An Example of Systems Thinking
A State agency implemented an incentive payment program that was directly linked to case closures.
![Page 11: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/11.jpg)
Incentive Program Goals
Improve Rehabilitation Counselor Retention, Decrease Turnover
Increase Rehabilitation Counselor Compensation
Increase the quantity of Successful closures, Productivity
It was originally presented as a Trial Program
![Page 12: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/12.jpg)
Since Implementation
Voluntary terminations have increased from an average of 15 per year to 21 per year
Voluntary resignations have not declined with the existence of the incentive program
![Page 13: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/13.jpg)
Compensation
An average of 25 counselors receive the incentive in a given year ($894 before taxes)
On average 91 counselors receive no incentive
The average increase in compensation across the board is $180 before taxes per counselor for the 3 year period
![Page 14: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/14.jpg)
Increase Productivity
Average 26's Per Counselor
23.5
24
24.5
25
25.5
26
26.5
27
27.5
28
28.5
2002 2003 2004 2005 2006 2007
![Page 15: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/15.jpg)
Asking the Program Evaluation Question
Identify the ProblemDefine the problem in terms of processes and
systems Draw out the process Identify the way that processes influence the system
![Page 16: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/16.jpg)
Models and Theories
Models and theories provide the context for making the questions concrete.
Provide the structure for understanding the variables.
Don’t assume that models and theories do not apply to your unique situation.
![Page 17: JARED C. SCHULTZ, PH.D., CRC UTAH STATE UNIVERSITY Using Quantitative Data for Effective Program Evaluation.](https://reader034.fdocuments.us/reader034/viewer/2022051619/56649e405503460f94b318a4/html5/thumbnails/17.jpg)
References
Deming, W.E. (2000). Out of the crisis. Cambrige, MA: The MIT Press.
Mezirow, J. (2000).Learning to think like an adult: Core concepts of transformation theory. In J.Mezirow and Associates (Eds.). Learning as transformation: Critical perspectives on a theory in progress. San Francisco: Jossey-Bass.
Neave, H.R. (1990). The Deming dimension. Knoxville, TN: SPC Press.
Rossi, P.H., Lipsey, M.W., & Freeman,H.E. (2004). Evaluation: A systematic approach (7th Ed.). Thousand Oaks, CA: Sage.