Hierarchical Topic Models and the Nested Chinese Restaurant Process
Improving evaluations and utilization with statistical edge nested data designs and hierarchical...
-
Upload
cestoronto -
Category
Technology
-
view
483 -
download
3
description
Transcript of Improving evaluations and utilization with statistical edge nested data designs and hierarchical...
![Page 1: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/1.jpg)
Improving Evalua/ons and U/liza/on with Sta/s/cal Edge: Nested Data Designs and
Hierarchical Linear Modeling (HLM)
CES Conference -‐ June 10 2013
Marci Pernica -‐ Ministry of Community and Social Services Judith Godin – J Godin Consul7ng
![Page 2: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/2.jpg)
• The goal of this presenta0on is to introduce the concept of HLM and explain how it can be used in program evalua0on
Introduc0on
![Page 3: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/3.jpg)
What is Hierarchical Linear Modeling (HLM)
• HLM is a sta0s0cal technique to analyze data that is structured in hierarchies (or “nested”) • To account for the fact that people that are “clustered” or
“nested” within the same group have more in common than if they were independent random samples
Classroom 1 Classroom
2 Classroom 3
Student 4
Student 2
Student 5
Student 3 Student 8 Student 7 Student 6 Student 9 Student 10
Student 11
Student 12
Nested data designs
![Page 4: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/4.jpg)
Student
Class
School
District
Hierarchical Structure – mul0-‐level
age I.Q
Measuring test scores (dependent variable)
Independent variables
![Page 5: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/5.jpg)
• HLM enables a more robust analy0c approach for nested data (than regression or ANOVA) • Data in evalua0on are oZen nested
• To determine success condi*ons for the program – e.g. is the program more suitable for certain sub-‐popula0ons or more successful if delivered in a certain way
Why Use HLM in Evalua0on
![Page 6: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/6.jpg)
Program design structure
Data structure
Evalua0on ques0ons -‐ Which par0cipant or site-‐level characteris0cs are most influen0al in
explaining the varia0on in test scores among the program par0cipants?
-‐ What program delivery characteris0cs (site level prac0ces) seem to be having the most posi0ve impact on the par0cipants’ test scores?
-‐ Are some program features more suited to certain sub-‐popula0ons (e.g. gender, age group, ethnic or cultural group)
Applying HLM in Evalua0on
![Page 7: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/7.jpg)
Par0cipant 1
Site 1 Site 2 Site 3
Par0cipant 4
Par0cipant 2
Par0cipant5
Par0cipant 3
Par0cipant 8
Par0cipant 7
Par0cipant 6
Par0cipant 9
Par0cipant 10
Par0cipant 11
Par0cipant 12
Example of levels of a hierarchical model
Par0cipants (level 1) nested within sites (level 2)
![Page 8: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/8.jpg)
Assessing test scores by age from site to site
Test Score
Age
Four different program sites
Although the test scores differ from site to site, the rela0on between age and test score is the same at different sites
![Page 9: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/9.jpg)
Student 1 Student 2 Student 3
Baseline
Month 7
Month 1
Month 12
Month 6 Month 9 Month 8 Month 6 Baseline Month 2 Month 4 Month 6
Assessing change over 0me
Assessments across 0me (level 1) are nested within individuals (level 2) (i.e. repeated measures design)
![Page 10: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/10.jpg)
Assessing improved performance over 0me
Test Score
Time
Four different study par0cipants
Although some individuals have higher test scores to start with, the rate of change (improved performance) is comparable among the par0cipants
![Page 11: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/11.jpg)
Tradi0onal Methods
Test Score
Age
Rela0on between age and test score es0mated once for all sites together
![Page 12: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/12.jpg)
Advantages of HLM
Test Score
Age
Four different program sites
Here, the rela0on between age and test score varies across sites.
Are there any site level variables associated with the strength of this rela0on?
![Page 13: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/13.jpg)
Design Considera0ons for Using HLM
• Sample size – Par0cipant level – Site level – Repeated measures
• Missing Data – Can be easy or difficult to deal with
• Number of variables – Comprehensive coverage – Parsimony
![Page 14: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/14.jpg)
Final Thoughts
• Applying HLM in evalua0ons with nested data enables more robust results and conclusions
• U0liza0on-‐focused – Iden0fy evidence-‐based success factors or condi0ons for improving the program delivery model, to ul0mately achieve beger program effec0veness
– Promo0ng the value in evalua0on (gathering the evidence to determine the ‘success factors’ for the interven0on to be effec0ve)
![Page 15: Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling](https://reader034.fdocuments.us/reader034/viewer/2022051515/553911b0550346f53d8b4909/html5/thumbnails/15.jpg)
Ques0ons?
Marci Pernica [email protected] Ministry of Community and Social Services Judith Godin sta/s/[email protected] Independent Consultant