ECEC Research, Evaluation and Monitoring What should be ...
Transcript of ECEC Research, Evaluation and Monitoring What should be ...
![Page 1: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/1.jpg)
ECEC Research, Evaluation and Monitoring What should be monitored, why, and how?
![Page 2: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/2.jpg)
Building on SSIII
“…information is lacking across all areas of ECEC provision…”
“…we not only need to develop and build extensive data bases that can connect information across ECEC sectors, but we must ensure that these are connected with rigorously designed research studies if we are to use the data to inform programme improvements leading to increasing the effectiveness of our early learning efforts globally…”
![Page 3: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/3.jpg)
Public Policy
![Page 4: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/4.jpg)
Public Policy
Research can point us towards what is optimal…
…but public policy must deal in trade-offs
Where to spend dollars, euros, or political capital is easy
Where to spend your next dollar is more important
…and how you spend your existing dollars is probably more important than that
![Page 5: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/5.jpg)
Public Policy
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
5
Level 2
Level 3/4
Level 5
ECERS-E Literacy Sub-Scale Scores by Manager Qualification
Derived from EPPE (Quality in Early Childhood Settings, Kathy Sylva)
![Page 6: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/6.jpg)
Public Policy
0
5.000
10.000
15.000
20.000
25.000
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Qualified
Unqualified
Numbers of New Zealand ECE Teachers, Qualified and Unqualified
![Page 7: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/7.jpg)
Public Policy
Poor Fair Good Very good
05
10
15
20
25
2004
2006
2009
Overall quality shifts 2004 to 2006 to 2009
![Page 8: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/8.jpg)
Public Policy
Expenditure, $ per full time equivalent child
$0
$2.000
$4.000
$6.000
$8.000
$10.000
$12.000
2001/02 2002/03 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12
![Page 9: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/9.jpg)
Public Policy
% of services with more than 80% of teachers qualified, by SES decile
40%
42%
44%
46%
48%
50%
52%
54%
56%
1 (highest SES)
2 3 4 5 6 7 8 9 10 (lowest SES)
![Page 10: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/10.jpg)
Policy Questions
What works?
What works best?
How do Governments make it work?
How are the gains distributed?
Does our existing body of knowledge enable us to
answer the key policy questions?
![Page 11: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/11.jpg)
Where we get our information – what it tells us
![Page 12: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/12.jpg)
What We Know at the Moment
Research – a number of significant studies have highlighted, across a wide variety of contexts, the importance of the early years
e.g. Strengths and Needs in the Early Years NZ study found statistically significant improvements in reading scores for children attending ECE (but no correlation with disruptive behaviour)
Evaluation – large scale evaluations have considered and highlighted important factors for delivering gains (access, quality)
e.g. the NZ evaluation of the ECE Strategic Plan found correlations between observed process quality and improved structural quality where more qualified teachers were present (but no consistent effect for better ratios)
![Page 13: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/13.jpg)
What Does ECEC Look Like?
Intervention (e.g. Perry, Abecedarian)
Programme (e.g. Head Start, Sure Start)
System (e.g. schools)
![Page 14: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/14.jpg)
What Does ECEC Look Like?
Intervention (e.g. Perry, Abecedarian)
Programme (e.g. Head Start, Sure Start)
System (e.g. schools)
Very small scale, experimental design delivered with high control / fidelity
Broader scale, tends to be targeted, delivered with a range of variability
Near-universal, commonly understood, features of boundary object, politically significant
![Page 15: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/15.jpg)
What Does ECEC Look Like?
Intervention (e.g. Perry, Abecedarian)
Programme (e.g. Head Start, Sure Start)
System (e.g. schools)
Research
Evaluation
Monitoring
![Page 16: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/16.jpg)
Monitoring
![Page 17: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/17.jpg)
Monitoring
Frequent
Long-term
High cost
Financial
Compliance
Highly complex systems required
Difficult to focus – e.g. easy to collect what’s easy, not what’s useful
![Page 18: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/18.jpg)
Best Practice
www.ecedata.org
![Page 19: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/19.jpg)
Early Childhood Data Collaborative
Are children, birth to age 5, on track to succeed when they enter school and beyond?
Which children have access to high-quality early care and education programs?
Is the quality of programs improving?
What are the characteristics of effective programs?
How prepared is the early care and education workforce to provide effective education and care for all children?
What policies and investments lead to a skilled and stable early care and education workforce?
![Page 20: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/20.jpg)
Early Childhood Data Collaborative
Unique statewide child identifier
Child-level demographic and program participation information
Child-level data on child development
Ability to link child-level data with K–12 and other key data systems
Unique program site identifier with the ability to link with children and the ECE workforce
Program site data on structure, quality and work environment
Unique ECE workforce identifier with ability to link with program sites and children
Individual ECE workforce demographics, including education, and professional development information
State governance body to manage data collection and use
Transparent privacy protection and security practices and policies
![Page 21: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/21.jpg)
Examples of data collection structures
![Page 22: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/22.jpg)
Types of national data collection by subject area
0 1 2 3 4 5 6 7
Overall
Demographic
Attendance
Service quality
Workforce
Outcomes / achievement
Health
Individual
Aggregate
![Page 23: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/23.jpg)
Case Study – New Zealand
Current state –
two data collections, few linked data, no child-level data
paper-based collection
most data from one-week census
Problems include -
Infrequent, inaccurate data; hard to react or assess impact of policy
We don’t know how many children there are or what they’re up to
![Page 24: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/24.jpg)
Case Study – New Zealand
Early Learning Information System
Improve Knowledge Help to link ECE information
Help to target resources better Achieve 98% participation
- $25m project - Idea to implementation – 6 years
- Hugely complex - Enormous change programme for
5,000 providers
![Page 25: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/25.jpg)
Case Study – Singapore
Target of 98.8% participation of 6yos by 2021
92
93
94
95
96
97
98
99
100
2006 2007 2008 2009 2010 2011
6YO children attending pre-school (%)
![Page 26: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/26.jpg)
Case Study – Singapore
Strong reliance on data use and matching – use of cross-Government administrative data matching
Matched with other strategies:
Outreach (home visiting)
Awareness, information and direct contact
Pre-school placement
![Page 27: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/27.jpg)
International Monitoring
![Page 28: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/28.jpg)
International Monitoring
Focus mostly on indicators from national administrative data
Or trans-national studies (PISA, PIRLS, TIMMS)
Increases reliability and comparability
Extends sample size, e.g.
![Page 29: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/29.jpg)
International Monitoring
Focus mostly on indicators from national administrative data
Or trans-national studies (PISA, PIRLS, TIMMS)
Increases reliability and comparability
Extends sample size, e.g.
Ratios for 3 year olds
In NZ, I can experiment on a range between 1:7 and 1:15
Across the OECD, I can experiment between 1:6 and 1:25
![Page 30: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/30.jpg)
International Indicators
How good is our ECEC system?
![Page 31: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/31.jpg)
International Indicators
0
10
20
30
40
50
60
70
80
90
100
New Zealand Country A Country B Country C
How good is our ECEC system?
Excellent
Rubbish
![Page 32: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/32.jpg)
International Indicators
0
10
20
30
40
50
60
70
80
90
100
New Zealand Country A Country B Country C
How good is our ECEC system?
Excellent
Rubbish
Is this a useful, representative measure of how good the system is? (What works?)
![Page 33: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/33.jpg)
International Indicators
0
10
20
30
40
50
60
70
80
90
100
New Zealand Country A Country B Country C
How good is our ECEC system?
Excellent
Rubbish
Is this country really half as good as New Zealand? (What works best?)
![Page 34: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/34.jpg)
Improving Indicators
0
10
20
30
40
50
60
70
80
90
100
New Zealand Country A Country B Country C
How good is our ECEC system?
Excellent
Rubbish
Should this country be trying to get to 100? Or is 80 good enough? Are the greatest gains between 0 and 50, or 90 and 100? (What works best?)
![Page 35: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/35.jpg)
Improving Indicators
0
10
20
30
40
50
60
70
80
90
100
New Zealand Country A Country B Country C
How good is our ECEC system?
Excellent
Rubbish
How many children in this country get 40? Do some get 20 and some 60? If so, who, and how much? (How are the gains distributed?)
![Page 36: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/36.jpg)
Improving Indicators
0
2
4
6
8
10
12
14
16
18
20
22
24
26
Chart C2.3. Ratio of students to teaching staff in early childhood education (2010)Public and private institutions
Countries are ranked in descending order of students to teaching staff ratios in early childhood education.
Source: OECD. Argentina and Indonesia: UNESCO Institute for Statistics (World Education Indicators Programme). Table C2.2. See Annex 3 for notes (www.oecd.org/edu/eag2012).
Student to teaching staff ratio
![Page 37: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/37.jpg)
Improving Indicators - conclusions…
We can probably collect quite a lot of comparative data regularly:
Levels of qualification
Mixture of qualification
Group size and ratio
Space and facilities
But it’ll need additional contextual data to make it really useful
![Page 38: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/38.jpg)
Challenges – Answering Policy Questions
How do Governments make it work?
Outcomes
![Page 39: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/39.jpg)
Challenges – Answering Policy Questions
Marginality
Impacts
Balance of interventions
Funding
Taxation
Regulation
Information
Relationships
Choice architecture
![Page 40: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/40.jpg)
Challenges – Answering Policy Questions
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
90,0
100,0
$0
$200.000
$400.000
$600.000
$800.000
$1.000.000
$1.200.000
$1.400.000
$1.600.000
$1.800.000
2001/02 2002/03 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12
Government Expenditure % of 4YOs participating
New Zealand – participation and expenditure
![Page 41: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/41.jpg)
Challenges – Answering Policy Questions
![Page 42: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/42.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
![Page 43: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/43.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
• Responsive interactions
• High quality attachments
• Safe and welcoming environments
![Page 44: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/44.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
• Responsive interactions
• High quality attachments
• Safe and welcoming environments
• Child wellbeing
• Child learning
![Page 45: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/45.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
• Responsive interactions
• High quality attachments
• Safe and welcoming environments
• Child wellbeing
• Child learning
(Structural Quality) (Process Quality)
![Page 46: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/46.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
• Responsive interactions
• High quality attachments
• Safe and welcoming environments
• Child wellbeing
• Child learning
To find out what works and what works best, we need information on all of these stages
![Page 47: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/47.jpg)
Challenges – Answering Policy Questions
• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum
• Responsive interactions
• High quality attachments
• Safe and welcoming environments
• Child wellbeing
• Child learning
(Structural Quality) (Process Quality)
Almost all of our current data sit here
![Page 48: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/48.jpg)
Challenges – Answering Policy Questions
350
370
390
410
430
450
470
490
510
530
550
Did not attend One year or less More than one year
Pasifika
Non-Pasifika
PISA 2009 – reading point score difference from attending ECE after controlling for SES
![Page 49: ECEC Research, Evaluation and Monitoring What should be ...](https://reader034.fdocuments.us/reader034/viewer/2022042114/6257799fe38422304d5da5dd/html5/thumbnails/49.jpg)
Challenges – Answering Policy Questions
PISA 2009 – reading point score difference from attending ECE after controlling for SES
400
420
440
460
480
500
520
540
560
Did not attend One year or less More than one year
Māori
Non-Māori