Accountability for Quality: Policy Innovation in

Out-of-school Time Programs

November 2, 2007

Introduction

• Why focus on quality?

– Quality matters

– Quality is measurable

– Quality is malleable

Introduction

A New Guide to Quality Metrics

• Assessing Afterschool Program Practices Tool (APT)National Institute on Out-of-School Time and the MA Department of Education

• Out-of-School Time Observation Instrument (OST)Policy Studies Associates

• Program Quality Observation (PQO)Deborah Vandell and Kim Pierce

• Promising Practices Rating Scale (PPRS)WI Center for Education Research and Policy Studies Associates, Inc.

• Program Observation Tool (POT)National Afterschool Association

• Quality Assurance System (QAS)Foundations Inc.

• Program Quality Self-Assessment Tool (QSA)New York State Afterschool Network

• School-Age Care Environment Rating Scale (SACERS)Frank Porter Graham Child Development Center, UNC

• Youth Program Quality Assessment (YPQA)High/Scope Educational Research Foundation

Introduction

Exemplar: The Youth Program Quality Assessment (YPQA)

• YPQA is a good example of new class of tools that:

– Produce data that leads to real change in staff/teacher performance

– Provide continuity that is place-based, not silo-based

– Link accountability policy with workforce development policy

– Offer a more efficient and effective use of resources than a sole focus on measuring child outcomes

Part I. About the YPQA

– Characteristics• Observation at point of service• Takes 1-2hours, training available but not required• Applies to across content areas

– Purposes:• Metric for most important part of education and

human service programs – point of service• Increase access to experiences that motivate

clients to attend and engage • Staff learning and performance change• Foundation for more effective accountability

Part II: How YPQA Works in Programs High/Scope YPQA Pilot Site

Safe Environment

Supportive Environment

Peer Interaction

Youth Engagement

•Psychological & emotional safety

•Physically safe environment

•Emergency procedures•Program space and furniture

•Healthy food and drinks

•Welcoming atmosphere

•Session flow •Active engagement

•Skill building•Encouragement

•Reframing conflict

•Experience belonging

•Be in small groups•Lead and mentor

•Partner with adults

•Make choices

•Plan•Reflect

1.33

2.17

4.01

4.92

Winter 2007 Youth Program Quality Assessment 5

2.53

2.96

3.78

4.41

North

Branch

Norm

Scores*

* The norm column represents mean scores for approximately 635 offerings within 170 programs collected during the years 2001-2006.

Part II. YPQA in Programs

• North Branch Area Community Education

• Provide a comprehensive learning program for all ages;

• Seek out human resources in our community;

• Develop citizen advisory councils;

• Strive toward minimum use of facilities

Part II: YPQA in Programs Youth Classes 2006-2007

We offered 178 youth classes in our

Fall, Winter, and Summer sessions, having

a total of 3482 participants.

Part II: YPQA in Programs North Branch Area SAC Program

• GoalsSchool Age Care (SAC) program will provide appropriate activities which will positively contribute to the total development of each youth in areas of:

personal discipline

social development

physical skills

health and safety skills

Part II: YPQA in Programs Program, Plan and Staff Procedures Handbook

• Program Curriculum Examples provided by State Department

• Behavior Guidance and Procedure

• Procedures for Tornado/Fire/Lock Down Drills

• Accidents/First Aid

• Staff Guidelines for Working with Children

Part II: YPQA in Programs So what? Now what?

“If you always do what you always did, then you will always get what you have always got.”

Our youth need us to do better than we currently do.

Program Quality Defined in North Branch:

Atmosphere + Activities = Quality

- But how do we get more specific?

Part II: YPQA in Programs Quality Scores in North Branch

Safe Environment

Supportive Environment

Peer Interaction

Youth Engagement

•Psychological & emotional safety

•Physically safe environment

•Emergency procedures•Program space and furniture

•Healthy food and drinks

•Welcoming atmosphere

•Session flow •Active engagement

•Skill building•Encouragement

•Reframing conflict

•Experience belonging

•Be in small groups•Lead and mentor

•Partner with adults

•Make choices

•Plan•Reflect

1.33

2.17

4.01

4.92

Winter 2007 YPQA Scores

Part II: YPQA in Programs Data-Driven Improvement Goals

Activities support active engagement. (YPQA quality scale II-H)

- Creative new science lessons to supplement current themes.

Part II: YPQA in Programs Data-Driven Improvement Goals

Youth have an opportunity to develop a sense of belonging. (YPQA quality scale III-L)

Youth have an opportunity to participate in small groups. (YPQA quality scale III-M)

Part II: YPQA in Programs Data-Driven Improvement Goals

Youth have opportunities to act as group facilitators and mentors. (YPQA quality scale III-N)

Part II: YPQA in Programs Challenges and Opportunities

Staff and student habits are hard to change.

Time and funds for staff training on new strategies.

New lens through which to see our program.

Creative, purposeful planning.

Engaged kids!

Part III. YPQA in State Accountability Policy

Context

• Child Care ServicesSchool Age Child Care

• County Youth BureausNYC Out of School Time

• State Education Department21st Century Community Learning Centers

• TANF - Advantage After School Program

Part III. YPQA in State Accountability Policy

• Three similar quality “trains” are moving in NY state, all focused on raising quality and building more effective forms of accountability:

– NYSAN Quality Self Assessment– Accreditation for School Age Child Care– YPQA

Part III. YPQA in State Accountability Policy

• Key roles for state policy

– Set clear priorities about quality– Fund provider networks to support

implementation of quality policy– Meet provider-level demand for more

effective forms of accountability

Part III. YPQA in State Accountability Policy

• How YPQA is used in Advantage After School Programs

Part IV. Scope of Use

• YPQA is part of state and county accountability policies:

– Cross sector (DHS& DOE) snapshots: IA, WA– Statewide 21st Century: MI, ME, MN, RI– Cities and Counties: Rochester, Detroit, Grand

Rapids, Palm Beach

STEP 1Decide to

build system

STEP 2aProgram Self-assessment

STEP 2bExternal

assessment

STEP 3Plan for improve-

ment

STEP 4Targeted

T&TA

STEP 5External

assessment with criterion

IMPROVEMENT Sequence

Accountability Policy Elements:External Data Collection, Reporting & Data Analysis, Training, Coaching

MONITORING & ACCOUNTABILITY Sequence

Part IV. Scope of Use

Generic Acct Policy Elements Using Quality Metric

STEP 6Re-assess and move forward!

Part IV. Scope of Use Leveraging Impact from Existing “Change” Resources

POS QualityAccess to key

developmental and learning

experiences

Professional Development

Local EvaluatorsAccountabilityMeasurement

Summary

• The field is looking for more effective ways to spend resources that are already dedicated to accountability, evaluation, and professional development. Observational quality assessment metrics are increasingly the answer

• Youth PQA (and tools like it) allow us to measure the most important elements of programs – and raises the possibility of increased returns on existing investments

• With the right kind of supports, this form of accountability can be effective and sustainable part of programs that serve children and youth

• Constituencies of state government want to move in this direction – want to be accountable in ways that actually and demonstrably change the quality of services available to youth

• State and local governments are already experimenting with new accountability policies based on the youth PQA – and they are affordable