American Educational Research AssociationAnnual Conference

New York – March 24-28, 2008

Noelle Griffin, Ph.D.

Evaluation of an Arts-Based Instructional Program for Primary Grades:  Preliminary Findings and Lessons Learned from WebPlay

• Section 1: WebPlay Evaluation Overview• Section 2: Survey Design and Analysis• Section 3: Pre-/Post- Comparison Results

Overview

Section 1WebPlay Evaluation

Overview

WebPlay Program

• Internet-enhanced arts education project

• Integrates theater arts in school curriculum• Performing arts, literacy, social studies, and

technology

WebPlay Curriculum

• Two weekly lessons• Use of internet: Connections with partner classrooms• WebPlay curriculum involves activities in the

following four areas:Performing Arts

English Language Arts

History/Social Studies

Technology

Evaluation

• 3 year evaluation beginning 2005-06• 3rd and 5th grade classes/large urban school

district• Quantitative methods/small scale• Both summative and formative purposes

Evaluation Questions: WebPlay Program

1. Will WebPlay improve student performance?2. What is the impact of WebPlay on student skill

development?3. Do effects persist across different student/site

cohorts?4. Do different student/site characteristics interact

with treatment?

Sampling and Methodology

• Sample (2006-07 school year) 18 WebPlay schools

12 matched comparison schools (for survey)

• Student survey (pre-/post-)• Standardized state test data (English Language

Arts) – future analysis

Evaluation Roadblocks

• Selecting outcome measures• The focus on standardized data• Lack of resources• Comparison participation

Section 2Survey Design and Analysis

Key Constructs in WebPlay Curriculum

1. Theatrical knowledge2. Internet knowledge and safety3. Engagement4. Efficacy and academic esteem5. Collaboration and broadening horizons

Sample Items by ConstructAreas Items

1. Theatrical knowledge

The location or place (“setting”) of a play is where the play is being presented.

2. Internet knowledge and safety

E-mails as well as the documents sent via e-mail (“attachments”) can have viruses.

3. School engagement I think the activities I do in school are boring.

4. Efficacy and academic esteem

I would be able to do a good job if I had to write a story for school.

5. Collaboration and broadening horizons

I can learn new things from students who live in other countries.

Factor Analysis

• Pre- surveys• N = 590 (424 WebPlay, 166 comparison)• Exploratory and confirmatory factor

analysis techniques

Exploratory Factor Analysis

• Based on Likert-style items• Three higher order factors determined

Theatrical engagement/interest

General academic confidence/engagement

External connections

Confirmatory Factor Analysis

• Initial fits indices mixed (i.e., not all indices reached cut-offs)

• Initial model revised (allow for cross-loadings, correlated residuals)

• Consistent with survey theoretical design• Fit indices improved

WebPlay Student Survey: Final CFA Model

Section 3 Pre/Post Comparison

Results

Survey Pre-/Post- Analyses

• Comparing pre/post results for WebPlay and comparison students

• Based on only students with link-able pre/post data (i.e., smaller sample)

• Hierarchical linear modeling techniques used (due to nested nature of data)Two-level model (student and school)

Applicability of Hierarchical Linear Models (HLMs) to Evaluation Studies

• HLM allows analysis of outcome variable relationship to predictors at multiple data levels

• Evaluation studies often involve: Intact clusters

Nested data

Background characteristics of individuals that may vary appreciably across different clusters

Overall Results

• Two sets of outcomes analyzedKnowledge

Attitude/Engagement

• Knowledge: No significant group differences (p = .41)

• Engagement/attitude: WebPlay students increased pre-/post- relative to comparison students (p = .02)

HM Results for Knowledge Outcome

Fixed Effects Coefficient SE p Value

Intercept 7.10 0.21 <.0001

WebPlay -0.22 0.26 0.41

Pre-test 0.20 0.06 0.00

Random EffectsVariance Compone

ntSE p Value

Adjusted means 0.18 0.10 0.03

Student residual 2.12 0.17 <.0001

HM Results for Engagement/Attitude Outcome

Fixed Effects Coefficient SE p Value

Intercept 3.80 0.07 <.0001

WebPlay 0.19 0.08 0.02

Pre-test 0.57 0.05 <.0001

Random Effects

Variance Componen

tSE p Value

Adjusted mean 0.00 0.01 0.43Student residual 0.36 0.03 <.0001

Sub-scale Analyses

• Descriptive/exploratory in nature• No variation in impact between types of

Knowledge items (theater, internet)• Some apparent variation in impact on

attitude/engagement sub-itemsTheatrical engagement/interest WebPlay vs.

comparison group differences most pronounced

Attitude Outcome sub-area scores

by treatment group

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

1 2

Implementation (Pre and post)

Ave

rage

sco

res

Theatrical, Control

Academic,Control

Collaboration, Control

Theatrical, Treatment

Academic, Treatment

Collaboration,Treatment

Next Steps

• Integration of state standardized test data• Revision of Knowledge items (multiple

choice)• Collection and integration of teacher-level

data

griffin@cse.ucla.edu