Evaluating Online Learning

8/14/2019 Evaluating Online Learning

1/80


2/80


3/80

Prepared by WestEdWith Edvance Research, Inc.

For

U.S. Department of Edu cation

Office of Innovation and Improvement

2008

Evaluating Online LearningChallenges and Strategies for SuccessI n n O v a t I O n S I n E d u C a t I O n


4/80

this repor ws proce er u.S. deprme o Ecio Corc no. Ed-01-CO-0012, tsk Orer

d010, wih WesE. Shro Kiey Hor sere s he corcig ocers represeie. the coe

o his repor oes o ecessrily refec he iews or policies o he u.S. deprme o Ecio, or

oes he meio o re mes, commercil procs or orgizios imply eorsemes by he u.S.

goerme. this pblicio lso cois uRLs or iormio cree miie by prie org-

izios. this iormio is proie or he reers coeiece. the u.S. deprme o Ecio

is o resposible or corollig or greeig he ccrcy, relece, imeliess, or compleeess o

his osie iormio. Frher, he iclsio o iormio or uRL oes o refec he imporce o

he orgizio, or is i iee o eorse y iews expresse, or procs or serices oere.

U.S. Department o Education

Mrgre Spelligs

Secretary

Ofce o Innovation and Improvement

dog Mesecr

Assistant Deputy Secretary

Technology in Education Programs

Cheryl Gree

Director

Jly 2008

this repor is i he pblic omi. ahorizio o reproce i i whole or i pr is gre. While

permissio o repri his pblicio is o ecessry, he ciio shol be: u.S. deprme o

Ecio, Oce o Ioio Improeme, Evaluating Online Learning: Challenges andStrategies or Success, Wshigo, d.C., 2008.

to orer copies o his repor (orer mber Ed004344P),

write o: Ed Pbs, Ecio Pblicios Ceer, u.S. deprme o Ecio, P.O. Box 1398,

Jessp, Md 20794-1398;

or ax yor reqes o: 301-470-1244;

or e-mailyor reqes o: [email protected];

or calli yor reqes oll-ree: 1-877-433-7827 (1-877-4-Ed-PuBS). those who se elecommi-

cios eice or he e (tdd) or eleypewrier (ttY), shol cll 1-877-576-7734. I 877 ser-

ice is o ye ilble i yor re, cll 1-800-872-5327 (1-800-uSa-LEaRn; ttY: 1-800-437-0833);or order online : hp://www.epbs.e.go.

this repor is lso ilble o he deprmes Web sie :

hp://www.e.go/mis/le/cemic/elolie/

O reqes, his pblicio is ilble i lere orms, sch s Brille, lrge pri, or comper

iskee. For more iormio, plese coc he deprmes alere Form Ceer 202-260-0852

or 202-260-0818.


5/80

iii

ContentsList o Illustrations and Tables iv

Foreword v

Acknowledgments vii

Introduction 1

Commo Chlleges o Elig Olie Lerig (1)

the Fere Elios (2)

Whs i this Gie (3)

Part I: Challenges o Evaluating Online Learning 7

Meeig he nees o Mliple Skeholers (7)

Bilig o he Exisig Bse o Kowlege (12)

Elig Mlicee Olie Resorces (20)

Fiig approprie Compriso Grops (26)

Solig d Collecio Problems (33)

Ierpreig he Impc o Progrm Mriy (40)

trslig Elio Fiigs Io acio (43)

Part II: Recommendations or Evaluating Online Learning 49

Geerl Recommeios (49)

Recommeios or Meeig he nees o Mliple Skeholers (50)Recommeios or uilizig Bilig o he Exisig Bse o

Kowlege (50)

Recommeios or Elig Mlicee Resorces (52)

Recommeios or Progrms Cosierig Compriso Sy (52)

Recommeios or Gherig vli Elio d (53)

Recommeios or tkig Progrm Mriy Io acco (54)

Recommeios or trslig Elio Fiigs Io acio (54)

Featured Online Learning Evaluations 55

Appendix A: Resources 59

disce Lerig (59)Elio Mehos tools (60)

Resorces From Higher Ecio (61)

Appendix B: Research Methodology 63

Glossary o Common Evaluation Terms 65

Notes 67


6/80

iv

Illustrations and TablesFIGURES

1. Excerpt From Appleton eSchools Online Program Perceiver Instrument 17

2. Excerpt From Digital Learning Commons Meeting 21st Century Learning Challenges

in Washington State 24

3. Example o Tabulated Results From an Arizona Virtual Academy Online Parent SurveyAbout the Quality o a Recent Student Workshop 38

4. Excerpts From AZVAs Next Steps Plan, In Response to Recommendations

From the K12 Quality Assurance Audit 47

TABLES

1. Selected Variables o Profled Online Learning Evaluations 4

2. Evaluation Design Characteristics 28


7/80

v

Foreword

Ecio i his cory hs eole rmiclly rom he ys o oe echer i oe-room

schoolhose. toy, se lerig is o loger coe o physicl spce. Compers he

Iere he broke hrogh school wlls, giig ses greer opporiies o persolize heir

ecio, ccess is resorces, receie exr help or more-chllegig ssigmes, egge

i lerig i ew iqe wys.

alhogh olie lerig is reliely ew eerprise i he K12 re, i is expig rpily,

wih icresig mbers o proiers oerig serices more ses choosig o pricipe. as

wih y ecio progrm, olie lerig iiiies ms be hel ccoble or resls. ths,

i is criicl or ses heir press well s miisrors, policymkers, erso

he iormig hem bo progrm se ocomes , i rele, bo how well

priclr progrm compres o riiol ecio moels. to his e, rigoros elios re

esseil. they c ieiy wheher progrms olie resorces re perormig s promise, ,

eqlly impor, hey c poi o res or improeme.

the elios highlighe i his gie represe bro specrm o olie opios, rom pro-

grms h proie olie corses o Web sies h ere ecio resorces. the elios

hemseles rge rom ierl ssessmes o exerl, scieic reserch sies. all emosre

how progrm leers elors he bee ble o impleme srog elio prcices espie

some chlleges ihere o exmiig lerig i olie eirome.

this gie complemes oher pblicio, Connecting Students to Advanced Courses Online, pb-

lishe ls yer by he u.S. deprme o Ecio. Boh re pr o he Ioios i Ecio

series, which ieies exmples o ioie prcices rom cross he cory h re helpig

ses chiee.

My hope is h his gie will ssis elors progrm leers who seek o se o gie

progrm improeme ime chieig posiie ocomes or or ios ses.

Mrgre Spelligs, Secrery

u.S. deprme o Ecio


8/80


9/80

Acknowledgments

this gie ws eelope er he spiceso he u.S. deprme o Ecios Oce oIoio Improeme. Shro Hor wsprojec irecor.

a exerl isory grop proie eebcko ree he sy scope, ee he seleciocrieri, clriy he ex. Members icleRober Blomeyer, presie, Blomeyer & Clem-ee Coslig Serices; tom Clrk, presie,ta Coslig; tri dis, presie, Ier-iol Sociey or techology i Ecio, ssis proessor irecor o eEcio,texs a&M uiersiy; Helee Jeigs, icepresie, Mcro Ieriol, Ic.; Liz Ppe,presie chie execie ocer o virlHigh School Globl Cosorim; Lis Es,irecor, Ceer or Elio Cpciy Bil-ig, Wexor, Ic.

S i he deprme o Ecio who proi-e ip reiewe rs icle Se Bek,

Cyhi Cbell, tom Corwi, Ke deie,di d, Lorezo Esers, Mereih Frce,See Frie, Cheryl Gree, virgii Geles,Robi Gilchris, dog Herber, neie Lser,Bri Leker, Megh Lerch, tim Mger, Li Wilso. this gie wol o be possi-ble wiho he sppor o tim Mger, irecor,Oce o Eciol techology.

the see olie progrms h pricipe ihe eelopme o his gie he cse s-ies o which i is bse were geeros wih boh

heir ime eio.

Alabama Connecting Classrooms, Educators, &

Students Statewide Distance Learning

albm deprme o Ecio50 n. Ripley S.P.O. Box 302101Mogomery, aL 36104hp://ccessl.se.l.s

vii

Algebra I OnlineLoisi virl SchoolLoisi deprme o EcioP.O. Box 94064Bo Roge, La 70804-9064hp://www.loisiirlschool.e/?lgebr

Appleton eSchool2121 E. Emmers drie

appleo, WI 54915hp://www.s.k12.wi.s/Eschool

Arizona Virtual Academy4595 S. Plo vere RoSie 517/519tcso, aZ 85714hp://www.z.org

Chicago Public Schools Virtual High School125 S. Clrk S.Chicgo, IL 60603hp://cler.cps.k12.il.s/ohsp/isce_

lerig.hml

Digital Learning CommonsCorbe Bilig4507 uiersiy Wy nESie 204Sele, Wa 98105hp://www.lerigcommos.org

ThinkportMryl Pblic teleisio11767 Owigs Mills Bl.

Owigs Mills, Md 21117hp://www.hikpor.org

i prership wih:

Johs Hopkis Ceer or techology iEcio6740 alexer Bell drieSie 302Colmbi, Md 21046hp://ce.jh.e/iex.hm


10/80


11/80

Introduction

This guide is designed as a resource or leaders and evaluators o K12 online learning programs. In

this guide, the term online learning is used to reer to a range o education programs and resources

in the K12 arena, including distance learning courses oered by universities, private providers, or

teachers at other schools; stand-alone virtual schools that provide students with a ull array o online

courses and services; and educational Web sites that oer teachers, parents, and students a range o

resources.1 The guide eatures seven evaluations that represent variety in both the type o program or

resource being evaluated, and in the type o evaluation. These evaluations were selected because they

oer useul lessons to others who are planning to evaluate an online learning program or resource.

O corse, elig olie lerig is o lo-

geher iere rom ssessig y oher ype o

ecio progrm, , o some egree, el-

ors my ce he sme ki o esig

lysis isses i boh isces. Sill, olie

progrm elors my ecoer some -

icipe chlleges i he irl re owig,

or exmple, o he isce bewee progrm

sies ses, pricips miliriywih he echology beig se, lck o

rele elio ools. this gie exmies

rge o chlleges h olie progrm el-

ors re likely o mee, some h re iqe

o olie seigs ohers h re more ge-

erl. I lso escribes how olie eiromes

c someimes oer ges o elors

by preseig opporiies or sremlie

collecio lysis, or exmple.

the gie speciclly ocses o chlleges

respose sregies. all o he elios

escribe here illsre srog ssessme prc-

ices robs igs, hey re moels

or emosrig how progrm leers

elors c hle he chlleges o el-

ig olie lerig.

Common Challenges o Evaluating

Online Learning

Olie lerig is reliely ew eelopme

i K12 ecio b is rpily expig

i boh mber o progrms pricips.

accorig o repor by he norh americ

Cocil or Olie Lerig (naCOL), as o

Sepember 2007, 42 ses he [h] sigi-c spplemel olie lerig progrms (i

which ses erolle i physicl schools ke

oe or wo corses olie), or sigic ll-

ime progrms (i which ses ke mos or

ll o heir corses olie), or boh.2 I iio,

he Iere hoses eer-expig mber

o Web sies wih bro rge o ecio

resorces or ses, pres, echers.

Gie his expsio erh o exisig

reserch o he opic, i is criicl o cocrigoros elios o olie lerig i K12

seigs o esre h i oes wh people hope

i will o: help improe se lerig.

Howeer, hose erkig sch elios

my well ecoer mber o echicl

mehoologicl isses h c mke his ype

1


12/80

2

Evaluating Online Learning: Challenges and Strategies or SuccessInnOvatIOnS In EduCatIOn

o reserch icl o exece. For exmple, he

sc reserch lierre o K12 olie lerig

elio proies ew exisig rmeworks o

help elors escribe lyze progrms,

or ools, sch s sreys or rbrics, hey c

se o collec or ssess progrm qliy.

aoher commo chllege whe ses re

syig olie is he icly o exmiig

wh is hppeig i mliple, geogrphiclly

is lerig sies. a mlicee ec-

io resorcessch s s Web sies oerig

wie rge o eres or irl schools h

oer corses rom mliple eorsre lso

hr o ele, s re progrms h ilize

echologies isrciol moels h re

ew o sers.

Frhermore, elios o olie lerig o-

e occr i he coex o poliiclly lo-

e ebe bo wheher sch progrms re

worh he iesme how mch ig

is eee o r high-qliy progrm; bo

wheher olie lerig relly proies ses wih high-qliy lerig opporiies;

bo how o compre olie riiol

pproches. uersbly, ers pol-

icymkerso o meio ses heir

presw h show wheher olie

lerig c be s eecie s riiol ec-

iol pproches which olie moels re

he bes. these skeholers my or my o

hik bo elio i echicl erms, b ll

o hem re ierese i how ses perormi hese ew progrms. a he sme ime, my

olie progrm leers he mliple gols i

mi, sch s icrese se eggeme or

icrese se ccess o high-qliy corses

echers. they rge h es scores loe

re ieqe mesre or cprig impor-

iereces bewee riiol olie

lerig seigs. a, like ecors i y

seigriiol or oliehey my eel

rl repiio bo iiig elors o

ke criicl look heir progrm, erig h

i will hmper he progress o heir progrm,

rher h sreghe i.

this gie will iscss how elors he

rie o compre riiol olie lerig

pproches, wh chlleges hey he eco-

ere, wh lessos hey he lere.

The Featured Evaluations

this gie ieiolly eres riey o o-

lie progrms resorces, iclig irl

schools, progrms h proie corses olie,

Web sies wih bro eciol resorces.

Some sere eire se, while ohers sere

priclr isric. this gie lso icles is-

ic kis o elios, rom ierlly le

ormie elios (see Glossry o Commo

Elio terms, p. 65) o scieic reserch

sies by exerl expers. I some cses, pro-

grm isiers iiie he elios; i ohers,

here were exerl resos or he elio.

the ere elios icle wie rge

o collecio lysis ciiiesrom

ormie elios h rely primrily o sr-

ey, ieriew, obserio , o scieic

experimes h compre ocomes bewee

olie riiol seigs. I ech isce,

elors chose he mehos crelly, bse

o he prpose o he elio he specic

se o reserch qesios hey sogh o swer.

the gol i choosig rge o elios or

his gie ws o oer exmples h col be

isrcie o progrm leers elors i

ierse circmsces, iclig hose i ry-


13/80

3

grm or resorce, wheher he progrm seres

isric- or se-leel iece, sge o

mriy. I selecig he ere elios,

he reserchers rew rom s wie rge o

chrcerisics s possible while keepig he

qliy crieri high. a ll escripio o he

mehoology se o sy he elio(s) o

he selece sies c be o i ppeix B:

Reserch Mehoology.

the l selecio icle elios o he

ollowig olie progrms resorces: al-

bm Coecig Clssrooms, Ecors, & S-

es Sewie disce Lerig, opere by

he albm deprme o Ecio; algebr

I Olie, opere by he Loisi deprme

o Ecio; appleo eSchool, opere by

Wiscosis appleo are School disric; ari-

zo virl acemy, pblic chrer school;

Chicgo Pblic Schools virl High School;

digil Lerig Commos i Wshigo se;

thikpor, Web sie opere by Mryl

Pblic teleisio Johs Hopkis Ceer or

techology i Ecio. aiiol iorm-

io bo ech progrm is elio(s) is

icle i ble 1.

Whats in This Guide

this gie ws eelope s resorce or el-

ors, wheher exerl, hir-pry reserchers,

or progrm miisrors oher s whore cosierig cocig heir ow ierl

elio. Some o he elios highlighe

here were crrie o by exerl elors,

while ohers were coce by progrm s

or he s o pre orgizio. I ll cses,

he reserch ws erke by experiece

proessiols, his pblicio is ime

ig sges o mriy, wih ryig egrees o

ierl cpciy mos o ilble -

ig. the ere elios re o wiho

fws, b hey ll illsre resoble sregies

or cklig commo chlleges o elig

olie lerig.

to selec he ere elios, reserchers

or his gie compile iiil lis o c-

ies by serchig or K12 olie lerig

elios o he Web i pblishe oc-

mes, he expe he lis hrogh reer-

rls rom six-member isory grop (see lis

o members i he ackowlegmes secio,

p. ii) oher kowlegeble expers i he

el. Fory orgizios were o he l lis

or cosierio.

a mrix o selecio crieri ws re

reise bse o eebck rom he isory

grop. the hree qliy crieri were:

the elio cosiere mliple ocome

mesres, iclig se chieeme.

the elio igs were wiely comm-

ice o key skeholers o he progrm orresorce beig sie.

Progrm leers ce o elio resls.

Reserchers wre sies p o hree pois o

ech o hese hree crieri, sig pblicly il-

ble iormio, reiew o elio repors,

gp-llig ieriews wih progrm leers.

all he icle sies score les six o he

possible ie pois cross hese hree crieri.

Sice gol o his pblicio ws o showcse

riey o ypes o elios, he poeil

sies were coe s o sch iiol chrcer-

isics s: ierl s. exerl elor, ype o

elio esig, ype o olie lerig pro-


14/80

44


Table 1. Selected Variables o Profled Online Learning Evaluations

Name of Program

or Resource

Type of Program or Resource/

Year Initiated

Type of Evaluation

Featured in GuideaYear

Evaluation

Started

Evaluation

Objective

Alabama Connect-ing Classrooms,Educators, &Students StatewideDistance Learning

Online courses and interactivevideo-conerence classes orstudents across state / Piloted inspring 2006; statewide imple-mentation in all 2006

External; ormative &summative; includes com-parisons with traditionalinstructional settings

2006 Monitoring programimplementation; programimprovement; sharing bestpractices

Algebra I OnlineLouisiana

Online algebra courses or stu-dents across the state / 2002

External and internal;ormative and summative;includes comparisons with

traditional instructionalsettings

2003; com-parative studyin 200405

Determine i program is e-ective way to provide stu-dents with certifed algebra

teachers and to support thein-class teachers certifca-tion eorts

Appleton eSchool Online courses or students en-rolled in districts high schools(some students take all coursesonline) / 2002

Internal; ormative; evalua-tion process based on inter-nally developed rubric

Rubric pilotedin 2006

Program improvement;sharing best practices

ArizonaVirtualAcademy

Virtual charter school or stu-dents enrolled in public schoolsand homeschool students (no

more than 20%) / 2003

Formative and summative;external and internal

2003 State monitoring; qualityassurance; program im-provement

Chicago PublicSchools VirtualHigh School

Online courses or students en-rolled in districts high schools /2002

External; ormative 2002 Assess need or mentortraining and other studentsupports; identiy ways toimprove completion andpass rates

Digital LearningCommons

Web site with online coursesand a wide array o resources orteachers and students / 2003

External and internal; or-mative

2003 Understand usage o site;assess impact on studentachievement and collegereadiness

Thinkport Maryland PublicTelevision withJohn Hopkins

Web site with a wide array oresources or teachers and stu-dents / 2000

External and internal;ormative and summative,including randomized con-trolled trial

2001; ran-domizedcontrolled trialin 2005

Understand usage o site;assess impact o elec-tronic feld trip on studentperormance

a See Glossary o Common Evaluation Terms on page 65.b Run by the nonprot College Board, the Advanced Placement (AP) program oers college-level course work to high school students. Many institutions o higher education

oer college credits to students who take AP courses.c North Central Regional Educational Laboratory.


15/80

55

Cost of Evaluation Funding Source for

Evaluation

Data

Collected

Data

Collection Tools

Improvements Resulting

From Evaluation

$60,000 in 2007;$600,000 in 2008

Specifc allocationin program budget(originating romstate o Alabama)

Student enrollment, comple-tion, grades; APb coursepass rates; student andteacher satisaction; de-scription o implementationand challenges

Surveys, interviews,observations

Teacher proessional devel-opment; improvements totechnology and administrativeoperations

$110,000 or themost labor-intensivephase, including the

comparative analysisduring 200405

General programunds, grants romNCREL,c

BellSouth Founda-tion, and U.S. De-partmento Education

Student grades and statetest scores; pre- and post-tests; student use and satis-

action data; ocus groups;teacher characteristicsand teachers certifcationoutcomes

Pre- and posttestsdeveloped by eval-uator, surveys

Teacher proessional develop-ment; increased role or in-classteachers, curriculum improve-

ments, new technologies used;expansion to middle schools

No specifc allocation;Approx. $15,000 tomake the rubric andevaluation processavailable in Webormat

General programunds (originatingrom charter grantrom state o Wis-consin)

Internal descriptions and as-sessments o key programcomponents (using rubric);mentor, student, and teachersatisaction data; coursecompletion and grades

Internally developedrubric and surveys

Mentor proessional develop-ment; course content improve-ments; expanded interactivity incourses; improvements to pro-gram Web site and printed mate-rials, sharing o best practices

No specifc allocation General programunds (originatingrom state o Ari-

zona)

Student enrollment, grades,and state test scores; par-ent, teacher, and student

satisaction data; internal& external assessments onkey program components

Electronic surveys;externally devel-oped rubric

Wide range o operational andinstructional improvements

Approx. $25,000 Dist ricts Ofce o Technology Services

Student enrollment, coursecompletion, grades, andtest scores; student use andsatisaction data; mentorassessments o needs

Surveys, interviews,ocus groups

Designated class periods oronline learning; more onsitementors; training or mentors

Approx. $80,000 orthe college-readinessstudy

Bill & Melinda GatesFoundation

Student transcripts; studentgrades and completionrates; use and satisactiondata

Surveys Improvements to student ori-entation; curriculum improve-ments; development o schooluse plans to encourage bestpractices

Estimated $40,000or the randomizedcontrolled trial (parto a comprehensiveevaluation)

Star Schools Grant Student test scores oncustom-developed contentassessment; inormationabout delivery o curriculum;use and satisaction data

Test o content knowl-edge developed byevaluator, surveys,teacher implementa-tion logs

Changes to teaching materials;changes to online content andormat


16/80

66


primrily reers who re milir wih bsic

elio prcices. For reers who re less

milir wih elio, glossry o commo

erms is ilble o pge 65.

Pr I o his gie ocses o some o he likely

chlleges ce by olie progrm elors,

i is orgize io he ollowig secios:

Meeig he nees o Mliple Skeholers

Bilig o he Exisig Bse o Kowlege

Elig Mlicee Olie Resorces

Fiig approprie Compriso Grops

Solig d Collecio Problems

Ierpreig he Impc o Progrm Mriy

trslig Elio Fiigs Io acio

Ech secion o Pr I presens prcicl inor-

mion bou one o he chllenges o elu-

ing online lerning nd proides exmples o

how he eured eluions he ddressed

i. Pr II synhesizes he lessons lerned rom

meeing hose chllenges nd oers recom-

mendions bsed s well on reserch nd

conersions wih expers in eluing online

lerning. these re gered o progrm leders

who re considering n eluion nd o ssis

hem nd heir eluors s hey work ogeh-

er o design nd complee he process. Brie

profles o ech o he seen online progrms

cn be ound he end o he guide, nd de-

ils bou ech eluion re summrized in

ble 1.


17/80

77

P A R T I

Challenges ofEvaluating OnlineLearning

Meeting the Needs o MultipleStakeholders

Eery goo elio begis wih clerly s-

e prpose specic se o qesios o be

swere. these qesios rie he elio

pproch help eermie he specic

collecio echiqes h elors will se.

Someimes, howeer, progrm elors

hemseles i he icl posiio o eeig o

lll seerl prposes oce, or eeig o -

swer wie riey o reserch qesios. Mos

skeholers he he sme bsic qesioIs

i workig? B o eeryoe ees working

i he sme wy. While policymkers my be

ierese i gis i srize es scores,

progrm leers my be eqlly ierese i

oher iicors o sccess, sch s wheher he

progrm is ressig he ees o riio-

lly errepresee sbgrops, or procig

ocomes h re oly iirecly rele o es

scores, like se eggeme.

nrlly, hese qesios will o be swerei he sme wy. For exmple, i skeholers

w cocree eiece bo he impc o

se chieeme, elors migh coc

romize corolle rilhe gol s-

r or ssessig progrm eecsor qsi-

experimel esig h compres es scores o

progrm pricips wih ses i mche

compriso grops. B i skeholers w o

kow, or exmple, how progrm hs bee

implemee cross my sies, or why i is

leig o priclr ocomes, he hey migh

op or escripie sy, icorporig sch

echiqes s sreys, ocs grops, or obser-

ios o progrm pricips o gher qli-

ie process .

Whe mliple skeholers he ierig i-

eress qesios, how c elors mee

hese rios expecios?

to sisy he ems o mliple skehol-ers, elors oe combie ormie

smmie compoes (see Glossry o Com-

mo Elio terms, p. 65). I he cse o

albm Coecig Clssrooms, Ecors, &

Ses Sewie disce Lerig (aCCESS),

escribe below, he elors he bee ery

procie i esigig series o elios

h, colleciely, yiel iormio h hs il-

iy boh or progrm improeme or -

ersig progrm perormce. I he cseo arizo virl acemy (aZva), he schools

leership em hs me he mos o he my

elio ciiies hey re reqire o com-

plee by sig igs rom hose ciiies

or heir ow improeme prposes pig-

gybckig o hem wih collecio eors

o heir ow. I ech isce, progrm leers


18/80

8


wol likely sy h heir elios re rs

oremos iee o improe heir pro-

grms. Ye, whe clle po o show chiee-

me resls, hey c o h s well.

Combine Formative and Summative Evaluation

Approaches to Meet Multiple Demands

From he begiig o heir ioleme wih

aCCESS, elors rom he Ieriol Sociey

or techology i Ecio (IStE) he ke

combie smmie ormie pproch

o syig his se-r progrm h oersboh Web-bse iercie ieocoerec-

ig corses. the origil eelopme proposl

or aCCESS icle ccobiliy pl h

clle or ogoig moiorig o he progrm

o ieiy res or improeme o geer-

e sel iormio h col be shre wih

oher schools hrogho he se. I iio,

rom he begiig, progrm leers se

policymkers expresse ieres i gherig

bo he progrms impc o se lerig.to ccomplish hese mliple gols, IStE com-

plee wo sccessie elios or albm,

ech o which h boh ormie smmie

compoes. a hir elio is er wy.

the rs elio, rig he progrms pilo

implemeio, ocse o proiig eebck

h col be se o moiy he progrm, i

ee be, o geerig iormio o shre

wih albm schools. Elio ciiies his sge icle lierre reiew ob-

serio isis o six romly selece pilo

sies, where elors coce ieriews

sreys. they lso r ocs grops, or

which reserchers ieriewe respoes i

grop seig. Elors chose hese mehos

o geere qliie iormio bo how

he pilo progrm ws beig implemee

wh chges migh be eee o sreghe i.

the seco elio ook more o smm-

ie pproch, lookig o see wheher or o

aCCESS ws meeig is oerll objecies. Firs,

elors coce sreys ieriews

o ses echers, s well s ieriews

wih school miisrors persoel he

progrms hree Regiol Sppor Ceers. I

iio, hey ghere se erollme

chieeme , sewie corse erollme

compleio res, oher progrm o-come , sch s he mber o ew isce

corses eelope he mber o prici-

pig schools.

this seco elio lso se qsi-exper-

imel esig (see Glossry o Commo El-

io terms, p. 65) o proie iormio o

progrm eecs. Elors compre chiee-

me ocomes bewee aCCESS pricips

ses sewie, bewee ses iiercie ieocoerecig corses s-

es i riiol seigs, bewee s-

es who pricipe i olie corses

hose who ook corses oere i he ierc-

ie ieocoerecig orm.

as o erly 2008, he elors were coc-

ig hir elio, iegrig he rom

he rs wo sies ocsig o se

chieeme. Lookig he, aCCESS leers

pl o coie gherig lly i -

icipio o cocig logiil sies h

will ieiy aCCESSs ll impc o se

progress chieeme.

togeher, he crelly ple elio c-

iiies coce by aCCESSs elors he

geere seerl igs recommeios


19/80

99

Reasons and Contexts for Formative Versus Summative Evaluations

Formative and summative evaluations can each serve important unctions or programs. Formative evaluations,

sometimes called process evaluations, are conducted primarily to fnd out how a program is being implemented

and how it might be strengthened. Summative evaluations, also called outcome evaluations, are appropriate or

better-established programs, when program leaders have settled on their best policies and practices and want to

know, or example, what results the program is yielding.

Ideally, ormative evaluations are developed as partnerships that give all stakeholders a hand in planning and help-

ing conduct the evaluation. Explicitly raming a ormative evaluation as a collaboration among stakeholders can

help in more ways than one. Practitioners are more likely to cooperate with and welcome evaluators rather than eel

wary or threateneda common reaction. In addition, practitioners who are invited to be partners in an evaluation

are more likely to eel invested in its results and to implement the fndings and recommendations.

Even more than ormative evaluations, summative evaluations can be perceived by practitioners as threatening

and, in many cases, program sta are not eager to welcome evaluators into their midst. Even in these situations,

however, their reaction can be mitigated i evaluators work diligently to communicate the evaluations goals. Evalu-

ators should make clear their intention to provide the program with inormation that can be used to strengthen it, or

to give the program credible data to show unders or other stakeholders. In many cases, summative evaluations do

not uncover fndings that are unexpected; they merely provide hard data to back up the anecdotes and hunches o

program leaders and sta.

Program leaders who are contemplating an evaluation also will want to consider the costs o whatever type o study

they choose. Some ormative evaluations are relatively inormal. For example, a ormative evaluation might consist

primarily o short-term activities conducted by internal sta, like brie surveys o participants, to gather eedback about

dierent aspects o the program. This type o evaluation is inexpensive and can be ideal or leaders seeking ongoinginormation to strengthen their program. In other instances, ormative evaluation is more structured and ormal. For

instance, an external evaluator may be hired to observe or interview program participants, or to conduct feld surveys

and analyze the data. Having an external evaluator can bring increased objectivity, but it also adds cost.

In many cases, summative evaluations are more ormal and expensive operations, particularly i they are using

experimental or quasi-experimental designs that require increased coordination and management and sophisticated

data analysis techniques. Typically, external evaluators conduct summative evaluations, which generally extends

the timeline and ups the costs. Still, experimental and quasi-experimental designs may provide the most reliable

inormation about program eects.

Finally, program leaders should consider that an evaluation need not be exclusively ormative or summative. As

the ACCESS case illustrates (see pp. 710), sometimes it is best or programs to combine elements o both, either

concurrently or in dierent years.

h lrey he bee se o sreghe he

progrm. For exmple, heir igs sgges

h ses pricipig i he isce ler-

ig corses re compleig corses high res

, i he cse o he College Bor ace

Plceme (aP) corses,* re chieig scores

comprble o ses gh i riiol

* Run by the nonprot College Board, the Advanced Placement program oers col-

lege-level course work to high school students. Many institutions o higher education

oer college credits to students who take AP courses.


20/80

10


10

seigs. these kis o col be criicl o

miiig ig poliicl sppor or he

progrm i he re. aCCESS lso is bilig

logiil bse o proie core o

or se i re elios , progrm le-

ers hope, o help eermie aCCESSs log-erm

impc o se progress chieeme.

Wih hese collecio ools processes

i plce, aCCESS hs rme isel o ress he

ees expecios o rios skeholers

isie osie he progrm.

Make the Most o Mandatory Program

Evaluations

While ll leers s o olie ecio

progrms re likely o w o ers heir

ifece o se lerig, some he o

choice i he mer. My olie progrms ms

elier smmie se ocome becse

er or reglory boy ems i. I he

cse o arizo virl acemy (aZva), K12

sewie pblic chrer school, he progrmms comply wih seerl mory elio

reqiremes: Firs, school leers re reqire

by he se o arizo o sbmi l e-

ecieess reiew, which is se o eermie

wheher or o he schools chrer will be re-

ewe. For his yerly repor, aZva s ms

proie o se erollme, reeio,

mobiliy, se es perormce. the repor

lso ms icle ppil pre siscio

, which aZva collecs olie he e oech corse, eile sel-elio o

operiol miisrie eciecy.

aZva lso ms swer o K12 Ic., he ec-

io compy h spplies he progrms cr-

riclm or ll gre leels. K12 Ic. hs is ow

ieres i elig how well is crriclm

procs re workig i esrig h i is

prere wih high-qliy school. aZvas i-

recor, Mry Gior, sys h rom he seco

yo ope yor school, here is expecio

[o he pr o K12 Ic.] h yo will collec

, lyze hem, se hem o mke eci-

sios. K12 Ic. hs esblishe bes prcices or

cemic chieeme. they ke gre prie i

beig -rie compy, she s. I co-

cs qliy ssrce is aZva pproxi-

mely eery wo yers, which cosis o sie

isi coce by K12 Ic. persoel ex-

esie qesioire, complee by aZva, h

ocmes rios specs o he progrm, sch

s isrcio, orgiziol srcre, pr-

e-school relios. K12 Ic. lso reqires aZva

o proce eile l School Improe-

me Pl (SIP), which coers progrm oper-

ios s well s se chieeme. the pl

ms icle lysis o se perormce

o srize se ess, iclig compri-

so o he perormce o aZva ses o he

perormce o ll ses cross he se.

Ech o hese meshose o he se

hose o aZvas crriclm proierhs im-

por prpose. B he mliple reqiremes

p o wh col be see s sbsil

bre or y smll orgizio. aZvas smll

cerl s chooses o look i ierely.

alhogh he reqiremes geere yer-ro

work or aZva employees, hey he me he

mos o hese ciiies by sig hem or heirow prposes, oo. Ech o he my me

elio ciiies seres ierl prpose:

S members pore oer es scores, corse

compleio , ser siscio o

eermie how hey c improe heir progrm.

the SIP is se s giig ocme o or-

gize iormio bo wh specs o he


21/80

1111

progrm ee xig o moior he schools

progress owr is se gols. alhogh he

process is ime-cosmig, eeryoe bees:

K12 Ic. is ssre h he school is oig wh

i shol, aZva hs srcre pproch o

improig is progrm, i c emosre

o he se ohers h se perormce

is meeig expecios.

aZva is ble o mke he mos o is mory

elios becse he schools clre sp-

pors i: S members icorpore collec-

io lysis io heir eeryy resposibil-

iies, rher h iewig hem s exr bres

o heir worklo. Frhermore, aZvas le-

ers iiie collecio lysis eors

o heir ow. they reqely coc olie

sreys o pres o gge he eecieess o

priclr serices. More recely, hey lso he

beg o srey echers bo heir proessio-

l eelopme ees heir siscio wih

he riigs proie o hem. Were ryig o

o sreys er eery sigle proessiol e-

elopme [sessio], o o wh ws mos

eecie, sys Gior. do hey w more o

his, less o his? Ws his oo mch ime? Ws

his eogh ime? th eebck hs bee ery

goo. aZvas K8 pricipl, Brige Schleier,

corms h echers resposes o he sreys

re ke ery seriosly. Wheeer srey

comes p we see ee, she sys, we

will eiely p h o he ge or he

ex moh o proessiol eelopme.

togeher, hese my eors proie aZva wih

comprehesie iormio h helps he school

ress exerl ccobiliy ems, while

lso serig ierl progrm improeme ob-

jecies. Js s impor, aZvas rios el-

io ciiies re iegre sppor ech

oher. For isce, he SIP is bse o he -

igs rom he elio ciiies me

by he se K12 Ic., he lers i

process icles pe o progress me

owr SIP gols. More broly, he ormie

elio ciiies help he school leers o

se specic cemic gols eelop pl

or rechig hem, which limely helps hem

improe he chieeme ocomes ssesse by

he se. Oe lesso aZva illsres is how o

mke he mos o elios h re iiie

exerlly by reig eery collecio c-

iiy s opporiy o ler somehig l-

ble h c sere he progrm.

Summary

as he boe progrm elios emosre,

someimes he bes pproch o meeig he

ees o mliple skeholers is beig proc-

ie. the seps re srighorwr b criicl:

Whe cosierig elio, progrm le-

ers shol rs ieiy he rios skehol-ers who will be ierese i he elio

wh hey will w o kow. they migh

cosier cocig ieriews or ocs grops

o collec his iormio. Leers he ee

o si hrogh his iormio prioriize

heir ssessme gols. they shol eelop

cler isio or wh hey w heir elio

o o work wih elors o choose

elio ype h will mee heir ees. I i

is me o sere seerl iere skeholergrops, elors progrm leers migh

ecie o coc mli-meho sy h

combies ormie smmie elio

ciiies. they migh lso cosier eelop-

ig mliyer elio pl h resses

sepre gols i iere yers. I he repor-

ig phse, progrm leers elors c


22/80

1212


cosier commicig igs o iere

skeholer ieces i wys h re ilore

o heir ees ieress.

I isces where olie progrms prici-

pe i mory elios, progrm leers

shol seek o bil o hese eors se

hem or ierl prposes s well. they c

leerge he iormio lere i smm-

ie elio o improe he progrm, cqire

ig, or esblish he progrms creibiliy.

Elors c help progrm leers piggybck

o y mory ssessme ciiies by se-

lecig complemery elio mehos h

will proie o js he reqire b lso

iormio h progrm s c se or heir

ow improeme prposes.

Building on the Existing Base o

Knowledge

uer orml circmsces, elors re-qely begi heir work by reiewig ilble

reserch lierre. they lso my serch or

cocepl rmework mog similr sies or

look or exisig collecio ools, sch s

sreys rbrics, h c be borrowe or

pe. Ye, compre o my oher opics i

K12 ecio, he boy o reserch lierre

o K12 olie lerig is reliely ew

rrow. ailble escripie sies re oe

ery specic oer igs h re o es-ily geerlize o oher olie progrms or re-

sorces. Empiricl sies re ew. Oher kis

o ools or elors re limie, oo. Rece

eors he le o mliple ses o srs or

K12 olie lerig (see Srs or K12

Olie Lerig, p. 13). Howeer, here sill re

o wiely ccepe ecio progrm ocome

mesres, mkig i icl or elors o

gge sccess relie o oher olie or ri-

iol progrms.

Gie he exisig bse o kowlege o K12

olie lerig, how shol elors procee?

O corse, elors will rs w o cosl

he K12 olie lerig reserch h oes ex-

is. alhogh he el is compriely limie, i

is growig ech yer lrey hs geere

mber o sigic resorces (see ppeix

a, p. 59). amog oher orgizios, he norh

americ Cocil or Olie Lerig (naCOL)

hs eelope collece ozes o pblic-

ios, reserch sies, oher resorces se-

l or elors o K12 olie lerig pro-

grms. I some cses, elors lso my w

o look o higher ecio orgizios, which

he richer lierre o olie lerig el-

io, iclig seerl pblicios h ieiy

srs bes prcices (see ppeix a).

I some cses, hese resorces c be pe

or K12 seigs, b i oher cses, reserchers

he o, hey o o rsle well.

aoher pproch is o eelop elio ools

echiqes rom scrch. as escribe be-

low, his my be s simple s eig s-

rizig he ocome mesres se mog

mliple schools or eors, like he elors

o digil Lerig Commos i. Or i my be

mch more mbiios eor, s whe apple-

o eSchools leers eelope ew moelor elig irl schools wih olie sys-

em or compilig elio . Filly, some

elors respo o limie kowlege bse

by ig o i. For exmple, he elors o

Loisis algebr I Olie progrm he pb-

lishe heir elio igs or he bee

o oher elors progrm miisrors.


23/80

1313

I iere b eqlly helpl shio, he

leers o appleo eSchool he coribe o

he el, by eelopig Web sie h llows

miisrors o olie progrms o shre heir

bes prcices i pblic orm.

Clearly Defne Outcome Measures

Elors ms be ble o clerly ricle key

progrm gols, ee ocomes h lig wih

hem, he ieiy specic ocome me-

sres h c be se o rck he progrms

progress i meeig he gols. Presely, how-

eer, ocome mesres or elig olie

lerig progrms re o cosisely ee,which mkes i icl or skeholers o gge

progrms sccess, compre i o oher pro-

grms, or se improeme gols h re bse

o he experiece o oher progrms. Frher-

more, he lck o cosise ocome mesres

crees echicl heches or elors. a re-

ce ricle co-hore by Liz Ppe, presie

chie execie ocer o virl High School

Globl Cosorim, opro ework o o-

lie schools, escribes he problem his wy:

Although standards or online course andprogram eectiveness have been identied,

data-driven yardsticks or measuring

against those standards are not generally

agreed upon or in use. There is no general

agreement about what to measure and

how to measure. Even or measures

that most programs use, such as course

completion rates, there is variation in the

metrics because the online programs that

measure course completion rates do not

measure in the same manner.3

the elors o Wshigo ses digil

Lerig Commos (dLC) ecoere js sch

problem whe empig o clcle he

mber o olie corse-kers sere by he

progrm. dLC is cerlly hose Web porl

h oers wie rge o olie corses rom

Standards for K12 Online LearningAs online learning in K12 education has expanded, there has been an eort to begin to codiy current best prac-

tices into a set o standards that educators can look to in guiding their own perormance. Several organizations

have released sets o these emerging standards based on practitioner input to date.

In late 2006, the Educational Technology Cooperative o the Southern Regional Education Board (SREB) issued

Standards for Quality Online Courses. These standards are available along with other key resources rom SREB at

http://www.sreb.org/programs/EdTech/SVS/index.asp.

A year later, NACOL published National Standards of Quality for Online Courses, which endorsed SREBs standards

and added a ew others. The national standards cover six broad topic areas: course content, instructional design,

student assessment, technology, course evaluation and management, and 21st-century skills.

NACOL also developed National Quality Standards for Online Teachingin 2008 and is currently working on pro-gram standards. The standards or online courses and teaching can be ound at http://www.nacol.org along with

many other resources.

The National Education Association also has published standards or online courses and online teachers, both

available at http://www.nea.org.


24/80

14


meros prie eors. Whe elors

rom Cohe Reserch Elio rie o -

lyze dLCs corse-kig compleio res,

hey o rge o reporig prcices mog

he eors: Some rcke se pricipio

hrogho he corse, while ohers repore

oly o he mber o ses who complee

corse receie l gre. also, i some

cses, eors i o iereie bewee s-

es who wihrew rom corse ses

who receie F, cofig wo iere s-

e ocomes h migh he isic implic-

ios or progrm improeme.

Ppe e l. oe iiol problems i he wys

h corse compleio is ee:

When does the measure begin? How is

completion dened? Do students have a

no penalty period o enrollment in the

online course during which they may

drop rom the course and will not be

considered when calculating the course

completion rate? Is completion denedas a grade o 60 or 65? How are students

who withdrew rom the course ater the

no penalty period counted, especially i

they withdrew with a passing grade?4

Followig he recommeios o heir el-

ors, dLC s me eors o commice e-

iios o corse compleio wihrwl h

were ierlly cosise me sre ech

eor ws reporig ccre bse o

hese coersios. the resl ws higher leel

o cosisecy ccrcy i reporig.

there is growig eio o he problem o -

ee ocome mesres i he el o el-

ig olie lerig. a 2004 repor by Chy

Cgh e l., speciclly recommee

h srs be eelope or reporig he

cemic progrmmic ocomes o is-

ce lerig progrms.5 a naCOL eor is

er wy o eelop bechmrks or mesr-

ig progrm eecieess oerll srs

or progrm qliy. Mewhile, he bes el-

ors c o is o esre ierl cosisecy i

he ocome mesres se cross ll o heir

ow sorces. alhogh he reserch bse

is limie, elors my be ble o similr

sies or ies o how o ee ocomes.

Moig orwr, olie progrm elors c

prociely wys o shre reserch mehos eiios rech cosess o he

bes wys o mesre progrm eecieess.

Work Collaboratively to Develop New

Evaluation Tools

I he erly yers o appleo eSchool, school

leers becme wre h hey eee oer-

ll elio sysem o eermie he schools

sreghs wekesses. aer ilig o

exisig comprehesie ool h wol heirees, Be vogel, appleos pricipl Go-

erce Bor chir, Coie Rke, apple-

os progrm leer, beg o eelop heir

ow elio process. their gol ws o e-

sig isrme h wol ieiy he core

compoes ecessry or ses o be sc-

cessl i olie progrm. I iio, hey

we o cree process h col be se o

promp iloge mog progrm leers, s,

goerce bor members, exerl col-leges, bo he compoes o sccessl

olie lerig experiece, i orer o proie

irecio or re growh ehceme.

throgh exesie ierl iscssios

coslio wih exerl colleges, icl-

ig s rom sch olie progrms s virl


25/80

15

High School Flori virl School, vo-

gel Rke ieie eigh key progrm

compoes eelope rbric or me-

srig hem clle he Olie Progrm Percei-

er Isrme (OPPI). (See Key Compoes o

appleo eSchools Olie Progrm Perceier

Isrme, p. 16.) vogel sys, [Or gol ws]

o gre o hose iersl core compoes

h re ecessry i K12 olie progrm o

llow ses o be sccessl. We were ble

o shre [or iiil hoghs] wih oher people

i he K12 relm, sy, Wh re we miss-

ig? a wh oher pieces shol we ? I

js ki o eelope rom here.

the core compoes ieie i he OPPI re

wh appleo leers see s he esseil bil-

ig blocks or spporig ses i olie

eirome. the eigh compoes ress he

olie progrm sers eire experiece, rom

rs lerig bo he corse or progrm o

compleig i.

Whe eelopig he OPPI, vogel Rke

reserche my exisig olie progrm el-

ios i higher ecio, b o hem

iscie or bilig comprehesie rbric

he K12 leel. vogel oes, or exmple, h

hig ce-o-ce meors or coches or s-

es kig olie corses is criicl he K12

leel, wheres i is o cosiere so impor

or oler ses who re syig olie i

higher ecio. to cpre his progrm ele-me, vogel Rke icle Progrm Sp-

por s key compoe i he OPPI rbric, o-

csig o he riig gie o meors (slly

pres i he appleo eSchool moel) s well

s riig or locl school cocs who sp-

por coorie locl se ccess o o-

lie corses (see g. 1, Excerp rom appleo

eSchools Olie Progrm Perceier Isrme,

p. 17). to ssess meor percepio o progrm

qliy, he elors sreye hem ollowig

he compleio o ech olie corse.

as pr o he OPPI process, vogel Rke

eelope hree-phse pproch o ierl

elio, which hey reer o s sel-is-

coery process. I he discoery Phse, pro-

grm persoel ll o repor h escribes

he schools prcices i ech o he eigh res

ieie i he OPPI. the progrm ecisio-

mkers se he rbric o eermie wh leel o

progrm perormce is beig ie or ech

eleme: ecie, eelopig, procie, or ex-

emplry. I iio, progrm leers e-mil sr-

eys o ses, meors (slly pres),

echers he e o ech corse, giig hem

opporiy o comme o he progrms

perormce i ech o he eigh OPPI res.

I he Ocome Phse, resls rom he disco-

ery Phse repor sreys re smmrize,

geerig mericl rig i ech progrmre. a he sme ime, iormio o se

ocomes is reiewe, iclig se gres,

gre poi erges, corse compleio

res. Progrm ecisio-mkers syhesize ll

o his i ocome shee se i o

se gols or re growh eelopme.

Filly, i he Shrig o Bes Prcices Phse,

progrm leers my selec priclr prcices

o shre wih oher progrms. appleo hs pr-

ere wih oher olie progrms o orm heWiscosi eSchool nework, cosorim o ir-

l schools h shre resorces. the neworks

Web sie icles Bes Prcices Porolio

schools sig he OPPI re iie o sbmi ex-

mples rom heir elios. 6 Prcices h re

eermie o be i he procie or exem-

plry rge re cosiere or plceme i he


26/80

16


Key Components of Appleton eSchools Online Program

Perceiver Instrument (OPPI)

Practitioners and program administrators use the OPPI to evaluate program perormance in eight dierent areas:

1. Program Inormation: System provides and updates inormation necessary or prospective users to understand the

program being oered and determine whether it may be a good ft or students.

2. Program Orientation: System provides an introduction or orientation that prepares students to be successul in the

online course.

3. Program Technology: System provides and supports program users hardware and sotware needs in the online

environment.

4. Program Curriculum: System provides and supports an interactive curriculum or the online course.

5. Program Teaching: System provides and supports teaching personnel dedicated to online learning and their

online students.

6. Characteristics and Skills Displayed by Successul Online Students: System identifes and provides opportunities

or students to practice characteristics necessary or success in an online environment.

7. Program Support: System provides and supports a system o support or all online students and mentors (e.g.,

parents) and coaches.

8. Program Data Collection: System collects data and uses that data to inorm program decision-makers and share

inormation with other programs.

porolio, pricipig schools re cie or

heir coribios. the eire elio sysem

is Web-bse, llowig or sremlie col-

lecio, lysis, shrig.

appleos ecisio o eelop is ow elio

rbric process proies seerl ges.

Besies reslig i perecly ilore elio

process, appleo leers lso he he biliy o

ele heir progrm y ime wiho wiigor ig or relyig o hir-pry elor.

Sill, eelopig elio process is ypiclly

expesie my o be prcicl opio or

my progrms. appleos leers spe my

hors reserchig eelopig ocome me-

sres (i.e., he escripios o prcice or ech

progrm compoe er ech leel o progrm

perormce). they lso iese bo $15,000

o heir progrm gr s o py Web e-

eloper o esig he olie sysem or compil-

ig isplyig elio . For ohers -

empig o eelop his ype o ilore rbric

process, ccessig osie experise is criicl

o ll gps i kowlege or cpciy. appleo

leers collbore exesiely wih experiece

colleges rom oher irl schools, priclrly

s hey were eelopig heir rbric.

Share Evaluation Findings With Other

Programs and Evaluators

alhogh he OPPI rbric ws eelope speci-

clly or appleo, rom he begiig vogel

Rke iee o shre i wih oher progrms.


27/80

1717

Figure 1. Excerpt From Appleton eSchools Online Program Perceiver Instrument*

Compoe 7. Progrm Sppor. the prpose o he sppor ework is o proie iiol sppor

o he se h complemes he isrcor. this icles o oly sppor perso rom home,

b lso oher school resorces h my icle coselor, socil worker, ec.

ELEMENT

LEVEL OF PROGRAM PERFORMANCE

DEFICIENT DEVELOPING PROFICIENT EXEMPLARY

Overall StudentSupport Structure Plan

No student supportstructure plan in placeor the plan is poorly

written and/or incom-plete. Students are notreceiving necessaryresources.

Student support struc-ture plan is in placebut parts o the planare incomplete or maybe unclear. The planprovides or resourcesnecessary or studentsuccess but some gapsin the plan may exist.

Student support structureplan is in place and isclear in its purpose andobjectives. The planprovides or resourcesnecessary or studentsuccess.

Exemplary student sup-port structure plan is inplace and is a provenmodel or assuring stu-dents have necessarysupport and ongoingsupport.

Mentor responsibilitieshave been developedand shared

Mentor responsibilitieshave not been devel-oped or are incompleteand/or unclear or havenot been appropriatelyshared.

Mentor responsibilitieshave been developedand shared but partsmay be unclear and/or all mentors have notreceived the inormation.

Mentor responsibilitiesare well written, clear,and have been shared

with all mentors.

Mentor responsibilitiesare extremely well writ-ten, clear, and shared innumerous ormats withall mentors.

Mentors are providednecessary trainingand support

No mentor training pro-gram provided or thereare no written objectivesthat clearly outline thepurpose o the men-tor training program.No ongoing support isavailable.

Mentor training programis provided and thereare written objectivesthat outline the purposeo the mentor training.Some objectives maybe unclear. Ongoingsupport is available, butmay be inconsistent.

Mentor training programis provided and thereare objectives that out-line the purpose o thementor training. Fewobjectives are unclear.Ongoing support isavailable.

Mentor training programis provided and thereare extremely clear andconcise objectives thatoutline the purpose othe mentor training pro-gram. Ongoing supportis consistently provided.

Mentors providepositive support orstudent

No system in place tomonitor mentor supportand/or concerns thatmay arise.

System in place tomonitor mentor supportand/or concerns thatmay arise, but systemmay break down romtime to time.

System in place to mon-itor mentor support and/or concerns that mayarise. System is reliablemost o the time.

Proven system in placeto monitor mentor sup-port and/or concernsthat may arise. Thesystem provides op-portunities or two-waycommunication.

Mentors have the abil-ity to communicate

with teacher

No system in place toallow mentors to com-municate with teacher ina timely manner.

System in place toallow mentors to com-municate with teacherin a timely manner butsystem may break downrom time to time.

System in place to allowmentors to communi-cate with teacher in atimely manner. Systemis reliable most o thetime.

Proven system inplace to allow men-tors to communicate

with teacher in a timelymanner. The systemprovides check-ins withteacher on a regularbasis.

* The U.S. Department o Education does not mandate or prescribe particular curricula or lesson plans. The inormation in this gure was provided by the

identied site or program and is included here as an illustration o only one o many resources that educators may nd helpul and use at their option. The

Department cannot ensure its accuracy. Furthermore, the inclusion o inormation in this gure does not refect the relevance, timeliness, or completeness o

this inormation; nor is it intended to endorse any views, approaches, products, or services mentioned in the gure.


28/80

18


echer who my o be cerie o elier l-

gebr isrcio. B oce i his clssroom,

ech se hs his or her ow comper

pricipes i olie clss eliere by

highly qlie (i.e., cerie) lgebr echer.

the i-clss echer gies ses ce-o-ce

ssisce, oersees lb ciiies, procors ess,

is geerlly resposible or miiig

mosphere h is cocie o lerig. the

olie echer eliers he lgebr isrcio,

swers ses qesios i olie is-

cssio bor, gres ssigmes i e-mil,

proies ses wih eebck o homework

ess, sbmis gres. the olie

i-clss echers commice reqely wih

ech oher o iscss ses progress col-

lbore o how o help ses ler he pr-

iclr coe beig coere. this iercio

bewee echers o oly bees ses; i

lso seres s orm o proessiol eelop-

me or he i-clss isrcors. I iio o

proiig ll ses wih high-qliy lgebr

isrcio, secory gol o he progrm

is o icrese he isrciol skills o he i-

clss echers sppor hem i erig heir

mhemics echig cerice.

alhogh is oers beliee h he alge-

br I Olie moel oere gre promise or

ressig Loisis shorge o mhem-

ics echers, whe he progrm ws lche

i 2002 hey h o eiece o bck p his

belie. the key qesio ws wheher sch progrm col proie ses wih lerig

opporiies h were s eecie s hose

i riiol seigs. I i were s eecie,

he progrm col proie imely cos-

eecie solio or he mhemics echer

shorge. Loisi eee hr eiece o

show wheher he progrm ws creible.

this rbric is crrely ccessible ree o chrge

hrogh he Web sie o he Wiscosi eSchool

nework, escribe boe.* vogel explis h

he OPPI is mbrell elio sysem re

reily pble o oher progrms: Ierlly,

his sysem oes sk people o he oer-

whelmig mo o kowlege. I llows peo-

ple o mke weks s eee or heir pric-

lr progrms, b hey o he o cree he

whole wheel oer gi [by esigig heir ow

elio sysem]. the OPPI sysem lso llows

or ggregio o resls cross mliple pro-

grms mechism h wol llow grops o

schools i se, or exmple, o lyze heir

combie . to ssis schools sig he OPPI

or he rs ime, appleo oers coslio

serices o ech oher sers how o ierpre

commice key igs. throgh heir eors

o shre heir elio ool cree he o-

lie orm, appleo leers he eelope

ecie ioie wy o bil he kowl-

ege bse o olie lerig progrms.

I Loisi, elors rom Ecio deel-

opme Ceer (EdC) he se more coe-

iol chels or shrig igs rom he

elio o he ses algebr I Olie pro-

grm. this progrm ws cree by he Loisi

deprme o Ecio o ress he ses

shorge o highly qlie lgebr echers, es-

pecilly i rb rrl seigs. I iio,

isrics esirig o proie cerie echers

ccess o pegogy riig meorig sohey c bil cpciy or srog mhemics

isrcio re eligible o pricipe. I alge-

br I Olie corses, ses physiclly e

clss i sr bricks--morr clssroom

heir home school, which is mge by

* Registration is required to access the OPPI and some consultation with its developers

may be needed to implement the process ully.


29/80

19

Followig mber o ierl elio cii-

ies rig he progrms rs wo yers, i 2004

he progrms leers egge exerl el-

io em cosisig o Rebecc Crey o EdC,

orgizio wih experiece i reserchig

olie lerig; Lr Odwyer o Boso Col-

leges Lych School o Ecio; Gle

Kleim o he Friy Isie or Eciol

Ioio norh Croli Se uiersiy,

College o Ecio. the elors were im-

presse by he progrm leers willigess o

ergo rigoros elio. We i he

o o lo o coicig, sys EdCs Crey.

they we i o be s rigoros s possible,

which ws gre , I hik, lile bi sl.

the progrm lso ws gie boos i he orm

o gr rom he norh Cerl Regiol E-

ciol Lborory (nCREL), eerlly e

ecio lborory. the gr e primry

reserch o he eecieess o olie lerig

proie he projec wih $75,000 beyo is

iiil $35,000 elio bge rom he se

legislre. the iiol ig llowe EdC

o ocs grops i-clss obserios, s

well s o gme is ow elio cpciy

by hirig exerl cosl wih exesie

experise i reserch mehoology lysis.

the EdC elors chose qsi-experime-

l esig (see Glossry o Commo Elio

terms, p. 65) o compre ses erolle i

he olie lgebr progrm wih hose syig

lgebr oly i riiol ce-o-ce clss-room orm. to exmie he impc o he alge-

br I Olie corse, hey se hierrchicl lier

moelig o lyze poses scores oher

collece rom he reme corol

grops. to eermie i ses i olie ler-

ig progrms egge i iere ypes o peer-

o-peer iercios i hey perceie heir

lerig experieces ierely h ses i

riiol clssrooms, he elors sreye

ses i boh eiromes coce

obserios i hl o he reme clssrooms.

I ol, he elors sie algebr I Olie

corses ce-o-ce corses i six isrics.

aer compleig heir ssessme, he elors

proce l repors or he Loisi depr-

me o Ecio nCREL ler wroe

wo ricles bo he progrm or proessiol

jorls. the rs ricle, pblishe i he Jour-

nal o Research on Technology in Education,7

escribe algebr I Olie s ible moel or

proiig eecie lgebr isrcio. I he

sy, olie ses showe comprble (

someimes sroger) es scores, sye o sk,

spe more ime iercig wih clssmes

bo mh coe h ses i riiol

clssroom seigs. the elors specle

h his ws resl o he progrms iqe

moel, which brigs he olie ses ogeh-

er wih heir peers reglrly schele ime.

the elors o ew res or cocer s

well. For exmple, higher percege o o-

lie ses repore h hey i o he

goo lerig experiece, ig h is boh

sppore corice by reserch sies

o olie lerig rom higher ecio. the

elio lso o h he algebr I Olie

ses el less coe i heir lgebr skills

h i riiol ses, ig he el-

ors eel is priclrly ripe or rher resercheors. (For iiol iscssio, see Ierpre-

ig he Impc o Progrm Mriy, p. 40.)

the algebr I Olie elors wroe pb-

lishe seco ricle i he Journal o Asyn-

chronous Technologiesh ocse o he pro-

grm s proessiol eelopme moel or


30/80

20


Gie he lck o commoly se ocome me-

sres or olie lerig elios, iiil

progrms shol les srie or ierl cosis-

ecy, s dLC hs. I workig wih mliple e-

ors or school sies, olie progrm leers ee

o ricle cler se o bsiess rles or wh

re o be collece how, isribig

hese gielies o ll pries who re collecig

iormio. Lookig he, wiho hese com-

mo gielies, elors will be hr presse

o compre heir progrms ocomes wih oh-

ers. Some o he elors ere i his gie

he me coribios o he el o olie

lerig elio, like appleos leers, who

eelope elio moel h c be bor-

rowe or pe by oher progrms, he

elors o algebr I Olie, who pblishe

heir sy igs i proessiol jorls.

Evaluating Multiaceted Online

Resources

Like my riiol ecio progrms, o-

lie lerig resorces someimes oer pr-

icips wie rge o lerig experieces.

their mlicee oerigs re boo or s-

es or echers wih ierse ieress, b c

be ilemm or elors seekig iorm

igs bo eecieess. I he cse o

eciol Web sie like Wshigos dLC, or

exmple, iere ypes o sers will explore

iere resorces; some ses my ke olie corse while ohers my be reserchig

colleges or seekig meor. virl schools h

se mliple corse proiers prese similr

corm, ee he sme olie corse

my oer iereie lerig experieces i,

or exmple, ses iiie more or less co-

c wih he corse isrcor or receie ryig

cerie or iexperiece mh echers.8

I his piece, he elors escribe he pro-

grms pirig o olie i-clss echers s

ible olie moel or proiig [he i-clss]

echers wih eecie moel or heic

embee proessiol eelopme h is

rele o heir clssroom experieces.

O corse, o ll progrms will he he re-

sorces o coribe o he reserch lierre i

his mer. I Loisis cse, he elors

h exesie experise i olie elio

ook he ime iiiie reqire or pblish-

ig heir igs i cemic jorls. I so

oig, hey sere wo prposes: proiig he

progrm leers wih he eiece hey eee

o coely procee wih algebr I Olie

pblishig mch-eee reserch o ses

h migh be cosierig similr pproches.

Summary

the lierre o K12 olie lerig is grow-ig. Seerl pblicios resorces oc-

me emergig bes prcices policies i

olie lerig (see ppeix a). For progrm

leers elors who re eelopig

elio, he qliy srs rom SREB

naCOL proie bsic rmework or lookig

he qliy o olie corses echers. there

lso is growig boy o sies rom which

elors c rw lessos p mehos;

elors ee o reie he wheel. a hesme ime, hey ms exercise cio whe p-

plyig igs, ssmpios, or mehos rom

oher sies, s olie progrms resorces

ry remeosly i whom hey sere, wh

hey oer, how hey oer i. Wh works

bes or oe progrm elio my o be

pproprie or oher.


31/80

21

egrees o ce-o-ce sppor rom pre

or coch. (a similr lck o iormiy c be

o i riiol seigs wih iere i-

srcors sig ryig isrciol moels.)

Whe ce wih mlicee resorce, how

is elor o ers ocme he

olie lerig experiece, mch less eer-

mie wh le i s?

Seerl o he elios ere i his gie

ecoere his isse, lbei i isic wys.

dLC elors were chllege o ssess howses experiece beee rom he

Web sies bro rge o resorces. Elors

o Mryl Pblic teleisios thikpor Web

sie, wih is exesie echer se re-

sorces rom my proiers, similrly srggle

o ssess is impc o se chieeme. I

ery iere exmple, he elors or he

arizo virl acemy (aZva) ce he chl-

lege o elig hybri corse h icl-

e boh olie ce-o-ce compoes i which ses iiil experieces

rie cosierbly.

Combine Breadth and Depth to Evaluate

Resource-rich Web Sites

Wih is wie rge o serices resorces

or ses echers, dLC is sprwlig,

ierse projec o ele. throgh his ce-

rlly hose Web sie, ses c ccess oer

300 olie corses, iclig ll core sbjecs rios elecies, pls ace Plceme

(aP) corses Eglish s Seco Lgge.

dLC lso oers ses olie meors, college

creer plig resorces, exesie

igil librry. I iio, dLC oers oher re-

sorces ools or echers, iclig olie

crricl, ciiies, igosics. For schools

h sig p o se dLC, he progrm proies

riig or school persoel o ssis hem i

implemeig he Web sies resorces.

Iiilly, dLCs elio sregy ws o col-

lec bro iormio bo how he Web sie

is se. Ler, progrm leers shie heir

sregy o ocs o ewer rrower opics

h col sbsie he progrms eccy.

the elors ocse o se chiee-

me i he olie corses o school-leel

sppors or ecors o help hem mke he

mos o dLCs resorces. togeher, he series o

dLC elioshere he bee les e

isic eors o ecombie breh

eph, he bil o ech ohers igs rom

yer o yer, he proce impor or-

mie smmie igs (see Glossry o

Commo Elio terms, p. 65).

I he projecs rs yer, debr Friem, le

miisror he uiersiy o Wshigo (

dLC prer orgizio), coce el-io h sogh iormio o whom dLC

seres wh school coiios poli-

cies bes sppor is se. to swer hese qes-

ios, he elors selece mehos esige

o elici iormio irecly rom pricips,

iclig iscssios wih dLC miisrors,

bor members, school leers, echers, s

well s se echer sreys h ske

bo heir se o compers he Iere

bo he iliy o he dLC riig. theelor lso looke ew iicors o s-

e chieeme, sch s he gres h s-

es receie or dLC olie corses.

the elio yiele bro igs bo

operiol isses oe he ee or dLC

o prioriize mog is my prposes


32/80

22


miisrors (primrily o help eelop srey

qesios). to gi isigh io how well s-

es were perormig i he clsses, he el-

ors lyze gres compleio res or

ses erolle i dLC corses. the el-

io ciiies coce i he seco yer

gi poie o he ee or more school-leel

sppor or sig dLC resorces. the el-

ors o h some schools were excie

commie o sig he Web sies resorces,

b were erilizig i becse hey lcke

scie srcres, sch s riig ime

or echers o ler bo is oerigs, ier-

l commicio mechisms o rck se

progress, eqe echicl sppor.

Whe dLCs leers beg o coemple

hir-yer elio, hey we more h

bsic ocome , sch s se gres

compleio res. We c co how my

corses, we kow he orie sbjecs, we

kow he gre erges ll o h, sys

Jy Mrgrh-Hge, dLC presie chie ex-

ecie ocer. Wh hey eee, she explis,

ws o ge he so wh, meig hey we

o ers wh ierece [dLC] mkes.

the elio em kew h i is eor were

o proce relible iormio bo dLCs i-

fece o se chieeme, i wol ee

o zero i o oe, or js ew, o he Web

sies my compoes. Some eresdLCs

s igil librry, or exmplesimply wereo goo cies or he ki o sy hey

ple o coc. as Krl nelso, dLCs i-

recor o echology operios, explis,

I is ery icl o ele he eecieess

o o swer so wh qesio bo

librry bse, or exmple. Is js hr o

poi o ki sig librry bse he

ieces. I lso coere impor -

ig bo se chieeme i he olie

corses: the grees percege o ses

(52 perce) receie Fs, b he ex grees

percege (37 perce) receie as. to expli

hese ocomes, he elor poie o he lck

o iormiy i ses moiio ees,

he ype o cemic sppor ilble o

hem. the elor lso oe he ryig ql-

iy o he eors who proie corses, ig

h some eors re fexible resposie

o ses ees; ohers re obly ifexible.

Some re highly proessiol operios, ohers

less so.9 this elio lso escribe sbs-

il riio i how well schools were ble o

sppor he se o dLC resorces. the igs

helpe progrm miisrors who, i respose,

seppe p heir eors o ri ecors bo

he Web porls resorces how o se i

wih ses. the elio lso ws jmp-

ig-o poi or re ssessmes h wol

ollow p o he hemes o se chieeme

i he olie corses sppors or ecors

o help hem ke ge o dLCs oerigs.

I he projecs seco yer, projec leers

lche oher elio. this eor co-

sise o se ocs grops o ieiy s-

es expecios o dLCs olie corses, s-

e pre-corse preprio, oerll experiece,

sggesios or improig he corses

proiig beer sppor. as sepre eor,

hey lso corce wih iepee el-or, Cohe Reserch Elio, o ler

more bo he behior moiios o

ses oher sers, sch s school librr-

is, echers, miisrors. this spec o

he elio cosise o olie sreys wih

ses, echers, school librris; i-

eriews wih selece echers, librris,


33/80

23

es score goig p. ulimely, sys nelso,

dLCs leers chose o look primrily he o-

lie corses, belieig h his ws he ere

hey col bes ele.

dLC leers hire osie elors, Fos &

associes, o help hem rill ow io spe-

cic spec o se chieemeeermi-

ig he role h dLC olie corses ply i: 1)

eblig ses o gre rom high school

2) helpig ses become eligible l-

ly prepre or college. I his elio, he

reserchers ieie smple o 115 gre

seiors rom 17 schools who h complee dLC

corses. the elors isie he schools o

beer ers olie corse-kig policies

grio reqiremes o ieiy dLC

corses o he rscrips o hese 115 ses.

a he school sies, elors ieriewe school

cooriors exmie se chieeme

, se rscrips, dLC ocmes.

the elio ge dLCs leers wh hey we: cocree eiece o he impc o

dLC olie corses. this sy showe h 76

perce o ses who ook olie corse

hrogh dLC i so becse he clss ws o

ilble heir school h oe-hir o he

ses i he sy wol o he gre

wiho he creis rom heir olie corse. this

oher elio igs, show h we re

meeig or missio, sys Mrgrh-Hge. We

re ccomplishig wh we were se o o c-complish. a is relly impor or s o be

ble o s elier hose kis o messges

wih h ki o behi s. dLC hs se

is elio igs i mliple wys, icl-

ig whe mrkeig he progrm o osiers,

o emosre is rge o oerigs is e-

ec o ses (see g. 2, Excerp rom digil

Lerig Commos Meeting 21st Century Learn-

ing Challenges in Washington State, p. 24).

I wol be impossible o coc compre-

hesie elio o eeryhig h dLC hs

o oer, b cerily he elio sregy

o combiig breh eph hs gie i

gre el o sel iormio. dLCs leers

he se he igs rom ll he elios

o improe heir oerigs o emosre

eecieess o ers oher skeholers.

I Mryl, thikpor elors ce similrchllege i ryig o sy s Web sie h

compiles eciol resorces or echers

ses. a rs, he elio em rom Mc-

ro Ieriol, reserch, mgeme,

iormio echology rm, coce sch c-

iiies s gherig siscio , reiewig

Web sie coe, ocmeig how he sie

ws se. B oer ime, projec leers were

ske by ers o proie more cocree ei-

ece bo thikpors impc o se per-ormce. the elio ( he projec isel)

h o eole o mee his em.

I respose, he em ecie o rero he

elio i 2005, selig o wo-pr el-

io h wol oer boh breh eph.

Firs, he elors sreye ll regisere s-

ers bo sie sge siscio, sec-

o, hey esige romize corolle

ril (see Glossry o Commo Elio terms,p. 65) o sy how oe o he sies mos pop-

lr eres elecroic el ripece

ses lerig. Seerl el rips h bee

eelope er his gr; he oe selece ws

Phwys o Freeom, bo slery he u-

ergro Rilro. this priclr proc ws

chose or mber o resos: mos mile


34/80

24


Figure 2. Excerpt From Digital Learning Commons Meeting 21st Century

Learning Challenges in Washington State*

* The U.S. Department o Education does not mandate or prescribe particular curricula or lesson plans. The inormation in this gure was provided by theidentied site or program and is included here as an illustration o only one o many resources that educators may nd helpul and use at their option. TheDepartment cannot ensure its accuracy. Furthermore, the inclusion o inormation in this gure does not refect the relevance, timeliness, or completeness o thisinormation; nor is it intended to endorse any views, approaches, products, or services mentioned in the gure.


35/80

25

school socil sies crricl icle he op-

ic; he elors h obsere he el rip i

clssrooms i erlier ormie sy were

wre o ses high ieres i is opics

ciiies; Web sie sisics iice h i

ws heily iewe ilize resorce.

the elors ge pre- posess o coe

kowlege bo slery he uergro

Rilro o ses whose echers se he

elecroic el rip corol grops o s-

es whose echers se riiol isrcio.

they o h he elecroic el rip h ery sbsil posiie impc o se ler-

ig, priclrly mog ses whose echers

h preiosly se i: these ses o experi-

ece echers score 121 perce higher o he

coe kowlege es h he ses whose

echers se riiol isrcio.

Like he dLC elio, he thikpor el-

io proe sel boh or ormie sm-

mie prposes. thikpors leers lereh echers who were ew o he elecroic

el rip eee more riig experiece

o sccesslly icorpore i io heir clss-

rooms. they lso lere h oce echers

kew how o se he ool, heir ses lere

he is coe r beer h heir peers i

riiol clsses. the elors wo-pr pl

ge thikpors leers wh hey eee:

bro iormio bo sge siscio

creible eiece h reqely seere hs rel impc o ses.

Use Multiple Methods to Capture Wide-ranging

Student Experiences in Online Courses

I he exmples boe, elors srggle o

wrp heir rms ro sprwlig Web resorc-

es h lcke iormiy. Someimes similr

chllege is o he micro leel, s whe

ses he heerogeeos experieces i

he sme olie clss. the leers o arizos

aZva srggle wih his problem whe hey se

o o ele oe o heir olie corses.

I 2006, aZva school leers beg o ex-

perime wih hybri corsesreglr olie

clsses spplemee wih weekly ce-o-ce

lessos rom clssroom echer. the i-per-

so compoe ws origilly esige i

ery srcre wy: Ses receie clss-

room isrcio eery week specic ime

locio, hey h o commi o his

weekly isrcio or eire semeser. I -

iio, ses col pricipe i he hybri

clss oly i hey were workig eiher o gre

leel or o more h oe leel below gre

leel. these resricios llowe he ce-o-ce

echers o oer he sme lessos o ll ses

rig he weekly sessio. School leers spe-

ciclly esige his srcre o brig ior-

miy o ses experieces mke i esier

o ele he clss. as aZvas irecor, Mry

Gior, explis, We we he hybri expe-

riece o be he sme or ll he kis so we

col clly eermie wheher or o i is

icresig se chieeme.

Howeer, whe progrm leers sreye pr-

es he semeser brek, Gior sys, Pr-

es oere some ery specic eebck. they

i like he semeser-log, oce--week com-mime, hey rge h he srcre

preee ses rom workig heir ow

pce. Ise, pres we rop-i moel

h wol oer ses more fexibiliy

ilore ssisce. I respose, she sys, We

olly oerhle he corse or he seco se-

meser me i iere ki o moel.


36/80

26


I he ew orm, echers o o elier pre-

pre lessos, b, ise, work wih ses

oe-o-oe or i smll grops o y opic wih

which se is srgglig.

While he fexibiliy o he ew moel mees

se ees, ses rlly will he more

rie experieces sig i school leers

will o be ble o isole is eec o se

chieeme. I oher wors, obsere gis

col be e o y mber o cors, sch s

how reqely he se rops i, wheher

se works oe-o-oe wih echer ori smll grop, wh coe is coere

rig he rop-i sessio. I his isce, he

ees o progrm pricips ecessrily o-

weighe hose o he elio.

Howeer, becse he school hs mber o

oher collecio eors i plce, Gior

her colleges will sill be ble o gher ior-

mio bo wheher he hybri moel is help-

ig ses. School miisrors rck howreqely ses e he rop-i clss

chr ses cemic progress hrogh he

crriclm boh beore er hey prici-

pe i he hybri progrm. aZva lso sepre-

ly exmies se es scores or ses who

e he hybri progrm o reglr bsis. I

iio, aZva reqely ses sreys o pr-

es, ses, echers o gher iorm-

io bo he eecieess o my specs o

heir progrm, iclig he hybri clss. these

kis o ciiies c proie impor isighs

whe corolle sies re impossible.

Summary

thogh mlicee resorces c mke i i-

cl or elors o gge eecieess, goo

eliosespecilly hose sig mliple,

complemery reserch mehosc ieiy

he circmsces er which he progrm or

resorce is mos likely o sccee or il c

geere sel recommeios or sregh-

eig wek pois. Elors who re syig

mlicee resorces shol cosier sr-

egy h combies boh breh eph.

I syig eciol Web sie h oers

rry o resorces, elors migh collec bro

iormio bo sie sge he selec oe

or wo priclr eres o exmie i more

eph. Progrm leers c cilie his process

by clerly riclig wh ech resorce is i-

ee o o, or wh ocomes hey wol hope

o see i he resorce ws beig se eeciely.

From his lis, progrm leers elors

c work ogeher o eermie wh o sy

how. I some isces, i migh be logicl

o esig mliyer elio h ocses o

isic progrm compoes i iere yers,

or collecs bro iormio i he rs yer,

rrows i ocs i sbseqe yers.

I elig priclr corse or resorce h

oers ses wie rge o experieces,

elors migh cosier sig mix o qi-

ie qliie mehos o proie well-

roe ssessme o i. Rich, escripie i-

ormio bo ses experieces wih he

corse or resorce c be sel whe ryig o

ierpre bo se ocomes.

Finding Appropriate Comparison

Groups

Whe elors he reserch qesios bo

olie progrms impc o se chiee-

me, heir bes sregy or swerig hose


37/80

27

qesios is oe experimel esig, like

romize corolle ril, or qsi-experi-

mel esig h reqires hem o mche

compriso grops. these wo esigs re he

bes, mos wiely ccepe mehos or eer-

miig progrm eecs (see ble 2, p. 28).

Elors who se hese mehos ms procee

crelly. they ms rs esre h compri-

sos re pproprie, kig io cosierio

he poplio sere he progrms gols

srcre. For exmple, olie progrms

eice o crei recoery wol w o com-

pre heir se ocomes wih hose o oher

crei recoery progrms becse hey re ser-

ig similir poplios. Elors ms lso be

scrplos i heir eors o goo compri-

so gropsoe chllegig sk. For pr-

iclr olie clss, here my be o correspo-

ig ce-o-ce clss. Or i my be icl o

oi sel-selecio bis i ses (or ech-

ers) he chose o pricipe i olie pro-

grm. Or he olie progrm migh sere wie

rge o ses, possibly rom mliple ses,

wih bro rge o ges, or wih iere le-

els o preprio, mkig i icl o compre

o riiol seig or oher olie progrm.

Elors empig o coc romize

corolle ril c ecoer ee greer

chllege i eisig wy o romly ssig

ses o receie reme (see Glossry o

Commo Elio terms, p. 65) olie.

How migh elors work ro hese com-

priso grop iclies?

Seerl o he elios ere i his gie

sogh o compre isce lerig progrms

wih ce-o-ce lerig seigs, hey

ook rios pproches o he ihere echi-

cl chlleges o oig so. despie some i-

clies log he wy, he elors o olie

projecs i Loisi, albm, Mryl ll

sccesslly coce comprie sies h

yiele impor igs or heir progrms.

Identiy Well-matched Control Groups or

Quasi-experimental Studies

I esigig he elio or Loisis alge-

br I Olie, progrm leers i he Loisi

deprme o Ecio we o ress he

boom lie hey kew policymkers cre bomos: wheher ses i he progrms olie

corses were perormig s well s ses

syig lgebr i riiol clssroom.

to impleme he qsi-experimel esig o

heir elio (see Glossry o Commo El-

io terms, p. 65), he progrms miisrors

eee o ieiy riiol lgebr clssrooms

o se s corols. the ie ws o ieiy s-

r lgebr corses serig ses similr o

hose pricipig i he algebr I Olie pro-

grm he gie pre- pos-corse ess o

boh grops o compre how mch ech grop

lere, o erge. O corse, i esig sch

s his, mber o cors besies he clss

orm (olie or ce-o-ce) col ec s-

es perormce: se-leel cors, sch s

iiil biliy home eirome; ech-

er-leel cors, sch s experiece skill i

echig lgebr; school-leel cors, sch

s o-o-clss cemic sppors or ses.

Clerly, i ws impor o he elio o

he closes mches possible.

uer igh imelie, progrm miisrors

worke wih ll heir pricipig isrics o

ieiy riiol clssrooms h were mche

emogrphiclly o he olie clsses , he,


38/80

28


miisere prees o geerl mhemics

biliy o ses i boh ses o clsses. Exerl

elors rom he Ecio deelopme Ce-

er (EdC) joie he eor his poi. alhogh

impresse by he work h he progrm s h

ccomplishe gie ime bge cosris,

he EdC elors were cocere bo he

qliy o mches bewee he reme

corol clsses. Fiig goo mches is i-

cl sk er he bes o circmsces , ihis cse, i proe ee more icl or o-

elors, h is, progrm school miis-

rors. the qliy o he mches ws especilly

problemic i smll isrics or opblic schools

h h ewer corol clssrooms rom which o

choose. EdC elor Rebecc Crey sys, to

heir crei, [he progrms miisrors] i he

bes hey col i he mo o ime hey h.

Sill, she s, he mches were ecessrily

h gre cross he corol he experime-

l. H we bee iole rom he begiig,

we migh he bee lile bi more srige

bo how he corol schools wol mch o

he iereio schools , mybe, he me

he selecio process lile bi more rigoros.

ulimely, he elio em se ses

prees scores o gge wheher he reme

corol grop ses sre he corse

wih comprble skills kowlege em-

ploye ce sisicl echiqes o help

corol or some o he poor mches.

I heir repor, he elors lso proie

o iereces bewee he corol reme

Table 2. Evaluation Design Characteristics

Design Characteristics Advantages Disadvantages

Experimental

design

Incorporates random assignment o participants

to treatment and control groups. The purpose o

randomization is to ensure that all possible ex-

planations or changes (measured and unmea-

surable) in outcomes are taken into account,

randomly distributing partic

Evaluating Online Learning

Documents

Transcript of Evaluating Online Learning