Improving Low-Performing High Schools: Searching for ... · Searching for Evidence of Promise Steve...

Improving Low-Performing High Schools:Searching for Evidence of Promise

Steve Fleischman and Jessica Heppen

SummaiyNoting that many of the nations high schools are beset with major problems, such as low studentreading and math achievement, high dropout rates, and an inadequate supply of effective teach-ers, Steve Fleischman and Jessica Heppen survey a range of strategies that educators have usedto improve low-performing high schools.

The authors begin by showing how the standards-based school reform movement, togetherwith the No Child Left Behind Act requirement that underperforming schools adopt reformssupported by scientifically based research, spurred policy makers, educators, and researchers tocreate and implement a variety of approaches to attain improvement.

Fleischman and Heppen then review a number of widely adopted reform models that aim tochange "business as usual" in low-performing high schools. The models include comprehen-sive school reform programs, dual enrollment and early college high schools, smaller learningcommunities, specialty (for example, career) acadeinies, charter high schools, and educationmanagement organizations. In practice, say the authors, many of these improvement effortsoverlap, defying neat distinctions. Often, reforms are combined to reinforce one another.

The authors explain the theories that drive the reforms, review evidence of their reforms' effec-tiveness to date, and suggest what it will take to make them work well. Although the reforms arepromising, the authors say, few as yet have solid evidence of systematic or sustixined success.

In concluding, Fleischman and Heppen emphasize that the reasons for a high school's poorperformance are so complex that no one reform model or approach, no matter how powerful,can turn around low-performing schools. They also stress the need for educators to implementeach reform program with fidelity to its requirements and to support it for the time requiredfor success. Looking to the future, the authors suggest steps that decision makers, researchers,and sponsors of research can take to promote evidence-based progress in education.

www. futureofchildren. org

Steve Fleischman is vice president, Pubiic Affairs and Communications, as weii as director. Scientific Evidence in Education Forums,at the American institutes for Research. Jessica Heppen is a senior research analyst and deputy director of the Nationai High SchooiCenter at the American institutes for Research.

VOL. 19 / NO. 1 / SPRiNG 2009 IOS


mproving the nations high schools,particularly those that are low-performing, is a task whose challengesare far easier to catalogue than to

, surmount. Readers familiar with thecurrent state of American high schools, andefforts to improve them, can cite their ownfavored grim statistics and stories that illus-trate the extent ofthe problem. Many of thosestories and statistics may be gleaned fromcompanion pieces in this volume.

In this article we take a cautiously optimisticapproach that highlights promising—but notproven—reform programs and strategies forturning around what many consider a failededucation system. Our optimism is based onour own research review work, in which wehave identified several comprehensive schoolreform models that we found to be demon-strating notable improvements, particularlyin high-poverty, low-performing schools.As well, we are heartened by the increasingamount of evidence being produced throughhigh-quality studies, including randomizedtrials. One example is the strong evidenceof positive effects now available for careeracademies. Finally, we are impressed bythe increase in options available to thoseinterested in high school improvement.We leaven our optimism with a call for allreformers to consider the evidence andtake into account the particular needs andcircumstances they confront before adoptingany models.

In this article we identify some approachesthat may help to achieve the goal that allstudents will attend, stay and succeed in, andthen graduate from high school well preparedfor further leaming, successful careers, andengaged citizenship. In particular, we focuson reforms targeted at the nation s lowest-performing high schools, although the same

approaches could be used in many of thecountry's more than 18,000 high schools.

We begin by placing the search for effectiveprograms to improve high schools within thecontext of the two-decade evolution of thestandards-based reform movement, a move-ment that simultaneously exposed the flawsof the education system and helped policy-makers and educators create a road mapfor improving it. With reformers constantlydefining, demanding, and measuring betterperformance, educators set about imaginingand implementing a variety of approaches tomeet this goal.

Against the backdrop of standards-basedreform, we review the promise of someleading reform models, such as comprehensiveschool reform, charter schools, andj smallerleaming communities. By model we mean aset of specified practices or ideas that havebeen, or are intended to be, replicated widely.Models typically have a group of coherentelements, driven by an expectation itliat theseelements—^when well executed—^will accom-plish a desired goal, such as to reduce drop-outs or improve student achievement.

Some models are instmctional in focus."Ninth-grade academies," for example,provide special "catch-up" courses andcurricula for students who arrive at highschool academically unprepared. Othermodels, such as smaller leaming communi-ties, in a general sense seek to reform the wayin which the high school is structured. Stillothers—such as charter schools, educationmanagement organizations, and some com-prehensive school reform programs—focus atleast in part on how schools are govemed. Inpractice, many reform efforts overlap, defyingneat distinctions. Often, reforms are com-bined to reinforce one another, as when a

106 THE FUTURE OF CHILDREN

Iniprooing Low-Performing High Schools: Searching for Evidence of Promise

charter high school seeks to be a small school,or introduces a ninth-grade-academy formatand extends the school day or school year.

We identify some approachesthat may help to achieve thegoal that all students willattend, stay and succeed in,and then graduate fromhigh school well preparedfor further learning,successful careers, andengaged citizenship.

We explain the theories that drive thesestructural and programmatic reforms, reviewevidence of their effectiveness to date, andsuggest what it will take to make them workwell. When reviewing the evidence, we dis-cuss its quality and quantity and point out anypotential problems that make it difficult todraw conclusions regarding the effectivenessof an individual model or the class of inter-vention it represents.

We stress from the outset that no one reformmodel or approach, no matter how powerful,can turn around low-performing schools. Thereasons for a high school's poor performanceare complex and cannot be addressed piece-meal. Furthermore, as many experts havepointed out, school-based reform can haveonly limited effects on improving educationalattainments and reducing societal inequalities.As a group of leading sociologists pointed outrecently, narrowing the nation s educationgaps "requires reducing poverty, as well as

improving the schools that poor childrenattend."' Because no one reform can get thejob done by itself and schoohng cannot attainall the improved outcomes we seek, we hopethat the efforts we review here will be seen asparts of the solution, to be used judiciouslyand in comprehensive reform efforts that givedue consideration to the contexts in which thechanges are implemented.

Understanding ImprovementOptionsBy exposing the failure of the nation s schoolsystem to provide all students with acceptablelevels of education, the assessment andaccountability measures of standards-basedreform—including those embodied in the NoChild Left Behind Act (NCLB)—have servedas a dynamic engine, driving Û\e search fordemonstrably more effective programs andpractices for low-performing schools. Thedesire to find evidence-based solutions hasbeen further energized by the NCLB require-ment that underperforming schools adoptreforms supported by "scientifically basedresearch."

Particularly in high schools, however, thesearch for and implementation of effectivereforms are complicated by the many chal-lenges that schools face, such as the lowreading and math achievement of enteringstudents, the high dropout rates, the growingnumbers of English-language learners, thelack of safety in some schools, the inadequatesupply of effective teachers in the neediestschools, and the intense focus and effortrequired to restructure complex organizations.

It is not surprising that in this environment agreat many reform approaches have arisen,each promising to address the challenge ofimproving high schools. In this section, wereview a range of reform models. Our

VOL. 19 / NO. 1 / SPRING 2009 107

Steve Fleisehman and Jessica Heppen

definition of "model" is intentionally loose andreflects our experience with how both educa-tion decision makers—such as school boardmembers, superintendents, central officeadministrators, principals, and school staff—and federal and state policy makers thinkabout their improvement options.

When decision makers consider effectivereform approaches, they do not make thefine distinctions that researchers might make.Decision makers with whom we have workedover the past decade are much more eclecticin their consideration of options. Thus, theymay at the same time explore whether toadopt a program such as First Things First, oran education management organization, or ahomegrown professional-development initia-tive to promote greater literacy for enteringhigh school students, or whether to split alarge school into smaller academies or con-vert it into a charter school. In fact, NCLBmay promote this eclectic approach by outlin-ing five equivalent options for "restructuring"the schools that are most persistently thelowest performing.-

Our review of reforms is not exhaustive. Ourselection of approaches to highlight is basedon our research regarding key challengesfaced in improving high schools and on ourprofessional judgment regarding whiclioptions are most prevalent across the countryand which models decision makers are mostlikely to consider in the coming years.'̂ Wehave examined research on evidence-basedreforms as well as "gold standard" researchreviews produced by organizations such asthe What Works Clearinghouse.

We seek to avoid the "either-or" thinking thatoften prevails in education. For example,reformers seeking to improve high schoolsneed not choose between improved

professional development or smaller schools.Furthermore, we argue that all schools musthave strong curricula and instmction in place,as well as ways to meet the nonacadeinicsocial and emotional needs of students. Eachmodel we review has strengths and limitations.A combination of several models may beneeded for success. In our conclusion, wereturn to the need for coordinated systemicsolutions.

Making Evidence MatterMore than ever, education decision makersconsidering reform approaches are asking twoquestions. Does it work? How do we know?As yet, the growth of evidence on the effec-tiveness of reform models has not :caught upwith educators' understandable desire to havemultiple research-proven options. A numberof randomized controlled trials, cqnsideredthe "gold standard" in evaluation research,have already provided evidence regarding diepromise of some approaches, such as careeracademies.'' Other rigorous studies are nowunder way. Through our own work at theAmerican Institutes for Research's Compre-hensive School Reform Quality Center wehave rated the quality and effectiveness ofeighteen leading middle school and highschool comprehensive school reform modelsand examined factors that may contribute totheir success.̂ In addition, the Wliat WorksClearinghouse continues to review evidenceregarding the effectiveness of dropout-prevention programs and may focus on otherhigh school topics in the future."̂

But despite the encouraging growth ofresearch on the effectiveness of high schoolreform models, the evidence is still quitelimited both in quantity and quality. Forexample, in our report on ¡middleland highschool comprehensive school reform models,we identified more than 1,500 potential


Improving Low-Performing High Schools: Searchingfor Evidence of Promise

studies to review; of these studies, onlyforty-two met our standards for quality andrigor. Even when studies are conducted usingrigorous methods, judging a model's impactcan be very difficult, given the complexity ofthe reform models, the variability of settingsin which they are implemented, and theimportance of implementation as a significantvariable affecting outcomes. Furthermore, inmany evaluations of school reforms, themeasures used to evaluate the impact are notaligned with the outcomes that the reformmodel seeks to affect. An additional challengeis timing—school reforms are dynamic, andeven the best research studies are able tocapture them only at distinct points in time.

Recognizing the pressure to improve highschools and the current limits of the evidencebase, we recommend that poficy makers takea "best available evidence" approach toselecting from among viable alternatives.Policy makers should judiciously weighexisting rigorous evidence, along with otherimportant considerations, such as the supporttliat the model has in the local educationcommunity, the "readiness" of a school ordistrict to incorporate a reform into otherefforts already under way, and the commit-ment and ability of an external serviceprovider or the district to provide the long-term implementation support necessary forsuccess. In the end, ignoring rigorous evi-dence means risking disappointment, butwaiting until "all the evidence is in" does notmeet the urgently felt need for positive action.

Reviewing the ModelsIn what follows, we review a number ofwidely adopted approaches to changing"business as usual" in low-performing highschools. The models include comprehensiveschool reform (CSR) programs, dual enroll-ment and early college high school (ECHS),

smaller learning communities, specialty (forexample, career) academies, high schoolcharter schools, and education managementorganizations (EMOs). Although theseapproaches represent many of the mostprevalent whole-school efforts to reformlow-performing high schools today, we stressthat this is not a comprehensive review of allof the high school reform models available.For example, many dropout-preventionprograms exist, and some, such as the Check& Connect program, show compellingevidence of effectiveness. Other programsfocus on improving literacy (for example,supplemental literacy programs such as SRACorrective Reading and Language! andliteracy programs across content areas, suchas the Strategic Instruction Model) and onreducing school violence (for example.Positive Behavior Supports). Other modelshave been locally developed.

These approaches vary widely in theirassumptions about how they will bring aboutimprovement. Few have solid evidence ofsystematic or sustained success. For example,high school CSR models—such as America'sChoice, Coalition of Essential Schools, FirstThings First, High Schools That Work, ProjectCRAD, and Talent Development HighSchools—present themselves to schools aseffective programs. But in our 2006 review ofleading middle and high school whole-schoolreform models (that includes those listedabove), we found just five widely adoptedmodels that we felt had a solid body of evi-dence regarding their effectiveness.'

To succeed, policy makers must matchcarefully the models they choose to theoutcomes they seek to promote. To helporient readers and support policy makers'matching process, we propose a two-partdecision-making framework. First, decision

VOL. 19 / NO. 1 / SPRING 2009 109

Steve Fleisehman and Jessiea Heppen

makers should consider five outcomes thatany chosen model should be designed in part,or in whole, to help achieve. Second, theyshould consider the instructional, structural,and governance elements within each model.

The five reform outcomes, drawn fromresearch on secondary school reform con-ducted by MDRC, can be thought of asmediators of improvement.^ Decision makerscan use these five outcomes as a compre-hensive road map for reform and as a wayto consider which models help to meet theirneeds, singly or in combination.

To succeed, policy makersmust match carefully themodels they choose to theoutcomes they seekto promote.

The first outcome is a personalized andorderly learning environment. Researchershave pointed out the importance of creating aschool atmosphere that supports effectivelearning for all students. Such an atmospheremay be particularly important in large,comprehensive high schools where studentscan get lost in the crowd and thereby fail toreceive the academic support they need.Impersonality may also contribute to behav-ioral problems and increased violence inschools. Efforts in this area are in line withthe growing realization that successfulschools focus on academic, as well as socialand emotional, learning.^

The second outcome is the capability toassist students who enter high school with

poor academic skills. Scores on thé NationalAssessment of Educational Progress confirmthat a significant percentage of students enterhigh school poorly prepared for academicsuccess. Particularly troubling is weakness inthe literacy and readiiig skills that form thefoundation of most academic endeavors."*Almost all high school reform models recog-nize and seek to address these challenges.

The third outcome is improved instructionalcontent and practice. Leading experts instandards-based reform consistently identifythe lack of a strong instructional focus andeffective practice as one of ¡the centraldeficiencies in low-perforniing high schools."The academically neediest students are ofteneducated by the least well-prepared and leastexperienced teachers, in terms of t:eachingout-of-field and having fewer than five yearsof experience.'^ Some models reviewed belowaddress this issue by providing new curriculaand by offering extensive, targeted profes-sional development.

The fourth outcome is the capability toprepare students for the world beyond highschool. Many high schools are failing to pre-pare students well either for postsecondaryeducation or for careers."' In today's globaleconomy, students with only a high schooleducation face far lower career earningsand greater chances of being unemployedthan their college-educated peers.''' More-over, most of the good jobs being createdin the new economy, particularly the best-paying ones, require postsecondary educa-tion. Some, but not all, of the models belowaddress this desired outcome directly.

The fifth outcome is positive change inoverstressed high schools. All highi schools,being complex systems, are difficult tochange. Low-performing schools offer all the

110 THE FUTURE OF CHiLDREN

¡mproving Low-Performing High Schools: Searching for Evidence of Promise

expected challenges of reforming an alreadycomplex organization, with the additionaldifficulty of having to do it in a setting withdiminished administrative and instmctionalcapacity and heightened physical disruptionand psychological pressure. These challengesare often exacerbated by accountabilitydemands and adverse publicity. Comparedwith reforming other schools, creating posi-tive change in these low-performing schoolsmay take more skilled leadership and time,greater moral and fiscal support from thedistrict, efforts by external reform organiza-tions, cultural changes in terms of expecta-tions and behavior, and more staff leaming ofnew habits, skills, and ways of doing things.Models reviewed below address the need tosupport change at the school level in a varietyof ways, including by creating new or smallerschools.

In what follows, we describe reform modelsin ways that allow decision makers to com-pare how the models seek to achieve the fivedesired outcomes or mediators of improve-ment, what outcomes the models do notdirectly help to achieve, and whether themodels do or do not now show evidence ofeffectiveness in helping improve high schools.

The second part of the decision-makingframework involves the instructional, struc-tural, and governance elements within eachmodel. Models with strong instmctionalcomponents focus on improving teaching andleaming through refining a school's existingcurriculum, introducing new and often morestructured curricula, and providing profes-sional development and other supports thatenhance teacher quality. These models seekto strengthen both the content and deliveryof instruction to provide a rigorous and rel-evant learning experience for all students.

Models with strong structural elements tendto focus their attention on how a school isorganized to deliver educational services tostudents. Some model variants may, forexample, extend instructional time in keysubject areas, lengthen tlie school day orschool year, create smaller leaming communi-ties within the larger school, reduce the sizeof the school, offer new ways to improveconnections to the community, or break downthe barriers between high school and collegethrough strategies such as "dual enrollment."

Models with strong governance elementsdirectly address the operations and manage-ment of schools and change how high schoolsare run—usually by creating new authoritystructures to mn the schools. Leading exam-ples of governance models are charters orthird-party education management organiza-tions that mn schools. Changes often includepersonnel policies.

In practice, most models combine all threeelements, but some focus predominantly onone. Because changes in all tliree may berequired to achieve improved outcomes,decision makers should have clearly in mindwhich changes a model seeks to make and howwell those changes align witli local improve-ment plans. For example, high schools tliatstmggle witli student performance in particu-lar areas such as literacy or mathematicsshould orient clearly toward improvementoptions witli a strong instmctional focus. Highschools such as some of tlie nation s "dropoutfactories," which are stmggling in all areas andare seeking to restore order, might considermodels or third-party providers that addressschool govemance. Finally, some of thelowest-performing high schools consideringrestructuring options might tend toward tlieprimarily structural approaches, such asconverting into smaller leaming communities.

VOL. 19 / NO. 1 / SPRING 2009 111

Steve Fleischman and Jessica Hejipen

Characterizing models in tliis way givesdecision makers a frame of reference that canhelp them matcli their most acute needs withpotential solutions.

In what follows we provide summaries ofa representative and illustrative set of highschool reform options. For each, we indicatewhich of the reform elements (instructional,structural, governance) are typical featuresof the model, summarize its theory of actionor approach to achieving the five desirableoutcomes, and re\ne\v the current evidence ofits effectiveness.

CSR is intended to besystemic and to addressevery aspect of a school, fromcurriculum to scheduling tomanagement to family andcommunity involvement.

Although our comments regarding effective-ness are tentative, they are guided by thebest available rigorous evidence and well-conducted evidence reviews. We rely heavilyon the findings of several studies that wepublished in 2006 and 2007, which reportedon systematic reviews of the evidence ofeffectiveness and quality of leading highschool models and education managementorganizations. When the evidence base is stillemerging, we point out the limits of what isknown. In addition, although we seek to gen-eralize the evidence of effectiveness of thesemodels as a set or approach (for example,comprehensive school reform or specializedacademies as a whole), individual programsthat represent a model type vary widely in

effectiveness. Thus, for example, one typeof specialized academy may be much moreeffective than another. Policy makers shoulduse the following information as a way to ori-ent their thinking about which reform optionsto pursue. They should also keep in mind thateffective reform involves programmatic andnonprogrammatic changes in schools and thatthese changes are often beyond the scope ofany model.

Comprehensive School RefonnComprehensive school reform came intobeing during the 1980s but grew in impor-tance during the late 1990s with the supportof Congress, which created the Comprehen-sive School Reform Demonstration Program(later the Comprehensive School ReformProgram), and with the sponsorship of NewAmerican Schools.''' Although not a center-piece of No Child Left Behind, it neverthe-less remains a reform approach that has thesupport of major foundations and remains ofinterest in the education community. Oneindication of its prevalence is that; more than5,000 schools had implemented the eighteenmodels that we rexnewed in 2006.

Whether implemented with the support of anexternal provider or through the efforts ofindividual schools or districts, CSR isintended to be systemic and to address eveiyaspect of a school, from curriculum toscheduling to management to family andcommunity involvement. Its integration ofresearch-based practices into a unifiedprogram is designed to give a school's reformeffort coherence—iristructionally, organiza-tionally, and culturally—leading to improvedstudent achievement. Depending on theirdesign philosophy, individual representativesof this approach vary in the level of curricularor structural support̂ that they provide to aschool. With some exceptions, such as the


Improving Low-Performing High Schools: Searching for Evidence of Promise

School Development Program describedbelow, CSR programs typically do not changegovernance structures in the schools. Some ofthe best-known CSR high school programsare America's Choice, Coalition of EssentialSchools, First Things First, High SchoolsThat Work, Project CRAD, and TalentDevelopment High Schools.̂ ''

As one might expect from its name, compre-hensive school reform seeks to achieve allfive ofthe desirable high school outcomes.'"Individual CSR programs differ, however, inhow they meet these objectives; our discus-sion illustrates the range of solutions diatthey provide.'**

To achieve personalized learning environ-ments, for example. First Things Firstfeatures theme-based smaller learning com-munities that bring a core group of studentsand teachers together for all four years ofhigh school. To meet the same objective, theTalent Development High Schools programcreates both a "Ninth-Crade Success Acad-emy" and career academies at the upper highschool grades. The America's Choice programorganizes its high schools into small schoolsand "houses." "* Taking a different approachto meeting this same goal, the Coalition ofEssential Schools focuses on helping schoolsdesign their own approaches—through pro-fessional development, the creation of learn-ing communities, and so forth—to meet tencore program principles, including "personal-izing teaching and learning."^"

Talent Development's ninth-grade academiesalso serve as an example of how some CSRprograms seek to address the needs ofstudents who enter high schools with weakacademic skills. These academies offercatch-up courses and a "Freshman Seminar"to support the development of academic and

social skills necessary for high school success. '̂Another example is America's Choice, whichoffers "ramp up" courses in math and readingto accelerate progress for students who enterhigh school behind academically.̂ ^

CSR programs differ widely in whetherthey provide support to improve high schoolinstructional content and practice. Whilemany programs focus on professional devel-opment and creating teacher learning com-munities to improve instruction, few providea curriculum. One model that does provideinstructional content is America's Choice,which offers its own curriculum in reading,writing, and mathematics.̂ '̂ Another is theTalent Development High Schools, with thepreviously noted catch-up curriculum.

Talent Development High Schools andAmerica's Choice also include "careeracademy" components intended to preparestudents for the world beyond high school.In another approach, the High Schools ThatWork program merges the requirements forcompleting a college-preparatory academiccore with those of completing a plannedsequence of career courses or further aca-demics. In this way, the program seeks toprepare students well for whichever postsec-ondary options they choose.

Leading CSR programs take a variety ofapproaches to help make positive change inlow-performing high schools. In fact, many ofthese programs were created because reform-ers recognized that overstressed schools needexternal support to improve. Models providesuch support in many ways. They providetraining, professional development, change-process consulting, school-based coaching,and implementation visits. They promoteinnovative structures, such as academies orhouses, and teacher professional learning

VOL. 19 / NO. 1 / SPRING 2009 H 3


communities. They foster changes in struc-ture, such as the introduction of blockschedules, different forms of student assign-ment, and common planning time. Sometimesthey foster changes in the way the schools aregoverned. For example, the School Develop-ment Program, created by James Comer,offers a structure and process for schoolimprovement based on mobilizing teachers,administrators, and community members tosupport students' maturation along six devel-opmental pathways: physical, cognitive,psychological, language, social, and ethical.Schools that adopt the program must altertheir organization and governance to createthree key structures to run the school: a schoolplanning and management team, a studentand staff support team, and a parent team.

Making overall statements about the effec-tiveness of CSR as an improvement approachfor high schools is difficult. Nevertheless,several pieces of evidence suggest its prom-ise.̂ "* Geoffrey Borman's 2002 meta-analysisof the evidence of effectiveness of twenty-nine leading CSR programs, including thoseoperating at the high school level, synthesized232 studies and concluded that the overalleffects of CSR are significant and meaningfulrelative to effects of other interventions usedin similar contexts.^ Borman found that asignificant factor in the strength of CSRmodels' effects is the maturity of the pro-grams; that is, models in place for more thanfive years yielded the strongest effects.Because many experts believe that it is harderto improve high schools than elementaryschools, we take a closer look at CSR out-comes at the high school level.

In our own 2006 systematic review of eigh-teen secondary CSR models, we gave four(America's Choice, First Things First, SchoolDevelopment Program, and Talent

Development High Schools) a rating of"Moderate" in the category of "evidence ofpositive effects on student achievement." Wederived this rating from our review of findingsreported in studies in which we have confi-dence based on their research designs (that is,studies with comparison groups arid longitudi-nal designs). In most cases, the findingsreported in tliese studies are a mix of positiveeffects and no significant differences instudent achievement for students in schoolsimplementing these CSR models comparedwith students in schools that are not. It isimportant to note that a rating of "Moderate"is the second highest̂ rating achieved by any ofthe nearly fifty models we have reviewed.̂ "Civen the difficulty of improving low-per-forming schools and the relative newness ofsome of these models, we consider the findingthat four programs are moderately effective inraising student achievement to be promisingevidence for high school CSR.

Despite its promise, though, some cautionis necessary. Although four programs didreceive a "Moderate" rating, we gave severalother widely adopted programs a "Zero" rat-ing, indicating that we could find no evidencethat theyihad positive effects on studentachievement.^^ Decision makers inust there-fore choose carefully among CSR programoptions. And even the most comprehensive ofthese programs has gaps that must be identi-fied and addressed to provide an ¡effectivetotal package of reforms for a school.

Dual Enrollment and Early CollegeHigh SchoolsDual enrollment programs allow high schoolstudents to take college courses and earncredits toward an associate's or bachelor'sdegree. Once available only to studentsperforming well beyond grade level, todaydual enrollment is becoming increasingly


Improving Low-Performing High Schools: Searehingfor Evidence of Promise

popular as a way both to reach a wider pool ofstudents who can benefit from college course-work and to decrease the need for remediationin college.^ Research has shown that postsec-ondary success is predicated on both rigorousacademic preparation and a clear understand-ing of the expectations in college. Thus, statepolicy makers are increasingly turning to dualenrollment to accelerate learning and tobridge the transition to postsecondary successafter high school graduation.̂ '̂

Making overall statementsahout the effectiveness of CSRas an improvement approachfor high schools is difficult.Nevertheless, several pieces ofevidence suggest its promise.

State and local dual enrollment policies varysubstantially in terms of tuition and eligibilityrequirements, funding, and program charac-teristics. Dual enrollment is primarily a struc-tural reform approach, in that its focus is onaligning systems in K-12 with postsecondarygoals. Unlike traditional high schools, manyschools with dual enrollment opportunitiesoperate on college campuses (approximately80 percent in 2005).̂ ^ Other dual enrollmentprograms are implemented in high schools orthrough distance-learning providers.^' Thesestructural differences produce variations inthe instructional elements of reform—specif-ically, in the ways that high school studentsreceive college-level instruction, includingtaking classes at the high school taught bycollege-accredited teachers and taking classesdirectly at the college.̂ ^ Finally, implement-ing dual enrollment programs can also

involve a change in school governance. Eorinstance, many dual enrollment high schoolsare also charter schools (for example, aboutone-third of schools participating in the EarlyCollege High School Initiative are charterschools), and the success of all dual enroll-ment programs requires partnerships withlocal community colleges and universities.^

Depending on their structure, dual enroll-ment programs seek to improve studentachievement through all five of the desiredoutcomes for high school reform models.Most seek to create a personalized learningenvironment that is part of a college-goingculture. Eor example, they often incorporatethe use of "advisories" and other formalmentoring structures. Although some dualenrollment programs have entrance require-ments, many assist students with pooracademic skills by serving students who aretraditionally underrepresented in postsecond-ary education.^'' To improve instruction andto prepare students for the world beyondhigh school (the third and fourth desiredoutcomes), the dual enrollment approachenables students to earn credits toward a highschool diploma and toward a college degreeconcurrently, thus providing access to morerigorous curricula and instruction. By align-ing the content and pedagogy with collegeexpectations, the approach aims to helpstudents become better positioned to succeedin college and beyond.^ Einally, all of thereforms associated with implementing dualenrollment approaches seek to achieve thefifth desired outcome, eliciting positivechange in overstressed high schools. Inparticular, implementation of dual enrollmentprograms emphasizes cultural changes interms of expectations of students and adoptionof new ways of supporting student success inincreasingly challenging course settings.

VOL. 19 / NO. 1 / SPRiNG 2009 115


No definitive evidence shows that dual enroll-ment programs are consistently achieving theobjectives identified above, and there is notyet strong evidence of the overall effect ofdual enrollment on student achievement andpostsecondary outcomes. Correlational studiessuggest that dual enrollment opportunities areassociated with increased academic achieve-ment and educational attainment.^^ For exam-ple, a U.S. Department of Education studyreported in 2004 tliat earning college creditswhile in high school increases the likelihoodof graduation and reduces the average time ittakes to eam a college degree.^" Because thisstudy is a descriptive analysis of longitudintJdata and does not include a control group,however, we have limited confidence in thesefindings, which may be explained by students'own self-selection into college credit-bearingcourses while in high school.

Two specific dual enrollment programs thatliave been evaluated are middle college highschools and the Early College High SchoolInitiative. Middle college high schools(MCHSs) are alternative high schools,located on college campuses, whose goal is toincrease access to college among traditionallyunderrepresented students. They maintainsmall enrollments, aiming to personalizelearning, and offer relevant, career-relatedcourse experiences to students. Instructionalstrategies include collaborative, peer-assistedlearning groups, team teaching, and the useof alternative assessments, including port-folios.̂ ** One rigorous, experimental studyevaluated a MCHS program implemented inthe Seattle Public Schools in the early toniid-1990s.*'' The study found that dropoutrates for students with access to the programwere statistically equal to those of students inthe control group (36 percent vs. 33 percent)and that a similar share of students in bothgroups earned a diploma or CED (40 percent

vs. 38 percent). Based on this study, evidenceis not strong that middle college high schoolsare effective at keeping at-risk students inschool. However, it is worth noting that sincethe mid-1990s, the model has been revisedand aligned with the early college high schoolmodel and to our knowledge, there are nonewer or more definitive studies of currentevidence of its effectiveness.

The Early College High School Initiative isa dual enrollment program developed by theBill & Melinda Cates Foundatiori. Schools inthe initiative adhere to an established set ofcore principles that includes providing stu-dents with the opportunity to earn up to anassociates degree or'two years' worth of col-lege credits toward a baccalaureate degree,finding public resources to cover the cost ofthe college credits, and rewarding masteryand competence in high school classes withenrollment in college-level courses. Targetinga student population that includes those whoare traditionally underrepresented in postsec-ondaiy education, the initiative encompassesthe broad goal of serving these students withmore rigorous instruction^ relevant curricula,and supportive relationships.""'

A descriptive, longitudinal study is now exam-ining the implementation and outcomes ofthis national initiative. The students attend-ing the schools under study are recruited bythe schools, all of which generally seek toenroll low-income students, students of color,and English language learners. Some oftheschools have explicit selection criteria thatinclude minimum (and maximum) achieve-ment requirements for entrance."" Findingsto date suggest that schools in the initiativeare recmiting and enrolling low-incomestudents and are serving student populationswith minority compositions that exceed thoseof their feeder districts; however, survey



results also indicate that students attendingearly college high schools are more likelyto have college-educated parents than thenational average (in 2006-07, 33 percent ofECHS tenth-grade students had parents whograduated from college, compared widi 17percent of tenth-grade students on a nation-ally representative survey)."*̂

The research also provides suggestive evi-dence that early college high schools canestablish personalized learning communitiesinvolving students and teachers (based in parton high average attendance rates and othersurvey-based and qualitative measures ofpersonalization). It also seems that studentsattending these schools are engaged academi-cally and are taking college courses in sizableproportions, particularly in schools thatare new "start-ups" (compared with thoseconverted from existing schools) and schoolsthat are physically located on the campusof a two- or four-year institution of highereducation (compared with those not locatedon a college campus)."*̂ Because there is nocomparison group, however, it is not possibleto discern the extent to which the FCHSmodel as a whole produces positive outcomesfor students. Students who attend theseschools are clearly motivated to do so, as theyself-select into the programs, and they wouldneed to be compared to similarly motivatedstudents not attending ECHSs to generatestronger evidence of effectiveness. Despite alack of definitive evidence, the national-leveldescriptive studies of this widespread andgrowing high school reform model provideuseful information about implementation andtrends over time.

In general, dual enrollment programs arenow widely used to increase access to collegecourses for a broader range of high schoolstudents. Exposing students, particularly

at-risk students, to college campuses andcollege-going culture can potentially ease thetransition to postsecondary education, as wellas improve outcomes for students while theyare still in high school. Although the full ben-efits of these programs are as yet unknown,the continued study of specific programs suchas middle college high schools, early collegehigh schools, and other programs that operatein states across the country should providea clearer picture of the extent to which andthe conditions under which dual enrollmentapproaches achieve the five desired outcomesof high school improvement.

Smaller Learning CommunitiesSmaller learning communities (SLCs)include a variety of school redesign initia-tives intended to create smaller theme-basedunits of organization, including schools withinschools, academies within buildings, andfree-standing small schools. These commu-nities include structures such as freshmanacademies organized around career interestsor other themes, "houses" in which smallgroups of students are taught by a cadre ofcore-subject teachers and remtün togetherthroughout high school, and semiautonomousschools within a school.

This approach to high school reform isprimarily structural in focus, although it canresult in governance and instructionalchanges. Smaller learning communities areformed in differing ways, depending onfunding sources and political and physicalconstraints. While small learning environ-ments have many structural variations, amongthe most common and practical approaches isto divide an existing large high school intosmall units. These "conversion" strategiesinclude schools within schools, which oftentake the form of subprograms within a hostschool, and schools within a building, such as

VOL. 19 / NO. 1 / SPRING 2009 117

Steve Fieischman and Jessica Heppen

academies with career themes or curricularfocus areas. The autonomy over scheduling,staffing, and budgeting varies for conversionSLCs. In some cases, even schools within abuilding can have their own principal; inother cases, administrators are shared. Afree-standing small school is typically locatedin its own building and has its own principaland autonomy over budget. There is nouniversal agreement about the optimal sizefor small high schools, but free-standing smallschools usually enroll fewer than 600 stu-dents.'''' Unlike redesigned large high schoolsbroken into smaller units, free-standing smallschools are often started up from scratch,typically beginning with one grade (forexample, ninth grade) and adding grades overtime. Furthermore, in many cases, high schoolreform into smaller leaming communitiesalso includes or emphasizes an instructionalelement. For example, the establishment ofthese communities with themes may requirecurricular reform that includes a shift incontent and pedagogy.

The U.S. Department of Education's SmallerLeaming Communities program, authorizedunder NCLB, awards grants for up to sixtymonths to local education agencies to planand implement SLCs in high schools withmore than 1,000 students. A recent reporton schools in the first cohort (a total of 119schools first funded in 2000) shows that theSLC structures most commonly implementedare freshman and career academies, followedby non-themed schools within schools.''̂

Of the five desired outcomes of high schoolimprovement, personalization is the primarygoal in creating SLCs. The underlyingrationale is that the educational experience forstudents—particularly at-risk students—^willimprove when they attend smaller, moreintimate schools where they feel known and

cared for by their teachers and become moreengaged in leaming. While SLCs tiike manydifferent structural forms, all share theobjective of personalization for high schoolstudents. Personalization strategies enacted inthe 119 schools in the federal SLCs programinclude the use of individual assessments,integration of a cooperative leaming focusinto the curriculum, mentoring programs suchas teacher advisories and formal mentoring,and interdisciplinary teaming.''̂

Although changing the school structure tocreate a more personalized learning environ-ment is a primary objective of SLCs, achiev-ing this outcome is intended to be a catalystfor the other four desired outcomes for highschool improvement. The idea is that chang-ing the culture by decreasing the size of highschools will create the enabling conditionsfor schools and teachers to provide bettersupports for students who enter below gradelevel, to improve instmction, and in so doing,to better prepare students for postsecondarysuccess. Together, these reforms are designedto elicit positive change in overstressed highschools (the fifth outcome), by promotingstmctural and cultural changes in low-performing high schools.

Some evidence is emerging that students insmall high schools do experience benefits.Research on small schools over the past twodecades generally indicates that smaller highschools can achieve the goal of personaliza-tion. Findings, from mostly descriptive andmatched comparison studies, indicate thatSLCs can provide more personal leamingenvironments that reduce alienation ofstudents and teachers, increase school safety,improve working conditions for teachers,and foster greater student engagement inschool.-*'



Some studies suggestencouraging findingsabout the benefits of SLCson student achievementoutcomes, while otherssuggest mixed or evennegative results.

Consistent with these findings, an analysis ofseven-year trends among schools participat-ing in the federal SLC program suggests posi-tive trends in terms of student participationin extracurricular activities and ninth-gradepromotion rates and downward trends inschool violence, disciplinary action, and theuse of drugs and alcohol.'"* The research fur-ther suggests that the size of the high schoolmatters most for minority and low-incomestudents.*"* In particular, several studies ofhigh schools redesigned into SLCs in largeU.S. cities—including Chicago, Baltimore,Boston, and New York—have reportedimprovements in school climate, culture,and student attitudes and short-term studentoutcomes including ninth- to tenth-gradepromotion, in comparison with students inlarge comprehensive high schools.*'

However, the effect of implementing SLCson student achievement, graduation rates,and postsecondary success has not beendefinitively established with rigorousresearch. Two primary challenges emergefrom a review of the evidence regarding theeffect of school size on these student out-comes. First, many of the studies on schoolsize are correlational in design, often basedon large national databases. These studies

may use sophisticated methods, but they areunable to remove the possible bias thatresults from the facts that students andteachers self-select, rather than beingassigned randomly, into schools and programsand that attrition from these programs is alsononrandom. Second, as noted, SLC is not asingle program but rather a term that repre-sents a variety of possible approaches, oftenin combination with other reform strategies,making it difficult to make overall statementsregarding effects. So, although the researchsuggests that creating smaller learningenvironments can, indeed, foster morepersonalization, a definitive link from thesechanges to effects on student achievement inSLCs has not been clearly established withrigorous research.

Civen these limits in the research, somestudies suggest encouraging findings aboutthe benefits of SLCs on student achieve-ment outcomes, while others suggest mixedor even negative results. For example, arecent study in New York City reports thatstudents in the New Century High Schoolswere more likely to graduate on time thanstudents citywide.^' Another recent evalu-ation of the formation of SLCs throughthe "Focus on High Schools" initiative inBoston Public Schools uses an interruptedtime series design to examine outcomes forstudents before and after implementation ofthe initiative, over a twelve-year period. Thekey features of the initiative are the breakingdown of Boston's twelve large comprehensivehigh schools into "educational complexes" ofSLCs and a curricular and instructional focuson Fnglish and language arts. The estimatedeffects showed positive trends over time foroutcomes related to student engagementsuch as absences, suspensions, and ninth- totenth-grade promotion. But language artsand mathematics scores on the Massachusetts

VOL. 19 / NO. 1 / SPRING 2009 119

Stave Fleïselinian and Jessica Heppen

Comprehensive Assessment System testsactually declined relative to the projectedtrend in the absence of the initiative.^-

The Bill & Melinda Cates Eoundation,among the many supporters of high schoolredesign strategies that focus on reducingschool size, has invested more than $900million in improving U.S. high schools since200L The foundation's High School CrantsInitiative has provided grants to intermediaryorganizations tasked with redesigning existingschools and starting new high schools. Smallschool size is considered in this program tobe a "necessary but not sufficient" conditionfor creating effective learning environmentsfor students. A comparative, longitudinalevaluation of the initiative from 2001 through2005 demonstrates that, as the foundationexpected, implementing new schools is easierthan converting existing schools. That is,free-standing small schools that start up fromscratch seem better able to create conditionsfor learning that are consistent with the attri-butes of high-performing high schools thanare schools within schools or schools withinbuildings that are converted from large, com-prehensive high schools. Eindings from thisresearch indicate that students in foundation-supported new schools, but not in redesignedschools, exhibited positive trends (in atten-dance and in ninth- to tenth-grade progres-sion rates). With some exceptions, however,average test scores in both new and rede-signed high schools remained below districtaverages.'^ These findings are similar to thosein the Boston study and were also replicatedÍ71 a longitudinal evaluation of foundation-supported small school reform in BaltimoreCity Schools. There, students at new (called"innovation") schools outperformed compari-son students in conversion high schools andlarge comprehensive high schools on stateassessments in English and algebra.'"*

The evaluation of the foundation's HighSchool Crants Initiative suggests, consistentwith other research, that explicit attention toimplementing instructional changes is vitalto the effectiveness of SLCs.̂ ^ This findingis echoed and emphasized in a recent reportto the Cates Eoundation that synthesizescurrent research and discusses the chal-lenge of converting large high schools toSLCs.̂ ^ It may be that instractional changeis particularly difficult to achieve when largehigh schools are converted into smaller highschools, and this may partly explain why con-version SLCs are less likely to succeed.

It is important to note that the implemen-tation of SLCs is a key program feature inseveral of the comprehensive reform modelsreviewed above, including Talent Develop-ment, Eirst Things Eirst, and High SchoolsThat Work. Thus, the results demonstrated inevaluations of these models may be, at leastin part, attributed to the use of SLC structureas a fundamental element. However, no studyhas established the contribution that SLCsmake to the outcomes of comprehensiveschool reform models.

Specialty AcademiesThe formation of specialty academies, includ-ing career academies and academies with acurricular focus such as science, technology,engineering, and math, is often part of astructural change into SLCs. The units thatresult from downsizing a large, comprehen-sive high school often are formed aroundparticular themes, although in many casesstudents and teachers are able to cross SLCboundaries. Eor the purpose of distinguish-ing this approach to high school refonn fromSLCs more broadly, we define specialtyacademies here as schools that are largelyself-contained and committed to the careeror curricular theme, so that most of the

120 THE FUTURE OF CHiLDREN


experiences of the students are related to thattheme. However, it is worth noting that mostschools implementing SLCs are using careeracademies as their model.''' Specialty acad-emies are most focused on making changes tothe instruction and structure of schools.

Specialty academies are designed to achievenearly all of the five desired outcomes of highschool improvement. In particular, they seekto create personalized leaming environments,often with small enrollments and stablestudent-teacher groupings across grades.They also seek to address instructionalcontent and pedagogy, focusing on particularcurricular areas with increased rigor in somecases (for example, STEM academies, whichfeature science, technology, engineering, andmath), and increased relevance in others (forexainple, career academies). This approachperhaps most specifically seeks to addressdirectly the challenge and desired outcomesof preparing students for the world beyondhigh school, both for postsecondary educa-tion and for tlie world of work. Some typesof specialty academies do include compo-nents of support for students who enter highschool with poor academic skills; however,it is important to note that, depending ontheir eligibility and selection policies, selec-tive academies may not address this desiredoutcome.

Career academies as a program have been inexistence since 1969 and are now operatingin more than 2,500 schools in the UnitedStates.̂ ** Career academies operate as aschool-within-a-school structure, wherestudents have the same teachers acrossgrades, teachers have common planning timeto share in decision making, and studentstake at least one occupational course eachyear related to their academy's career theme.Partnerships with local businesses are a key

feature of career academies. Local employersprovide internship opportunities for studentsand help schools in developing curricula foroccupational courses.

A relatively strong body of evidence isavailable for the effect of career academeson student outcomes. Studies (main''quasi-experimental) conducted between 1985and 2000 suggest that students in careeracademies outperform non-academy studentson measures of academic success in highschool, although differences in postsecondaryeducation and employment are less consis-tently positive and statistically significant.̂ **However, it is important to note that thesestudies, although they use analytic techniquesto control for observed differences betweenacademy and non-academy students (forexample, prior achievement), are not basedon random assignment of students to careeracademies. For example, studies by DavidStern and several colleagues found thatstudents attending ten career academies inCalifornia posted higher attendance andgrades, earned more credits, and were morelikely to stay in school than matched compari-son students."" Using propensity scorematching. Marc Elliott, Lawrence Hanser,and Curtis Cilroy reported similar findings ina comparison of students in different types ofacademies located in large cities.®' Althoughthe outcomes after high school examined inthis research are mixed, some positivefindings reported in some studies includehigher participation in postsecondaiy educa-tion for academy students, lower rates ofcollege remediation, and higher rates ofbachelor's degree completion, compared withstatistically similar non-academy students. '̂-

Because of its rigorous research design usingrandom assignment, we have high confidencein the findings reported in a fifteen-year-long

VOL. 19 / NO. 1 / SPRING 2009 121


evaluation of career academies conducted byJames Kemple at MDRC.^ In the early yearsof the study, the researchers found that thecareer academies model provided studentswith more support, career guidance, opportu-nities to take technical classes, and workexperience than the schools attended bystudents not in career academies. Retentionrates among high-risk students were higheramong career academy students comparedwith their counterparts while still in highschool. Although no effect was found onachievement scores while students were inhigh school or on postsecondary educationattainment after high school, the analysis oflong-term labor market outcomes revealssignificant effects. The ten- and fifteen-yearfollow-up reports indicate that career acad-emies produced positive and sustained effectson labor market outcomes, particularly foryoung men. Young men—even those at thehighest risk of dropping out of high school—who attended career academies postedearnings 18 percent higher than non-academystudents four years after they left high school.Eight years after leaving high school, careeracademy students (women and men) earned11 percent more than non-career academystudents; for men, real earnings for academystudents were 17 percent higher (earningswere $3,731 higher per year on average overthe eight-year period) than those for non-academy students.**̂

Based in part on the findings reported inMDRC's experimental, longitudinal study ofcareer academies, the What Works Clearing-house review of the effectiveness of careeracademies as a dropout-prevention interven-tion in 2006 concluded that the career acad-emies model has "potentially positive" effectson staying in school and progressing in schoolbut "no discernible effects" on completingschool.^

Thus, promising evidence shows that thecareer academies approach can improveoutcomes for students, particularly in thelonger term. Not much evidence yet exists,however, on the potential effect of othertypes of specialty academies in attaining thegoals of improving instruction in high schoolsand preparing students for the world beyondhigh school.

Thus, promising evidenceshows that the careeracademies approach canimprove outcomes forstudents, particularly inthe longer term.

Charter Schools and EducationManagement OrganizationsCharter schools and education managementorganizations (EMOs) epitomize an approachto improvement that focuses on how schoolsare run. Their approach suggests that, byaltering their governance, schools will havegreater opportunities to make requiredinstructional and structural changes that canlead to improvement. The approximately 900charter high schools* around the coun-try reflect this approach, which is focusedon governance reform and most directlyaddresses the desire to bring posit:ive changeto overstressed high schools.

The underlying rationale for charter schools isthat autonomy and flexibility in governance,and the creation of market competitionamong schools, will allow charter schools todevelop the attributes of effective schools.



Public charter schools are exempt from manystate regulations but are held accountable forimproving student achievement. This meansthat charter schools generally have greaterfiscal control, more discretion over hiring andfiring of teachers and school staff, and morefreedom to implement programs (such asthose reviewed in this chapter) dian dotraditional public high schools. In exchangefor these exemptions, charter schools haveagreements or contracts with state-approvedauthorizing agencies that make explicit theschools' accountability to demonstrateimproved student achievement.̂ ^

Although charter high schools vary extensivelyin focus and operation, many share a missionthat, in theory, addresses nearly all fivedesired outcomes for high school improve-ment. To foster a personalized and safeleaming environment, many charter highschools are small in size and use strategiessuch as advisory programs to support stu-dents and improve student engagement.Often located in inner cities, charter highschools' primary goal is typically to create asafe environment for learning that providessocial and academic support for traditionallyunderserved students.

EMOs are either for-profit or nonprofit edu-cation organizations that contract with new orexisting public, charter, or private schools andschool districts to provide comprehensive ser-vices to schools. These services include, butare not limited to, educational programmingand administrative services. Educational pro-gramming includes curriculum design, pro-fessional development, and tools for studentassessment. Administrative services includeoperation-management (for example, stu-dent enrollment, school marketing), financialmanagement (for example, payroll assistance,budget oversight), facilities management

(maintenance and use of facilities), andhuman resources management (hiring andtraining staff, staff benefits). Many of the ser-vices provided by EMOs are comparable tothose offered by whole-school improvementproviders, such as comprehensive schoolreform models.^ EMOs are included in thisdiscussion because they often mn charterschools and, in addition, often manage low-performing schools for districts. Although notall charters are mn by EMOs and EMOs domore than run charters, charters and EMOsshare in common the fundamental premisethat schools will be more successful if theyare govemed differently.

Many EMOs focus narrowly on administrativeoperations, but some take a more compre-hensive approach and also address some orall of the five desired outcomes definedearlier. For example, the organization^structure of Edison's Whole School Manage-ment model is designed to create small,fiexible schools within schools, known asacademies ("Senior Academies" for studentsin grades nine to ten; "Collegiate Academies"for students in grades eleven to twelve), forthe purpose of fostering a more personalizedenvironment.^^

Charter high schools and EMOs vary widelyin the extent and ways in which they assiststudents who enter high school with pooracademic skills. Charter high schools oftenimplement strategies to involve parents andcommunity members, some of whom becomepart of tutoring and mentoring programs.

Vast differences exist in how charter highschools and EMOs address explicitly thecurriculum and instmctional challenges thatmust be met to achieve the goal of improvingcontent and pedagogy in high schools. Somecharter high schools seek to make the

VOL. 19 / NO. 1 / SPRING 2009 123


instruction rigorous, relevant, and innovative;many struggle to meet this goal.™ Manycharter schools implement other whole-school refonn models. As noted, aboutone-third of current schools participating inthe Early College High School Initiative arecharter schools."'

Some EMOs do not address content andpedagogy at all; instead, they focus squarelyon improving how die school is run. OtherEMOs do address classroom practice. TheEdison School design, for example, includescurricular programs, either selected or devel-oped by Edison, for all core academic sub-jects. In the Senior Academy, curricula aredesigned to prepare students for advancedplacement (AP) courses that are offered inthe Collegiate Academy. To prepare studentsfor college, Edison partners with PrincetonReview to focus on SAT or ACT preparationand provides college and career counseling.

Research comparing outcomes of charterschool students with those of students attend-ing traditional public schools is emerging, butstudies on the overall effectiveness of charterhigh schools are lacking. For example, a studyon California charter high schools finds that,after adjusting for enrollment size and stu-dent characteristics, charters that are "class-room based" score higher than non-charterson performance indicators, including the per-centage of students proficient or above on theCalifornia High School Exit Exam in Englishand Math.'^ These findings are suggestive butnot at all definitive because students self-select into the charter schools, and factorsother than their charter school experiencemay explain their higher performance.

As with the other models we discuss, charterschools come in many forms. An importantquestion, therefore, is what features of

charter high schools are likely to producepositive outcomes for students. A recentanalysis of charter high schools took on thisquestion by identifying schools with goodtrack records in terms of graduating studentswho go on to postsecondary success. Theanalysis reports that successful charter highschools seem to maintain a focus on highereducation and foster a safe, orderly learningenvironment and positive school culture."^The practices observed in these high-performing charter high schools are alignedto all five desired outcomes of high schoolimprovement; however, it is important tonote that this study sample is small and hasno comparison group.

As with charter high schools, we lack definitiveevidence about the effectiveness of EMOs asa whole for high school improvement.Through the Comprehensive School ReformQuahty Center, we conducted a systematicreview of the effectiveness and quality ofseven widely implemented EMO models, fiveof which serve students in K-12 and none ofwhich exclusively serves high school students.Our rating for the overall effectiveness onstudent achievement of the EMO EdisonSchools was "Moderate." "'' The rating wasderived in part from our review of a five-yearquasi-experimental evaluation of EdisonSchools conducted by RAND, which reportedmixed results for reading and math achieve-ment for Edison School students comparedwith students not in Edison Schools."^ Westress that our rating appUes to Edison's K-12model and none of the studies we reviewedisolated the effects of Edison Schools on highschool student outcomes. The other sixmodels in our review of EMOs received arating of either "Zero " or "No rating," bothsignifying a lack of strong research thatdemonstrates positive effects on studentoutcomes. Tlierefore, we conclude that, as a


Improving Low-Petforming High Schools: Searehingfor Evidence of Promise

whole, there is not yet reliable evidence thatEMOs can have a positive impact at the highschool level.

Although it is as yet impossible to assess theeffectiveness of charter high schools andEMOs in improving high school studentoutcomes, including achievement andpostsecondary success, both approaches areimportant options for education decisionmakers to consider. By focusing on schoolgovernance, charter high schools and EMOscan address head-on the organizational andinstitutional capacity issues of low-performinghigh schools, thereby potentially stimulatingchange in overstressed high schools, the fifthdesired outcome in high school improvement.

Implementation Is CrucialOne piece of evidence regarding research-based reform is probably more consistentand may be more important than all theothers. Implementation is a critical factor inreform success. This observation may seemso obviously a matter of common sense thatit hardly needs to be stated or supported withevidence. But educators, for all their goodintentions, habitually defy both the strongresearch evidence and the common sensebehind this observation by implementingeducation reforms with neither the fidelitynor the long-term support required to allowthem to succeed and sustain themselves.

The authoi's of a recent large-scale synthesisof research concerning the iinplementationof evidence-based practices and programsacross a number of industries and socialservice arenas observe that it is often moredifficult to implement an effective model suc-cessfully than to design it.™ This observationpoints not only to the difficulty of implemen-tation but also to how crucial it is in gettinggood results.

Eurthermore, a program or practice is worthimplementing only when it is likely to havethe desired results. Again, this may seemobvious, but anyone who has spent any timein the education arena has heard peopleobserve that "If you do anything well, youwill get results. " But as the authors point out,"Desired outcomes are achieved only wheneffective programs are implemented well." "

The field of CSR lends further support to theneed to implement programs with fidelityand then to support them for the timerequired for their success. In their recentmultiyear, quasi-experimental study of CSRimplementation and impact involving 650elementaiy and middle schools in twenty-onedistricts across seventeen states, DanielAladjem and several colleagues found apositive relationship between the level offidelity of implementation and the level ofstudent achievement.'** The study identifiedseveral conditions associated \\dth higherachievement gains among the CSR studyschools than their matched comparisons.CSR must be implemented with high fidelityto the model generally, fidelity must be highduring later years of the model's introduction,and fidelity must be consistently high acrossthe numerous model components and notjust in a few.'̂

The finding that implementation fidelity is asignificant factor for success should notobscure a number of important consider-ations. Amanda Datnow and Sam Stringfieldhave pointed out that, based on their reviewof the findings of sixteen studies and morethan 300 school case studies conducted fromthe mid-1980s into the late 1990s, implemen-tation of external reform models is a complexprocess, which requires the model providersto work together with schools and districts to"co-construct" the reform's implementation.**"

VOL. 19 / NO. 1 / SPRiNG 2009 125


No model is adopted wholesale and imple-mented in exactly the same way in all loca-tions. Simply put, implementation is not amechanical process but an adaptive one thatmust be conducted taking due account of thestructural, financial, political, and culturalenvironments of each school and district.

Taken together, these studies offer significantpolicy lessons. First, precious time andresources will be wasted by selecting ineffec-tive models to improve high schools. Somemodels are likely to be more effective thanothers, regardless of the contexts in whichthey are implemented. However, the modelsthat are most likely to have significant impactin any given situation are those effectivemodels that engender the commitment ofschool and district staff and leaders. Finally,once effective models are selected andimplemented at schools, additional time andresources will be wasted by failing to givethem the time and support they need tosucceed.

Conclusion: Evidence-BasedModels Are Necessary butInsufficient for ChangeOverall, evidence for the effectiveness of thehigh school improvement models we havereviewed is sparse. There are, however,glimmers of hope. Some models have anemerging evidence base of effectiveness, andmore research is under way that can help toidentify the models and approaches thatdemonstrate the most promise. But havingsolid, research-based evidence of modeleffectiveness is just the first step in improvinghigh schools. The second, and equally impor-tant step, is to implement effective modelswith care and with fidelity to the requirementsofthe models. Finally, improving high schoolsrequires taking a holistic view—focusingsimultaneously on the desirability of a number

of outcomes and recognizing that high schoolscan be improved not by adopting piecemealprograms or actions but through systemic,coordinated action that may involve combin-ing many approaches.

Looking to the future, we suggest steps thatdecision makers, researchers, and sponsors ofresearch can take to promote evidence-basedprogress in education. We believe that deci-sion makers should demand rigorous evidenceof effectiveness before they consider a modelfor widespread adoption. They can get thisevidence by consulting reliable third-partyreview organizations, such as the What WorksClearinghouse, or by seeking advice directlyfrom researchers and organizations withexpertise in judging research quality. If, as islikely, no models or approaches underconsideration have strong evidence of effec-tiveness, decision makers should implementprograms on a pilot basis—and engage in arigorous evaluation of effects—beforeproceeding to widespread scale-up.

Researchers should join with decision makersin helping to design and execute small-scale,cost-effective tests of promising models. Theyshould also design larger studies that createthe types of planned variation in the imple-mentation of models that make it possible toidentify program elements that seem particu-larly critical to success. Identifying thesecritical elements would help to assure fidelityof implementation to the "required" elementsof the model and those areas available for"co-construction" in which individualsimplementing the model have greater leewayto experiment. These planned variations willalso produce the evidence necessary todesign future, more effective models.

Finally, sponsors of research should considerfunding the types of studies suggested above.


Improving Low-Performing High Schools: Searehingfor Evidence of Promise

They should also commit resources to long- improve high schools is stronger today thanterm funding of a stream of research that can ever before, it is still not yet robust enough toresult in more definitive answers about which truly promote evidence-based practice inmodels do work, under what conditions, and high school reform. Over the coming years,for which types of students. policy makers must continue to demand, and

researchers to supply, better evidence. If theyAlthough the knowledge base regarding do, we believe tliat high school students willpromising strategies and programs to be able to look toward a brighter future.

VOL. 19 / NO. 1 / SPRING 2009 127

Steve Fleiseltman and Jessica Heppen

Endnotes

1. Alan R. Sadoviiik and others, "Sociological Perspectives on NCLB and Federal Involvement in Education,"

in No Child. Left Behind, and the Reduction ofthe Achievement Gap: Sociological Perspectives on Federal

Education Policy, edited by Alan R. Sadovnik and others (New York and London: Routledge, 2008), p. 361.

2. Center on Education Policy, Managing More than One Thousand Remodeling Projects: School Restructur-

ing in California (VVasliington: Center on Education Policy, Eebruar)' 2008).

3. See, for example. Comprehensive School Reform Quality Center, Works in Progress: A Report on Middle

and High School Improvement Program's (Washington: Comprehensive School Reform Quality Center,

American Institutes for Researcii, January 2005); and Chris Dolejs and others. Report on Key Practices

and Policies of Consistently Higlier Performing High Schook (Washington: National High School Center,

American Institutes for Research, October 2006).

4. James J. Kemple, Career Academies: Long-Tenn Impacts on Labor Market Outcomes, Educational Attain-

ment, and Transitions to Adulthood (New York: MDRC, June 2008).

5. Comprehensive School Reform Quality Center, CSRQ Center Report on Middle and High School CSR

Models (Washington: Comprehensive School Reform Quality Center, American Institutes for Research,

October 2006).

6. "Dropout Prevention" section ofthe What Works Clearinghouse (ies.ed.gov/nceeAN'WC/reports/topic.

aspx?tid=06 [accessed March 3, 2008]).

7. Comprehensive School Reform Quality Center, CSRQ Center Report on Middle and. High School CSR

Models (see note 5).

8. Janet Quint, Meeting Five Critical Challenges of High School Refonn: Lessons from Research on Three

Refonn Models (MDRC, May 2006). This framework and the number of models used to develop it were

further expanded by MDRC researchers in a brief produced for the National High School Center Corinne

M. Herlihy and Janet Qnint, Emerging Evidence on Improving High School Student Achievement and.

Graduation Rates: Tlie Effects of Four Popular Improvement Programs (Washington: National High School

Center, American Institutes for Research, November, 2006). Many ofthe same challenges were identified

by the CSRQ Center; see CSRQ Center, Works in Progress (see note 3).

9. See for example, CASEL Lfpdate, "The Benefits of School-Based Social and Emotional Leaming Programs:

Highlights from a Forthcoming CASEL Report" (Chicago: Collaborative for Academic, Social and Emo-

tional Leaming, December 2007).

10. CSRQ Center, Works in Progress (see note 3), pp. 45^9 .

11. Jennifer O'Day, "NCLB and the Comple.xity of School Improvement," in No Child Leß Behind and. the Re-

duction ofthe Achievement Gap, edited by Sadovnik and others (New York and London: Routledge, 2008),

pp. 27, 40, 46.

12. American Institutes for Research, Research Retrospective: Teacher Quality Research in 2007 (Washington:

American Institutes for Research, n.d.), pp. 2, .5. Tricia Coulter, "Implementing NCLB: State Plans to

Implement tlie Challenge of Equitable Distribution of Effective Teachers," in America's Challenge: Effective

Teachers for At-Risk Schools and Students, edited by Carol A. Dwyer (Washington: National Comprehensive



Center for Teacher Quality, Learning Point Associates, 2007), pp. 55-70. R. M. Ingersoll, Out-of-Field.

Teaching, Educational Inequality, and. the Organization of Schools: An Exploratonj Analysis (Seattle:

Center for the Study of Teaching and Policy, 2002).

13. See for e.xample, C. Rouse, "Labor Market Consequences of an Inadequate Education," paper presented

at the Symposium on the Social Costs of Inadequate Education, Teachers College at Columbia University,

September 2005. Available at: http://devweb.tc.columbia.edu/nianager/symposium/Files/77_Rouse_paper

. pdf [July 28, 2008].

14. U.S. Department of Labor, Bureau of Labor Statistics, "Annual Averages—Household Data: Employment

Status of the Civilian Noninstitutional Population 25 Years and Over by Educational Attainment, Sex, Race,

and Hispanic or Latino Ethnicity," Employment and Earnings 52 (2005): 204; U.S. Department of Labor,

Bureau of Labor Statistics, "Usual Weekly Earnings of Wage and Salary Workers: The Second Quarter

2008" (Washington: Bureau of Labor Statistics, July 2008); Thomas J. Kane and Cecelia E. Rouse, "Com-

ment on W. Norton Crubb: The Varied Economic Returns to Postsecondary Education: New Evidence

from the Class of 1972,'" Journal of Human Resources 30, no. 1 (Winter 1995): 205-21.

15. The term "coniprehensive school reform" (CSR) is often used interchangeably with "wliole school reform."

For further background on comprehensive sciiool reform, see: wwvv.csrq.org/aboutcsrasp [July 28 2008].

16. CSRQ Center Report on Middle and High School CSR Models (see note 5), pp. 25-26.

17. It is interesting to note that, despite their comprehensiveness, none of four CSR programs (ATLA8 Com-

munities, Eirst Things First, High Schools That Work, Talent Development High Schools) reviewed by tlie

C8RQ Center in its report on programmatic responses to key "hot topic" issues in liigli schools featured a

formal violence-reduction component. CSRQ Center, Works in Progress (see note 3), p. 79.

18. The CSRQ Center Report on Middle and High School CSR Models, cited above (see note 5), provides

detailed reviews of each of the models mentioned in this chapter In addition to rating their evidence of

effectiveness in five key outcome domains of interest to policymakers, the report provides a thorougli sum-

mary of the prograin's mission, goals, costs, organization and operation, and key considerations regarding its

implementation. These reports are available onUne at www.csrq.org/MSHSreport.asp [July 28, 2008].

19. CSRQ Center Report on Middle and High School CSR Models (see note 5), p. 51.

20. Ibid., p. 68.

21. Quint, Meeting Five Critical Challenges of High School Reform (see note 8), pp. 10, 30-32.

22. CSRQ Center Report on Middle and High School CSR Models (see note 5), pp. 53-54.

23. Ibid., pp. 52-54.

24. Overall statements are made difficult because of the wide array of individual CSR programs that take

different approaches to improvement, because the evidence of effectiveness for the overall CSR approach

is often provided for grades K-12 without differentiating the high school outcomes, and because no meta-

analysis has been undertaken of outcomes for all schools using high school CSR programs.

25. C. D. Borman and others, Coniprehensive School Refonn and Student Achievement: A Meta-Anab/sis (Bal-

timore: Center for Research on the Education of Stndents Placed at Risk, Johns Hopkins University, 2002),

VOL. 19 / NO. 1 / SPRING 2009 129

Steve Fleisehman and Jessiea Heppen

p. 34. Another recent large-scale study of CSR, conducted at the elementary school and middle school

levels, also concluded that, when well implemented, CSR models experience higher academic achievement

gains than comparison schools. D. K. Aladjem and others. Models Matter: The Final Report of the National

Longitudinal Evaluation of Comprehensive School Refonn (Washington: American Institutes for Research,

September 2006), p. 6.

26. To arrive at its ratings, the center weighed the strength and quality of a program's evidence of effectiveness

and the size of the overall impact as computed from the studies that met the center's standards! For more

on the centers approach to program rating, see the CSRQ Center Report on Middle and High School CSR

Models (see note 5), pp. 17-20.

27. Ibid., pp. 20-21.

28. M. M. Karp and others. State Dual Enrollment Policies: Addressing Access and Quality (Washington: U.S.

Department of Education, Office of Vocational and Adult Education, 2004).

29. Ibid.

30. B. Kleiner and L. Lewis for National Center for Education Statistics, Dual Enrollnient of High School Stu-

dents at Postsecondary Institutions: 2002-03, NCES 2005-08 (Washington: U.S. Department of Education,

2005).

31. C. Krueger, Dual Enrollment: Policy Issues Confronting State Polictjinakers (Denver: Education Commis-

sion of the States, 2006).

32. M. Martinez and S. Klopott, The Link between High School Refonn and College Access and Success for

Low-Incoine and Minority Youth (Washington: American Youth Policy Forum and Pathways to College

Network, 2005).

33. A. Berger and others. Early College High School Initiative: 2003-05 Evaluation Rejiort (Washington:

American Institutes for Research and SRI International, 2006).

34. A. Berger and others, 2003-07 Early College High School Initiative: Emerging Patterns and Relationships

(Washington: American Institutes for Research and SRI International, 2008).

35. N. Hoffman, Add and Subtract: Dual Enrollment as a State Strategy to Increase Postsecondary Success for

Underrepresented Students (Boston: Jobs for the Future, 2005); National High School Center, Findings

from the Early College High Schools Initiative: A Look at Best Practices and Lessons Learned Regarding a

Dual Enrollment Program (Washington: American Institutes for Research, 2006).

36. Krueger, Dual Enrollment (see note 31).

37. U.S. Department of Education, Principal Indicators of Student Academic Histories in Postsecondary Edu-

cation 1972-2000 (Washington: U.S. Department of Education, 2004).

38. M. Dynarski and others. Impacts of Dropout Prevention Programs: Final Report, A Research Report from

the School Dropout Prevention Denwnstration Assistance Program Evaluation (Princeton, N.J.: Matliematica

Policy Research, 1998).

39. Ibid.



40. A. Berger and others. Evaluation ofthe Early College High School Initiative: Select Topics on Implementa-

tion (Washington: American Institutes for Research and SRI International, 2007).

41. Berger and others. Early College High School Initiative: 2003-05 (see note 33); Berger and others,

2003-07 Early College High School Initiative: Emerging Patterns and Relationships (see note 34).

42. Berger and others, 2003-07 Early College High School Initiative: Emerging Patterns and Relationships (see

note 34).

43. Ibid.

44. Martinez and Klopott, The Link between High School Reform and College Access and Success (see note 32).

45. L. Bernstein and others. Implementation Study of Smaller Leaming Communities: Final Report (Cam-

bridge, Mass.: Abt Associates for U.S. Department of Education Office of Planning, Evaluation, and Policy

Development, Policy and Program Studies Service, 2008).

46. Ibid.

47. K. Cotton, School Size, School Climate, and Student Performance (Portland, Ore.: Northwest Regional

Educational Laboratory, 1996); K. Cotton, New Small Leaming Communities: Findings from Recent Lit-

erature (Portland, Ore.: Northwest Regional Educational Laboratory, 2001); L. Page and others. National

Evaluation of Smaller Leaming Communities, Literature Review, Executive Siimmary (Cambridge, Mass.:

Abt Associates, 2002); V. E. Lee and others, "Inside Large and Small High Schools: Curriculum and Social

Relations," Educational Evaluation and Policy Analysis 23, (2000): 147-71.

48. Bernstein and others. Implementation Study of Smaller Leaming Communities (see note 45).

49. V. E. Lee and J. B. Smith, "High School Size: Which Works Best and for Whom?" Educational Evaluation

and Policy Analysis 19 (1997): 205-27.

50. J. Kahne and others. Small High Schools on a Larger Scale: The First Three Years ofthe Chicago High

School Redesign Initiative (Chicago: Consortium on Chicago School Research, 2006); D. Rhodes and

others. Getting Results: Student Outcomes in New and Redesigned High Schools (Washington: American

Institutes for Research and SRI International, 2005); B. Smerdon and J. Cohen, Baltimore City's High

School Refonn Initiative: Schools, Students, and Outcomes (Washington: Urban Institute, 2007); S. James-

Burduiuy, I. Perez-Johnson, and S. Vartivarian, High School Refonn in Boston Puhlic Schools: The Effect

of Focus on High Schools on Student Academic Outcomes (Princeton, N.J.: Mathematica Policy Research,

Inc., 2008); E. M. Foley, A. Khnge, and E. R. Reisner, Evaluation of New Centunj High Schools: Profile of

an Initiative to Create and Sustain Small, Successful High Schools (Washington: Policy Studies Associates,

Inc., 2007).

51. Foley and others. Evaluation of New Century High Schools (see note 50).

52. James-Burdumy and others. High School Refonn in Boston Public Schools (see note 50).

53. American Institutes for Research and SRI International, Evaluation ofthe Bill and. Melinda Gates Founda-

tion's High School Grants Initiative: 2001-05. Final Report (Washington: American Institutes for Research

and SRI Internationa], 2006).

54. Smerdon and Cohen, Baltimore City's High School Reform Initiative (see note 50).

VOL. 19 / NO. 1 / SPRING 2009 131

Steve Fleischman and Jessica He])pen

55. American Institutes for Rcsearcli and SRI International, Evaluation of the Bill and Melinchi Gates Founda-

tion's High School Grants Initiative: 2001-05 (see note 53).

56. J. T. Fouts and others. Leading the Conversion Process: Lessons Learned, and Recommendations for Con-

verting to Small I.^an^ing Communities, prepared (or the Bill and Melinda Gates Foimdation (Fouts and

As.sociates, L.L.C., 2006).

57. V. E. Lee, D. D. Ready, and D. J. Johnson, "The Difficulty of Identifying Rare Samples to Study: The Case

of High Schools Divided into Schools-Within-SchooLs," Educational Evaluation and Policij Analysis 23, no.

4 (2001): 365-79; Bemstein and others. Implementation Study of Smaller I.j¡a m ing Coinnmnities (see note

45).

58. Quint, Meeting Five Critical Challenges of High School Refonn: Lessons from Research on Three Refonn

Models (see note 8).

59. D. Stem and others, "Learning by Doing Career Academies," in Improving School-to-Work Transition,

edited by D. Nenmark (New York: Russell Sage Foundation, 2007), pp. 134-68.

60. D. Stern and others, "Benefits and Costs of Dropout Prevention in a High School Program Combining

Academic and Vocational Education: Third-Year Results from Replications o( the California Partnership

Academies," Educational Evaluation and Policij Analysis 11 (1989): 405-16.

61. M. N. Elliott, L. M. Hanser, and C. L. Gilroy, Evidence ofPo.sitive Student Outcomes in jROTC Career

Academies (Santa Monica, Calif.: RAND Corporation, 2000).

62. N. L. McLwvell and V. Rubin, High Scliool Career Academies: A Pathway lo Edticalional Refomi in Urban

Schools? (Kalamazoo, Mich.: VV. E. Upjohn Institute for Employment Research, 2000); N. L. Maxwell,

"Step to College: Moving from the Higli School Career Academy through tlie Four-Year University,"

Evahiation Review 25, no. 6 (2001): 619-54.

63. Kemple, Career Academies (see note 4).

64. Ibid.

65. See the What Works Clearinghouse report on Career Academies (http://ies.ed.gov/ncee/wwc/reports/drop

out/career_academic/ [July 28, 2008]).

66. Center for Education Reform, "Charter Schools by the Numbers: Research Fact Slieet" (Wasliington: Center

for Education Reform, 2007) (vvww.edreform.com/charter_directory/charters-by-niuiiber.pdf [July 28, 2008]).

Also note that another 860 charter schools combine middle school and high school grades.

67. U.S. Department of Education, Office of Innovation and Improvement, Charier High Schools: Closing the

Achievement Gap (Washington: U.S. Department of Education, 2006).

68. Comprehensive School Reform Quality Center, CSRQ Center Report on Education Service Providers

(Washington: American Institutes for Research, 2006).

69. Ibid.

70. U.S. Department of Education, Charter High Schools (see note 67).

71. Berger and others. Early College High School Initiative (see note 33).

1 3 2 THE FUTURE OF CHILDREN


72. B. Edwards and others, California's Charter Schools: 2008 Performance Update (Mountain View, Calif.:

EdSource, 2008.

73. U.S. Department of Education, Charter High Schools (see note 67).

74. It should be noted that on June 30, 2008, Edison Schools became EdisonLearning. It is not clear at the

time of publication whether Edison Leaming will continue to offer EMO services. The information regard-

ing the effectiveness of the Edison model is included in this article to illustrate the potential that may be

offered by the EMO approach to school improvement.

75. B. P. Cill and others. Inspiration, Perspiration, and Time: Operations and Achievement in Edison Schools

(Arlington, Va.: RAND Corporation, 2005).

76. D. L. Fixsen and others. Implementation Research: A Synthesis ofthe Literature, FMHI Publication 231

(Tampa: University of South Florida, Louis de la Partner Florida Mental Health Institute, The National

Implementation Research Network, 2005).

77. Ibid., p. 12.

78. Aladjem and others. Models Matter (see note 25), p. 6.

79. Ibid.

80. A. Datiiow and S. Stringfield, "Working Together for Reliable School Reform," J(;(/n)«Z of Education for

Students Placed at Risk 5, no. 1 (2000): 183-204.

VOL. 19 / NO. 1 / SPRING 2009 133

Improving Low-Performing High Schools: Searching for ... · Searching for Evidence of Promise Steve...

Documents

Transcript of Improving Low-Performing High Schools: Searching for ... · Searching for Evidence of Promise Steve...