Adapting to large-scale changes in Advanced Placement

Adapting to large-scale changes in Advanced PlacementBiology, Chemistry, and Physics: the impact of online teachercommunities

Kim Frumin a, Chris Dede b, Christian Fischerc, Brandon Fosterd, Frances Lawrenze,Arthur Eisenkraft f, Barry Fishman g, Abigail Jurist Levyd and Ayana McCoy h

aGraduate School of Education, Harvard University, Cambridge, MA, USA; bGraduate School of Education,Harvard University, Cambridge, MA, USA; cSchool of Education, University of Michigan, Ann Arbor, MI, USA;dSTEM Portfolio, Education Development Center, Inc., Waltham, MA, USA; eDepartment of EducationalPsychology, University of Minnesota, Minneapolis, MN, USA; fCenter of Science and Math in Context,University of Massachusetts Boston, Boston, MA, USA; gSchool of Information, University of Michigan, AnnArbor, MI, USA; hCenter of Science and Math in Context, University of Massachusetts Boston, Boston, MA, USA

ABSTRACT

Over the past decade, the field of teacher professional learning hascoalesced around core characteristics of high quality professionaldevelopment experiences (e.g. Borko, Jacobs, & Koellner, 2010.Contemporary approaches to teacher professional development.In P. L. Peterson, E. Baker, & B. McGaw (Eds.), Internationalencyclopedia of education (Vol. 7, pp. 548–556). Oxford: Elsevier.;Darling-Hammond, Hyler, & Gardner, 2017. Effective teacherprofessional development. Palo Alto, CA: Learning Policy Institute).Many countries have found these advances of great interestbecause of a desire to build teacher capacity in science educationand across the full curriculum. This paper continues this progressby examining the role and impact of an online professionaldevelopment community within the top-down, large-scalecurriculum and assessment revision of Advanced Placement (AP)Biology, Chemistry, and Physics. This paper is part of a five-year,longitudinal, U.S. National Science Foundation–funded project tostudy the relative effectiveness of various types of professionaldevelopment in enabling teachers to adapt to the revised APcourse goals and exams. Of the many forms of professionaldevelopment our research has examined, preliminary analysesindicated that participation in the College Board’s online APTeacher Community (APTC) – where teachers can discuss teachingstrategies, share resources, and connect with each other – hadpositive, direct, and statistically significant association with teacherself-reported shifts in practice and with gains in student AP scores(Fishman et al., 2014). This study explored how usage of theonline APTC might be useful to teachers and examined a morerobust estimate of these effects. Findings from the experience ofAP teachers may be valuable in supporting other large-scalecurriculum changes, such as the U.S. Next Generation ScienceStandards or Common Core Standards, as well as parallelcurricular shifts in other countries.

ARTICLE HISTORY

Received 31 August 2017Accepted 4 January 2018

KEYWORDS

Online teacher learningcommunities; AdvancedPlacement; professionallearning

© 2018 Informa UK Limited, trading as Taylor & Francis Group

CONTACT Kim Frumin [email protected] Graduate School of Education, Harvard University, 424 Long-fellow Hall, 13 Appian Way, Cambridge, MA 02138

INTERNATIONAL JOURNAL OF SCIENCE EDUCATION, 2018

VOL. 40, NO. 4, 397–420

https://doi.org/10.1080/09500693.2018.1424962

Introduction

Many countries across the developed and developing world have realised the importance

of teacher capacity building. For example, the Goal 4 (Education) targets in the United

Nations Sustainable Development Goals includes the following: ‘By 2030, substantially

increase the supply of qualified teachers, including through international cooperation

for teacher training in developing countries, especially least developed countries and

small island developing states’ (United Nations, 2016). The Organization for Economic

Cooperation and Development (OECD) has shown that, across the world, professional

learning is a powerful lever for improving educational outcomes (Schleicher, 2016).

Over the past decade, the study of teacher professional learning has made important

progress along a number of fronts. Internationally, the field has coalesced around core

characteristics of high quality professional development experiences (e.g. Borko, Jacobs,

& Koellner, 2010; Darling-Hammond, Hyler, & Gardner, 2017). While these high quality

professional development characteristics have been developed in the U.S. context, they

have been implemented in interventions in countries around the globe including

Germany (e.g. Roesken-Winter, Schüler, Stahnke, & Blömeke, 2015; Weißenrieder,

Roesken-Winter, Schueler, Binner, & Blömeke, 2015), the United Kingdom (Pedder,

Opfer, McCormick, & Storey, 2010), and Australia (Askell-Williams & Murray-Harvey,

2016), among others. Models have been proposed describing how commonly defined pro-

fessional development characteristics relate to desired student learning outcomes (Desi-

mone, 2009) and various channels and technologies that can be used to provide access

to professional development opportunities have been explored (Fishman & Dede, 2016).

This paper continues this progress by examining the role and impact of an online pro-

fessional development community within a top-down, large-scale curriculum revision

associated with high-stakes assessments in a United States context: the Advanced Place-

ment (AP) examinations offered by the College Board in Biology, Chemistry, and

Physics. This context provides a unique opportunity to examine teacher learning for

several reasons. First, AP examinations present a common outcome measure associated

with curricula that are taught in thousands of secondary schools across the United

States, allowing teacher learning to be related to a desired student-learning outcome

measured with a validated, reliable assessment. Second, because the AP examination

scores hold real-world value for students (high-scoring secondary students are eligible

for college-level credit on the summative exam), teachers have a natural incentive to

engage in professional development related to the curriculum revision to provide their stu-

dents with better chances of success. Third, because AP is so widespread, a broad variety of

professional development options are available to teachers, from the College Board and

from third-party providers, allowing for the study of role and impact of the online com-

munity option within a complex, real-world teacher learning ecosystem. Fourth, the forms

of professional development effective in this context are generalisable to similar situations

in many countries across the world, in which teachers are asked to prepare pre-college stu-

dents with knowledge and skills that bridge to higher education (Schleicher, 2016).

Although AP courses are intended to promote rigorous, college-level instruction for

high school students, until recently the AP science tests have been criticised for privileging

breadth over depth (National Research Council, 2012; Sadler, Sonnert, Tai, & Klopferstein,

2010). In response to that criticism and to a 2002 U.S. National Research Council report

398 K. FRUMIN ET AL.

on science, technology, engineering, and math education, the AP Biology, Chemistry, and

Physics curricula and exams were redesigned in 2013, 2014, and 2015 respectively. These

courses now emphasise scientific inquiry and reasoning, as well as depth of understanding

for big ideas in science, over fixed, broad content coverage of facts (Harlen, 2015; Honey,

Pearson, & Schweingruber, 2014). Subsequent National Research Council science, tech-

nology, engineering, and math reports (National Research Council, 2011, 2012), the Fra-

mework for K-12 Science Education (National Research Council, 2012), and the Next

Generation Science Standards (NGSS Lead States, 2013) have emphasised the importance

of this shift in science curriculum standards, which parallel changes in other countries to

emphasise scientific inquiry, labs, and big ideas as opposed to extensive coverage of low-

level content knowledge (Achieve, 2010; Harlen, 2015; Liang, Liu, & Fulmer, 2017; Wind-

shitl & Barton, 2016).

These are sweeping changes to the long-standing AP instructional programme.

Approximately 400,000 AP Biology, Chemistry, and Physics students – and 20,000 tea-

chers – are now challenged to succeed on a high-stakes test focused on the inquiry

process, real-world applications of scientific principles, and synthesis of complex

content knowledge. To help teachers learn about the revisions, the College Board and

other providers offer a range of professional development, from weeklong summer work-

shops to short face-to-face courses, online self-paced courses, downloadable resources, and

online peer-learning communities.

This paper is part of a five-year, longitudinal, U.S. National Science Foundation–funded

project to study the relative effectiveness of these various types of professional development

in enabling teachers to adapt to the revised AP course goals and exams. Of the many forms

of professional development our research has examined, preliminary analyses indicated

that participation in the College Board’s online AP Teacher Community (APTC) –

where teachers can discuss teaching strategies, share resources, and connect with each

other – had positive, direct, and statistically significant association with teacher self-

reported shifts in practice and with gains in student AP scores (Fishman et al., 2014).

Therefore, it is important to explore how usage of the online APTC aids teachers in

helping their students succeed on the revised AP tests, as well as to provide a more

robust estimate of the effects. Findings from the experience of AP teachers may be valuable

in supporting other large-scale curriculum changes, such as the U.S. Next Generation

Science Standards or Common Core Standards and their counterparts in other countries.

This paper examines the following research questions:

1. What are teachers’ APTC usage patterns?

a. What types of teachers are using – or not using – the APTC; how and why are

teachers using the APTC?

b. For those participating teachers, what patterns, if any, of teacher and school

characteristics exist among those who post at least once in the online forums

(referred to as ‘posters’) versus those who engage without ever posting (referred

to as ‘lurkers’)?

2. What is the association of teacher APTC participation and student performance?

a. What is the relationship between teachers’ APTC participation and students’ AP

science performance?

INTERNATIONAL JOURNAL OF SCIENCE EDUCATION 399

b. How, if at all, does the frequency and duration of teachers’ APTC participation

relate to students’ AP science scores?

3. In what ways, if any, does the APTC, whose content and processes are driven by

bottom-up peer interactions, complement and extend top-down forms of the pro-

fessional development offered by the College Board?

Context for the study

The AP programme is offered by the U.S.-based College Board as a means of introducing

rigorous, college-level material to high school students across a broad range of subject

areas (College Board, 2015). The College Board defines curriculum standards for AP

courses and offers corresponding examinations that are administered in centralised

locations under controlled conditions and graded centrally for quality control and

norming. The examinations are scored with whole numbers on a 1 to 5 scale. Students

who earn a 3 or higher, depending on the discretion of their institution of higher edu-

cation, can use their scores toward college credit and/or as a way to place into advanced

courses upon arrival at college. Increasingly, U.S. colleges view AP courses and AP exam

performance as important information in the admissions process (Geiser & Santelices,

2006). There are no official College Board curriculum materials, though in each subject

there is a range of well-regarded texts created by third parties. Thus, AP teachers create

and submit their own curriculum plans that must be reviewed and certified by the

College Board before the course can officially be listed as AP on high school transcripts.

In 2013, 2014, and 2015 respectively, AP Biology, Chemistry, and Physics curricula and

exams were redesigned to align with recommendations from the U.S. National Research

Council (2002). The changes stress scientific inquiry and reasoning, reduce the emphasis

on broad content coverage, and focus on depth of understanding, in alignment with both

the Framework for K-12 Science Education (National Research Council, 2012) and the

U.S. Next Generation Science Standards (NGSS Lead States, 2013). The major facets of

the AP science redesigned curricula include (a) a refining of key concepts and content

into Big Ideas, which encompass core scientific principles, theories, and processes;

(b) identification of scientific skills that students should know and be able to do; and

(c) student-directed inquiry labs, which provide students with opportunities to take

risks, apply inquiry skills, and direct and monitor their own progress (College Board,

2015). In order to assess these new components, the three-hour redesigned AP science

exams consist of a 90-minute section of multiple-choice questions and a 90-minute

section of open-ended, free-response questions. Both sections evaluate students’ under-

standing of the Big Ideas and the ways in which this understanding can be applied

through science practices, such as the use of modelling and/or mathematical processes

to explain scientific principles, the manipulation and interpretation of data, and the

making of predictions and justification of phenomena (College Board, 2015).

The online APTC, a place where teachers can discuss teaching strategies, share

resources, and connect with each other (College Board, 2015), began approximately 15

years ago as an independent, teacher-created listserv and evolved into a College Board-

hosted electronic discussion board. In 2013, the College Board upgraded the APTC to a

web-based platform, allowing for posting, commenting, and sharing resources within


each subject-specific online community. Each subject-specific APTC comprises six web

pages:

. Landing page or Home page, which features the latest Discussion Board posts and most

recent community activity;. Discussions Board page, which features the online community and allows for subscrip-

tion to certain discussion threads;. Resources page, which features documents that members have uploaded and shared;. Curriculum Framework page, which features the subject-specific AP Big Ideas and

Science practices;. My Library page to upload personal resources and bookmark resources that others have

posted; and. Members page, which lists all of the APTC members.

Also notable is a search box, which is featured prominently at the top right on every APTC

web page. Although the APTC is most directly accessed by logging into the web portal

[https://apcommunity.collegeboard.org/], there is also an option to receive daily and

weekly email digests of all discussions, particular topics, and threads that a user wants

to follow, as well as updates on when members post new resources and comments.

As of October 2015, the Biology, Chemistry, and Physics APTCs had 11,585, 8,867, and

6,675 unique users respectively, with only approximately 7.5%, 6.4%, and 5.4% of regis-

tered users actively logging in weekly to the Biology, Chemistry, and Physics communities

(J. Clewley, College Board, personal communication, October 15, 2015). When disaggre-

gated, those users that are actively logging in are comprised of approximately 75% of

returning users (those who have logged in at least once before) and 25% of new users

(those who are logging in for the first time) in the Biology, Chemistry, and Physics APTCs.

Literature review

Professional development in the United States

Although there are many possible goals for teacher professional development, our research

begins with the premise that the ultimate goal of teacher professional development is to

increase student achievement, with interim objectives that involve shifts in instructional

practices (Fishman et al., 2003; Mundry, Spector, Stiles, & Loucks-Horsley, 1999).

Darling-Hammond et al. (2017) found that professional development correlated with

student gains when it was ongoing, connected to practice, focused on specific academic

content, linked to school initiatives, and collaborative. Similarly, Borko et al. (2010) con-

sidered the content, processes, and structure of high-quality professional development,

and concluded that the best professional development offerings were both situated in tea-

chers’ practice and focused on students’ learning. Borko et al. (2010) found that effective

professional development was ongoing and sustainable, integrated with other aspects of

change within the school, and focused on modelling preferred instructional strategies

by building communities of practice [defined below].

Unfortunately, most teachers across the globe do not have access to professional devel-

opment at this level of quality. Cohen and Moffit (2009) argued that one reason


professional development does not work in the United States is due to a strong commit-

ment to localism, which precludes national common curricula or exams and a shared

instructional context for teachers to collaborate. Mehta (2013) further emphasised that

this commitment to localism makes it difficult for centralised authorities to consistently

support and produce high-quality schooling and, therefore, results in wide variation in

performance levels.

This study examines a kind of national ‘subsystem’ – of AP Biology, Chemistry, and

Physics teachers – within an otherwise decentralised American school system – which

increases relevance for other countries with nationalised curricula. All AP teachers

across the United States must submit their course syllabi for accreditation review by the

College Board to ensure course guidelines and resources are in place, and must prepare

their students to take the same national exam. Therefore, the community of AP teachers

provides an unusual example of professional development that transcends local school

systems. This study hopes to extend theory by investigating a professional development

community of teachers with shared curricula and assessments. The online AP learning

communities parallel similar peer-based professional learning communities across many

countries (OECD, 2012).

Communities of practice and technology

The term ‘community of practice’ is usually attributed to Lave and Wenger (1991), who

described knowledge as a ‘property’ enacted by groups of people over time in shared prac-

tices, rather than in the minds of individual learners. Over time, the term ‘community of

practice’ came to be defined as a ‘group of people who interact, learn together, build

relationships, and in the process, develop a sense of belonging and mutual commitment’

(Wenger, McDermott, & Snyder, 2002, p. 34). In order to build and sustain a community

of practice, Wenger et al. (2002) advocated for seven principles: (1) design for evolution,

(2) open a dialogue between inside and outside perspectives, (3) invite different levels of

participation, (4) develop both public and private community spaces, (5) focus on value,

(6) combine familiarity and excitement, and (7) create a rhythm for the community (p.51).

These scholars emphasise that the benefit of group learning is not limited to periods of

success; collective reflection is preferable to individual reflection because exposure to

the mistakes of others catalyzes creative solutions to complex problems (as cited in

Noonan, 2014). Practically, DuFour, Dufour, Eaker, and Many (2006) concluded that suc-

cessful communities of practice are characterised by a shared mission (and values and

goals), collaborative focus on learning, collective inquiry, action orientation and exper-

imentation, commitment to continuous improvement, and a results orientation.

Wenger’s more recent work with White and Smith (2010) emphasised the role of tech-

nology in a community of practice; technology can be used to either support the commu-

nity (i.e. communication among members) or the practice itself. Typically, scholars have

investigated the role of technology in supporting communication rather than practice

(Hoadley, 2012). Hoadley and Kilner (2005) identified three ways – content, process,

and context – that technology can support practice. Technology allows for the ability to

store, manipulate, and transmit information [content] in a variety of formats; for

example, in the online APTC, teachers share lesson plans, resources, and instructional

materials that they use with their students. Technology allows for scaffolding [process]


of a particular task, activity, or sequence; for example, video links posted within the APTC

allow teachers to reflect on and share their implementation of inquiry labs. In addition to

content and process, technology enables the user to shift social context, by allowing for

interactions across distance, location, and time (Hoadley, 2012); for example, by partici-

pating in the online APTC, teachers across the country engage in discussion around

their shared curricula and assessments.

Online learning communities

Online learning communities are groups of people who gather in an online space to ‘learn,

interact, and build relationships, and through this process develop a sense of belonging

and mutual commitment’ (Wenger et al., 2002 in Booth, 2012). Barab, MaKinster, and

Scheckler (2003) further defined an online community as ‘a persistent, sustained social

network of individuals who share and develop an overlapping knowledge base, set of

beliefs, values, history, and experiences focused on a common practice or mutual enter-

prise’ (p. 238). Specifically, online teacher communities potentially allow educators to

learn while they are actively applying new ideas in their own work settings, to provide

and receive sustained coaching and feedback, and to cultivate a reflective, collaborative,

professional community (Dede, 2006). Many online learning communities can be

described as communities of practice (Wenger et al., 2002). Based on the supposition

that online communities can provide a form of ongoing professional learning for teachers,

this study examines how online communities complement and extend other forms of pro-

fessional development offered by the College Board.

While the field has made good progress in understanding the design elements and

structure of high-quality professional development, there have been few opportunities

to examine the relative effectiveness of alternative professional development offerings,

such as online learning communities related to a specific curriculum (Fishman et al.,

2013) and in particular, to explore online learning communities in relation to teacher

and student learning, particularly during a large-scale change in curricula and tests.

This study strives to extend theory by examining an online community whose teacher par-

ticipation has been associated with increases in student scores (Fishman et al., 2014).

Dede and Eisenkraft (2016) differentiate between professional development and teacher

learning, the key distinctions of which are agency and formality. Professional development

may be mandated for teachers and is generally a formal experience with a fixed duration,

curriculum, instructional strategy, and expected outcomes. Teacher learning, by contrast,

is typically begun by an educator as a voluntary activity and may be informal, with dur-

ation, content, form of learning, and eventual impact uncertain at its inception. Models of

professional development fall along a continuum, with mandated, formal professional

development at one end and voluntary, informal teacher learning at the other, with

many approaches somewhere in between (Borko, 2004; Borko et al., 2010); the richest

forms of professional learning lie toward the middle of the continuum, combining theor-

etical and research-based insights with the wisdom of practice. These can foster teacher

ownership by providing some autonomy while accomplishing lasting impacts via expert

modelling of innovative practices and teacher-led discussions about overcoming chal-

lenges in implementation. Exemplary online learning communities strive to achieve this

balance.


Methodology

This study employed a mixed-method approach, using survey and case study data to

investigate how APTC usage supports teachers in navigating the AP redesign and in

helping their students succeed on the revised AP tests. The primary quantitative data

sources were a web-based survey emailed to all AP Biology and Chemistry teachers in

late May 2014 and to all AP Biology, Chemistry, and Physics teachers in May 2015

(Table 1), as well as, students’ corresponding AP scores.

The survey collected data on demographics; professional development participation;

general attitudes toward professional development; characteristics of each teacher’s AP

science course, instruction, and school context; and levels of concern regarding the rede-

sign. Specific survey questions related to the online APTC inquired about the frequency of

site visits, duration of average visit, degree to which the community is responsive to

teacher needs, degree to which student work was a focus, degree to which teaching was

modelled, degree to which there were opportunities to build relationships with colleagues,

degree to which there was effective support for teaching the redesigned course, the reasons

for participation in the online community, the types of activities teachers do while in the

community, and an open-response item on how teachers would improve the online

community.

Data preparation strategies include the computation of composite variables using stan-

dardised Bartlett factor scores (DiStefano, Zhu, & Mindrila, 2009) based on conceptual

consideration, exploratory and confirmatory factor analysis. These composite variables

describe teachers’ knowledge and experience, self-efficacy, professional development incli-

nation, challenges with the AP redesign, enactment of AP practices, enactment of AP cur-

riculum, administrative support, and AP workload (Table 2).

Qualitative data was compiled through a multiple case study approach (Stake, 1995,

2000; Yin, 2014) to understand the experiences and perspectives, as they prepared for

and implemented the newly redesigned AP courses, of 34 teachers who volunteered

within the aforementioned survey and were selected based on diversity of teaching experi-

ence and geographic location (Table 3). Teachers were also asked to submit examples of

student assessments, student lab work, and course syllabi for their AP courses. These arte-

facts formed the basis of individual discussion with each teacher, and provided additional

evidence of their classroom practices. Over a two-year period (January 2014, June 2014,

January 2015, and June 2015), a series of four semi-structured, 60-minute telephone inter-

views were conducted individually with 24 AP Biology and 10 AP Chemistry teachers. All

interviews were digitally recorded and transcribed. During these individual interviews,

case study teachers were asked about their use (or lack thereof) of the APTC, how often

they visit the APTC (frequency), how long they spend in the APTC (duration), why

they visit, what they do when they visit, the role of the APTC in their adapting to the

Table 1. Descriptive statistics for the dependent variables.

Survey N [all] N [non-user] N [APTC user] N [lurker] N [poster]

Biology Year 2 (2014) 2,231 964 1,267 715 516Biology Year 3 (2015) 2,240 942 1,298 737 561Chemistry Year 1 (2014) 2,271 1,092 1,179 747 432Chemistry Year 2 (2015) 2,038 969 1,069 620 449Physics Year 1 (2015) 1,733 1,003 730 410 320


redesign, their participation in ‘lurking’ or posting within the community, and ‘pros’ and

‘cons’ of the community.

Research Question 1a reports descriptive data and proportional odds logistic regression

models on the full data set to predict teachers APTC usage (Harrell, 2015). Research

Table 2. Single-indicator variables included in computations of composite variables.

Included variables

Teachers’ knowledge andexperience

(a) years teaching high school scienceC, (b) years teaching AP scienceC, (c) number ofscience teaching-related professional organizationsC, (d) number of conferenceattendances within past three yearsC, (e) years serving as AP Reader, (f) years serving asAP ConsultantC, (g) time of assignment to teach AP scienceO

Self-efficacy (a) Students perform better because of my extra effort×, (b) student scores improvebecause of my teaching×, (c) my teaching can overcome student backgrounds×, (d)extra effort in teaching produces little change×

Professional developmentinclination

(a) importance of professional development to instructional performance×, (b)importance of professional development to student performance×, (c) effectiveness ofself-teaching compared to formal professional development participation×, (d) efficacyof professional development participation×, (e) enjoyment of participation in face-to-face professional development s×

Challenges with the APredesign

Challenges with (a) science content×, (b) organisation of science content×, (c) labs×, (d)inquiry labs×, (e) format of questions/problems/exam×, (f) application of sciencepractices×, (g) developing new syllabi×, (h) understanding the ‘exclusion statements×,’(i) designing new student assessments×, (j) using the textbook×, (k) working with new/different textbooks×, (l) pacing of course×, (m) moving students to conceptualunderstandings of science×

Enactment: AP practices (a) Students work on laboratory investigations×, (b) provide guidance on integratedcontent questions×, (c) provide guidance on open/free response questions×, (d)students report laboratory findings to another×, (e) students perform inquiry laboratoryinvestigations×

Enactment: AP curriculum (a) Refer to the ‘Big Ideas’×, (b) use science practices outside of the classroom×, (c) referhow enduring understandings relate to the ‘Big Ideas×,’ (d) refer to learning objectivesfrom AP curriculum×, (e) refer to the curriculum framework×

Administrative support (a) Principal understands challenges for AP science students×, (b) principal understandschallenges for AP science teachers×, (c) principal supports professional development×,(d) lighter teaching load for AP science teachers×, (e) fewer out-of-class responsibilitiesfor AP science teachers×, (f) AP science is given additional funding×, (g) availability ofequipment to perform labs◊, (h) availability of expendable (consumable) supplies toperform labs◊

AP workload (a) Number of students across all AP science sectionsC, (b) number of AP science sectionsC,(c) weekly number of prepsC

Note: ×: 5-point Likert scale item, ◊: 4-point Likert scale item, O: Ordinal variable, C: Continuous variable.

Table 3. Case Study Participants.

Biology(n = 24)

Chemistry(n = 10)

Total(n = 34)

GenderFemale 20 7 27Male 4 3 7

Years AP Science experience0–2 5 3 83–5 5 2 76–9 8 0 810–15 1 4 516+ 5 1 6

Geographic RepresentationMidwest 6 3 9Northeast 5 3 8Pacific 4 2 6Rocky Mountains 1 0 1Southeast 6 2 8Southwest 2 0 2


Question 1b is a subgroup analysis (APTC participants) using proportional odds logistic

regression analysis to predict teacher lurking and posting behaviour (Harrell, 2015).

Research Question 2a uses doubly robust entropy weighted regression (Hainmueller,

2012) to understand treatment effects from a quasi-causal analytic perspective. This

assumes that conditional on the included covariates, students’ AP scores are independent

of the potential treatment assignment. Entropy weights and balance statistics are generated

to reasonably balance teachers on treatment and control conditions. These weights are

applied in weighted multilevel models to predict students’ AP scores. Research Question

2b utilises hierarchical linear modelling (HLM) with robust standard errors to analyze

associations of teachers’ frequency and duration towards students’ AP scores on the sub-

sample of APTC participants controlling for student, teacher, teaching, and school charac-

teristics (Raudenbusch & Bryk, 2002). Research Question 3 applies qualitative

methodologies using Atlas.ti software, in an iterative process whereby a sample of tran-

scripts were read and initial codes were generated using an emic approach; these codes

were applied, reviewed, and revised, yielding emerging themes, which were tested and

refined through an etic approach.

Limitations

There are several limitations in this work. Most notably, the College Board does not track

demographic data for their overall teacher population, which limits the understanding of

how representative this study’s sample is to the overall AP Biology, Chemistry, and Physics

populations. In addition, the College Board does not record individual user behaviour (log

files) on their APTC platform; it is therefore unknown how often, for what duration, and

which elements of the site individual teachers are accessing. Therefore, APTC teacher use

(e.g. behaviour, frequency, and duration) had to be gleaned from survey and case study

data based solely on teacher self-report. Self-report data is inherently characterised by

selection bias, which might reduce the overall generalizability of the implications. It is

important to note that the AP redesign was released as a staggered rollout; for example,

2014 data captures implementation for AP Chemistry year one and AP Biology year

two and 2015 data captures implementation for AP Physics year one, AP Chemistry

year two, and AP Biology year three. A final limitation is that the correlation of teachers’

APTC participation with increases in student AP scores may be due to the disposition of

the teachers who frequent the online community, rather than the effectiveness of the site

itself. The data available do not permit resolving these uncertainties.

Findings

Research question 1: what are teachers’ APTC usage patterns?

About half of the survey respondents used the APTC. Biology (57%) teachers had a higher

percentage of using the APTC more often than chemistry (53%) and physics (42%) tea-

chers. Overall, women (2014, 2015: 58%) had a higher percentage of using the APTC

than men (2014: 48%, 2015: 44%) and Whites (2014: 55%, 2015: 52%) had a higher per-

centage of using the APTC when compared to African-Americans (2014: 45%, 2015: 43%)

(Table 4). In 2014 and 2015, 2,561 and 2,553 self-identified APTC-users took the survey.


Of these, 73% of teachers who did not use the site in 2014 continued not to use the site in

2015; 86% of teachers who used the site in 2014 continued to do so in 2015; this is similar

for both biology and chemistry teachers. Also, the newer the revised test, the more fre-

quently teachers visited the site (so biology and chemistry teachers visited less frequently

in 2015 than in 2014).

The online APTC can be accessed by participants 24 hours a day, 7 days a week and

does not have a fixed duration or format. Therefore, participants in this type of pro-

fessional development do not adhere to identical forms of dosage (how often visits

occur) or intensity (how long visits last) (Figure 1). For instance, for 2014 Biology

survey respondents, the most common frequency of use was once a month, followed by

every few months, and then twice a month; once logged in, 2014 AP Biology survey

respondents, most frequently reported spending approximately 10 to 20 minutes in the

APTC, followed by 20 to 40 minutes.

Logistic regression models suggested that teachers with certain teacher, teaching, and

school characteristics were significantly more likely to participate in the APTC. Results

from the statistical models indicated that, for instance, female teachers, as well as, teachers

with greater knowledge and experience, are significantly more likely to participate in the

APTC. Also, teachers who reported a higher AP workload were significantly associated

with greater odds to participate in APTC. Furthermore, teachers in the first year of the

AP science redesign were more likely to participate in the APTC if they felt more chal-

lenged with the AP redesign or if they enacted more AP practice elements in their instruc-

tion (Table 5).

Case study participants reported similar patterns of APTC use as survey respondents,

illustrating that consistent use of the APTC is not a given. Only 18% (6 of 34) of case study

participants reported the same frequency and duration between the first and second inter-

views. Case study participants also noted that they are more likely to access the online

community on school days, rather than on weekends; the only notable exception is that

case study participants regularly mention the importance of accessing the APTC after

the College Board has scored the examinations and returned the results to students and

teachers.

The three top-rated threads, based on number of posts and number of views, are all

related to the exam. For example, even as of October 2015, in the biology community

Table 4. APTC Usage.

2014 2015

GenderFemale 58% 58%Male 48% 44%

EthnicityAfrican-American 45% 43%Asian-American 53% 52%Hispanic 49% 46%White 55% 52%Other 65% 58%Multi-racial 48% 49%

DisciplineBiology 57% 57%Chemistry 52% 53%Physics 42%


the top-rated threads based on number of posts were as follows: Exam 2013: After

Thoughts with 174 posts, AP Biology 2013 Scores: A Summary with 113 posts, and

Answers to the 2013 FRQs (free response questions) with 96 posts. The top-rated

threads based on number of views followed suit: Exam 2013: After Thoughts with 7,626

views, AP Biology 2013 Scores: A Summary with 5,879 views, and AP Bio Review Packet

with 4,602 views. For comparison, a low thread might yield zero responses and 38

views and be about starting a Biology Buddy/Mentor Program in one’s high school

with the pre-AP Biology teacher.

Logistic regression models suggested that none of the analyzed teacher, teaching, and

school characteristics consistently determined whether teachers are lurkers or posters

Figure 1. Frequency (top panel) and duration (bottom panel) of APTC use.


Table 5. Odd ratios of logistic regression analyses.

Biology year 2 Biology year 3 Chemistry year 1 Chemistry year 2 Physics year 1

M1 M2 M1 M2 M1 M2 M1 M2 M1 M2

Intercept (in log-odds) −54.209*** −45.034* −60.782*** −31.815* −41.632*** −2.189 −35.710*** −6.500 −13.571 −16.551DemographicsBirth year/100 15.644*** 11.486** 21.850*** 5.005* 8.280*** 1.246 6.109*** 1.327 1.944 2.139Female 1.732*** 0.846 1.822*** 0.921 1.327** 1.315 1.436*** 0.830 1.624*** 0.675*Racial background (vs. White)Native American or Islander 1.562 0.320 1.483 0.836 1.125 0.444 1.325 0.970 0.683 0.187Asian or Asian American 0.915 0.964 0.976 0.906 0.858 0.749 0.949 1.415 0.786 1.795Black or African American 0.582 0.979 0.573 0.855 0.627 0.541 0.690 0.828 0.452 1.595Hispanic 0.917 1.315 0.812 0.500 0.709 0.150 0.361 365.225 0.321 0.251Multiracial 0.861 1.730 0.703 1.160 0.604* 0.893 0.889 0.884 0.713 1.191

Knowledge and experienceC 1.347*** 1.264** 1.272*** 1.270** 1.247*** 1.086 1.258*** 1.129 1.242*** 1.155Teaching and school characteristicsSelf-efficacyC 1.040 1.112 0.960 1.024 1.017 0.935 1.096* 0.939 1.060 1.053Professional development inclinationC 1.212*** 0.892 1.121** 1.046 1.025 0.888 1.068 0.977 1.021 0.936AP redesign challengesC 1.067 1.049 1.053 0.947 1.129** 0.949 1.053 0.965 1.251*** 1.093Enactment: AP practicesC 1.059 0.925 1.166*** 1.052 1.142** 0.992 1.073 0.991 1.248*** 1.007Enactment: AP curriculumC 1.052 1.054 1.047 1.049 1.011 0.968 0.964 0.926 1.079 0.894AP workloadC 1.172*** 1.121 1.114** 1.020 1.142** 0.932 1.228*** 1.156* 1.123* 0.979Administrative supportC 1.021 0.948 1.040 1.022 0.993 1.008 0.994 0.989 1.089 0.965APTC usageFrequency (vs. once/month or less)Every other week 0.733 1.229 0.831 1.495* 1.207Once/several times a week 1.515* 1.130 0.860 1.370 1.394Almost every day 1.562 1.302 1.073 1.123 1.051

Duration (vs. <5 minutes)5–10 minutes 0.568 0.465* 1.178 1.199 1.93610–20 minutes 0.417 0.489* 1.227 1.497 1.98220–40 minutes 0.326 0.456* 1.335 1.469 1.887>40 minutes 0.799 0.286** 0.829 1.329 1.320

Professional development characteristicsResponsive agenda 1.111 1.023 0.909 1.042 0.976Focus on student work 1.085 0.876* 0.941 0.940 1.131Modelling teaching 1.035 0.896 1.000 0.998 0.945Building relationships 0.924 1.081 0.998 0.967 1.169Effective support 1.088 0.954 1.238 1.006 0.946

(Continued )

INTERNATIONALJO

URNALOFSCIENCEEDUCATION

409

Table 5. Continued.

Biology year 2 Biology year 3 Chemistry year 1 Chemistry year 2 Physics year 1

M1 M2 M1 M2 M1 M2 M1 M2 M1 M2

Participation reasonsAccess resources 0.357*** 1.057 0.337*** 0.964 0.909Ask questions 8.210*** 4.367*** 6.642*** 5.686*** 6.420***Advice for AP redesign 0.058*** 1.288 0.069*** 0.635 0.715Share ideas/insights 2.694** 6.727*** 7.128*** 6.602*** 24.999***Social interactions 1.400 1.233 0.854 1.619 0.883

Note: M1: Non-users vs. APTC users; M2: Lurkers vs. posters; C: Composite variable; *p < 0.05, **p < 0.01, ***p < 0.001.

410

K.FRUMIN

ETAL.

across all years and disciplines. In addition, APTC usage characteristics such as the fre-

quency, duration, and teachers’ rating of the Desimone-inspired (2009) characteristics

of high-quality professional development did not consistently predict whether teachers

are lurkers or posters across all years and disciplines. However, teachers’ self-reported

reasons for APTC participation strongly predicted whether teachers are lurkers or

posters. For example, teachers who participated to ask questions or to share their own

ideas and insights were significantly more likely to be posters instead of lurkers (Table 5).

According to APTC users, most engaged with the APTC in passive ways. Across dis-

ciplines and years, APTC users reported spending an average of approximately 8% of

their time on the site engaged in active participation (creating new threads, commenting,

or posting resources), and the remaining average 92% of their time in passive pursuits,

such as finding resources and reading. Only around 30% of users spent more than 5%

of their time in active participation, while close to 85% spent more than 80% of their

time in passive interaction. Men spent close to 10% of their time engaged in these

active forms of participation, while women spent closer to 7.5%.

Although these APTC users may not have initially planned to lurk, they attributed their

lurking, or lack of posting, to the desired content already being posted. Therefore, APTC

users searched for specific items and lurked, rather than posted. Case study participants

described the phenomenon in which plentiful information leads to lurking, rather than

posting:

I didn’t really have to ask [post] a question because somebody else had already asked it.Or, they’d ask a question I hadn’t even thought yet to ask so it was really beneficial. (CaseStudy Participant #17)

When conducting specific searches, case study APTC users most frequently mentioned

that they are seeking lesson support (e.g. resources, answers to questions, advice) and com-

munity support (e.g. emotional support, sharing challenges and setbacks). Searching for

specific topics and lurking, the most common uses cited by APTC case study teachers dis-

tinguish the APTC from other examined forms of professional development by tailoring

the experience to the user. The ability to find plentiful existing information often leads to

lurking, rather than posting.

Research question 2: what is the association of teacher APTC participation

and student performance?

The results for the balance of covariates used to generate entropy weights showed that,

across all disciplines and years, the sample of APTC and non-APTC users were

matched. Controlling for student, teacher, teaching, professional development, and

school characteristics, the multi-level regression models supported the assessment that

teachers’ APTC participation is predictive of students’ AP science scores across all disci-

plines and years, with the exclusion of the second year of the AP Chemistry redesign

(Table 6). Teachers who participated in the APTC teach students who scored significantly

higher on the AP science examinations; Biology 2015: 0.20, p < .001; Biology 2014: 0.08, p

< .05, Chemistry 2015: 0.07, n.s.; Chemistry 2014: 0.09, p < .01; Physics 2015: 0.16, p < .001.

HLM analyses indicated the absence of consistent associations of the frequency and

duration of APTC participation on students’ AP scores for teachers participating in the


Table 6. Results for APTC effect for multilevel models with entropy weightsapplied.

Discipline APTC Effect SE

Biology Year 3 0.20*** 0.05Biology Year 2 0.08* 0.03Chemistry Year 2 0.07 0.04Chemistry Year 1 0.09** 0.03Physics Year 1 0.16*** 0.04

Note: Variables used to generate the entropy weights are years of the AP redesign, yearsAP teaching experience, years science teaching experience, hours of AP instruction,knowledge and experience composite, duration of time in conventional professionaldevelopment activities, professional development inclination composite, importanceof professional development for students’ performance, and school percentage of stu-dents eligible for enrolment in free- or reduced-priced lunch programmes; covariatesused in the models included students’ PSAT scores, school percentage of students eli-gible for enrolment in free- or reduced-priced lunch programmes, administrativesupport composite, enactment of AP practices composite, enactment of AP curriculumcomposite, self-efficacy composite, knowledge and experience composite, professionaldevelopment inclination composite, importance of professional development for stu-dents’ performance, years AP teaching experience, years science teaching experience,years of AP redesign experience, hours of AP course instruction, number of labs,number of conventional professional development, number of supplemnetary pro-fessional development, average professional development effective support rating,average professional development active learning rating, average professional develop-ment student work rating, average professional development responsive agenda rating,average professional development modelling teaching rating, average professionaldevelopment relationship building rating, and of duration of conventional professionaldevelopments; *p < 0.05, **p < 0.01, ***p < 0.001.

Table 7. HLM examining the influences of frequency and duration of APTC participation on studentscores.

Bio Year 2 Bio Year 3 Chem Year 1 Chem Year 2 Phy Year 1

b SE b SE b SE b SE b SE

Frequency of APTC participation (vs. Only once/every few months)About once per month −0.002 0.035 0.056 0.036 −0.059 0.054 −0.065 0.053 0.063 0.066Every other week 0.014 0.040 0.050 0.041 −0.031 0.057 0.046 0.062 0.001 0.081Once a week 0.050 0.041 0.012 0.043 0.012 0.061 −0.045 0.070 −0.009 0.082Several times per week −0.003 0.051 0.056 0.053 0.000 0.080 0.138 0.077 0.044 0.093Almost every day 0.082 0.069 −0.008 0.075 0.095 0.087 0.055 0.133 0.133 0.153

Duration of APTC participation per session (vs. less than 5 minutes)5–10 minutes 0.042 0.077 −0.112 0.063 0.259** 0.076 0.047 0.101 0.053 0.09910–20 minutes 0.052 0.075 −0.187** 0.058 0.169* 0.072 0.017 0.099 0.103 0.09820–40 minutes 0.081 0.075 −0.160** 0.060 0.101 0.080 −0.012 0.102 0.071 0.102More than 40 minutes 0.026 0.089 −0.124 0.068 0.075 0.099 −0.045 0.125 0.080 0.132

N (level 1) 15,219 13,057 14,976 12,189 8,627N (level 2) 727 715 761 632 348Variance (level 2) 0.0770 0.0644 0.1788 0.1596 0.1166χ2 5.59 11.56 22.29 13.40 3.87df 9 9 9 9 9p 0.78 (n.s.) 0.24 (n.s.) 0.008 0.15 (n.s.) 0.92 (n.s.)

Note: Log likelihood measures and likelihood ratio tests are based on models observed information matrix standard errors;χ2 tests based on the addition of frequency and duration of APTC participation variables; table only reports coefficients ofthe independent variables, covariates on the student-level included students’ PSAT scores, mothers’ educational attain-ment, covariates on the teacher/school level included years of the AP redesign, years AP teaching experience, hours of APinstruction, duration of time in conventional professional development activities, importance of professional develop-ment for students’ performance, and school percentage of students eligible for enrolment in free- or reduced-pricedlunch programmes, district-level school funding, administrative support composite, enactment of AP practices composite,enactment of AP curriculum composite, interaction of both enactment composites, self-efficacy composite, number ofstudents, number of sections, number of preps, number of labs, number of conventional professional developments,number of supplementary professional developments, professional development effective support rating, professionaldevelopment active learning rating, professional development student work rating, professional development responsiveagenda rating, professional development modelling teaching rating, and professional development relationship buildingrating; *p < 0.05, **p < 0.01, ***p < 0.001.


APTC across all years and disciplines (Table 7). Including APTC duration and frequency

variables in the models did not significantly explain more variance on the school/teacher

level across all years and disciplines, except for the first year of the AP Chemistry redesign.

In the first year of the AP Chemistry redesign, teachers who used the APTC on average for

about 5–10 minutes for each session, as well as 10–20 minutes, taught students who scored

better on the AP examination compared to teachers who used the APTC on average for

less than 5 minutes per session. Overall, however, this analysis indicated that correlation

of the APTC participation on students’ AP scores is mostly independent from teachers’

frequency and duration of APTC participation.

Research question 3: in what ways, if any, does the APTC, whose content

and processes are driven by bottom-up peer interactions, complement

and extend top-down forms of the professional development offered by

the College Board?

In an effort to support teachers in shifting to the redesigned AP Biology, Chemistry, and

Physics curricula and tests, the College Board offers a range of professional development

options. Among the mix of face-to-face weeklong workshops, one-day workshops, and

online courses, the online APTC is the only College Board offering that is co-constructed

by teachers; the other professional development options are top-down professional devel-

opment sessions delivered by the College Board. Although each APTC has a unique mod-

erator, a current or former AP teacher who is paid modestly by the College Board, the

online community functions and is sustained by the original interactions, resources,

and momentum of AP teachers. In addition, compared to the other professional develop-

ment offerings, the APTC is the only option that is continuous throughout the calendar

year and is accessible 24 hours a day, 7 days a week without a fixed duration or frequency.

The others are finite professional development sessions with a one-day, four-day, five-day,

or a multi-week span. In addition, most multi-day, face-to-face professional development

sessions are held over the summer; whereas APTC participation can happen within and

proximate to the school day and school year.

In addition to being ‘bottom-up’ and ongoing, the APTC has the following two primary

attributes, based on survey and case study data, that complement and extend top-down

forms of professional development: 1) personalisation of content and 2) a shared, affective

community.

Personalisation of content. All survey participants were asked to rate each identified

professional development experience that they engaged in on its responsiveness to partici-

pant needs, focus on student work, modelling of teaching practices, opportunities to build

relationships with colleagues, and effectiveness in supporting needs with respect to the

revised AP course. Of all sampled professional development offerings in 2014 and 2015,

the APTC is among the top three highest rated professional developments on average

for ‘responsiveness to your needs as a participant,’ which was defined by the survey as

‘for example, was the agenda flexible or customisable to accommodate your (and

others’) varying interests and needs? Or was the agenda fixed and followed rigidly?’

The responsiveness of the APTC, as rated by the survey participants, is not surprising

given the inherent set-up of the APTC, in which users can search for and find specific dis-

cussions and resources of interest, as well as initiate these discussions. A case study


participant articulated the difference between the APTC and other professional develop-

ment offerings with respect to personalisation of content by stating:

[In the APTC], you get that discussion going back and forth as far as, have you thought aboutthis or what about this or how do you handle it with this kind of special needs student andthings like that. Whereas, if you just had a book or at times, if you had a teacher up in front ofa classroom giving a presentation, there isn’t time to get into those details. It’s not that theywouldn’t want to. It’s just the time isn’t there. But with the AP Community, as long as you’rewilling to put the time in, that discussion can happen. (Case Study Participant #23)

In addition to responsiveness, survey participants ranked ‘to access resources’ as the

second most popular reason for visiting the APTC. Resources within the APTC can be

accessed at any time of day and from any device (computer, tablet, smartphone) with

Internet capability. A case study teacher noted that the sheer number of participants

ensures a variety of rich resources, which are perceived to be more relevant than those

that can be found outside of the community:

The vast number of the people that are a part of a group means there are a vast number ofresources. So, if you need an activity for a certain concept because the kids aren’t getting it,Googling it will get you some things, going on the resource board will get you more and morerelevant things that you can use. (Case Study Participant #25)

As case study participants noted, sharing resources was particularly useful in the early

implementation of the redesign when few exam questions had been formally released.

Yeah, I mean, it [the APTC] was honestly the best thing I had last year, you know, see whatother people are doing, and get advice. When it came time for the test, people were sharingreview materials and, you know, questions… That sort of stuff was invaluable to my kidsbecause I would then share with them… especially at the end of the year; I pretty muchlived on that thing [APTC]. (Case Study Participant #8)

The APTC provides personalised access to content in a way that most forms of top-

down professional development cannot.

A shared, affective community. Connecting with off-site colleagues is important given

that most AP Biology, Chemistry, and Physics teachers are likely the only teacher of that

particular curriculum in the school. As described by a case study participant:

I am really an island. I am the only AP Biology teacher at my school. There is no connectionbetween AP Biology teachers in the county really. We get together for a 20-minute sessiononce a year and brainstorm, but there is no sharing. (Case Study Participant #21)

The APTCmay serve to foster collegial relationships across distance, which reduces iso-

lation and provides encouragement to teachers. Moreover, the APTC may support an

affective community for teachers, as one case study teacher explained:

It is very helpful to hear when other people have had the same problems or obstacles as you.Like a lab that does not work out, what are some possible reasons? How do people overcomethe problem? So, you do not feel so alone. (Case Study Participant #21)

This peer-to-peer component of the community seems to be something that dis-

tinguishes the APTC from the other top-down forms of professional development,

which usually do not allow time for the sharing of emotions and personal experiences.


The following case study participant confirmed the importance of the affective component

and acknowledges that emotional sharing reduces isolation and provides encouragement:

I think in the year leading up to the new exam… it [APTC] was vital. Just to hear otherpeople’s frustration, knowledge, and understanding… to know that okay, other people arehaving the same challenges or questions that I have… It kind of reduced the sense of iso-lation. (Case Study Participant #6)

In complement to more traditional, top-down forms of professional development, the

APTC allows for teachers to search peer-shared resources and discussions tailored to their

needs, allowing for personalisation of content responsive to interests and curiosities. In

addition to resources, the peer-to-peer aspect of the APTC creates a space for sharing

affective components of teaching that may not be featured or addressed in top-down

forms of professional development, which are usually more focused on imparting

content knowledge. Case study participants who use the APTC report that their isolation

is reduced and they are encouraged through these peer exchanges.

Making the AP teacher community more effective in serving participants

Although participation in the APTC is correlated with positive student outcomes, one of

the survey questions queried respondents in an open-ended format – ‘What, if anything,

would improve your experience with the APTC?’ Interestingly, the majority of survey

respondents commented on moderation and on user interface, which specifically com-

prised search functionality and organisation of information. The following paragraphs

report on these findings with respect to moderation and user interface.

Survey respondents note that the online community does not always feel safe given the

employed moderation techniques (or lack thereof) and/or the domination (or bullying) by

a few strong voices.

I think that [the moderator] is incredibly responsive and helpful. I do sometimes cringe atharsh replies to folks (newbies) who ask questions that are searchable. I wonder if thatturns some off from participating or asking. I also know it must be frustrating from [mod-erator’s] standpoint.

Some individuals, including the moderator, have too strong of a voice. The moderator shouldmoderate, not be the ‘know it all’ of the group. But instead, 2,000 posts later, he is often theend of a thread, ceasing discussion. Given that, he is also an advocate for those that believe inthe new course, which is refreshing and much appreciated. But too often, it is a site for bully-ing or being bullied.

Similarly, some case study teachers note that the APTC feels like an echo chamber of

the same voices or an insider’s club; these teachers describe, ‘you’ll notice that there is a

small core of people really talking’ (Case Study Participant #11) and ‘the forum is generally

dominated by lots of strong voices’ (Case Study Participant #25). Far fewer survey respon-

dents describe the moderation as successful, with comments such as: ‘The moderator does

an excellent job of keeping the group professional, respectful, and focused on student

learning. Thanks!’

With regard to user interface, survey respondents ask for better search functionality and

better organisation of information. Survey respondents repeatedly note that the search


engine functionality is less than ideal for locating desired resources and that it is challen-

ging to find one’s way back to a particular resource:

When you are looking for a specific topic, you often get all sorts of hits from all differentareas. For example, if you put in ‘photosynthesis’ and search for an activity, you might geta couple of photosynthesis, some genetics, some ecology… it’s annoying.

It’s so vast. Researching resources in the search option can be arduous. A simple search canyield numerous non-related threads and resources. I wish there was a way to streamlineeverything.

My experience with the search bar for the APTC, I was not able to find the resources I waslooking for by typing in a keyword search. Because of the way replies to posts are organized, itwas hard to find the starting post that many times contained the resource I was looking for. Aredesign of either how the threads are organized or an update of the search function to be ableto find posts with resources attached in a more efficient way would be extremely helpful.

According to the survey, there are also specific calls for ‘live online chat with other users

who are online,’ ‘shared test question banks,’ and opportunities to ‘interact more with tea-

chers on successful teaching strategies’.

Delayed response time was not directly mentioned by any of the survey respondents, but

it is something that we have observed within the community. Research points to the impor-

tance of receiving a timely response, especially for new users (Frumin & Dede, 2016).

Discussion

This report set out to explore the role of the online APTC in supporting teacher learning in

the face of the large-scale redesign of AP science courses, and arrived at three main find-

ings: (1) teachers who use the APTC teach students who subsequently score higher on the

AP exam, (2) the APTC can provide a personalised, peer-generated, affective environment

that complements and extends top-down forms of professional development, and

(3) based on teacher self-reported needs, the APTC might be further improved with

changes to moderation and user interface.

Teachers who reported participation in the APTC had students who scored higher on

the AP science examinations, compared to teachers who did not indicate using the APTC

for their professional development. This could be interpreted as APTC participation

having a positive effect on teacher learning and thus student outcomes. However, since

the design of our study does not allow for causal interpretations, this finding could indicate

that unobserved characteristics, which were not in the model, could account for this

correlation.

With regard to the second finding, while professional development is inherently

focused on teacher learning, professional development in the face of a redesigned curricu-

lum or assessment (or both) also requires ‘unlearning’ of prior practices, beliefs, and values

about the nature of teaching, learning, and/or schooling. In other words, teachers must

‘unlearn’ lecturing that provides broad, shallow content coverage (which is what the

older versions of the AP tests required), and revise their instructional practice so that stu-

dents have opportunities to actively practice inquiry and learn big ideas in that scientific

discipline. According to Dede (2005), unlearning requires higher levels of emotional and

social support than traditional forms of professional development that are focused on


intellectual and technical dimensions. Dede (2005) also notes that unlearning, ideally,

should take place in ‘distributed learning communities, so that the learning process is con-

sistent with the knowledge and culture to be acquired’ (p.15). Therefore, the APTC might

complement and extend top-down, multiday, face-to-face professional development

workshops by providing a continuous context, to be accessed when needed, where teachers

can collectively unlearn skills and support each other emotionally in re-learning new prac-

tices in their respective environments. Survey and case study participants confirmed this

when describing how participation in the APTC reduces isolation, provides emotional

support, and encourages new types of teaching practices.

In addition, the challenge with the top-down, face-to-face workshops is that teachers

generally are able to attend only when they are farthest removed from their classroom con-

texts during vacation weeks or months. The APTC provides a form of unlearning and

reflection that is more proximate to teacher practice during the school year. As a result

of this proximity to teacher practice, the APTC may complement and extend top-down

forms of professional development by being more responsive and personalised to

teacher needs in the moment during the school year, as evidenced by survey and case

study participant descriptions of the search function to find activities and resources for

planning and test-preparation.

The third finding from this study focuses on improving moderation and user interface

within the APTC. A review of the scholarly literature highlights the importance of mod-

erators in the effective functioning of online learning communities, in part because of the

critical role that moderators play as facilitators of community discussions and champions

of member engagement (Frumin & Dede, 2016). For example, Booth (2012, p. 24) states,

‘Without the social artistry and guidance of a single leader or a team of leaders, online

communities will unlikely reach their full knowledge sharing potential.’ In order to

foster collaboration and knowledge sharing, moderators should employ intentional

language to prime the members of the community to the norm of reciprocity. Phrases

such as ‘pay it forward’ or ‘return the favor’ can activate members’ altruism and

promote their development of commitment to the community. To accomplish this, mod-

erators might consider highlighting what benefits certain members have received, as well

as any opportunities for them to ‘give back’ to the community that supports them.

In addition to altruistic language, moderators also need to pay attention to how the

information within an online community is organised. It is critical that members of an

online community are able to easily discover where to add their voices to the discussion

and what contributions need to be made. A well-organised community will lead to

increased contributions by lowering the barrier to participation.

In sum, the APTC survey and case study data illustrate that teachers adapting to a large-

scale change in curricula and tests want an ongoing environment in which to share feel-

ings, emerging expertise, and experiences. A well-moderated community ensures

emotional and social safety for all members, and a user-friendly interface and efficient

search function ensure effective retrieval of resources.

Conclusion

As discussed in the introduction to this paper, the findings from this study have impli-

cations for teacher capacity building in many countries across the world. By examining


a professional development community of teachers with shared curricula and assessments,

this study explored a unique ‘subsystem’ – of AP Biology, Chemistry, and Physics teachers

adapting to a large-scale redesign in 2013, 2014, and 2015 respectively. While there were

many forms of professional development offered to support AP Biology, Chemistry, and

Physics teachers in navigating this monumental shift (toward depth and scientific inquiry,

reasoning, and application), the online APTC is the only offering that is ongoing through-

out the calendar year, proximate to all stages of teacher learning, and co-constructed by

teachers – sustained by their unique participation, resources, and momentum. In addition

to being continuous and ‘bottom-up,’ the APTC provides personalization of content and a

shared, affective community. The search function within the APTC allows users to locate

specific discussions and resources of interest and therefore to customise their professional

development experience. Survey and case study participants also report that the APTC

reduces isolation and provides emotional support in navigating curricular changes and

possibly in ‘unlearning’ prior practices and beliefs. According to Dede (2005), unlearning

demands higher levels of emotional and social support than traditional forms of pro-

fessional development that usually target intellectual and technical dimensions. Further

research is needed to better understand effective design for online professional learning

communities and the value they contribute to professional development ecosystems.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by National Science Foundation [grant number DRK-12 1221861].

ORCID

Kim Frumin http://orcid.org/0000-0002-4695-8578Chris Dede http://orcid.org/0000-0003-0322-2461Arthur Eisenkraft http://orcid.org/0000-0001-5928-7972Barry Jay Fishman http://orcid.org/0000-0003-2464-1999Ayana McCoy http://orcid.org/0000-0002-0573-0294

References

Achieve. (2010). International science benchmarking report. Washington, DC: Achieve. Retrievedfrom https://www.achieve.org/files/InternationalScienceBenchmarkingReport.pdf

Askell-Williams, H., & Murray-Harvey, R. (2016). Sustainable professional learning for early child-hood educators: Lessons from an Australia-wide mental health promotion initiative. Journal ofEarly Childhood Research, 14(2), 196–210.

Barab, S. A., MaKinster, J. G., & Scheckler, R. (2003). Designing system dualities: Characterizing aweb-supported professional development community. The Information Society, 19(3), 237–256.

Booth, S. E. (2012). Cultivating knowledge sharing and trust in online communities for educators.Journal of Educational Computing Research, 47(1), 1–31.

Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. EducationalResearcher, 33(8), 3–15.


Borko, H., Jacobs, J., & Koellner, K. (2010). Contemporary approaches to teacher professionaldevelopment. In P. L. Peterson, E. Baker, & B. McGaw (Eds.), International encyclopedia of edu-cation (Vol. 7, pp. 548–556). Oxford: Elsevier.

Cohen, D., & Moffit, S. (2009). The ordeal of equality: Did federal regulation fix the schools?Cambridge, MA: Harvard University Press.

College Board. (2015). Retrieved from https://www.collegeboard.org/Darling-Hammond, L., Hyler, M. E., & Gardner, M. (2017). Effective teacher professional develop-

ment. Palo Alto, CA: Learning Policy Institute.Dede, C. (2005). Planning for ‘neomillennial’ learning styles: Implications for the investment in

technology and faculty. In D. Oblinger, & J. Oblinger (Eds.), Educating the net generation(pp. 15.1–15.22). Boulder, CO: Educause.

Dede, C. Ed. (2006). Online professional development for teachers–emerging models. Cambridge,MA: Harvard Education Press.

Dede, C., & Eisenkraft, A. (2016). Introduction: Online and blended teacher learning and pro-fessional development. In C. Dede, A. Eisenkraft, K. Frumin, & A. Hartley (Eds.), Teacher learn-ing in the digital age. Cambridge: Harvard Education Press.

Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Towardbetter conceptualizations and measures. Educational Researcher, 38(3), 181–199. https://doi.org/10.3102/0013189X08331140

DiStefano, C., Zhu, M., &Mindrila, D. (2009). Understanding and using factor scores: Considerationsfor the applied researcher. Practical Assessment, Research & Evaluation, 14(20), 1–11.

DuFour, R., Dufour, R., Eaker, R., & Many, T. (2006). Learning by doing: A handbook for pro-fessional learning communities at work. Bloomington, IL: Solution Tree.

Fishman , B., & Dede, C. (2016). Teaching and technology: New tools for new times. In D. Gitomer& C. Bell (Eds.), Handbook of Research on Teaching, 5th ed. Washington, DC: AmericanEducational Research Association.

Fishman, B., Fischer, C., Kook, J. F., Levy, A., Jia, Y., Eisenkraft, A. … McCoy, A. (2014).Professional development for the redesigned AP biology exam: Teacher participation patternsand student outcomes. Paper presented at the American Education Research AssociationAnnual Meeting, Philadelphia, PA.

Fishman, B., Konstantopoulos, S., Kubitskey, B., Vath, R., Park, G., Johnson, H.J. … Edelson, D.(2013). Comparing the impact of online and face-to-face professional development in thecontext of curriculum implementation. Journal of Teacher Education, 64(5), 426-438.doi:10.1177/0022487113494413.

Fishman, B., Marx, R. W., Best, S., & Tal, R. T. (2003). Linking teacher and student learning toimprove professional development in systemic reform. Teaching and Teacher Education, 19(6), 643-658. doi:10.1016/S0742-051X(03)00059-3.

Geiser, S., & Santelices, V. (2006). The role of Advanced Placement and honorscourses in collegeadmissions. In P. Gandara, G. Orfield, & C. L. Horn (Eds.), Expanding opportunity in higher edu-cation: Leveraging promise (pp. 75–114). Albany, NY: SUNY Press.

Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method toproduce balanced samples in observational studies. Political Analysis, 20(01), 25–46. https://doi.org/10.1093/pan/mpr025

Harlen, W. (Ed.). (2015). Working with the big ideas in science education. Trieste: Interacademies.Retrieved from https://www.ase.org.uk/documents/working-with-the-big-ideas-in-science-education/

Harrell, F. E. (2015). Regression modelling strategies: With applications to linear models, logistic andordinal regression, and survival analysis (2nd ed.). Cham: Springer.

Hoadley, C. (2012). What is a community of practice and how can we support it? In D. Jonassen &S. Land (Eds.), Theoretical foundations of learning environments (2nd ed., pp. 286–299).New York, NY: Routledge.

Hoadley, C. M., & Kilner, P. G. (2005). Using technology to transform communities of practice intoknowledge-building communities. ACM SIGGROUP Bulletin, 25(1), 31–40.

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K-12 education:Status, prospects and an agenda for research. Washington, DC: The National Academies Press.


Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge,UK: Cambridge University Press.

Liang, L. L., Liu, X., & Fulmer, G. W. (Eds.). (2017). Chinese science education in the 21st century:Policy, practice, research. New York: Springer.

Mehta, J. (2013). The allure of order: High hopes, dashed expectations, and the troubled quest toremake American schooling. New York, NY: Oxford University Press.

Mundry, S., Spector, B., Stiles, K., & Loucks-Horsley, S. (1999). Working toward a continuum ofprofessional learning experiences for teachers of science and mathematics. Madison, WI:National Institute for Science Education, University of Wisconsin-Madison.

National Research Council. (2002). Learning and understanding: Improving advanced study ofmathematics and science in U.S. high schools. Washington, DC: The National AcademiesPress. doi:10.17226/10129

National Research Council. (2011). Successful K-12 STEM education: Identifying effectiveapproaches in science, technology, engineering, and mathematics. Washington, DC: TheNational Academies Press. doi:10.17226/13158

National Research Council. (2012). A framework for K-12 science education: Practices, crosscuttingconcepts, and core ideas. Washington, DC: The National Academies Press. doi:10.17226/13165

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC:The National Academies Press.

Noonan, J. (2014). Efficiency or deliberation? An analysis and re-imagination of professional devel-opment’s promise and purpose. (Unpublished qualifying paper). Harvard Graduate School ofEducation, Cambridge, MA.

OECD. (2012). Teaching practices and pedagogical innovations: Evidence from TALIS. Paris: OECDPublishing.

Pedder, D., Opfer, V. D., McCormick, R., & Storey, A. (2010). “Schools and continuing professionaldevelopment in England – state of the nation” research study: Policy context, aims and design.Curriculum Journal, 21(4), 365–394. https://doi.org/10.1080/09585176.2010.529637

Raudenbusch, S. W., & Bryk, A. S. (2002).Hierarchical linear models: Applications and data analysismethods (2nd ed.). Thousand Oaks, CA: Sage Publications.

Roesken-Winter, B., Schüler, S., Stahnke, R., & Blömeke, S. (2015). Effective CPD on a large scale:Examining the development of multipliers. ZDM Mathematics Education, 47(1), 13–25. https://doi.org/10.1007/s11858-014-0644-5

Sadler, P. M., Sonnert, G., Tai, R. H., & Klopferstein, K. (2010). AP: A critical examination of theadvanced placement program. Cambridge, MA: Harvard Education Press.

Schleicher, A. (2016). Teaching excellence through professional learning and policy reform: Lessonsfrom around the world, international summit on the teaching profession. Paris: OECD Publishing.http://doi.org/10.1787/9789264252059-en

Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage Publications.Stake, R. (2000). Case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative

research (2nd ed., pp.435–454). Thousand Oaks, CA: Sage Publications.United Nations. (2016). United Nations sustainable development goals – education. Retrieved from

http://www.un.org/sustainabledevelopment/education/Weißenrieder, J., Roesken-Winter, B., Schueler, S., Binner, E., & Blömeke, S. (2015). Scaling CPD

through professional learning communities: Development of teachers’ self-efficacy in relation tocollaboration. ZDM Mathematics Education, 47(1), 27–38. https://doi.org/10.1007/s11858-015-0673-8

Wenger, E., McDermott, R., & Snyder, W. (2002). Cultivating communities of practice: A guide tomanaging knowledge. Boston, MA: Harvard Business Review Press.

Wenger, E., White, N., & Smith, J. D. (2010). Digital habitats: Stewarding technology for commu-nities. Portland, OR: CPSquare.

Windshitl, M., & Barton, A. C. (2016). Rigor and equity by design: Locating a set of core teachingpractices for the science education community. In Handbook of research on teaching (5th ed., pp.1009–1158). Washington, DC: American Educational Research Association.

Yin, R. (2014). Case study research: Design and methods. Los Angeles, CA: Sage Publications.


Adapting to large-scale changes in Advanced Placement

Documents

Transcript of Adapting to large-scale changes in Advanced Placement