8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
1/27
State Evaluation Policies in Gifted Education:
Examining Options for Policymakers
Kristina Ayers Paul
B.S., Eastern University, 2000
M.A., Rosemont College, 2003
A Dissertation Proposal
Submitted in Partial Fulfillment of the
Requirements for the Degree of
Doctor of Philosophy
at the
University of Connecticut
2009
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
2/27
Abstract
Program evaluation in gifted education is a highly recommended yet underutilized
practice for ensuring high quality, effective services for gifted learners. State policy plays an
influential role in the programmatic decisions and practices of gifted programs within each state,
including evaluation. It often serves as a bridge from recommendation to practice. This policy
analysis study will explore the extent to which states have policies that promote gifted program
evaluation and examine the different products that emerge from the interpretation of these
policies (e. g. monitoring systems, self-assessment tools, resource manuals). Policy documents
will be collected and analyzed to determine the prevalence and types of evaluation policies
currently in use among the 50 states, while purposefully selected case studies will explore the
features and quality of each type policy. The goal of this research is to enhance our
understanding of state evaluation policy in gifted education by offering a sweeping view of the
national landscape and a focused look at each type of existing state policy, providing an
overview of the options available to policymakers, decision-makers, and advocates of gifted
program evaluation.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
3/27
State Evaluation Policies in Gifted Education:
Examining Options for Policymakers
One of the conditions of a good policy analysis is that it is helpful to a decision maker.
A decision maker looks at it and finds he or she understands the problem better, understands the
choices better, or understands the implications of choice better. (Lynn, 1980, p. 85).
While program evaluation is highly recommended for ensuring high-quality gifted
programs and state policies provide a bridge from recommendation and practice, there is mystery
surrounding the use of evaluation in gifted programs. A number of state gifted education policies
include provisions that influence the evaluation of gifted programs, so we know that evaluation
is occurring at some level. What we do not know is the types of evaluations that programs are
asked to conduct, the usefulness of these evaluations, or the quality of these evaluations. With
the pivotal leveraging role that state policy plays in the programmatic decisions and practices of
gifted programs at the local level, it is pertinent to understand the nature of these policies. By
examining the evaluation practices promoted through state policy, we can begin to understand
the context within which local gifted programs are asked to engage in evaluation activities.
This policy analysis will build a taxonomy to describe the range of state evaluation policy
models currently in use. The features of each policy model will be demonstrated through case
studies that provide a focused look at the utility of the policy products that emerge from
interpretation at the state level. In addition, each model will be evaluated against a matrix of
indicators of quality gifted education policy and gifted program evaluation. The results of this
study will provide advocates, decision-makers, and policymakers with options for using state
policy to leverage the practice of gifted program evaluation at the local level.
Statement of the Problem
From the boardroom to the classroom, we are witnessing a social epidemic of
evaluation (Preskill, 2008, p. 129). Decision-makers across every sector have renewed their
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
4/27
focus on questions such as:How do we know if this program works?How can we ensure that the
time, money, and resources devoted to this program are well spent? How will we determine if
this program accomplishes what it is meant to do?
The best practice literature from the past three decades in gifted education
demonstrates a solid appeal for the use of program evaluation to answer these questions. Since
the publication of the first manual for gifted program evaluation (Renzulli, 1975), a number of
other manuals, service publications, book chapters, and articles have been published to
recommend how gifted and talented programs can use evaluation as a tool for making data-
driven decisions about program changes or improvements (e.g. Callahan, 2009; Callahan &
Caldwell, 1997; Fetterman, 1993; Park, 1984; VanTassel-Baska & Feng, 2004). In fact, program
evaluation standards have even been included in the National Association for Gifted Childrens
(NAGC) PreK-12 Gifted Program Standards (Landrum & Shakelee, 1998), signaling the value
that the field places on the use of program evaluation as an indicator of a high-quality gifted
education program.
Despite these recommendations for incorporating program evaluation as an important
practice in gifted programs, the actual frequency with which gifted programs engage in
evaluation is not well understood (Avery & VanTassel-Baska, 2001). Three meta-evaluations of
evaluation reports have provided some understanding of the nature of evaluation practice in
gifted programs (e.g. Callahan, Tomlinson, Hunsaker, Bland, & Moon, 1995; VanTassel-Baska,
2004, 2006a) and a handful of published articles describe case studies of gifted program
evaluations (e.g. Avery & VanTassel-Baska, 2001; Avery, VanTassel-Baska, & ONeill, 1997;
Hertzog & Fowler, 1999), yet the sum of this work provides a murky picture of the status of
gifted program evaluation across our nation. Some leaders in the field have concluded that
quality program evaluation rarely occurs (VanTassel-Baska, 2006b; Callahan et al., 1999),
although the paucity of research and published examples of gifted program evaluation limits the
degree to which such a criticism can be made.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
5/27
A number of state education agencies (SEAs) have policies that influence the use of
evaluation in the gifted programs within their borders (NAGC & Council of State Directors of
Programs for the Gifted [CSDPG], 2009). Brief descriptions of some of these policies are
included in Table 1. Based on what we know states are requiring local gifted programs to do, we
can conclude that program evaluation is happening at some level; nevertheless, we do not have a
comprehensive picture of what these policy-inspired evaluations entail.
Several researchers have briefly examined policies that influence program evaluation
practices within the context of larger studies of comprehensive state policies in gifted education
(e.g. Brown, Avery, VanTassel-Baska, Worley, & Stambaugh, 2006; Clinkenbeard, Kollof, and
Lord, 2007;). Gallagher (2002), a leading voice in advocacy for gifted education policy, suggests
that program evaluation is one of the four most needed new policies in gifted education.
However, there is no known study that fully explores the nature of existing state evaluation
policies in gifted education.
The purpose of this policy analysis study is to explore the various state policy models that
influence program evaluation practices at the local level. Through the lens of utility, it is my goal
to reveal the features of the various policy models currently used, explore the products that result
through the interpretation of the policies at the state level (e. g., guidelines, monitoring systems,
self-assessment forms), and evaluate the quality of each policy type against a matrix of criteria
gleaned from the best practices literature in gifted education policy and gifted program
evaluation. This study marks an important first step in understanding the policy context within
which gifted program evaluations occur.
Table 1:
Brief Descriptions of State Policies Influencing the Practice of Evaluation in Gifted Programs
State Data Sources Brief Description of Policy
Arizona State of the States 2009 Arizona House Bill 2552 (2006)
LEAs must conduct annual program evaluations andsubmit the results to the SEA; On-site monitoring isconducted as an integrated component of the Title Imonitoring schedule.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
6/27
Hawaii Program Guide for Gifted andTalented
School Gifted and Talented ProgramChecklist
Chapter 51 Regulations
Programs are to conduct annual evaluations for programimprovement, which are to be approved by the ComplexSuperintendent.
Maine Guidelines for the Gifted and TalentedApplication, Budget, Academic Plan,and Visual and Performing Arts Plan
Application for Gifted and TalentedEducational Program Approval andApproval of Program Costs for State
Subsidy Visual and Performing Arts Plan for
Gifted and Talented 2009-2010 Academic Plan for Gifted and
Talented 2009-2010
LEAs must annually submit a program plan/applicationthat must be approved by the SEA. The plan/applicationmust include a summary of the results of programevaluation and a plan for how the program will beevaluated in the upcoming year.
Mississippi Regulations for the Gifted EducationPrograms in Mississippi (2006)
Annual Self-Evaluation of the LocalGifted Education Program: Directionsand Sample Instrument (2007)
Mississippi Gifted EducationPrograms Program StandardsEvaluation & Monitoring (2004)
LEAs must annually submit the results of a programevaluation using the state-supplied self-assessment; Mustsubmit a plan for corrective action on the items that do notmeet the minimum standard set by the state.
Pennsylvania 22 Pa. Code Chapter 16 Basic Education Circular: Monitoring
Schedule for Gifted Education (2009) Gifted Education Program Self-
Assessment Instrument (2003)
At least ten on-site compliance-monitoring visits per yearare conducted annually using a facilitated self-assessment,resulting in improvement and/or corrective action plans.
Texas Texas State Plan for the Education ofGifted/Talented Students (2000)
Flyer: Level 3 School BoardTraining
Advanced Programs & Gifted andTalented Education Questions andAnswers on the State Plan
Local school boards are required to monitor programcompliance and are provided with a standards-based rubricand training opportunities for evaluating compliance of the
program.
Background
Definition of Policy
Definitions of policy abound, all intended to describe the money, rules, and authority
that are used to influence the actions of individuals and institutions (McDonnell & Elmore,
1987, p. 133). According to Gallagher (2002), social policy is defined by the allocation of
resources (e.g., money, human) to any given cause within the broad scope of social needs.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
7/27
Gallagher suggests that evidence for social policy should be derived from four sources:
legislation, courts, administrative rules, and professional standards.
However, within the context of the current study, I am aiming to describe the ways that
local gifted programs are influenced by the rules, regulations, and guidelines of their state, for as
VanTassel-Baska (2006b) suggested, The structure that holds gifted programs together is nested
in the policies that individual states have enacted. The direction and continuity of the field at the
grassroots level, then, are heavily influenced by the state one resides in and the strength of the
policy initiatives in the state (p. 250). Therefore, a more targeted definition of policy within the
context of this study is the rules, statutes, codes, and regulations adopted by state legislatures,
interpreted by state school boards of education and state departments of education, and
implemented by local school districts (Brown et al., 2006, p. 11).
Policy in Gifted Education
The body of knowledge regarding state policy in gifted education is not an unexplored
territory-maybe untended, but not unexplored (Clinkenbeard et al., 2007, p. 2). However, the
focus of state policy inquiry has centered primarily on: (a) comprehensive state gifted education
policy (Brown et al., 2006; Clinkenbeard et al.; NAGC & CSDPG, 2009; Passow & Rudnitski,
1993;Swanson, 2002); (b) focused examinations of state policies regarding the identification ofgifted students (Coleman & Gallagher, 1995); (c) focused examinations of state policies
regarding acceleration in schools (Institute for Research and Policy on Acceleration [IRPA],
NAGC, and CSDPG, 2009); and (d) the allocation of state funding for gifted education (Baker,
2001; Baker & McIntire, 2003).
In 2007 NAGC published a seminal document for gifted education policy, which
unveiled the efforts of the NAGC policy task force to collect sample policy language
(Clinkenbeard et al., 2007). This document provides a discussion of the four major types of
gifted education policy: (a) identification, (b) program and curriculum, (c) personnel preparation,
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
8/27
and (d) program management: assessment/evaluation. Within the discussion of each type of
policy, different models are explored and examples of exact state policy language are shared.
Additionally, hyperlinks to the policies are included. Policies related to program evaluation were
categorized under the umbrella of Program Management: Accountability/Evaluation. There are
four types of policies under this umbrella, which are outlined in Table 2.
Table 2:
Four Types of Policy forProgram Management: Accountability/Evaluation
(Clinkenbeard et al., 2007)
1. Review of local education agency (LEA) plans for gifted education by the state
education agency (SEA);
2. Requirements for LEA plans to include some or all of the following components:
a. screening, identification, and referral processes
b. program provisions employed at each grade level, K-12c. goals and student outcome assessment process for each program model
d. contact time for each model
e. pupil-teacher ratios for classes
f. gifted teacher planning time
g. professional development plans for educators working with gifted learners
h. counseling and guidance
i. program evaluation design
3. SEA monitoring plans that includes all LEAs and require on-site visits; and
4. Annual reports from the LEAs to the SEA to report progress and include student and
program evaluation data.
The NAGC Policy Guide touches on the subject of evaluation policies within the context
of the larger policy umbrella; however, a deeper analysis of this type of policy is warranted. As
previously stated, there are no known studies that specifically focus on state evaluation policy in
gifted education. Evaluation policy is defined as policies such as (but not only) high-level rules
embedded in legislation that are used to guide the practice of evaluation (Mark, Cooksy, &
Trochim, 2009, pp. 4-5). The lack of attention to evaluation policy is not surprising, given the
general paucity of literature on this topic (Mark et al.). Accordingly, a recent issue ofNew
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
9/27
Directions in Evaluation (Trochim, Mark, & Cooksy, 2009), a peer-reviewed, themed journal of
the American Evaluation Association, was dedicated to calling researchers attention to the void
of empirical knowledge on evaluation policy, recommending an agenda for furthering
knowledge, and highlighting several tools and models to aid researchers in developing this area
of research. One of these tools has been incorporated into the conceptual framework for this
study, which is discussed later in this proposal.
Policy Analysis
There are generally three choices for the evaluand in a policy analysis: process, product,
and performance (Gilbert & Terrell, 2005). When analysts examine policyprocess, they seek to
understand how and why the policy developed. Questions such as, What factors lead to the
development of such policy? and, What were the dynamics of the context and politics
surrounding the development of the policy? guide the study of policy process. When analysts
examine theproductof policy, they seek to understand the resulting language, systems,
documents, support structures, etc. that result from the policy itself and the interpretation of the
policy. Questions such as, What is the content of the policy statement? and What are the
resulting structures or systems that are created as a result of the policy?guide the study of
policy product. Finally, when analysts examine theperformance of policy, they seek to
understand how the influence that the policy has had in implementation. Questions such as, Is
the policy accomplishing what was intended? and What effect does the policy have on relevant
stakeholders?guide the study of policy performance.
Trochim (2009) outlined several key approaches to the analysis of evaluation policy.
Within this newly developing area of study, he indicated that the first priority should be for
developing a basic understanding of the types of evaluation policies that are currently in use.
We desperately need descriptive studies of current evaluation policies, and this would be
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
10/27
something we could get to work on immediately (p. 28). He recommended that systematic
reviews of policy documents be conducted to identify existing evaluation policies. In other
words, he recommended the evaluation policyproducts as the starting point, suggesting that
these products could be categorized within a taxonomy of evaluation policy. The taxonomy
could be used for, auditing the as-is situation and suggesting potential gaps (p. 28).
Two recent policy studies provide useful models for the study of state policy products in
the manner that Trochim (2009) suggested. Bocala, Mello, Reedy, and Lacireno-Paquet (2009)
and Stepanek and Peixotto (2009) studied state policies regarding response to intervention (RTI)
in schools. These studies elicited descriptions and features of the policy models that are used by
states in the Northwest and Northeast United States. While Bocala et al. used publicly available
documents to identify the features of the RTI policy models used by the Northeastern states,
Stepanek and Peixotto also incorporated interview data from key state education agency
personnel who managed the states RTI policies and initiatives. Each of these studies explored
the different models and compared them to core elements of RTI as described by the literature.
However, these researchers were careful to limit their analysis and conclusions to the face value
of the policy models, recommending that future research concentrate on the contextual factors
that influence these policies (policy process) and the implementation of these policies at the local
level (policy performance). Nevertheless, these studies mark an important first step in
understanding how SEAs are influencing the practice of RTI in schools.
Research Questions
In the spirit of Trochims (2009) recommendation for descriptive studies of existing
evaluation policies, this study will approach the analysis of state evaluation policy in gifted
education in terms of the product: that is, the content of the policy and the products that emerge
from the interpretation of these policies. Once this baseline of information has been established,
additional studies will be pursued to explore the implementation and effectiveness of the
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
11/27
different types of policies (policy performance), as well as the contextual factors that influence
the policy process.
This study will be guided by three questions:
1. What are the existing models of state evaluation policy in gifted education?
2. What are the key features of each model of state evaluation policy in gifted
education?
3. What resources and activities are in place at the state level to support the various
models of evaluation policy in gifted education?
4. How does each state evaluation policy model in gifted evaluation align with
indicators of utility and quality?
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
12/27
Conceptual Framework
A conceptual framework has been developed to guide this study and is depicted in Figure
1. This conceptual framework is rooted in several important elements, which provide lenses for
examining the features of each type of policy examined.
Figure 1: Conceptual Framework for the Analysis of State Evaluation Policies in Gifted
Education
The Evaluation Policy Wheel
Another of Trochims (2009) key recommendations for the study of evaluation policy is
to categorize the different types of policies using a taxonomy. He proposed a preliminary
taxonomy, called the Evaluation Policy Wheel (EPW), although he recommended that the
taxonomy could be adapted and modified as the body of knowledge grows.
The EPW is a multi-layered wheel that provides a system for categorizing and analyzing
different types, levels, and elements of evaluation policy. The EPW has eight pie slices that
describe a different type or focus of evaluation policy, including policies for: (a) evaluation
goals, (b) participation in evaluation, (c) evaluation capacity building, (d) evaluation
management, (e) roles in evaluation, (f) evaluation process and methods, (g) evaluation use, and
(h) meta-evaluation.
Another feature of the Evaluation Policy Wheel (EPW) is the ringed nature of the wheel,
similar to that of a bulls eye target. This serves to illustrate the range of policy generality or
specificity, with broad policy statements on the outside of the wheel and the more specific,
practical application of those policies in the center. As broad policies move through the
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
13/27
(sometimes) many layers of interpretation, trickling down to application at the local level, they
can be placed on the wheel in the appropriate ring of specificity within each pie-slice category.
The first layer of analysis in this study will result in a taxonomy that will characterize the
as-is situation of state evaluation policy in gifted education. Using the EPW as a starting point,
the taxonomy will be modified to address the specific context of policy in gifted education. The
pie slices, or categories, will reflect the different types of policies currently in use by states,
while the rings of the EPW will be used to pinpoint the movement of policy products from the
more general to the more specific.
Utilization-Focused Evaluation
As the field of program evaluation has evolved, the strategies, models, and approaches to
evaluation have become more sophisticated (Stufflebeam, 2001). Among the 22 most popular
approaches examined by Stufflebeam in his seminal meta-evaluation of evaluation approaches,
several emerged as particularly worthy of continued practice. One of these, Utilization-Focused
Evaluation (Patton, 2008), seems particularly appropriate to use as a lens through which the
usefulness of policy products can be examined.
The main aim of Utilization-Focused Evaluation (U-FE) is to keep the intended uses and
users of the evaluation central in all methodological decisions. The hallmarks of useful
evaluations, according to Patton (2008), are as follows:
Clear identification of the primary intended uses of the evaluation,
Clear identification of the primary intended users of the evaluation,
Clear identification of the evaluation criteria,
Establishment of a clear action framework,
Distinction between the empirical questions and the values questions,
Selection of methods that are appropriate for the questions, and
Facilitation of use of findings.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
14/27
The second layer of analysis in this study will examine each type of evaluation policy in
terms of its utility. Each type of policy and resulting policy products (e.g. documents, resources,
activities) will be examined through the lens of U-FE, using the aforementioned hallmarks of
useful evaluation.
High-Quality State Evaluation Policy in Gifted Education
The third layer of analysis will focus on elements of quality. There are two tools that will
be used to examine quality: VanTassel-Baskas Attributes of High-Quality Gifted Education
Policy (2006) and the NAGC PreK-12 Gifted Program Standards for Program Evaluation
(Landrum & Shakelee, 1998). Each policy type will be examined for these indicators of quality,
and a cross-case matrix analysis will be completed for cross comparison. Appendix A includes
the matrix that will be used for this analysis.
Methods
This policy analysis study will begin with a review of the State of the States in Gifted
Education 2008-2009 report (NAGC & CSDGP, 2009) and the NAGC Policy Guide
(Clinkenbeard et al., 2007). Those states that have no policies or mandates for gifted education
will immediately be excluded from the sample. The data from the remaining states will be
examined to determine which of them have some form of policy influencing evaluation of gifted
programs. These states will form the basis for the initial sample from which I will conduct a
search of all publicly available information and documents that represent gifted education
policies.
The next step will be a content analysis (Krippendorff, 1980) using deductive coding
(Patton, 2002). The codebook will be developed from the EPW. This analysis will result in a
taxonomy of state evaluation policy in gifted education similar to the EPW. This taxonomy will
provide a clear picture of the different models of evaluation policies currently in use by SEAs.
Once I have determined the models of evaluation policies in use, I will purposeful
sampling (Creswell, 2007) to select specific state policies as case studies (Yin, 2009) that will
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
15/27
illustrate each type of policy. I will conduct document analyses, surveys, and semi-structured
follow-up interviews to probe each case; however, the flexibility of an emergent design will be
retained to allow for additional methods of data collection to be used as the need for them
emerges (Patton, 2002). Sample questions that may be used in the survey and follow-up
interview are included in Appendix B and Appendix C.
As recommended by Miles and Huberman (1994), I will use a pre-structured case outline.
A pre-structured case outline is appropriate in case study research when research questions have
been well specified. It prevents the researcher from becoming overwhelmed by the amount of
data available in case study data collection. This pre-structured case outline will provide the
framework needed within the case narrative to describe the features of each policy model,
incorporating elements from level two of the conceptual framework. Rich, thick description
(Merriam, 2009) will be used to complete each case outline, which will then be shared with the
informant to ensure that the policy and policy products have been accurately described (i.e.,
member check).
Upon receipt and incorporation of feedback (as warranted) from the informant, I will
complete a matrix analysis of each case to examine the quality of the policy as described in the
conceptual framework. The draft matrix is provided in Appendix A. As an added layer of
trustworthiness, a critical friend (Merriam, 2009) will read the case descriptions and complete
the matrix analysis. Following each of our analyses, we will debrief and reconcile disparities
between the ratings through a consensus-building discussion (Miles & Huberman, 1994).
Cross-case data displays will be constructed, not to compare or rank the different models,
but rather to provide a visual display of the various features of each policy model. The intention
for these displays is to provide a clear picture of the options available to policymakers. Examples
of these data displays are included in Appendix A and Appendix D.
This study should result in improved understanding of the various state policy models
that influence the practice of evaluation in gifted programs. The intent is that this study will help
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
16/27
policy makers understand the problem better, understand the choices better, or understand the
implications of choice better (Lynn, 1980, p. 85).
Limitations and Future Directions
While the purpose of this policy analysis is to examine the policy models themselves,
additional research is needed to explore the effectiveness of each policy model. It would also be
important to conduct in-depth explorations of the context within which each of these models is
best suited, as research has suggested that the political culture of each state may play a mediating
role in the policy and leadership practices at the state, district, and local level (Louis, Thomas,
Gordon, and Febey, 2008).
Another important future direction within this line of research is a focus on the economic
factors that influence states gifted education evaluation policies. The recently published State of
the States report (NAGC & CSDGP, 2009) highlighted the tenuousness of funding in gifted
education policy. An important direction of this research might be to explore the ways in which
state-initiated gifted program evaluations might actually provide the evidence needed to
command the attention of decision-makers in charge of appropriating funds for educational
programs. Some theoretical literature has explored the signaling role that program evaluation
plays in alerting government to worthwhile programs and subsequent resource allocation (Henry
& Mark, 2003; Segerholm, 2003; Trochim, 2009). Attention to evaluation as a tool for
advocating for mandates and funding at the state level might provide valuable information to our
field.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
17/27
References
Avery, L. D., & VanTassel-Baska, J., (2001). Investigating the impact of gifted education
evaluation at state and local levels: Problems with traction.Journal for the Education of
the Gifted, 25, 153-176.
Avery, L. D., Vantassel-Baska, J., & ONeill, B., (1997). Making evaluation work: One School
Districts Experience. Gifted Child Quarterly, 41(4), 124-131.
Bocala, C., Mello, D., Reedy, K., and Lacireno-Paquet, N. (2009). Features of state response to
intervention initiatives in Northeast and Islands Region states (Issues & Answers Report,
REL 2009No. 083). Washington, DC: U.S. Department of Education, Institute of
Education Sciences, National Center for Education Evaluation and Regional Assistance,
Regional Educational Laboratory Northeast and Islands. Retrieved from
http://ies.ed.gov/ncee/edlabs .
Brown, E., Avery, L., VanTassel-Baska, J., Worley, B. B., & Stambaugh, T., (2006). A five-state
analysis of gifted education policies.Roeper Review, 29, 11-23.
Callahan, C. M. (Ed.). (2004). Program evaluation in gifted education (Vol. 11). In S. M. Reis
(Series Ed.),Essential readings in gifted education. Thousand Oaks, CA: Corwin Press.
Callahan, C. M. (2006). Developing a plan for evaluating a program in gifted education. In J. H.
Purcell & R. D. Eckert (Eds.),Designing services and programs for high-ability
learners: A guidebook for gifted education (pp.195206). Thousand Oaks, CA: Corwin
Press.
Callahan, C. M. (2009). Evaluating for decision-making: The practitioners guide to program
evaluation. In J. S. Renzulli, E. J. Gubbins, K. S. McMillen, R. D. Eckert, & C. A. Little
(Eds.), Systems and models for developing programs for the gifted and talented(2nd ed.).
Mansfield Center, CT: Creative Learning Press.
http://ies.ed.gov/ncee/edlabshttp://ies.ed.gov/ncee/edlabs8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
18/27
Callahan, C. M., & Caldwell, M. S. (1997).A practitioners guide to evaluating programs for
the gifted. Washington, DC: NAGC.
Callahan, C. M., Tomlinson, C. A., Hunsaker, S. L., Bland, L. C., & Moon, T. (1995).
Instruments and evaluation designs used in gifted programs (Research Report No.
95132). Retrieved from the National Research Center on the Gifted and Talented website:
http://www.gifted.uconn.edu/nrcgt/nrconlin.html#95132
Clinkenbeard, P. R., Kolloff, P. B., & Lord, W. E. (2007).A guide to state policies in gifted
education. Washington, DC: National Association for Gifted Children.
Fetterman, D. M. (1993).Evaluate yourself. (Report No. 9304). Retrieved from the National
Research Center on the Gifted and Talented website:
http://www.gifted.uconn.edu/nrcgt/nrconlin.html#9304
Gallagher, J. J. (2002). Societys role in educating gifted students: The role of public policy.
(Report No. RM02162). Retrieved from the National Research Center on the Gifted and
Talented website: http://www.gifted.uconn.edu/nrcgt/nrconlin.html#02162
Gilbert, N., & Terrell, P. (2005).Dimensions of social welfare policy (6th ed.). Boston, MA:
Allyn & Bacon.
Henry, G. T., & Mark, M. M., (2003). Beyond use: Understanding evaluations influence on
attitudes and actions.American Journal of Evaluation, 24, 293-314.
Hertzog, N. B., & Fowler, S. A., (1999). Perspectives: Evaluating an early childhood gifted
education program.Roeper Review, 21, 222-227.
Institute for Research and Policy on Acceleration, National Association for Gifted Children, &
Council of State Directors of Programs for the Gifted, (2009). Guidelines for developing
an academic acceleration policy. Retrieved from Institute for Research and Policy on
Acceleration website: http://www.accelerationinstitute.org/Resources/Policy_Guidelines/
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
19/27
Krippendorff, K., (1980). Content analysis: An introduction to its methodology. Thousand Oaks,
CA: SAGE.
Landrum, M. S., & Shakelee, B. D. (Eds.). (1998). The NAGC PreK-12 Gifted Program
Standards. Washington, DC: NAGC.
Loius, K. S., Thomas, E., Gordon, M. F., Febey, K. S., (2008). State leadership for school
improvement: An analysis of three states.Educational Administration Quarterly, 44,
562-592.
Lynn, L. E., Jr. (1980). Crafting policy analysis for decision makers [Interview with Laurence E.
Lynn, Jr.].Educational Evaluation and Policy Analysis, 2, 85-90.
National Association for Gifted Children and the Council of State Directors of Programs for the
Gifted. (2009). 2008-2009 State of the states: A report by the National Association for
Gifted Children and the Council of State Directors of Programs. Washington, DC:
Author.
Mark, M. M., Cooksy, L. J., & Trochim, W.M.K. (2009). Evaluation policy: An introduction and
overview. In W.M.K. Trochim, M. M. Mark, & L. J. Cooksy (Eds.).Evaluation policy
and evaluation practice. New Directions for Evaluation, 123, 3-11.
McDonnell, L. M., & Elmore, R. F., (1987). Getting the job done: Alternative policy
instruments.Educational Evaluation and Policy Analysis, 9, 133-152.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation (3rd ed.). San
Francisco: CA: Jossey-Bass.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook
(2nd ed.). Thousand Oaks, CA: SAGE.
Park, B. N. (Ed.). (1984). Evaluating programs for the gifted and talented [Special issue].
Journal for the Education of the Gifted. 7(1).
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
20/27
Passow, A. H., & Rudnitski, R. A., (1993). State policies regarding education of the gifted as
reflected in legislation and regulation. (Report No. CRS93302). Retrieved from the
National Research Center on the Gifted and Talented website:
http://www.gifted.uconn.edu/nrcgt/nrconlin.html#93302
Patton, M. Q., (2003). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks,
CA: SAGE.
Patton, M. Q., (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: SAGE.
Preskill, H., (2008). Evaluations second act: A spotlight on learning. American Journal of
Evaluation, 29, 127-138.
Renzulli, J. S. (1975).A guidebook for evaluating programs for the gifted and talented. Venture,
CA: Office of the Ventura County Superintendent of Schools.
Segerholm, C. (2003). Researching evaluation in national (state) politics and administration: A
critical approach.American Journal of Evaluation, 24, 353-372. doi:
10.1177/109821400302400305
Stambaugh, T., Worley, B., VanTassel-Baska, J., & Brown, E. (2005). Analysis of state gifted
education policy self-assessment forms: Results from the NAGC state policy workshop,
November, 2004. Washington, DC: National Association for Gifted Children.
Stepanek, J., and Peixotto, K. (2009). Models of response to intervention in the Northwest
Region states (Issues & Answers Re port, REL 2009No. 079). Washington, DC: U.S.
Department of Education, Institute of Education Sciences, National Center for Education
Evaluation and Regional Assistance, Regional Educational Laboratory Northwest.
Retrieved from www.ies. ed.gov/ncee/edlabs.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
21/27
Swanson, M. (2002).National survey on the state governance of K-12 gifted and talented
education. Summary report. Nashville: Tennessee Initiative for Gifted Education Reform.
(ERIC Document Reproduction Service No. ED471886)
Trochim, W.M.K. (2009). Evaluation policy and evaluation practice. In W.M.K. Trochim, M. M.
Mark, & L. J. Cooksy (Eds.).Evaluation policy and evaluation practice. New Directions
for Evaluation, 123, 13-32.
Trochim, W. M. K., Mark, M. M., & Cooksy, L. J., (Eds.). (2009). Evaluation policy and
evaluation practice.New Directions in Evaluation, 123.
VanTassel-Baska, J. (2004). Metaevaluation findings: A call for gifted program quality. In J.
VanTassel-Baska & A. X. Feng (Eds.),Designing and utilizing evaluation for gifted
program improvement(pp. 227-245). Waco, TX: Prufrock Press.
VanTassel-Baska, J. (2006a). A content analysis of evaluation findings across 20 gifted
programs: A clarion call for enhanced gifted program development. Gifted Child
Quarterly, 50, 199-215.
VanTassel-Baska, J. (2006b). State policies in gifted education. In J. H. Purcell & Eckert, R. D.,
(Eds.).Designing services and programs for high-ability learners: A guidebook for gifted
education. Thousand Oaks, CA: Corwin Press & The National Association of Gifted
Children.
VanTassel-Baska, J., & Feng, A. X. (Eds.). (2004). Designing and utilizing evaluation for gifted
program improvement. Waco, TX: Prufrock Press.
Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Thousand Oaks, CA:
SAGE.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
22/27
Appendix A
Quality Indicators Matrix
Table X:
Cross-Case Analysis of Quality Criteria
Key: Not observed
Criteria Not Met
Criteria Partial Met
Criteria Met
Case
A
Case
B
Case
C
Case
D
Case
E
I
n
di
c
at
o
rs
of
H
ig
h-
Q
u
alit
y
P
ol
ic
y
(
V
a
n
Tas
se
l-
B
as
k
a,
Clarity
C1 Inclusion of clear,
unambiguous language
C2 Promotes common
interpretation
C3 Easy determination of
compliance by LEAs
C4 Easy determination of actions
necessary for compliance
Connectedness
C5 Internal validity with other
policies in gifted education
C6 Compatibility with other
policies in gifted education
C7 Linkage with general education
policies
Feasibility for
Implementation
C8 Practicality of implementation
at local level
C9 Replicability in other states
Research-Based C10 Grounding in research-basedbest practices
C11 Congruency with new research
in gifted education
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
23/27
2
0
0
4)
N
A
G
C
P
reK
-
1
2
P
r
o
g
r
a
m
Sta
n
d
a
r
d
s
fo
r
P
r
o
g
r
a
m
E
v
al
u
at
io
n
(1
9
9
8)
Purposeful EvaluationC12 Information reflects needs of
constituency groups
Efficient and
Economical Evaluation
C13 Sufficient time, money, and
personnel
Competent and Ethical
Evaluation
C14 Competent, knowledgeableevaluator
C15 Evaluation design
C16 Instruments and procedures
C17 Formative and summative
designs
C18 Confidentiality and confidence
of participants
Evaluation Report
C19 Written report content and
design
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
24/27
Appendix B
Survey Items
Instructions:
We are investigating the different ways that state policies influence the practice of program
evaluation in gifted and talented programs. You have been selected to receive this questionnaire
because information gleaned from the recent State of the States (2009) report indicates that your
state has some type of policy that may influence gifted program evaluation (e.g., self-evaluation,compliance monitoring, approval of local plans). We are conducting case studies of different
types of policies and would like your state to serve as a illustrative case for one type of policy.
If you are interested in participating in this study, please complete the answer the following
questions. Upon receipt of your completed questionnaire, you will be contacted to schedule a 30-
minute follow-up interview by phone.
Definitions: Throughout this document I refer to the evaluation system. By this I mean any
process, mandate, or initiative that requires gifted and talented programs to conduct, or supply
information verifying that they have conducted, a program evaluation.
Survey Questions
1. Briefly describe the system that your state uses to encourage, influence, or mandate gifted
and talented programs to conduct program evaluations.
2. Who is responsible for overseeing the implementation of this evaluation system? What
role do they play in the oversight of this evaluation system?
3. We have collected the following documents from your state website as documents that
reflect the policies influencing gifted program evaluation in your state. Are there any
other documents that would be valuable for our study? If so, how might we receive a
copy of those items?
4. If available, please also provide any other documents that would demonstrate your states
evaluation system in action (e.g., a typical evaluation report from a local gifted, an on-site
monitoring report).
5. In the future we may conduct a follow-up study that explores various stakeholders
experiences with this evaluation system. Please provide names and contact information
for stakeholders in your state that could share their experiences with the evaluation
system. These might be regional technical assistance providers, reviewers of plans, gifted
program coordinators who have gone through the process, or on-site monitors.
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
25/27
Appendix C
Follow-up Interview Questions
Thank you for taking the time to talk with me today about your states policy for gifted program
evaluation. I have several questions that I would like to ask you about your states policy. Our
conversation should take about 25 to 30 minutes.
1. I recently sent you a description that Ive written to characterize the nature of your states
policy for the evaluation of gifted programs. Have I accurately described your states
evaluation policy? Are there any points that you would like to clarify for me?
2. What do you think this policy is achieving?
3. What would be an ideal result of this policy from the states perspective? Do you feel that
your state is achieving your ideal results? (Probe intended uses and intended users)
4. How do you use the information that is gathered through this evaluation system?
5. How are local gifted programs affected by this policy? What role do you think this policy
plays in the quality of local gifted programs?
6. What have been some of the successes of this policy? Challenges?
7. What changes would you make to this evaluation system if it were within your power?
8. What advice would you offer to a policymaker who was considering a similar system of
gifted program evaluation in his or her own state?
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
26/27
Appendix D
Sample Cross-Case Data Displays
Table D1:
State Evaluation Policy Types in Gifted Education with Intended Uses and Intended
Users
Evaluation
Policy Type Description
States
Using this
Policy
Type
Intended Uses
of Evaluation
Results
Intended Users
of Evaluation
Results
Type A
Type B
Type C
Type D
Type E
Key
Type A
Type D
Type B
Type E
Type C
8/14/2019 State Evaluation Policies in Gifted Education: Examining Options for Policymakers
27/27
Figure D1. Visual display of policy type by state. (Note: Those states in black represent the
states in Table X of the proposal.
Top Related