Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
-
Upload
ajaxtelamonio -
Category
Documents
-
view
219 -
download
0
Transcript of Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
1/23
MULTIVARIATE BEHAVIORAL RESEARCH 1
Multivariate Behavioral Research, 38 (1), 1-23
Copyright 2003, Lawrence Erlbaum Associates, Inc.
Validation of Cognitive Structures:
A Structural Equation Modeling Approach
Dimiter M. DimitrovKent State University
and
Tenko Raykov
Fordham University
Determining sources of item difficulty and using them for selection or development of test
items is a bridging task of psychometrics and cognitive psychology. A key problem in this
task is the validation of hypothesized cognitive operations required for correct solution of
test items. In previous research, the problem has been addressed frequently via use of the
linear logistic test model for prediction of item difficulties. The validation procedure
discussed in this article is alternatively based on structural equation modeling of cognitive
subordination relationships between test items. The method is illustrated using scores of
ninth graders on an algebra test where the structural equation model fit supports the
cognitive model. Results obtained with the linear logistic test model for the algebra test are
also used for comparative purposes.
Determining sources of item difficulty and using them for analysis,
development, and selection of test items necessitates integration of cognitive
psychology and psychometric models (e.g., Embretson, 1995; Embretson &
Wetzel, 1987; Mislevy, 1993; Snow & Lohman, 1984). The cognitivestructure of a given test is defined by the set of cognitive operations and
processes, as well as their relationships, required for obtaining correct
answers on its items (e.g., Riley & Greeno, 1988; Gitomer & Rock, 1993).
Knowledge about cognitive and processing operations in a model of item
difficulty prediction allows test developers to (a) construct items with
difficulties known prior to test administration, (b) avoid piloting of individual
items in study groups and thus reduce costs, (c) match item difficulties to
ability levels of the examinees, and (d) develop teaching strategies that target
specific cognitive and processing characteristics. Previous studies have
integrated cognitive structures with (unidimensional and multidimentional)
item response theory (IRT) models allowing prediction of item difficulty from
cognitive and processing operations that are hypothesized to underlie item
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
2/23
D. Dimitrov and T. Raykov
2 MULTIVARIATE BEHAVIORAL RESEARCH
solving (e.g., Embretson, 1984, 1991, 1995, 2000; Fischer, 1973, 1983; Spada,
1977; Spada & Kluwe, 1980; Spada & McGaw, 1985).
The validation of hypothesized cognitive components with IRT models is
focused on the accuracy of item difficulty prediction. While such tests are
important for prediction purposes, they do not tap into cognitive relationshipsamong items that logically relate to the validity of a cognitive structure.
Therefore, the purpose of this article is to discuss a method that (a) furnishes
an overall validation of cognitive structures and (b) provides diagnostic
feedback about cognitive relationships among items. It should be noted that
this method and previously used IRT methods do not compete, since they
address the validation of cognitive structures from different perspectives.
To illustrate this, a brief description of the IRT linear logistic test model
(LLTM; Fischer, 1973, 1983) is provided below. The LLTM is suitable forsuch anillustration due toits relative simplicity and frequent use inprevious
IRT studies on cognitive test structures (e.g., Dimitrov & Henning, in press;
Embretson & Wetzel, 1987; Fischer, 1973; Medina-Diaz, 1993; Spada, 1977;
Spada & Kluwe, 1980; Spada & McGaw, 1985; Whitely & Schneider, 1981).
In the LLTM, the item difficulty parameter ifor each ofn items (i = 1, ..., n)
is presented as a linear combination of cognitive operations (Fischer, 1973, 1983):
(1) @ =i ik k k
pw c= +
=
,1
wherep n-1,k
represents the impact of the kth cognitive operation on the
difficulty of any item from the target set of items, wik
(weight ofk) adjusts the
impact ofkfor item i, and c is a normalization constant (k= 1, ...,p). The matrix
W= (wik)
is referred to as the weight matrix [i = 1, 2, ..., n, k= 1, ...,p; in the
remainder, the symbol (.) is used to denote matrix]. The LLTM is obtained
by replacing the item difficulty parameter, i, in the Rasch measurement
model (Rasch, 1960) with the linear combination on the right-hand side of
Equation 1:
(2) P(Xij
= 1|j,
k, w
ik) =
exp
exp
,
K =
K =
j ik k
k
p
j ik k
k
p
w c
w c
+F
HG
I
KJ
L
NMM
O
QPP
+ +FHG I
KJL
NMM
OQPP
=
=
1
1
1
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
3/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 3
whereXij
is a dichotomous observed variable taking the value of 1 if person
j correctly solves item i and 0 otherwise, and P(.|.) denotes the probability
of a person with ability j
to answer correctly the ith item with difficulty i
as obtained from (1) (i = 1, ..., n,j = 1, ...,N;Nbeing sample size). In an
IRT context, the term ability connotes a latent trait that underlies thepersons performance on a test (e.g., Hambleton & Swaminathan, 1985).
The ability score, , of a person relates to the probability for this person to
answer correctly any test item. Ability and item difficulty aremeasured on
the same scale. The units of this scale, called logits, represent natural
logarithms of odds for success on the test items. With the Rasch model,
log[P/(1-P)] = - , where P is the probability for correct response on an item
with difficulty for a person with ability . Thus, when a person and an item
share the same location on the logit scale ( = ), the person has a .5probability of answering the item correctly. Also, using that exp(1) = e = 2.72
(rounded to the nearest hundredth), a difference of one point on the logit
scalecorresponds to a factor of 2.72 in theodds for success on the test items.
The LLTM relates test items to cognitive operations inferred from a
theoretical model of knowledge structures and cognitive processes. For
example, Embretson (1995) proposed seven cognitive operations required for
correct solution of mathematical word problems by operationalizing three
types of knowledge (factual/linguistic, schematic, and strategic) defined in thetheoretical model of Mayer, Larkin, and Kadane (1984). It should be noted that
the LLTM has been found to be very sensitive to specification errors in the
weight matrix, W (Baker, 1993). This implies the necessity of careful and
sound validation of the cognitive model that underlies an LLTM application.
It is important to note that the LLTM is a Rasch model with the linear
restrictions in Equation 1 imposed on the Rasch item difficulty parameter, i.
Therefore, testing the fit of an LLTM requires two steps: (a) testing the fit
of the Rasch model (which includes testing for unidimensionality) and (b)
testing the linear restrictions in Equation 1 (Fischer, 1995, p. 147). While
there are numerous tests for the Rasch model (e.g., Glas & Verhelst, 1995;
Smith, Schumacker, & Bush, 1998; Van den Wollenberg, 1982; Wright &
Stone, 1979), testing the restrictions in Equations 1 is typically performed
with the likelihood ratio (LR) test for relative goodness-of-fit of the LLTM
and Rasch models (e.g., Fischer, 1995, p. 147). The pertinent test statistic
is asymptotically chi-square distributed with degrees of freedom equal to the
difference in the number of independent parameters of the two compared
models. There is also a simple graphical test for the relative goodness-of-fit of the LLTM and Rasch model (Fischer, 1995). The logic of this test is
that if the LLTM fits well, the points with coordinates that represent
estimates of item difficulty with the LLTM and the Rasch model,
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
4/23
D. Dimitrov and T. Raykov
4 MULTIVARIATE BEHAVIORAL RESEARCH
respectively, should scatter around a 45line through the origin (see Figure
4). As Fischer (1995) noted, the LR test very often turns out significant in
empirical research, thus leading to rejection of the LLTM even when the
graphical goodness-of-fit test indicates a good match between Rasch and
LLTM item difficulties. Medina-Diaz (1993) applied the quadraticassignment technique for analysis of cognitive structures with the LLTM, but
her approach similarly failed to provide good agreement with this LR test,
and inorder to proceedrequired data on the response of each examinee on
each cognitive operation information that is difficult or impossible to obtain
in most testing situations.
While LLTM tests are important for accuracy in the prediction of Rasch
item difficulty from cognitive operations and processes, they do not tap into
cognitive relationships among items that are logically inferred from thecognitive structure. The idea underlyingthe methodproposed in this article
is to reduce the cognitive weight matrix, W, to a diagram of cognitive
subordination relationships between items, and then test the resulting model
for goodness-of-fit using structural equation modeling (SEM; Jreskog &
Srbom, 1996a, 1996b). Triangulations with logical fits in the diagram of
cognitive subordinations among items can provide additional heuristic
evidence in the validation process.
It is worthwhile stressing thatthe LLTM tests and the proposed SEMmethod do not compete, because they target different aspects of the
validation of cognitive structures. LLTM tests relate to the accuracy of
prediction of Rasch item difficulty from cognitive operations, while the SEM
method relates to the validation of cognitive relations among items for a
hypothesized cognitive structure. The proposed SEM method can be used
to justify, not replace, applications of LLTM or other models for prediction
of item difficulty such as multiple linear regression (e.g., Drum, Calfee, &
Cook, 1981) or artificial neural network models (Perkins, Gupta, &
Tammana, 1995). The information about cognitive relations among items
provided by the proposed SEM method can be used also for purposes other
than prediction of item difficulty (e.g., task analysis, content selection,
teaching strategies, and curriculum development).
A Structural Equation Modeling Based Procedure for Validation of
Cognitive Structures
Oriented Graph of Cognitive Structures
Given a set ofm cognitive operations required for solution ofn items
comprising a considered test, the weight matrix Wrepresents a two-way
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
5/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 5
table with elements wik
= 1 indicating that itemIirequires cognitive operation
k, or wik
= 0 otherwise; (i = 1, ..., n; k= 1 , ..., m). In this article, an itemIj
is referred to as subordinated to item Ii
if and only if the cognitive
operations required for the correct solution of item Ij
represent a proper
subset (in the set-theoretic sense) of the cognitive operations required for thecorrect solution of itemI
i. For purposes of symbolizing the subordination
relationships between items, the W matrix can be reduced to a matrix of item
subordinations, denoted S = (sij), with elements s
ij=1 indicating that itemI
j
is subordinated to itemIiand s
ij= 0 otherwise (i,j = 1, ..., n ). We note that
two arbitrarily chosen test items need not necessarily be in a relation of
subordination; if they are not, then sij
= 0 holds for their corresponding entry
in the matrix S.
One can now represent graphically the matrix S as an oriented graphwhere an arrow fromI
jto I
i(denotedI
j I
iin the graph) indicates that item
Ij
is subordinated to itemIi. Figure 1 presents a hypothetical weight matrix
W, its corresponding S matrix, and the oriented graph associated with them.
An examination of thematrixW shows, for example, that itemI1is subordinated
to itemI4
as the set of cognitive operations required by itemI1
(viz. operations
O1and O
3) is a proper subset of the cognitive operations required by itemI
4(viz.
operations O1, O
3, and O
4). Therefore, we have s
41= 1 in thematrix S and an
arrow from itemI1 to itemI4 in the graph diagram. In the matrix S, one canalso see that all entries in the column that corresponds to itemI2equal 0. This
is because itemI2
is not subordinated to any of the other items. Indeed, the
set of cognitive operations required by itemI2(viz. operations O
2and O
3) do
not represent a proper subset of the cognitive operations required by any
other item. Thus, si2
= 0 (i = 1, 2, 3, 4) and there are no arrows from item
I2to other items in the graph diagram. Similarly, one can check other entries
in the matrix S.
The transitivity of the subordination relationship holds when successive
arrows in the oriented graph connect more than two items. For example, the
three-item pathI3 I
1 I
4indicates that (a)I
3is subordinated toI
1,(b)
I1
is subordinated to I4, and by transitivity (c) I
3is subordinated to I
4.
Relationships of subordination by transitivity are not represented by arrows
in a diagram in order to simplify the graphical representation.
Relationship to Structural Equation Modeling
The presently proposed method of cognitive structure validation makesthe assumption that for each test item there exists a continuous latent
variable, denoted (appropriately sub-indexed if necessary), which
represents the ability required for correct solution of the item. Persons solve
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
6/23
D. Dimitrov and T. Raykov
6 MULTIVARIATE BEHAVIORAL RESEARCH
an item correctly if they posses in sufficient degree this ability, that is, if their
ability relevant to the item exceeds or equals a certain minimal level (denoted
) required for its correct solution. Thus, formally, the assumption is
(3) Xij = 1 ifiji,0 if
ij<
i,
where ij
denotes thejth persons level of ability required for solving the ith
item, and iis a threshold parameter above/below which a correct/incorrect
response results for the ith item (i = 1, ..., n,j = 1, ...,N). We further assume
that the n-dimensional vector of latent variables = (1, ...,
n) is
multivariate normal.
W matrix S matrix
Cognitive Operation Item I1
I2
I3
I4
I5
Item O1
O2
O3
O4
I1
0 0 1 0 1
I2
0 0 1 0 0
I1
1 0 1 0 I3
0 0 0 0 0
I2
0 1 1 0 I4
1 0 1 0 1
I3
0 0 1 0 I5
0 0 0 0 0
I4
1 0 1 1
I5
1 0 0 0
Figure 1
Graph Diagram of Item Subordinations for a Hypothetical W Matrix
Graph Diagram
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
7/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 7
The assumptions in this subsection are identical to those made when
analyzing ordinal data using structural equation modeling (SEM; Jreskog &
Srbom, 1996b, ch. 7). Thus we are now in a position to evaluate the degree
to which a set of cognitive subordination relationships among given items is
plausible, by using the corresponding goodness of fit test available in SEM.That is, we can validate the cognitive structure represented by an oriented
graph by applying the popular SEM methodology. Specifically, in this
framework we consider the item subordination relationships to be the
building blocks of a structural equation model relating the items.
Accordingly, the model of interest is
(4) = B +
where B = (rs) is the n n matrix of relationships between the abilities
required for successful solution of the rth and sth items while is the vector
of corresponding zero-mean residuals, each assumed unrelated to those
variables that are predictors of its pertinent ability variable (r, s = 1, ..., n).
Hence, in modeling subjects behavior on say the rth item it is assumed that
up to an associated residual rthe required ability for its correct solution,
r,
is linearly related to those of corresponding other items (r= 1, ..., n; existence
of the inverse of the matrix In - B is also assumed, as in typical applicationsof SEM; e.g., Jreskog & Srbom, 1996b).
Figure 2 represents the oriented graph of item subordinations for the
weight matrix W from the example in the next section. The structural
equation model corresponding to Equation 4 is represented in Figure 3. Its
path-diagram follows widely adopted graphical convention of displaying
structural equation models. In this diagram, each item is represented by a
circle denoting the latent ability required for its correct solution. Other than
this inconsequential detail for the following developments, the model in
Figure 3 symbolically differs from the oriented graph in Figure 2 only in that
residuals are associated with latent variables receiving one-way arrows that
represent the cognitive subordination relationship of the item at its end to the
item at its beginning. With the proposed SEM approach, this model is fitted
to the n n tetrachoric correlation matrix of the observed dichotomous
variablesXithat each evaluate whether the ith item has been correctly solved
or not (i = 1, ..., n; cf. Equation 3).
Thus, with the SEM method used in this article, the validity of a cognitive
structure is tested by the following three-step procedure. First, reduce theweight matrix W to a matrix S of cognitive subordinations. Second, as
described in the previous section, represent graphically the cognitive
subordinations among items in the matrix S as an oriented graph. Third, using
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
8/23
D. Dimitrov and T. Raykov
8 MULTIVARIATE BEHAVIORAL RESEARCH
the ordinal data SEM approach, fit the latent relationship model based on the
last graph (as its path diagram) to the tetrachoric correlation matrix of the
items (Jreskog & Srbom, 1996b, ch. 7). The goodness of fit test available
thereby represents a test of the plausibility of the originally hypothesized set
of cognitive subordination relationships among the studied items. As abyproduct of this approach, one can also estimate the thresholds
iof all items
(i = 1, ..., n), which allow comparison of their difficulty levels. Evaluation of
significance of elements of the matrix B in Equation 4 indicates whether,
under the model, one can dispense with the assumption of cognitive
subordination for certain pairs of items (viz. those for which the
corresponding regression coefficient is nonsignificant). Further, in case of
lack of model fit, modification indices may be consulted to examine if model-
data mismatch may be considerably reduced by introduction of substantivelymeaningful subordination relationships between items, which were initially
omitted from the model. (This use of modification indices would initiate an
exploratory phase of analysis; Jreskog & Srbom, 1996c.)
The discussed SEM approach is based on the tacit assumption of
subjects not guessing on any of the items. This is an important assumption
made also in many applications of item response models and is justified by
the observation that guessing would imply for pertinent item(s) more than one
trait underlying their successful solution. Further, the approach is best usedwhen the sample of subjects having taken all items of a test under
consideration is large (Jreskog & Srbom, 1996b). Moreover, for this SEM
approach, the assumption of an underlying multinormal latent ability vector
is relevant, as well as that of linear relationships among its components as
reflected in the underlying model Equation 4 that is tested in an application
of the discussed SEM method (Jreskog & Srbom, 1996a).
Triangulation with Logical Fits
Logical fits in the path diagram also lend support to the validity of a
cognitive structure. One possible logical fit, referred to here as path-wise
increase of item difficulty, is basedon the assumption that the difficulty of
items increases with the increase of processing tasks required for item
success. Under this assumption, the item difficulty is expected to increase
down the arrows (path-wise) for any path of the oriented graph. With the
example provided later in this article, the items are simple linear algebra
equations with their difficulty logically increasing with the increase ofproduction rules required for the correct solution of these equations (see
Appendix). For this example, there was an increase of Rasch item difficulty
down the arrows for each path of the oriented graph, thus providing
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
9/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 9
triangulation support to the validation based on the SEM goodness-of-fit test.
However, the assumption of path-wise increase of item difficulty may not be
appropriate with other cognitive structures. Depending on the W matrix,
cognitive subordinations among items may not translate into ordered difficulty
values. In such cases, other logical fits may better reflect the cognitiverelationships among items in the path diagram of the W matrix. One can even
decide to modify the working definition of cognitive subordinations among
items to accommodate the substantive context of the W matrix. The validation
of cognitive structures is, after all, a flexible logical process of collecting
evidence that may include, but is not limited to, an isolated statistical act.
Example
This example illustrates the proposed validation method for a cognitive
structure on an algebra test. The test consisted of 15 linear equations to be
solved (see Appendix) and was administered toN= 278 ninth-grade high
school students in northeastern Ohio. The students were required to show
work toward solution in order to avoid guessing on the test items. The
cognitive structure of this test was determined by 10 production rules
(PRs) required for the correct solutions of the 15 equations. An earlier
version of the test and the PRs were developed in a previous study (Dimitrov& Obiekwe, 1998) in an attempt to improve the system of production rules
for algebra test of linear equations proposed by Medina-Diaz (1993).
Table 1 presents the W matrix of 15 items and 10 production rules. Table
2 gives the S matrix of item subordinations determined from W. The path
diagram corresponding to the matrix S is given in Figure 2. As an illustration,
itemI2
is the algebra equation 5x- 3 = 7 + 4x. The production rules (see
Appendix) for the correct solution of this equation are: PR6 (balancing):
5x- 4x= 7 + 3, PR3 (collecting numbers): 5x- 4x= 10, and PR4 (collecting two
terms):x= 10. On the other hand, the correct solution of equation 2(x+ 3) =x
- 10" (itemI1) requires the same three production rules plus PR7 (removing
parenthesis with a positive coefficient). Thus, the PRs required by itemI2
represent a proper subset of the PRs required by item I1
(cf. Table 1). This
indicates that itemI2is cognitively subordinated to itemI
1:I
2 I
1(Figure 2).
The tetrachoric correlation matrix resulting from the data is presented in Table
3 (and obtained with PRELIS2; Jreskog & Srbom, 1996a).
We note that in the general case of application of the method of this article,
residuals are associated with each dependent latent variable unless theoreticalconsiderations allow one to hypothesize lacking such for a certain latent
variable(s). In the present case, in which an algebra test is used to illustrate the
method, no residual term is assumed to be associated with the ability
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
10/23
D. Dimitrov and T. Raykov
10 MULTIVARIATE BEHAVIORAL RESEARCH
Table 1
Matrix W for Fifteen Algebra Items and Ten Production Rules
Production Rules
Item PR1 PR2 PR3 PR4 PR5 PR6 PR7 PR8 PR9 PR10
I1
0 0 1 1 0 1 1 0 0 0
I2
0 0 1 1 0 1 0 0 0 0
I3
1 1 0 0 0 1 1 0 0 0
I4
1 0 1 1 0 1 1 0 0 0
I5
1 0 0 0 0 0 0 0 0 1
I6
1 1 1 1 0 1 0 0 0 0
I7
1 0 1 1 0 1 1 0 1 0
I8
0 0 1 0 0 1 0 0 0 0
I9
0 0 1 1 0 0 0 0 0 0
I10
1 0 1 1 1 1 0 0 0 0
I11
1 0 1 1 0 1 0 0 0 1
I12
1 0 1 1 0 1 1 1 0 0
I13
1 0 1 1 1 1 1 1 0 0
I14
1 1 1 1 1 1 1 1 0 0
I15
1 0 1 1 1 1 1 0 0 0
Table 2
Matrix S for the W Matrix of the Algebra Test
Item I1
I2
I3
I4
I5
I6
I7
I8
I9
I10
I11
I12
I13
I14
I15
I1
0 1 0 0 0 0 0 1 1 0 0 0 0 0 0
I2 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0I3
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
I4
1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
I5
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
I6
0 1 0 0 0 0 0 1 1 0 0 0 0 0 0
I7
1 1 0 0 0 0 0 1 1 0 0 0 0 0 0
I8
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
I9
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
I10
0 1 0 0 0 0 0 1 1 0 0 0 0 0 0
I11 0 1 0 0 1 0 0 1 1 0 0 0 0 0 0I12
1 1 0 0 0 0 0 1 1 1 0 0 0 0 0
I13
1 1 0 0 0 0 0 1 1 1 0 1 0 0 0
I14
1 1 1 1 0 0 0 1 1 1 0 1 1 0 1
I15
1 1 0 1 0 0 0 1 1 1 0 0 0 0 0
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
11/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 11
corresponding to itemI2. This is because the ability that underlies the correct
solution of itemI2is determined by the abilities required for solving itemsI
8and
I9
that are related to itemI2
in the considered model. Indeed (see Table 1), the
production rules required for the correct solution of itemI2(PR3, PR4, and PR6)
are fully represented by those required for the correct solutions of itemI8 (PR3and PR6) and itemI
9(PR3 and PR4). Similarly, no residual is assumed to be
associated with the latent variable corresponding to item I13
. Indeed, an
examination of the solution of itemI12
that is related to itemI13
in the model
suggests that the ability underlying correct solution of the latter is determined by
those required to solve correctly itemI12
. This is because itemsI12
andI13
require
the same PRs, with the exception that itemI13
requires slightly higher frequency
for two of these PRs (namely, collecting terms and removing parentheses).
SEM Test for Goodness-of-Fit
Figure 2 represents the path diagram of the initial structural equation model
resulting from the matrix S of item subordination relationships. This model was
Table 3
Tetrachoric Correlation Matrix of the Algebra Test Items (N= 278)
I1
I2
I3
I4
I5
I6
I7
I8
I9
I10
I11
I12
I13
I14
I15
I1
1.00
I2
0.78 1.00
I3
0.52 0.48 1.00
I4
0.54 0.53 0.63 1.00
I5
0.55 0.48 0.56 0.56 1.00
I6
0.51 0.45 0.77 0.63 0.71 1.00
I7
0.58 0.41 0.44 0.40 0.51 0.57 1.00
I8
0.54 0.58 0.31 0.57 0.33 0.60 0.49 1.00
I9 0.56 0.51 0.44 0.50 0.51 0.55 0.66 0.61 1.00I
100.56 0.53 0.56 0.64 0.50 0.61 0.53 0.58 0.60 1.00
I11
0.41 0.29 0.32 0.26 0.54 0.53 0.34 0.53 0.48 0.33 1.00
I12
0.63 0.66 0.68 0.65 0.48 0.64 0.53 0.72 0.68 0.71 0.44 1.00
I13
0.55 0.50 0.43 0.47 0.49 0.55 0.47 0.51 0.57 0.46 0.25 0.82 1.00
I14
0.29 0.34 0.56 0.56 0.45 0.61 0.51 0.33 0.44 0.64 0.21 0.81 0.34 1.00
I15
0.41 0.47 0.46 0.60 0.39 0.55 0.45 0.58 0.66 0.74 0.41 0.74 0.59 0.57 1.00
Note. N= sample size.
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
12/23
D. Dimitrov and T. Raykov
12 MULTIVARIATE BEHAVIORAL RESEARCH
fitted to the tetrachoric correlation matrix of thedichotomously scored test
items that is presented in Table 3, and found to be associated with a chi-square
value 2 = 162.81 with degrees of freedom (df) = 84 and a root mean square
error of approximation (RMSEA) = .058 with 90%-confidence interval (.045;
.071). In this model, the relationship between itemsI14 andI15 turned out tobe nonsignificant:
14,15= -.28, standard error = .29, t-value = -.96. Inspecting
the cognitive processes required for correct solution of the (model-assumed)
related itemsI3,I
13,I
14, andI
15, we noticed that the operations necessary for
solving itemsI3
andI13
were in fact those required for correct solution of item
I14
(see Table 1). In the context of correct solution of itemsI3andI
13, therefore,
the cognitive operations underlying itemI15
are inconsequential for correct
solution of item I14
. This suggests that we can dispense with the earlier
assumed relationship between itemsI14 andI15. Indeed, fixing the path fromitemI
15toI
14in the model displayed in Figure 2 resulted in a nonsignificant
increase of the chi-square value up to 2 = 163.89, df= 85, RMSEA = .058
(.044; .071) (2 = 1.08,df= 1,p > .10). In addition, these model fit indices
are themselves acceptable (e.g., Jreskog & Srbom, 1996b), and hence the
last model version can be considered a tenable means of data description and
explanation. The parameter estimates and standard errors associated with this
final model are presented in its path-diagram provided in Figure 3. We
comment next on the elements of this solution by comparing path estimates totheir standard errors (the ratio of parameter estimate to standard error equals
the t-value; Jreskog & Srbom, 1996b).
First, moving from top to bottom in Figure 3 we note the relatively strong
relationship between each of items I8
and I9
with item I2. This is not
unexpected, given the similarity of these items in content and production
rules they require (see Appendix). At the same time, we note the
nonsignificant relationships between itemI11
and each of itemsI2andI
5. One
possible explanation is the lack of a linear relationship between the abilities
to solve each of itemsI2
andI5, with that of itemI
11. Indeed, an inspection
of the students test work on these items showed that those who did not solve
correctly itemI11
failed at the initial step, PR10 (removing denominators),
because of difficulty with finding the least common denominator. Therefore,
success of these students on the subsequent steps (production rules PR1,
PR3, PR4, and PR6) did not lead to correct solution of itemI11
. When solving
itemI2, however, the successful application of PR3, PR4, and PR6 led to a
correct solution because PR10 (removing denominators) was not required
by itemI2. Further, the weak relationship between itemsI11 and I5 can beexplained by the fact that most students who failed on the least common
denominator with itemI11
avoided this problem with itemI5by applying the
cross-multiplication rule.
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
13/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 13
Moving to the lower parts of the model solution in Figure 3, we notice that
itemI2is strongly related to itemsI
1,I
6, andI
10. This indicates that the ability
associated with itemI2plays a central role for solving each of itemsI
1,I
6, and
I10
. This is not unexpected because the production rules required by itemI2
represent the major part of the production rules required by itemsI1,I6, andI
10. Similarly, the ability associated with itemI
1plays a central role for solving
correctly itemsI4,I
7, andI
12 the paths leading from itemI
1into each of these
three items are significant. In this context, the ability of solving itemI10
is
only weakly related to that of solving itemI12
. This indicates that there is no
unique contribution of the ability underlying itemI10
to the correct solution of
itemI12
after controlling for the contribution of the ability underlying itemI1
to solving correctly itemI12
.
The relationships of itemsI4 and I10 to itemI15 are significant and thusindicate that, in the context of correctly solving itemI
4, the ability to correctly
solve itemI10
is also relevant (i.e., has unique contribution) for successful
solution of item I15
. An explanation of this finding is that removing
parentheses, which is required by itemI15
, can be avoided in the solution of
itemI4
by collecting the two terms within parentheses: 5(2x- 5x) ... (see
Appendix). Also, collecting more than two terms is required by itemsI10
and
I15
, but not necessarily by itemI4.
Focusing on the lowest part of the model depicted in Figure 3, it isstressed that there is a strong relationship between the ability underlying item
I12
to that associated with itemI13
. Thus, according to the model, the ability
to solve the latter item is essentially proportional to that for solving itemI12
.
This is not surprising given the fact that the same production rules are
required by both itemsI12
andI13
with the exception that collecting terms
and removing parentheses is more frequently required with itemI13
.
Last but not least, the relationship between items I3
and I14
is
nonsignificant while that between itemsI13
andI14
is significant. Thus, in the
context of the ability necessary for handling item I13
, there is no unique
(linear) contribution of itemI3to successful solution of itemI
14. This can be
explained by the fact that, out of the eight production rules required for
solving correctly itemI14
, seven are required also by itemI13
. There is only
one production rule required by item I14
that is required also by itemI3but not
by item I13
(PR2; see Table 1).
For triangulation purposes, if one assigns to each item in Figure 2 its
Rasch difficulty reported in Table 4, there is a path-wise increase of item
difficulty for all paths. This can be illustrated, for example, with the four-itempathI
8(-2.487) I
2(-1.632) I
1(-1.305) I
7(0.793), where the Rasch
difficulty (in logits) is given in parentheses. This logical fit is consistent with
the results from the SEM goodness-of-fit test for the validation of the weight
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
14/23
D. Dimitrov and T. Raykov
14 MULTIVARIATE BEHAVIORAL RESEARCH
matrix W in Table 1. In this example, the path-wise increase of item difficulty
can be expected to occur due to the orderly structure of production rules
required for the solution of simple linear algebra equations (see Appendix).
However, as noted earlier in this article, this may not be true in the context
Figure 2
Graph Diagram of Item Subordinations for the W Matrix in Table 1
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
15/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 15
of other cognitive structures. Yet, depending on the W structure, one may
find other plausible heuristic triangulations of the SEM goodness-of-fit test
for the validation of the hypothesized cognitive structure.
Figure 3
Path Diagram (final model) for the SEM Analysis with the W Matrix in Table 1
* Statistically significant beta estimates (p < .01)
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
16/23
D. Dimitrov and T. Raykov
16 MULTIVARIATE BEHAVIORAL RESEARCH
The SEM treatment of the oriented graph of cognitive subordinations
may provide useful validation feedback to subsequent applications of the W
matrix in various measurement and substantive studies. In the next section,
this is illustrated with an application of the LLTM for the test of simple linear
algebra equations (see Appendix). The LLTM, if it fits the data well,provides estimates of the
kparameters in Equation 1 for the prediction of
difficulty of simple linear algebra equations. The accuracy of this prediction
is the LLTM criterion for validity of the W matrix. Evidently, the LLTM
validation can complement, but cannot substitute, the validation of the same
W matrix provided by the SEM method. Moreover, the latter will be the only
validation perspective in this study should the LLTM fail to fit the data.
LLTM Application
The LLTM was conducted with the W matrix in Table 1 using the
computer program for structural Rasch modeling LPCM-WIN 1.0 (Fischer &
Ponocny-Seliger, 1998). As noted earlier, the LLTM can be used only if the
Rasch model fits the data. The test of model fit [2(14) = 22.83, p > .05]
indicates a good fit of the Rasch model. The estimates of LLTM parameters
are provided in Table 4, with kindicating the contribution of the kth production
rule to the difficulty of any item that requires this production rule (k= 1, ..., 10).Estimates of the Rasch difficulty and the predicted (LLTM) difficulty for the
algebra test items are also provided in Table 4. The graphical goodness-of-fit
test (Fischer, 1995) in Figure 4 indicates very good fit, with a Pearson
correlation of .975 between the LLTM and Rasch item difficulties. However,
the LR difference in chi-square test does not lend support to the fit of the
LLTM relative to the Rasch model [2(4) = 46.91, p < .01]. This is not
surprising given the reported strictness of this LR test in the literature (e.g.,
Fischer, 1995, p. 147). Thus, while the graphical test supports the validation
of the W matrix in terms of accurate prediction of Rasch item difficulty with
the LLTM, the strict LR test does not.
In this situation, the SEM validation of the W matrix in the previous
section can moderate in support of subsequent applications of Equation 1
for the LLTM prediction of item difficulty in a test of simple liner algebra
equations from 10 production rules (see Appendix).
Discussion and Conclusion
In previous research, validation of cognitive structures was addressed
primarily with LLTM applications for predicting item difficulties. As is well
known, LLTM goodness-of-fit tests focus on the similarity in fit between the
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
17/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 17
Table 4
LLTM Analysis with the W Matrix of the Algebra Test
Parameter Estimates Item Difficulty
PRs k(SE)a Item LLTM Rasch
PR1 -0.735 ( 0.151)** I1
-1.5659 -1.3050
PR2 2.401 ( 0.205)** I2
-1.1484 -1.6323
PR3 -1.536 ( 0.271)** I3
1.0500 1.1704
PR4 -0.773 ( 0.175)** I4
-0.4557 -0.2654
PR5 0.426 ( 0.178)* I5
0.3042 -0.1923
PR6 -0.950 ( 0.115)** I6
0.7432 0.7265
PR7 -0.314 ( 0.110)** I7 1.0813 0.7931PR8 -2.328 ( 0.145)** I
8-2.9167 -2.4871
PR9 0.388 ( 0.157)* I9
-1.5659 -1.6323
PR10 0.458 ( 0.138)** I10
-0.4245 -0.1440
I11
1.9139 2.3913
I12
-0.3993 -0.6997
I13
1.0828 0.9262
I14
1.8541 2.0282
I15
0.4469 0.3225
aSE= standard error of the LLTM parameter k
(k= 1,..., 10).
*p < .05. **p < .01.
Figure 4
Graphical Goodness-Fit-Test for the LLTM with the W Matrix in Table 1
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
18/23
D. Dimitrov and T. Raykov
18 MULTIVARIATE BEHAVIORAL RESEARCH
LLTM and theRasch model. However, the associated LR difference in chi-
square test rather frequently turns out significant, thus leading to rejection of
the LLTM even when the graphical test for goodness-of-fit indicates a good
match between the estimates of the Rasch item difficulties and their LLTM
predicted values (e.g., Fischer, 1995). The quadratic assignment test for theLLTM (Medina-Diaz, 1993) requires data on the subjects responses on each
cognitive operation information that is difficult or impossible to obtain in most
testing situations. The LLTM tests are important for accuracy in the prediction
of Rasch item difficulties but they do not tap into cognitive relationships among
items for the validation of the hypothesized cognitive structure. In addition, the
LLTM works under relatively strong assumptions: (a) the test is
unidimensional, (b) the Rasch model fits the data well, and (c) the restrictions
in Equation 1 hold.The SEM approach discussed in this article provides an overall
goodness-of-fit test as well as valuable information about cognitive
subordinations among test items. An item is referred to here as cognitively
subordinated to another item if the cognitive operations required by the first
item represent a proper subset of the cognitive operations required by the
second item. With the proposed validation method, the W matrix of a
cognitive structure is reduced to a matrix Sof item subordinations. The
model corresponding to the path diagram associated with the matrix S is thensubmitted to SEM analysis for goodness-of-fit by fitting it to the tetrachoric
correlation matrix of dichotomously scored items.
It should be emphasized that the SEM validation model is not tied to
specific assumptions of the LLTM or other IRT models that focus on
prediction of item difficulty from cognitive operations. Lack of
unidimensionality for the test may preclude the use of the LLTM, but not
applicability of the SEM method proposed in this article. Conversely, lack
of fit ofthe SEM model has no direct implications for the Rasch model, but
may question the substantive validity of Equation 1 with the LLTM. Indeed,
just as accuracy of prediction with a multiple linear regression equation does
not exclude specification problems, accuracy of item prediction with
Equation 1 does not exclude possibility for misspecifications of cognitive
components.
The path diagram corresponding to the matrix S can also be used for
additional substantive and logical analyses of cognitive structures. With the
cognitive structure of some tests (e.g., orderly structured production rules in
solving algebra equations), it is sound to expect that the more processingoperations are required by the items, the more difficult the items become. Under
this assumption, the increase of item difficulty along the paths of the diagram
provides heuristic support to the validity of the cognitive structure. This was
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
19/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 19
illustrated in the previous section by associating the items of the path diagram in
Figure 2 with their Rasch difficulties. When the Rasch model does not fit the
data, one can associate the items in the path diagram with their difficulty
estimates produced by other IRT models that assume no guessing (e.g., the
two-parameter logistic model; see Embretson & Reise, 2000, p. 70). With somecognitive structures, however, cognitive subordinations between items do not
necessarily parallel the item difficulty estimates. In such cases, one may use
other logical fits in the path diagram to triangulate the validation provided by the
SEM goodness-of-fit test. As noted earlier, one can even modify the working
definition of cognitive subordinations among items to accommodate the
substantive context of the W matrix. Combining the SEM goodness-of-fit test
with logical fits for the path diagram is a dependable process of collecting
evidence about the validity of a hypothesized cognitive structure.It should be noted that the path diagram corresponding to the matrix S
displays paths of subordinated items generated from the matrix W, but does not
include partial overlaps of cognitive operations between items. Despite this
limitation, the SEM goodness-of-fit test and triangulation logical fits of the paths
have strong potential toprovide evidence for the validity of a cognitive structure.
As illustrated in the previous section with the LLTM for the algebra test, the strict
LR difference in chi-square test can be soundly counterbalanced by consistent
results of the LLTM graphical goodness-of-fit test, the SEM goodness-of-fittest, and logical fits for the path diagram. In addition, the joint analysis of
cognitive substance and SEM coefficients of paths in the oriented graph can
provide valuable feedback about linearity of relationships, redundancy, and
uniqueness of contribution for abilities that underlie the correct solution of items.
This was illustrated with the joint analysis of SEM path estimates and substantive
implications for abilities that underlie the correct solution of items in the algebra
test. Also, the information about cognitive relations among items provided by the
path diagram and its SEM analysis can be used, for example, in task analysis,
content selection, teaching strategies, and curriculum development.
When using the method of this article, we emphasize that repeated
modification of the initial structural equation model effectively represents
elements of an exploratory analysis along with the confirmatory ones that are
inherent in a typical utilization of SEM. In this type of application of the method,
it is important that researchers validate the final model on data from an
independent sample before more general conclusions are drawn. Similarly,
such a validation is recommended also when the model that was fitted turns out
to be associated with acceptable fit indices and presents substantivelyplausible means of description of the relationships between the analyzed items.
Even in the latter likely rare cases in empirical work, it is highly recommendable
to replicate the model in other studies aiming at cognitive validation of the same
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
20/23
D. Dimitrov and T. Raykov
20 MULTIVARIATE BEHAVIORAL RESEARCH
test items. We also stress that the method of this article necessitates large
samples (Jreskog & Srbom, 1996b). This essential requirement is justified
on two related grounds. One, the analytic approach underlying the method is
the SEM methodology that represents itself a large-sample modeling
technique. Two, since the analyzed data is dichotomous (true/false solution oftest items), application of the weighted least squares method of model
estimation and testing is necessary, which requires as a first step the estimation
of a potentially rather large weight matrix (Jreskog & Srbom, 1996b).
Further, with other than large samples, the polychoric correlation matrix to
which the structural equation model is fitted may contain entries with
intolerably large standard errors, thus weakening the conclusions reached with
the SEM approach. Since we are not aware of explicit, specific, and widely
accepted guidelines as to determination of sample size in the dichotomous itemcase of relevance here, based on recent continuous variable simulation
research reviewed and presented by Boomsma and Hoogland (2001; see also
Hoogland, 1999; Hoogland & Boomsma, 1998) we would like to suggest that
a sample size of at least several hundred is needed even with a relatively small
number of items (up to dozen, say). In this regard, we would generally
discourage use of the method of this article with a sample size of less than 200
subjects even with a smaller number of items; with more than a dozen of items
say, we would view samples considerably higher than this as minimally needed,and perhaps higher than a thousand as required with larger models.
In conclusion, the SEM method discussed in this article combines confirmatory
and exploratory procedures in a flexible process of collecting statistical and
heuristic evidence about the validity of hypothesized cognitive structures.
References
Baker, F. B. (1993). Sensitivity of the linear logistic test model to misspecification of the
weight matrix. Applied Psychological Measurement, 17(3), 201-210.Boomsma, A. & Hoogland, J. J. (2001). The robustness of LISREL modeling revisited. In
R. Cudeck, S. H. C. duToit, & D. Srbom (Eds.), Structural equation modeling:
Presenceand future. A festschrift in honor of Karl Jreskog. Chicago: Scientific
Software International.
Dimitrov, D. M. & Henning, J. (in press). A linear logistic test model of reading
comprehension difficulty. In R. Hashway (Ed.), Annals of the Joint Meeting of the
Association for the Advancement of Educational Research and the National Academy
for Educational Research 2000. Lanham, Maryland: University Press of America.
Dimitrov, D. M. & Obiekwe, J. (1998, February). Validation of item difficulty components
for algebra problems. Paper presented at the meeting of the Eastern EducationalResearch Association, Tampa, FL.
Drum, P. A., Calfee, R. C., & Cook, L. K. (1981). The effects of surface structure variables
on performance in reading comprehension tests. Reading Research Quarterly, 16,
486-514.
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
21/23
D. Dimitrov and T. Raykov
MULTIVARIATE BEHAVIORAL RESEARCH 21
Embretson, S. E. (1984). A general latent trait model for response processes.
Psychometrika, 49, 175-186.
Embretson, S. E. (1991). A multidimensional latent trait model for measuring learning and
change. Psychometrika, 56, 495-516.
Embretson, S. E. (1995). A measurement model for linking individual learning to process
and knowledge: application to mathematical reasoning. Journal of Educational
Measurement, 32, 277-294.
Embretson, S. E. (2000). Multidimensional measurement from dynamic tests: Abstract
reasoning under stress. Multivariate Behavioral Research, 35, 505-543.
Embretson, S. E. & Reise, S. P. (2000). Item response theory for psychologists . Mahwah,
NJ: Erlbaum.
Embretson, S. & Wetzel, C. D. (1987). Component latent trait models for paragraph
comprehension tests. Applied Psychological Measurement, 11, 175-193.
Fischer, G. H. (1973). The linear logistic test model as an instrument in educational
research. Acta Psychologica, 37, 359-374.
Fischer, G. H. (1983). Logistic latent trait models with linear constraints. Psychometrika,
48, 3-26.
Fischer, G. (1995). The linear logistic model. In G. H. Fischer & I. W. Molenaar (Eds.)
Rasch models. Foundations, recent developments, and applications (pp. 131-155).
New York: Springer-Verlag.
Fischer, G. & Ponochny-Seliger, E. (1998). Structural Rasch modeling: Handbook of the
usage of LPCM-WIN 1.0. Groningen, Netherlands: ProGAMMA.
Gitomer, D. H. & Rock, D. (1993). Addressing process variables in test analysis. In N.
Frederiksen, R. J. Mislevy, & I. Bejar (Eds.), Test theory for a new generation of tests
(pp. 243-268). Hillsdale, NJ: Erlbaum.
Glas, C. A. W. & Verhelst, N. D. (1995). Testing the Rasch model. In G. H. Fisher & I. W.
Molenaar (Eds.). Rasch models. Foundations, recent developments, and applications
(pp. 69-95). New York: Springer-Verlag.
Hambleton, R. K. & Swaminathan, H. (1985). Item response theory: Principles and
applications. Boston: Kluwer-Nijhoff.
Hoogland, J. J. (1999). The robustness of estimation methods for covariance structure
analysis. Unpublished doctoral dissertation. University of Groningen, The
Netherlands.
Hoogland, J. J. & Boomsma, A. (1998). Robustness studies in covariance structure
modeling: An overview and a meta-analysis. Sociological Methods and Research, 26,
329-367.
Jreskog, K. G. & Srbom, D. (1996a). PRELIS2: Users reference guide. Chicago, IL:
Scientific Software International.
Jreskog, K. G. & Srbom, D. (1996b). LISREL8: Users reference guide. Chicago, IL:
Scientific Software International.
Jreskog, K. G. & Srbom, D. (1996c). LISREL8: The SIMPLIS command language.
Chicago, IL: Scientific Software International.
Mayer, R., Larkin, J., & Kadane, P. (1984). A cognitive analysis of mathematical problem
solving. In R. Stenberg (Ed.),Advances in the psychology of human intelligence (Vol.
2, pp. 231-273). Hillsdale, NJ: Erlbaum.
Medina-Diaz, M. (1993). Analysis of cognitive structure using the linear logistic test model
and quadratic assignment. Applied Psychological Measurement, 17(2), 117-130.
Mislevy, R. J. (1993). Foundations of a new theory. In N. Frederiksen, R. J. Mislevy, &
I. Bejar (Eds.), Test theory for a new generation of tests (pp. 19-39). Hillsdale, NJ:
Erlbaum.
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
22/23
D. Dimitrov and T. Raykov
22 MULTIVARIATE BEHAVIORAL RESEARCH
Perkins, K., Gupta, L., & Tammana, R. (1995). Predicting item difficulty in a reading
comprehension test with an artificial neural network. In A. Davies & J. Upshur (Eds.)
Language testing, Vol. 12, No. 1 (pp. 34-53). London: Edward Arnold.
Rasch, G. (1960). Probabilistic models for intelligence and attainment tests. Copenhagen:
Danmarks Paedagogiske Institut.
Riley, M. S. & Greeno, J. G. (1988). Developmental analysis of understanding language
about quantities and of solving problems. Cognition and Instruction, 5, 49-101.
Smith, R. M., Schumacker, R. E., & Bush, M. J. (1998). Using item mean squares to
evaluate fit to the Rasch model. Journal of Outcome Measurement, 2, 66-78.
Snow, R. E. & Lohman, D. F. (1984). Implications of cognitive psychology for educational
measurement. In R. Linn (Ed.), Educational measurement (3rd ed., pp. 263-331).
New York: Macmillan.
Spada, H. (1977). Logistic models of learning and thought. In H. Spada & W. F. Kempf
(Eds.), Structural models of thinking and learning (pp. 227-262).
Spada, H. & Kluwe, R. (1980). Two models of intellectual development and their reference
to the theory of Piage. In R. Kluwe & H. Spada (Eds.), Developmental model of
thinking (pp. 1-32). New York: Academic Press.
Spada, H. & McGaw, B. (1985). The assessment of learning effects with linear logistic test
models. In S. Embretson (Ed.), Test design: New directions in psychology and
psychometrics (pp. 169-193). New York: Academic Press.
Van den Wollenberg, A. L. (1982). Two new statistics for the Rasch model .
Psychometrika, 47, 123-139.
Whitely, S. E. & Schneider, L. (1981). Information structure for geometric analogies: A
test theory approach. Applied Psychological Measurement, 5, 38 3-397.
Wright, B. D. & Stone, M. H. (1979),Best test design . Chicago: MESA Press.
Accepted April, 2002.
Appendix
Test of Algebra Linear Equations
ItemI1: 2(x+ 3) = x- 10
ItemI2: 5x- 3 = 7 + 4x
ItemI3: 2(x- n) = 5ItemI
4: 5(2x- 5x) = 20x
ItemI5: 3x/7 = 10
Item I6: nx- a = 5a
ItemI7: 4[x+ 3(x- 2)] = 10
ItemI8: -7 +x= 10
ItemI9: 20 = 5x- 5x
ItemI10
: 2x+ 7 - 10x+ 3 = 12 - 2x
ItemI11: 4x/5 + 2 + 2x/3 - 10 = 0ItemI
12: -5(8 - 2x) = 2x- 2
-
7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach
23/23
D. Dimitrov and T. Raykov
ItemI13
: 5 - 2(x+ 3) =x+ 5(2x- 1) + 10
ItemI14
:x- 4(5x- 4) + 5x= 10 - n + 2n
ItemI15
: 6(x- 4) + 2x = 5x - 10
Production rules (PRs) required for the correct solution of the algebraequations:
PR1: Solve for variable with only numeric coefficients
Example: If 5x= 20 thenx= 20/5
PR2: Solve for variable with nonnumerical coefficients
Example: Ifnx= a + 5 thenx= (a + 5)/n
PR3: Collecting numbers
Example: If 2x= 5 - 10 then 2x= -5PR4: Collecting two terms
Example: If 2x- 5x= 10 then -3x= 10
PR5: Collecting more than two terms:
Example: If 2x- 5x+ 8x= 10 then 5x= 10
PR6: Balancing
Example: If 5 + 2x= 3 then 2x= 3 - 5
PR7: Removing parentheses with a positive coefficient
Example: If 5(2x+ 3) = 5 then 10x+ 15 = 5PR8: Removing parentheses with a negative coefficient
Example: If - 5(2x+ 3) = 5 then -10x- 15 = 5
PR9: Removing brackets:
Example: If 4[x+ 3(x- 2)] = 10 then 16x- 24 = 10
PR10: Removing denominators
Example: If 3x/7 = 1/2 then 6x= 7
Note. The examination of items and PRs shows that (a) PR2 subsumes PR1,
(b) PR4 subsumes PR3, (c) PR5 subsumes PR4, and (d) PR8 subsumes PR7.
For example, PR4 subsumes PR3 because collecting terms (PR4) includes
also collecting numbers (PR3) that represent the coefficients of the terms
involved in PR4. Therefore, when an item requires PR4, this item
automatically requires PR3 (see Table 1).