Explicit and Implicit States of Model Mixing in …lbao/archive/papers/Model...Model Mixing (Bao,...

Model Mixing (Bao, 2002) 1

Explicit and Implicit States of Model Mixing in Conceptual Development

Lei Bao Department of Physics, The Ohio State University, 174 W 18th Ave., Columbus, OH 43210

Abstract

Researchers in science education have demonstrated that students can hold several different understandings of a single concept and apply them inconsistently to tasks related to the concept. Many studies have been conducted to investigate how these coexisting and competing understandings of concepts affect the process of conceptual development. Based on the context dependence of learning, we developed Model Analysis as a theoretical and measurement framework to capture the process of how different types of understandings of a concept evolve and develop in learning and to quantitatively assess the existences and changes of these different understandings. In Model Analysis, we use the term models to represent the layer of functional mental constructs and describe the uses of different constructs with model states. A model state gives the distribution of probabilities for a student to apply different types of models in given contexts. When a student holds multiple models and applies them inconsistently across contexts, the student is described with a mixed model state. With this method, students’ changes of their understandings can be represented quantitatively with the structures (in terms of probability distributions) of their model states. This paper reports a new progress in Model Analysis, which identifies two different mixed model states: explicit state and implicit state. The author will discuss the theoretical basis, empirical evidence, and a mathematical representation developed to use with quantitative instruments such as multiple-choice questions. Examples of research applications are also discussed.

I. Introduction Mixed Use of Different Types of Knowledge

In research on student learning of science and mathematics, it is well documented that students can use inconsistent reasoning in responding to tasks that are related to a single concept and considered equivalent by experts (Clough and Driver, 1986; diSessa 1993; Palmer, 1993; Bao and Redish, 2001). For example, in learning physics, when presented with a set of expert-equivalent questions, a particular student can use different (and often contradictory) types of understandings of a concept to solve these questions.

Many studies have been conducted to model this situation. Maloney and Siegler (1993) proposed a framework of conceptual competition. They suggested that a student could hold several different understandings of a concept, which would coexist and compete during learning. The understanding that wins the competition on a given problem will be used to represent that problem. The winning of the competition of a particular understanding is dependent on the nature of the understanding and features of the problem.

In a similar study, Thornton (1994) used the term student views to represent the different understandings of a physics concept and developed a phenomenological framework to identify student views of the physical world and to explore the dynamic process by which these student views are transformed in learning. In his research, students were found to have different views that are coexisting at the same time during instruction. Such mixing of student views was defined as a transitional state. His research also suggests that in the learning process, many students often move from a somewhat consistent but incorrect view, through a transitional state, towards a consistent and correct view.

The two studies discussed above provide good descriptive frameworks to understand the inconsistency of student reasoning and students’ mixed use of different conceptual understandings. In a study on children’s learning, Vosniadou (1994) develop a framework theory and used the term mental


models to represent the different types of conceptual understandings. In her definition, mental models are dynamic and generative representations, which can be manipulated mentally to provide causal explanations of physical phenomena and make predictions about the physical world. They can be generated on the spot to deal with the specific problem-solving situations and can also be stored in long-term memory for future uses. The formation of a mental model is constrained by one’s high-level fundamental beliefs and the domain-specific features of the problems.

With the mental model framework, Vosniadou investigated children’s development of the concept of earth and identified a unique situation: during learning children can develop hybrid (synthesized) mental models to reconcile the conflicts between children’s initial naïve models and evidences that support the correct model. A hybrid mental model incorporates some pieces of the correct understandings while keeping large part of one’s existing knowledge system unchanged. Since it has some part of the correct understandings, a hybrid model can provide explanations consistent with some of the physical phenomena underlined by the correct model. This creates a temporary solution for the conflict created by using one’s existing models and thus can be used as a quick (and easy) fix to one’s knowledge system when cognitive conflicts are encountered in learning. For example, Vosniadou and Brewer (1992) reported that children often have an initial model of a “rectangular earth” which reflects the common sense of land being flat. Through learning, a child started to recognize that the earth has to be a sphere. To reconcile the conflict between a spherical earth and the land being flat, children can develop a hybrid model of a “hollow sphere” where people can stand inside the sphere on a flat plane. Not only children develop such hybrid models, in a study of student understanding of quantum mechanics, senior level undergraduate students also develop hybrid models to reconcile quantum phenomena with their classical intuitive beliefs (Bao, 1999).

In a study to further understand how and why students behave in variety inconsistent ways, we developed Model Analysis (Bao, 1999, Bao & Redish, 2001), in which we also used the term mental models (or models) to describe the types of unique understandings of a concept. In Model Analysis, we particularly emphasize the context-dependence of learning (we considered mostly the context factors in the content knowledge involved in the learning of physics concepts – e.g. the structural features of a physics scenario). Both the student models and the contexts in which the models are cued and used are part of the analysis. A mathematical representation and numerical tools were also developed for quantitative assessment of how students were able to apply their models and how this application varies as the context changes. In the measurement of a student’s use of models, we prepare a sequence of questions or scenarios in which an expert would use a single, coherent model, which we refer to these as a set of expert-equivalent questions/scenarios. For a particular concept, a student’s use of different possible models is measured with a set of expert-equivalent scenarios and the result is represented as a model vector in a model space associated with the concept, which is spanned by a set of base vectors each representing a unique model that is identified through qualitative research as to be commonly used by the population. The term common models is used to represent the set of unique models that span the model space. These models include the ones that a student may have developed before instruction (initial models), the ones that a student is likely to develop during instruction (such as a hybrid model), and the ones that expert would like the students to learn (expert models). The different situation of a single student’s use of different models is represented with a vector in the model space defined as the model state vector. The vector’s coefficients reflect the probabilities that the student will use a particular model when presented with an arbitrary scenario chosen from a set of expert-equivalent physics scenarios involving the same underlying concept but with different context settings. The probabilistic character of the student state arises from the presentation of a large number of scenarios, not from the probing of multiple students. We view the context dependence of mental model generation as a fundamental probabilistic character built into the individual. The statistical treatment is a way of treating many “hidden variables” in the problem that are both uncontrollable and possibly immeasurable even in principle (Bao & Redish, 2001).


When a student use one model consistently in all context settings related to the concept, the model state of this student only has one non-zero component (which equals to 1) in the dimension corresponding to that particular model. In this case, the student is considered to have a pure model state reflecting consistent use of a single model (the particular model can be correct, incorrect or hybrid). When a student use different models inconsistently across contests, the student’s model state has many non-zero components and is referred as a mixed model state.

When interpreting models and model states, it is worth noticing the difference between a hybrid model and a mixed model state: a hybrid model is treated as any other type of model and is given an unique dimension in the model space. The mixed use of different ideas is described as a mixed model state. On the other hand, a hybrid model is a model (just like any other models), which is formed through a small scale restructuring of one’s knowledge system by combining some elements of the existing knowledge system with some newly incorporated elements of the correct understandings. It represents a unique new type of locally consistent package of knowledge (although still inconsistent at a general scale) different from both students’ previous knowledge and the new correct knowledge introduced by the instruction. Therefore, a hybrid model is considered in our research as an independent mental construct, which represents a completely different entity comparing to a mixed model state: a hybrid model is equivalent to any models defined by researchers to represent unique types of knowledge whereas a model state represents the ways that students’ use the different types of knowledge (rather than the types of knowledge used by students).

In the Model Analysis framework, the situations of students’ using different models either consistently or inconsistently is interpreted as a result of the context-dependence of learning, and the dynamical process of conceptual development is modeled using the context as the fundamental “reference frame”. All the mental constructs (such as memories, models, and beliefs) and their operations are organized and measured based on their interactions with contexts. The context-dependence is taken as the underlying mechanism of students’ behaviors during learning. When using Model Analysis to evaluate student understanding, one does not simply say that a student can or cannot apply a correct model of a given concept. Instead, one states that the student is likely to use a particular model with a quantifiable probability in diverse but expert-equivalent context situations related to the concept. Further, one can begin to assess the effects of specific features of contexts on students’ learning (Bao, Hogg & Zollman, 2002).

Context Dependence of Learning

The significance of context on learning has gained much attention in many fields of education research. One example is the situated cognition, which has its main focus on the interactions between people and the historically and culturally constituted contexts (Lave, 1988; Rogoff, 1990; Resnick, 1991; Greeno, 1992; Clancey, 1993). Learning is understood largely in terms of the achievement of appropriate patterns of behavior, which are strongly tied to and sustained by the contexts.

In the field of physics education, researchers have discovered that in problem solving, many contextual factors can affect students’ responses to problems (Fischbein et al. 1989, Palmer, 1997, Steinberg and Sabella, 1998, Whitelock, 1991). In our research (Bao, 1999; Bao & Redish 2001; Bao, et al., 2002), contexts are not only considered as factors that affect student learning, but are also treated as significant components of students’ knowledge and integrated into the representation framework to model and assess the conceptual development process.

As a continuous effort to understand the interactive relations between contexts and learning, we systematically investigated the effects of specific features of context scenarios on students’ use of mental models related to Newton’s Third Law (Bao et al., 2002). Through research, we were able to identify three types of contextual features, the object’s velocity, mass, and the source of the force (or who is pushing), which are significantly involved in college level students’ reasoning. We also developed instruments that can probe the effects of individual context features on students’ conceptual development


during instruction. Both quantitative and qualitative results indicate that the structures and the development processes of students’ models often vary with different context features. For example, before instruction, most students in introductory level physics courses at Kansas State University have consistent naïve models with all three context features (mass, velocity and pushing). (A naïve model with the context feature of the source of the force reflects the belief that the object that is exerting the force (pushes) exerts a larger force during interaction.) After traditional instruction, most students still hold a consistent naïve model with the context features of mass and velocity; but are in a mixed model state in using the models associated with the context feature of pushing. These results provide important evidence on the effects of context features at much finer grains. As indicated from research (Bao, et al, 2002; Thornton & Sokoloff, 1998; Maloney & Siegler, 1993), the mixed use of models or competing concepts appears to be a typical and important intermediate stage for a student to achieve a complete favorable conceptual change in learning physics. Therefore, measurements of such mixed states can have important values in assessment and instruction.

The study presented in this paper is a further development based on Model Analysis to capture further details of different types mixed use of models. Two different mixed model states, namely the explicit mixed model state and the implicit mixed model state, have been identified. Empirical evidence shows that these two states often reflect different stages of concept development and thus have practical values in both assessment and instruction. Further, a good understanding of the mechanisms of the model mixing can also facilitate the development of effective instruction. In the following parts of this paper, I introduce the theoretical basis, a mathematical representation, quantitative assessment methods, and discuss the design of multiple-choice measurement instruments. Empirical evidence and data analysis in support of this study will be discussed in detail.

It is acknowledged that there are multiple interpretations (perspectives) of what can be defined (or included) as context. In our research, we consider three major categories of context factors:

• Content-based context factors: these are the actual scenarios and specific features of context scenarios employed in or related to the learning of a particular piece of knowledge. For example, in teaching physics, the instructor can use a demo in a lecture. The specific objects (systems), the features of such objects (systems), and how these objects are put together to operate are considered as the content-based context feature of this learning activity. One can also use virtually posed scenarios such as a sample problem on the board or a question to the students. All the features of such examples and questions are considered as the content-based context factors.

• Learning environment context factors: these are specific education settings and features of such settings used in teaching and learning, which may include the class format, teaching/learning style, specific approaches used in learning activities, etc.

• Student-teacher internal cognitive status: these include the students’ and instructors’ general views and attitudes on the learning and teaching of a particular content of knowledge as well as the knowledge background on the content area. Specific items may include motivations, goals, attitudes, epistemological/philosophical views, affective status, conceptual understanding status, etc.

The content and learning environment based context factors are often physically determinable and can be manipulated during the instruction. The relations (differences) between the two types of contexts are that for a particular content-based context scenario (e.g. a physics demo), one can use it in very different learning settings (e.g., a lecture, an interactive lab, a tutorial, etc.). Similarly, one can also teach/learn different content topics through identical or different learning settings. The distinguishment between the two types of contexts allows clearer analysis of how the learning process may be affected by the variety context factors (both content and environment based) during the instruction.


Students and instructors’ internal mental states that are usually not physically determinable and controllable. However, these internal states may be activated or formed in relation with the content and learning environment contexts. As a result, these internal states may not be independent variables but often are not easily changed over a short period of time (e.g., in a class period). Thus, students will often maintain (either explicitly or implicitly) such views and attitudes through out a class unless significantly interfered (e.g. the class explicitly emphasize a particular view/attitude that are not nature to the students). Either way, the time constant for general views to change are often larger than that for students to change understandings of a piece of content knowledge. Therefore, during the learning of a particular piece of content knowledge, one can consider the general views and attitudes as a slow-changing background of the learning process, which can also have significant impact on the learning.

To fully understand the context dependence of learning, we may need to look into all the context issues simultaneously, which can be very difficult to realize in research. Therefore, it is a practical approach to first isolate certain variables and focus on a narrow slice of the problem. The research discussed in this paper emphasized the content-based context and its effects on learning.

II. Explicit and Implicit States of Model Mixing As indicated by research in neural science, features of context (or contextual cues) can produce a

variety of sensory inputs, which are incorporated all together to form episodic memories (Squire and Zola, 1996; Rolls & Treves, 1998). These memories can act as resources for the brain to further develop semantic and procedural knowledge where each piece of episodic memory can be activated partially or as a whole and associated with other pieces of knowledge. This process can be translated into the macroscopic level conceptual development where a student can initially form a type of association (often a naïve type) with a particular context and later learn to reinterpret the same context using a different type of understanding (form different associations). During the learning process, it is possible for a single context to activate a student with multiple mental associations to different pieces of knowledge (including the knowledge formed on the spot).

In cognitive research, we have very limited measurement ability: we can manipulate the actual settings of a context and can inferentially measure certain pieces of knowledge that are activated to one’s explicit awareness (implicit types of knowledge can also be inferred from measurement but such inferences often have much larger uncertainties). The intermediate layers of neural operations, which started from the sensory processing of context cues to the final mental representations that are brought to one’s conscious attention, are largely uncertain to both the external observers (the researchers) and the subjects (the students). Although such neural operations can be guided by back-projection (Rolls & Treves, 1998) from higher-level representations, precise controls of these intermediate layers are usually not achievable among average human beings. In this research, we treat these intermediate stages as a conditional random process, the outcome of which is represented with a distribution of probabilities for specific context settings to activate certain pieces of knowledge into the explicit awareness of the subjects.

In such processes, two unique situations can be observed: (1) different expert-equivalent context scenarios can cause a student to activate or to create different types models related to a concept but each context setting only results in one model; (2) a single context setting can cause a student into explicit consideration of multiple models of a concept.

To simplify the terminology, in the following discussion, the word “cue” will be used to represent the general process that a context setting can cause a student to activate and/or create certain types of knowledge and apply them in the context. So we would say that a context may cue a student into using a piece of knowledge.


Definition of Explicit and Implicit States of Model Mixing In our current research, we have studied students’ use of mixed models in many content areas

including Newtonian mechanics (Bao et al., 2002), probability (Bao, 2001), electricity and magnetism, and quantum mechanics (Bao & Redish, 1998). Through these studies, two unique situations of model mixing have been frequently observed through interviews, open-ended surveys, and multiple-choice tests.

First, consider the following experiment. Suppose we have a student who is in a stage of holding multiple coexisting models of a particular concept. To measure the student’s situation of model mixing, we can use a set of expert-equivalent tasks designed with different context settings. When a set of such tasks is presented to the student, two types of situations are observed in our research:

1. One task always cues the student into using a single model; however, different expert-equivalent tasks can cue the student into using different models. Then, the set of tasks as a whole can still cue the student into using different types of models and thus produce a mixed model state. This situation is defined as an implicit state of model mixing since for each task the student only uses one type of model and does not explicitly consider other models that are also held by the student but are not cued with this task. In addition, the student usually dose not experience any inconsistency in application of his/her models, since each task only involves one model, which provides a local consistency within the domain of the particular task. In this case, the coexistence of multiple models is largely in the eyes of the researchers and the students can find themselves only using one model at a time in perfectly consistent ways.

2. One task cues the student into explicit consideration with multiple models (the student may or may not be able to decide which one is more appropriate to use). In this case, a single task can bring the student into explicit awareness of multiple different models, and therefore is defined as an explicit state of model mixing.

In general, explicit use of multiple models in a single task does not necessarily bring a student into explicit awareness of the contradictory results by applying the different models. Depending on the specific design of the task, it is possible for a student to use different models in different pieces/events of the task and obtain results that does not show apparent conflict. If the task does not require the student to further analyze the potentially contradictory results, students usually don’t actively pursue such in-depth understanding. For example, in interview settings, when students are cued into an explicit mixing state, the researchers can often lead the students to explicitly see the conflict. Instruments such as open-ended surveys and multiple-choice questions often do not probe enough information that allows researchers to make inference on if a student is experiencing conflicts between multiple models on a single question. Such instruments are limited to the probe of students’ uses of certain types of models but cannot provide further information on if those models are the result of spontaneous activations/creations or the outcome of multiple cycles of intentional judgment and evaluation between the multiple simultaneously cued models. Therefore, depending on the goals of a measurement, one should carefully design the instruments as to whether the instruments should cause students into explicitly awareness of conflicts between different models. For example, if the goal of a measurement is to probe the existence of explicit mixing states, it is then not appropriate to use instruments that can cause students into explicit awareness of conflicts, which may “force” the students to choose (or guess) to use only one model. In this situation, individual students’ behaviors vary significantly depending on issues such as their personalities. There are students who never care about any conflicts and can conformable respond with contradictory ideas, whereas some other students may be very persistent in reconciling conflicts encountered in learning. Since the measured results don’t allow researcher to obtain detailed information on how a student may behave in responding to the measurement, the introducing of explicit awareness of conflicts can results in additional uncertainties in the analysis.

As for measurement using multiple choice instruments, a multiple-choice multiple-response (MCMR) type of instrument allows a student to respond with multiple answers that each corresponds to a single but


different model. This type of instrument can be used to measure the existence of both the explicit mixed model state and the implicit mixed model state. On the other hand, multiple-choice single-response (MCSR) type of instrument forces a student to respond with only one answer. Therefore, one can only measure implicit mixed model state using a MCSR instrument.

Values of Identifying the Differences in Model-Mixing States The significance of studying the two states of model-mixing can be considered from two directions.

First, from the research and assessment points of view, the two states may imply two different stages of conceptual development. Our research data has shown that explicit model mixing state often reflects a step of advancement comparing to the implicit state (results will be discussed in later parts of this paper). Therefore, in evaluation of a student’s progress in conceptual development, identification of the two different states can provide a rubric for researchers to evaluate a student’s status of knowledge.

From the instruction point of view, it has been well documented in the literature that a crucial step in helping students achieve correct concept change is to provide situations where students can be confronted by conflicting evidence for their existing models and be brought into dissatisfaction (Posner, 1982, McDermott, ). Such situations often cue students into an explicit state of using multiple models, and when properly designed, can also bring students into explicit awareness of conflicts, which is often a significant condition to create the need for students to change current conceptual understanding (Posner, 1982). In addition, explicitly cued mental constructs can be systematically attended to by students and integrated in their reasoning to create opportunities for modification or revision of such constructs in learning (mental constructs such as certain reasoning primitives that are implicitly involved in learning are often more difficult to be address). Therefore, in instruction, an assessment method that identifies the explicit mixing state as well as the context settings that cue such states can provide valuable information to instructors.

Evidence from Empirical Research Our empirical studies on students’ difficulties in learning physics have provided a large amount of

results showing that the students can be cued into the two types of mixing states. For example, in the study of the effects of context features on students’ understanding of Newton’s Third Law (Bao et al., 2002), we interviewed 9 students and also surveyed about 300 students using a model-based multiple-choice test. Both the interview protocol and the multiple-choice questions are designed to measure the independent effects of individual context features including the velocity of the object, the mass of the object, and the source of the force (or who is pushing), and to probe the possible situations of model mixing.

From the multiple-choice survey data of a calculus-based introductory physics class after lecture-based instruction, about 60% of the students used consistently their naïve models of Newton’s Third Law in six questions with context settings involving features of mass and velocity. Meanwhile, about 80% of the same population appeared to apply both the naïve model and the expert model with nearly equal probabilities on questions involving the feature of “pushing”. Since the format of the questions is multiple-choice single-response (MCSR), the data can only reflect the existence of implicit model mixing state. Still, the results demonstrate that strong model mixing can be cued by the context feature of “pushing”, and that different context features can have significantly different effects on students’ development of their understandings.

The evidence for explicit mixing states was found in the data of the nine interviews conducted with students in an algebra-based introductory physics class. In the interviews, five of the nine students used consistently the naïve models in all questions involving the three context features. Further analysis of these students’ reasoning shows extreme dominance of their naïve models and no signs of consideration of a different type of model in their reasoning. The remaining four students were found to use both the expert models and the naïve models. Two of them were in an implicit mixed model state where they used different models on different questions. The other two students were found to be in an explicit mixed


model state where a single question cues them to considering both the naïve model and the expert model simultaneously. The different effects of context features on students’ model states were also observed: For the two students with an implicit state, one is cued by the questions involving velocity. The other one is cued by the feature of “pushing”. On the other hand, the two students with explicit states were cued into consideration of both expert and naïve models by every single question that each involves one of the three context features.

It often seems unreasonable that two similar questions can cue a student into using different types of reasoning. But, this type of phenomenon is frequently observed in research and cannot be interpreted as errors but rather features of unique mental states (Bao & Redish 2001). For example, consider the two questions used in our interviews (see Figure 1 and 2). They both involve a single context feature, the velocity, and are virtually identical. In the interview, the two questions are separated by three other questions, which allows each question to have a more independent initiation of reasoning. This is an effort made to probe if students actively review previous tasks and pursue consistency in their reasoning. During the interview with Jane, she spontaneously applied the expert model in the first question (Figure 1) and came to the conclusion that both cars exert an equal amount of force. Later in the second question (Figure 2), she responded without any hesitation that the ball travels faster will exert a larger force. When asked if she compared the two questions, Jane responded with,

“I kind of have a little impression but I am not really thinking about that (the first question) when I am working with this question (the second question).”

She then recognized the similarity of the two questions and realized the conflict between the two responses she made before.

“Oh yah, they (the two questions) are the same … they (the two objects in the questions) have the same mass and one is going faster… I guess the force has to be equal and opposite then, because the force is equal and opposite, the professor showed it in class with two carts (collision experiment). But I am not sure. I guess the ball should also have equal and opposite force but I am really not sure.”

Apparently, Jane was initially in an implicit state where it seems likely that the context setting similar to the one used in the class was able to cue her to use the correct model. With some intervention by the interviewer, she was brought into an explicit awareness of her use of different models. In this particular example, we were able to observe a transition from an implicit state to an explicit state. That is after the intervention the second question (Figure 2) cues her into thinking of both models simultaneously. However, the first question (Figure 1) still cues only the correct model.

As shown in the figure, a collision happens between a small pickup truck and a car. The small truck has the same mass as the car does. At the time of collision, both vehicles travel at a constant speed but the small truck is moving at a slower speed than the car. Describe the forces at the moment they collide.

Figure 1. Interview question on Newton’s Third Law

Max’s Marble Jimmy’s Marble

Two boys, Jimmy and Max, are tossing identical marbles at each other. The Max is a little stronger than the Jimmy, so the marble from him goes faster than the marble from the Jimmy. During one nice shot, the two marbles, one from the Max and one from the Jimmy, collide in mid air. Discuss the forces at the moment they collide.

Figure 2. Interview question on Newton’s Third

Law (a different but equivalent context)


With the same two questions, Jennie (another student we interviewed) was cued into an explicit state in both questions. On the question shown in Figure 1, she responded with,

“I would think the car that travels faster will exert a larger force, but we know that the force has to be equal and opposite… but I think the velocity will matter. I am not sure.”

She also recognized the similarities between the two questions by herself.

“This question (the second question) is like the other one. So the force should be equal and opposite but I am not sure. The velocity should also matter.”

So for Jennie, each of the two questions cued her into explicit awareness of different models as well as the conflicting results. Other questions involving the feature of mass or “pushing” also cue her into explicit states similar to the situation with velocity.

Two additional issues were also implied from the interview results: A large number of students (7/9) didn’t actively seeking consistency of reasoning across different questions. The two students who did so all had explicit model mixing state. It is found that the students with implicit mixing states out-perform the students with consistent naïve models in terms of their grades (which is a natural result of having mixed models – more probabilities to produce correct answers) and especially the quality of their explanatory reasoning in interviews. The students with explicit model mixing states seem to have an even more advanced level of conceptual development with respect to the quality of their reasoning and their strategies of actively seeking consistency across contexts. This implication is consistent with the results observed by Thornton (1994). However, we only had a small number of students for interviews and the results may not represent the general features of a population.

Results from other studies have indicated that in learning of a particular concept, students often start with certain naïve models and go through a stage of mixed models before achieving a favorable concept change (Bao, 1999; Bao & Redish 2001; Bao et al. 2002; Thornton, 1994). With the results from this study, we can further identify two states within the stage of mixed models. Then using model-based assessment and Model Analysis, we can distinguish a single student’s level of conceptual development with four progressive marks: consistent naïve model state, implicit mixed model state, explicit mixed model state, and consistent expert model state. It is also observed that students with very weak background on the content knowledge may not even have a naïve model, in which case the measurement will not indicate any consistent naïve models but many tentatively created inconsistent ideas. Usually, the science major undergraduate population would start with much consistent naïve models. We have observe a slight shift toward inconsistent naïve models with non-science major undergraduate populations (Bao, et al, 2002).

In addition to the topics in mechanics, our research on students’ understanding of classical probability and quantum probability also reveals similar results (Bao and Redish, 2002). In a calculus-based introductory modern physics class at the University of Maryland, we implemented Tutorials to help students develop an understanding of classical probability and learn to use probabilistic representation to interpret physical system classically and quantum mechanically. After instruction, we gave two multiple-choice multiple-response (MCMR) questions in the final exam to probe students’ use of the classical and quantum models of probability. The data from the questions shows that about 20% of the students were cued into an explicit mixed state where they applied both the classical and quantum models to the situations that only one of the models applies. This result is consistent with the interviews we conducted with volunteer students from the same population.

Additional examples can be extracted from the studies on other topics of physics such as quantum wavefunction (Bao, 1999; Bao & Redish, 1998), conductivity and band structure (Redish & Bao, 1998), and mechanical waves (Wittmann, Steinberg & Redish, 1999). Based on these results, it can be inferred that the activation of implicit and explicit mixed model states represents a common phenomenon that occurs across many different content domains of physics.


III. Mathematical Representations and Assessment Tools To go beyond the qualitative description of the mixed model states, it is important to develop a mathematical representation for quantitative analysis and assessment. The methods developed in the existing framework of Model Analysis can represent mixed model states but do not distinguish between the explicit and implicit states. In a recent study (Bao, 2001a), a new extension of Model Analysis has been developed, which includes the methods to represent the explicit and implicit states. The following sections give a brief review of the new method and its applications.

Context-Explicit Model Analysis The new development of Model Analysis is built on an associative network structure in which the contexts are explicitly integrated in the representation and analysis. The new method is referred as Context-Explicit Model Analysis (CEMA). It provides a tool to study not only how students apply their models in contexts but also how those models are created in terms of students’ interactions with contexts in both the school environment and the real world. Again, the content-based contextual issues are emphasized when we use the term context. CEMA is developed based on the belief that an individual’s understanding of a particular concept is developed through generalization of a large number of concrete physical examples from experiences and other resources. In such generalization processes, one often needs to analyze the contextual features of examples and cue the relevant pieces of knowledge in long-term memory to develop an explanation for the events in the examples. Therefore, to understand and represent such processes in learning, we have to explicitly address three areas of issues: the specific physical examples (scenarios) that students use in their reasoning, the context features that students involve in their interpretations of the examples, and the models that students apply to generate explanations. For a particular concept, the associated context factors and mental constructs can be identified through systematic qualitative research using model-based instruments such as open-ended conceptual surveys and student interviews. With these components, we can build a representation to describe the different states of students’ conceptual development – a particular state of a student’s conceptual understanding is represented as a particular pattern of associative relations between different physical examples (scenarios), context features and models. For example, if a student hold a naïve model and apply it consistently across all relevant contexts, the association pattern will be in a form that all the physical examples are linked to the naïve model. Figure 3 shows four representative states of students’ associations between context scenarios and models: a) a consistent (pure) expert model state (M1 is assumed to be the expert model); b) a consistent (pure) naïve model state; c) an implicit mixed model state (each scenario only cue one model but different scenarios can cue different models); d) an explicit mixed model state (each scenario can cue multiple models).

When setting up the set of common models, each unique context feature (referred as a physical feature) can be

C1

C2

C3

C4

C5

Cn

M1

M2

Mw

Models Context

Scenarios

C1

C2

C3

C4

C5

Cn

M1

M2

Mw

Models Context

Scenarios

C1

C2

C3

C4

C5

Cn

M1

M2

Mw

Models Context

Scenarios

C1

C2

C3

C4

C5

Cn

M1

M2

Mw

Models Context

Scenarios

a) pure correct state b) pure incorrect state

c) implicit mixed state d) explicit mixed state

Figure 3. Patterns of associations between contexts and models


treated as an independent dimension that has its own sub-space of common models (Bao, 1999; Bao et al., 2002). The schematic diagram shown in Figure 4 is a simplified example that involves three common models and four context scenarios. It gives the representation for a particular dimension of physical feature (identified with η), where C1…C4 represent the set of context scenarios and Mη

0…Mη2 represent the

identified common models associated with the particular concept. The weights on the associations describe the probabilities of how a single student or a population is cued by specific context scenarios into using the different models. The associations can also be represented with an association matrix denoted as A. It gives the associatitive relations between the context scenarios, which can be represented with a context input vector C , and the non-normalized model probability vector M~ . For a particular

physical feature identified with η, Equation (1) gives the general format of the association matrix between L context scenarios and w models.

=

ηηη

ηµ

ηµ

ηµ

ηηη

η

wLwlw

Ll

Ll

aaa

aaa

aaa

��

��

��

��

��

1

1

1111

A (1)

Here, ηµla gives the association strength (in probability) between the µth context scenario and the lth

model. Each column of ηA represents the distributed probabilities of associations between a particular context scenario and different models. Therefore, all the elements in a column add up to 1.0, which can be

expressed as∑=

=w

iija

11(details of the normalization of the associative probabilities will be discussed in

later sections). The context input vector and model state vector for feature η are denoted with ηC and ηM~ respectively and are defined in equations (2) and (3).

=

=scenariocontext in theabsent isfeature0scenariocontext in thepresent isfeature1

where;th

th

1

ll

C

C

C

C

l

L

l ηηη

η

η

η

η

o

o

C (2)

=

η

ηµ

η

η

wM

M

M

~

~

~

~

1

o

o

M (3)

C1

C2

C3

M1η

M2η

M3η

a11

a21

a31 a33

a23

a13 a12

a22

a32

null

naïve

expert

C4 a34 a24

a14

Figure 4. Schematic of a 3 by 4

association matrix


Here, ηµM~ gives the non-normalized probability for a student to apply the µth model to the set of context

scenarios represented by ηC . The normalization can be calculated with equation (4)

∑=

= w

M

Mp

1

~

~

µ

ηµ

ηµη

µ (4)

where ηµp represents the normalized probability, which can be used to obtain the model state vector

=

η

η

η

η

wp

pp

�

2

1

u (5)

Obviously,

1=ηη uu (6)

The associative relations between the context scenarios and the models are described in equation (7).

ηηη MCA ~=⋅ (7)

Using the model state vector and the algorithms developed in Model Analysis (Bao & Redish, 2001), we can quantitatively analyze students’ learning in terms model states and changes of model states.

Matrix-based Analysis of Explicit and Implicit Mixed Model States The main difference between the explicit and implicit state is the number of models that can be cued by a context scenario for a single student. As an example, Figure 5 shows two types of association patterns corresponding to the two states: Figure 5a represents an explicit mixed state where each context scenario can cue a student into two models; Figure 5b shows an implicit mixed state where each context scenario only cue a student into one model.

Using the mathematical representation, denote the association matrices for the two states with Aη

1 (explicit) and Aη2 (implicit).

The numerical forms of the two matrices are obtained in Equations (8) and (9).

=5.05.00.00.00.00.05.05.05.05.05.05.0

1ηA (8)

0.50

0.50

0.50

0.50

0.50

0.50

a). Explicit mixed state

C1η

C2η

C3η

M1η

M2η

M3ηC4

η

0.50

0.50

null

naïve

expert1.0

1.0

1.0

1.0

b). Implicit mixed state

C1η

C2η

C3η

M1η

M2η

M3ηC4

η

null

naïve

expert

Figure 5. Examples of explicit and implicit mixed states


=0.10.00.00.00.00.10.00.00.00.00.10.1

2ηA (9)

As we can see, the association matrices for the two states show very different structures. In the implicit case (Aη

2), only one element of a column has a non-zero value which equals 1. On the other hand, depending on the mixing status, a column of Aη

1 can have several or all of its elements with non-zero values. Therefore, the association matrix contains direct quantitative information on the structure of a student’s model state (e.g. if a state is mixed, how it is mixed, and association probabilities between specific context scenarios and models).

Suppose in a particular measurement, all context scenarios are present. Then all components of the context input vector ηC are set to 1:

=

1111

ηC (10)

Using equation (7), we can calculate the non-normalized model probability vectors (the summation of the probabilities equals the number of scenarios used in the measurement):

=

⋅

=⋅=0.10.10.2

1111

5.05.00.00.00.00.05.05.05.05.05.05.0

~11

ηηη CAM (11)

=

⋅

=⋅=0.10.10.2

1111

0.10.00.00.00.00.10.00.00.00.00.10.1

~22

ηηη CAM (12)

The normalized model state vector as defined in Model Analysis can be obtained using equations (4) and (5):

==112

21

21ηη uu (13)

The model state gives the distribution of the amplitudes of the probabilities for a single student to be cued into using the different models by the whole set of context scenarios used in a measurement. It does not contain details of how specific context scenarios are involved and if a mixed state is explicit or implicit. However, a model state is much more compact (in terms of data size) than an association matrix, which makes it more efficient as an assessment variable for measuring large classes. For example, based on our experience (Bao, 2001a), the number of a typical set of commonly used context scenarios associated with a particular physics concept is between 7 and 12. Usually, there are about 3 to 5 different


models. Then the size of the association matrix for a single student is at least 3-by-7, which contains 21 data cells. Under the same situation, a student model state is a 3-by-1 vector with three data cells. Apparently, a model state vector is more efficient in data storage but at the cost of less information.

Assessment Instruments and Numerical Methods

For assessment purposes, a good method should balance between the richness of information and the efficiency of data storage. First, let’s consider the assessment instruments. When properly designed, most of the popular instruments such as multiple-choice questions, open-ended surveys, and interviews can be used to measure students’ models. In this study, the use of multiple-choice questions is emphasized due to three reasons: (1) multiple-choice questions can be easily implemented in a large class; (2) the analysis of the data is often more standardized (objective) than the results from parsing students’ responses in open-ended surveys and interviews by individual researchers; (3) the first two advantages also make it possible to use multiple-choice questions regularly in a large class to obtain rapid (or even real time) feedback on student learning during instruction.1

In general, there are two popular types of multiple-choice questions: multiple-choice single-response (MCSR) and multiple-choice multiple-response (MCMR). To measure the explicit and implicit mixed-states, we need to use the MCMR type, which allows the students to simultaneously respond with multiple answers that are associated with different models. In addition, a MCMR question also requires a student not only to decide which answers are correct but also to determine which answers are inappropriate. This often involves more extensive reasoning than what is required by a MCSR question. With a MCMR question, a student is likely to search for several possible models that he/she considers most relevant and apply them to obtain answers to the question. If a question is appropriately designed based on research, the choices of the question can capture the student’s cueing and application of his/her models.

A single MCMR question can measure students’ use of multiple different models where each model are mapped to one or several choices designed to match the typical outcomes of applying the model in the context scenario of the question.2 Since a question establishes a single context scenario, we consider that the spontaneous model-cueing process only occurs once with each question. However, this cueing process can simultaneously bring up several models. When a particular model is cued to the student’s mind, the related choices (which can be several) of the question are considered as the outcomes of possible ways of applying of the same model. As a result, in data analysis the number of choices picked by a student that correspond to a single model is usually not used as quantitative weights to calculate the probability components of the student’s model state. Therefore, with a single MCMR question, we only extract the information on which ones of the related models are cued by the particular context scenario and do not determine if some cued models may have larger probabilities than others – when several models are simultaneously cued, an identical probability is assigned to each one of the models and the total probability is normalized to 1.

For example, consider a single MCMR question that involves three models. For simplicity, only one dimension of context feature is considered in the following discussion (thus η is ignored from the model state vector and the related mathematical expressions). Then, a single student’s response to this question can be written as a three-dimension response vector kr :

=k

k

k

k

yyy

3

2

1

r (14)

where k = 1, …, N represents the kth student in a class of N students and kyµ (µ = 1, 2, 3) has a binary value (equal to 0 or 1) depending on if any of the choices corresponding to the µth model are selected. For


example, if the kth student didn’t pick any of the choices associated with the µth model, kyµ is set to 0 to indicate that the µth model was not cued. When the kth student picked at least one of the choices associated with the µth model, kyµ is then set to 1. With kr , the student model state vector can be calculated:

=

=

∑=

k

k

k

k

k

k

kk

qqq

yyy

y 3

2

1

3

2

1

3

1

1

µµ

u (15)

where ∑=

=3

1µµηηkkk yyq represents the normalized probability for the student to be cued into using the ηth

model. For example, if a student answers the question using only the first model, we have:

=

=001

001

kk and ur (16)

If a student uses both the first and the second model to answer the question, we then have:

=

=011

21

011

kk and ur (17)

Here, the interpretation of kr is straightforward: the question will cue both the first and the second

model with probabilities equal to 1. On the other hand, the model state vector ku reflects the normalized probabilities, which in this example gives:

05.05.0

(18)

Interpretations of the Probabilities Represented by a Model State The interpretation of the normalized probabilities involves a fundamental question of measurement

constraints and philosophical views on what is being measured. The philosophy here is much similar to that of quantum mechanics. A student is considered to have an internal mental structure that has intrinsic uncertainty to any observers (both internal and external) and has no well-defined static states. All the mental states that we have talked about are the results of measurements with specific contexts which collapse the mental “wavefunction” and produce certain observable states – i.e., when talking about any observables of students’ mental processes, we are explicitly constrained by the specific measurements that are to be conducted.

In this particular example, the probability distribution of students’ using different models is the result of the measurement. Without (before) the measurement, we don’t imagine any readily established probability distributions in the student’s mental structure. Then using multiple measurements with diverse context settings on a single concept, we may obtain an estimation, based on ensemble measurement results, of the probability distribution of how this student may use different models. This result can be


defined as the expectation value (actually a vector) of the probability distribution, which we assume to reflect features of the internal mental structure.

In addition, the measurement of the probability distribution vector has a resolution issue constrained by the number of questions used in a measurement (Bao, 1999). For example, the uncertainty for a two-dimension model space is m2π , where m is the number of questions used in the measurement, provided all questions are MCSR type. When only one MCSR question is used, it can only detect the existence of one of the two models and has no information on if at all a student may be in a mixed state. Therefore, the uncertainty is 2π − i.e., the student’s model state can be a vector anywhere between 0 and 2π . As for MCMR questions, they can each measure the existence of an explicit mixed state, and therefore, a single question has the resolution of 4π which has to be achieve with two MCSR questions. When a single MCMR question is use and produces the results of an mixed state as shown in Eq. (17), although the expected probability distribution vector can be anywhere in the model space, without further measurement, we cannot determine more precisely the expected distribution than making it at 4π in between the two base dimensions. This can ensure that the uncertainty between the measured results with a single question and the expected probability distribution vector (which is supposed to be obtained with ensemble measurements) is less than 4π .

It is then useful to make the MCMR instruments consistent with the MCSR instruments which reflect more like a quantum measurement – with a single measurement one usually gets one observable state from a single particle which can be in a mixed state. To do so, the probabilities reflected in any columns of the association matrix and the model state vector are normalized such that they no longer represent the actual likelihood for cuing a particular model by a single question (or a set of questions) but rather reflect how much likely if a student will use the model when only one answer is allowed. In the example discussed in equation (18), the normalized probabilities show that the student is equally probable to use the first two models if he/she has to make a choice. However, using a MCSR question does not measure the information on if a student is cued with multiple models by this question. By using MCMR questions and the format of normalized probabilities, we can measure the cueing of multiple models and keep the model state vector consistent with its original definition (Bao, 1999; Bao & Redish, 2001b).

Measurement and Analysis of a Single Student and a Population To more accurately determine a student’s model state, it is necessary to use a number of expert-

equivalent questions designed for measuring the same set of models with different context scenarios (Bao & Redish, 2001b). By analyzing a single student’s responses on the complete set of questions (usually 3~5 questions), we can obtain a good estimation of the student’s model state: If a student always uses one model consistently on all the questions, the student is then considered to have a “pure” model state. If a student uses multiple models on the set of questions but each question still cues one model, the model state will be an implicit mixed-state. When a single question can cue more than one models, the model state is then an explicit mixed-state.

As discussed earlier, using model state alone won’t distinguish between explicit and implicit mixed states. Therefore, a new treatment is implemented: for each question, a model state is estimated and a model density matrix is calculated. Then the model density matrix for the different questions is summed together and normalized to provide the model density matrix for the question set. This matrix retains all the information needed to represent the three different types of model states: pure states, implicit mixed state and explicit mixed state.

Suppose we measure a concept topic that involves three common models and we give a set of m questions to a class. The responses of the kth student on the ith question is mapped to a model state vector


represented with kiu where i = 1, …, m. Then we can form a single student model density matrix, denoted with DDDDk, using the student’s model state vectors obtained from all m questions:

∑∑

∑=

=

=

==m

i ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

ki

m

ikikik

yyyyyyyyyyyyyyy

ymm 132313

32212

31211

3

1

1

111

µµ

uuDDDD (19)

If the kth student has a pure model state, all kiu should be identical and have one of the three forms shown below:

100

or ,010

,001

(20)

The corresponding DDDDk is then:

100000000

or ,000010000

,000000001

(21)

Consider the situation for a student to have an implicit mixed state. Suppose the student was measured with five questions (m = 5) and responded two questions with the first model, two questions with the second model and one question with the third model. Then we obtain five model states as shown below:

=

=

=

=

=100

,010

,010

,001

,001

54321 kkkkk uuuuu (22)

Using equation (19), DDDDk is found to be:

=2.000

04.00004.0

kDDDD (23)

Now assume the student is in an explicit mixed state and he/she responded two questions with both the first and the second model, one question with both the first and the third model and two questions with all three models. The five model states can be found as:

=

=

=

=

=111

31,

111

31,

101

21,

011

21,

011

21

54321 kkkkk uuuuu (24)

Then the DDDDk is calculated to be


=

=233.0133.0233.0133.0333.0333.0233.0333.0433.0

7474101071013

301

kDDDD (25)

As shown from this example, DDDDk has very different structures corresponding to the different types of model states. When a student has a pure state, only one diagonal-element of DDDDk has a non-zero value that equals 1. The position of this diagonal element shows which model the student is using consistently. When a student has an implicit mixed state, DDDDk is a diagonal matrix and the values of the diagonal elements reflect the probabilities for the student to use the different models consistently within a single arbitrary question from the question set. If a student has an explicit mixed state, DDDDk has non-zero off-diagonal elements, which reflect the “strength” of the simultaneous cueing of the different models. The diagonal elements still give the overall probabilities for the student to use the different models but under the condition that multiple models are applied simultaneously. In this case, the ways a student apply his/her models cannot be seen directly from the density matrix and need to be extracted with additional methods such as eigenvector analysis. For all the situations discussed above, the diagonal elements always add up to 1 which reflects the total probability.

Extracting and Representing Student Model State • Single student model states

In real measurement, the model density matrices often show certain complicated mixing features. It is then difficult to obtain a direct quantitative evaluation with the density matrices on how students apply their models. To deal with this situation, we can perform eigenvalue analysis on the density matrices and use the eigenvectors and eigenvalues to evaluate the students’ ways of using their models. The detail of this method has been discussed extensively in the references (Bao, 1999; Bao & Redish, 2001b) and the following discussion will emphasize the interpretations of the results.

Using the example discussed in the previous section, suppose a student uses the second model consistently. The measured model density matrix will be:

000010000

(26)

Perform eigenvalue decomposition and we get one non-zero eigenvalue and one eigenvector:

=

==010

vvv

;0.1

31

21

11

11 vλ (27)

This can be interpreted as that the student has a probability of 1.0 to be cued with the second model by the set of questions.

For the case of implicit mixed-state (see equation-23), the eigenvalue decomposition will produce three eigenvalues and eigenvectors:


=

=

=

=

=

=

===

100

vvv

010

vvv

001

vvv

2.04.04.0

33

23

13

3

32

22

12

2

31

21

11

1

321

vvv

λλλ

(28)

Here, the eigenvectors represent the models that the student will apply consistently in a single question and the eigenvalues give the probabilities for the cueing of the corresponding models with the question set.

In the two simple cases discussed above, one can see directly the eigenvectors and eigenvalues from the density matrix. However, when a student has explicit mixed-state, the model density matrix becomes more complicated. With the density matrix shown in equation (25), the eigenvalues and eigenvectors are found to be:

−=

−=

=

===

41.058.070.0

82.058.001.0

40.058.071.0

02.014.084.0

321

321

vvv

λλλ

(29)

Here, the largest eigenvalue is close to 1, which implies that the student behaves rather similarly on the set of questions, e.g., the students applied the first two models on four out of five questions, and the first eigenvector alone contains most of the information of the student’s model state (Bao, 1999; Bao & Redish, 2001b). Based on equation (15), the square of a component of an eigenvector has the unit of probability. Therefore, 1v can be converted to a probability vector:

16.034.050.0

(30)

This particular state can be interpreted as that the set of questions can simultaneously cue the student into using all three models – an explicit mixed state. If the student has to choose only one model, we expect his/her choice with 50% chances to be the first model, 34% chances to be the second model and 16% chances to be the third model. The large eigenvalue of 0.84 in this example indicates that this model state represent the most significant features of the ways the student applies the different models.

When the eigenvalues are more evenly distributed, it indicates that the student is using very different sets of knowledge in responding to different questions. In this situation, we need to combine the eigenvalues with the model states in the analysis (similar examples on class model states are discussed in Bao, 1999).

• Population model states

The method discussed above can also be used to evaluate the performance of a class. To do so, we first construct a model density matrix for a class:

∑=

=N

kkE N 1

1DDDDDDDD (31)

Here, DDDDE is the explicit-state class model density matrix and is fundamentally different from DDDD used to represent the original definition of class model density matrix (Bao, 1999; Bao & Redish 2001b) and will


be called the general-state model density matrix. To calculate DDDDE, each individual question measures a model state and the questions set gives a single student model density matrix, which is then used to obtain the class model density matrix (see equations 20 and 31). In the case of DDDD, the whole set of questions together measure a model state of a single student. Suppose we use m questions in the measurement. Then for the kth student on the ith question, we can obtain a normalized probability vector k

iQ

=

=∑

=kwi

ki

ki

kwi

ki

ki

wki

ki

q

qq

y

yy

y ��

2

1

2

1

1

1

µµ

Q (32)

This vector can be obtained with both MCSR and MCMR instruments and follows the calculation described in equations (14) and (15). In the case of MCSR, only one of the components is non-zero and equals to 1. To obtain DDDD, we can first calculate kQ , the normalized probability vector for the whole questions set:

=

= ∑=

kw

k

k

m

ikwi

ki

ki

k

q

qq

q

qq

m ��

2

1

1

2

1

1Q (33)

The kth student’s model state vector for the question set can be obtained as:

=

kw

k

k

k

q

qq

o

2

1

u (34)

where qµk (µ = 1, …, w) gives the measured probability for the kth student to apply the µth model.

Obviously, we have

11

==∑=

wk

kk qµ

µuu (35)

Then DDDD can be obtained:

∑=

=N

kkkN 1

1 uuDDDD (36)

As we can see, the calculations to give DDDDE and DDDD are intrinsically different. In table 1, several examples are shown to illustrate the differences between the two calculations as well as the implications of the results (all the matrices are examples just to show the structural features – the significance of the matrix elements is either zero or non-zero values). When MCSR questions are used in the measurement, the method of DDDDE will not identify any model mixing of the population. As shown in table 1, DDDDE has the same structure when the class in pure multiple-model situation and in implicit mixed state. On the other hand, using DDDD with MCSR questions will show different results for the two situations of student model


state: in the case of pure multiple-model, DDDD is a diagonal matrix whereas in the case of implicit mixed-state, DDDD has non-zero off-diagonal elements (see table 1). However, the method of DDDD does not distinguish between explicit and implicit mixed state. To do so, MCMR questions are needed. As we can see, none of the two methods can cover the complete range of situations. Therefore, we need to use both types of calculations to analyze a class: use DDDDE to identify the explicit and implicit mixed state and use DDDD to distinguish between pure multiple-model situation and any types of mixed states. By combining the two methods, we can measure and analyze the full spectrum of the different situations of a population’s model states.

Table 1. Differences between DDDD and DDDDE

Student Model Types of Model States DDDD DDDDE

I. Single pure state – all students use one model consistently.

000000001

000000001

A single student always use one model for all the questions but different students may use different models.

II. Multiple pure states – a single student always use one model consistently on all questions (different students may use different models).

2.00003.00005.0

2.00003.00005.0

III. Implicit mixed-state – a single student use one model on one question but may use different models on different questions.

2.01.01.01.03.02.01.02.05.0

2.00003.00005.0

A single student can use multiple models. IV. Explicit mixed-state – a single

student can use multiple models on one question.

2.01.01.01.03.02.01.02.05.0

2.01.01.01.03.02.01.02.05.0

After obtaining the class model density matrix (either DDDD or DDDDE), we can again perform eigenvalue analysis. The resulting eigenvectors represent the model states of the class, which reflect the popular features of the individual students’ model states. Due to the differences in the calculations of the class model density matrices, the class model state vectors of DDDD and DDDDE have different interpretations: with DDDD, a pure state (one non-zero vector component) indicates a consistent use of a single model and a mixed state (multiple non-zero vector components) corresponds to an implicit mixed state if MCSR type of questions were used or an undetermined mixed state (could be explicit or implicit) when MCMR type of questions were used. In the case with DDDDE, one has to use MCMR questions. Then implicit mixed states would be represented with a set of eigenvectors that each has only one non-zero component. In this case, vectors with multiple non-zero components correspond to explicit mixed states.

Representing Student Model States with Model Plots

When studying students’ learning of a particular concept, it is often useful to monitor how students change from a common naïve model and toward an expert model during instruction. In such cases, using a graphic tool called model plot can easily represent the features of the students’ model states (Bao, 1999; Bao & Redish 2001b). A model plot is a two-dimensional graph whose two axes represent the probabilities that a student would use the corresponding models on one of the questions used in the measurement. For example, suppose we study the first two models in a 3-model situation. Then a model state, Vµ = (v1µ, v2µ,, v3µ)T, can be represented by a point (e.g. point A in Figure 6) on a model plot with λµv1µ

2 as the vertical component and λµv2µ2 as the horizontal component. A model plot can represent

student model states of three different types of measurements:


(1) Single student’s model states measured with MCMR questions and obtained with DDDDk

In this case, the student’s model states are obtained by eigenvalue decomposition, which produces three eigenvectors. We only need to plot two of the eigenvectors that have large values on the components of the two models being considered. The effects of the model state that is not plotted can be derived from the other two models (Bao, 1999). Using a model plot, we can graphically represent the following different situations of a single student’s model state:

(i) a consistent pure model state – a student uses one model consistently on all questions. In this case, only one non-zero eigenvalue will be found. The eigenvector is one of the base vector.

(ii) implicit mixed states – a student uses one model on some questions and another model on other questions (for each question, only one model is used). In this case, one can find multiple non-zero eigenvalues. The associated eigenvectors are the base vectors.

(iii) explicit mixed states – a student can use multiple models simultaneously on a single question. In this case, one can find multiple non-zero eigenvalues and the eigenvectors are superpositions of the base vectors.

In a three-model situation, Figure 6 shows the features of the model plot corresponding to the three types of model states. The definitions of the different model state regions shown in the diagram are meant to provide a qualitative sense for reading such plots.

(2) Class’s model states obtained with DDDD

In this situation, the states on a model plot represent the model states of the population, which reflect the common features of the individual student’s model states: the model plot can identify if the majority of the population hold consistent pure model states or mixed states in general (no difference between implicit and explicit mixed state). Much of the features in this case have been discussed in references (Bao, 1999; Bao & Redish 2001b) and are summarized in Figure 7.

1

1

0 Probability for using Model 2 (incorrect)

λµv1µ2

λµv2µ2

Prob

abili

ty fo

r usi

ng

Mod

el 1

(cor

rect

)

A

Explicit Mixed Region

Consistent Model 1 Region


Implicit Mixed Region

A pair of explicit mixed statesA pair of implicit mixed statesA pure state

Figure 6. Explicit-state model plot for showing single

student’s model states

1

1


Prob

abili

ty fo

r usi

ng

Mod

el 1

(cor

rect

)

Mixed Region



A pair of mixed statesA pair of pure states

Figure 7. General-State Model Plot for showing

class model states


(3) Class’s model states obtained with DDDDE

Again, the states on a model plot represent the model states of the population but also reflect the features of the individual student’s states. In this case, we can use the model plot to show the existence of explicit mixed states; however, the difference between two situations cannot be distinguished: (1) most students in the class have implicit mixed states; (2) there are several groups of students that each has a consistent pure mode state but different groups have different pure states. (see Figure 8).

From the above discussion, we can see that with MCMR questions we can measure all three different situations of a single student’s model states. To obtain the same information from a class’s data, we need to combine the different analysis methods: use DDDD to measure distinguish between pure and mixed states in general and use DDDDE to identify the explicit and implicit mixed states.

Model States Statistics

The methods of model density matrix and eigenvalues analysis of the density matrix are very good tools under two situations: (1) The population exists in groups of students each holding a consistent model. In this case, the eigenvalues represent the sizes of the groups and the eigenvectors are very close to the base vectors of the model space. (2) The majority of the population behaves very similarly with either a pure state or mixed states. In this case, the primary eigenvalue is very large (>0.8) and the corresponding eigenvector alone can represent the most significant features of the population.

When students are in mixed states but are in different mixed states (different but not orthogonal), the eigenvalues of the density matrix will be more evenly distributed. In this case, the primary eigenvector often represents the most common (average) features of the individuals’ mixed states which are also different from one another. The additional eigenvectors show the major differences between the individuals’ states and the primary eigenvector. If the differences between the individuals’ states are of the interests, the eigenvectors are not a good tool to extract such information, since the different individual states are not orthogonal and there exist a wide variety of differences among the individual states, which goes far beyond the dimensions of the model space. To extract such information, one may simply do statistic analysis or cluster analysis of the measured individuals’ model states.

First, we need to expand the dimensions of the analysis to account for the many differences among individual states. One way to do so is to establish a state-space, in which each base dimension represents a unique type of model state that is to be studied. Notice that a state-space is spanned by the different model states which are vectors in the model space. A model space is spanned by a set of common models. The structure of a state-space is highly dependent on the types of measurement (and the number of questions used) and the number of common models. For example, if there are only two common models and a single MCMR question is used to measure the model state, the components of the model state vector can only be 0 or 1 for pure states or 21 for an explicit mixed state. Then, the number of all the possible model states is 22 −1 = 3 (the subtraction of 1 reflects the removal of the empty vector with all zero components). If multiple questions are used to obtain the model states, there would be a range of possible values for each component of a single model vector. The number of the possible model states can be very large. In this case, one often needs to cluster certain types of model states together to reduce the

1

1


Prob

abili

ty fo

r usi

ng

Mod

el 1

(cor

rect

)

Explicit Mixed Region



Implicit Mixed or Multiple Pure States

A pair of explicit mixed statesA pair of implicit mixed states or two pure states

A pure state

Figure 8. Explicit-state model plot for showing class

model states


total dimensions of the state-space. Such clustering can be based on the similarities between different model states and/or on the significance of specific states in a particular study. Details of this method and specific treatment are shown in the next section.

IV. Application Examples Student Models of Mechanical Waves

This example is about the study of student conceptual understanding of mechanical waves (Wittmann, et al, 1999). The instrument used in the measurement is a single MCMR question, which is shown in Figure 11 in the Appendix. As we can see, this question probes the student understanding of how speed of waves is affected by various physical variables such as tension of the rope, the mass density of the rope, etc. Based on research, three common models were identified for this concept topic:

Model 1: The speed of the pulse depends on the properties of the media (density, tension, etc.). (Correct)

Model 2: Students treat the wave pulses as particles thinking that the speed of a traveling pulse is affected by the amplitude of the pulse and or the initial force exerted on the string to create the pulses. (Incorrect)

Model 3: Other less popular ideas and incomplete answers. (null model)

Table 2. Modeling scheme for the MCMR question on mechanical waves

Models Model 1 Model 2 Model 3 Responses e, f, g, h a, b, c, d, i, j k

The associations between the choices of the questions and the three models are described in Table 2. Since the choice corresponding to the third physical model is a “none of the above” type, it is logically impossible to have any mixing between this model and the first two. Therefore, in this case there are only four possible student model vectors:

001

,

010

,

100

, and

011

21

This question was used at the University of Maryland in four classes with similar instruction except that two classes had a tutorial on mechanical waves. Using the method discussed in equations (14) – (20), the class model density matrices and the population model states are calculated. Since there is only one question, the calculations of both DDDD and DDDDE give the same results and we can only identify if a single student has one of the pure model states or the explicit mixed state. The features of the model plot in this situation follow the explicit-state model plot shown in Figure 8 except that no implicit mixed states can be determined. The measurement of the existence of an implicit mixed state requires at least two questions.

The primary class model state vectors for each class (before and post instruction) are plotted in Figure 9 (each pre-instruction

0

0.5

1

0 0.5 1

Particle Pulse Model

Expert Model Post_Tut

Pre_All

Post_Trd

Figure 9. Plot of class model states on

students’ use of models of mechanical waves


model state is the results of the two classes with the same type of instruction). The results show that the initial situations are similar for all classes. Most students start with an explicit mixed state (almost equally mixed). The students in the two classes with tutorials have achieved a rather consistent expert model state after the instruction. The model states of the students without the tutorials remain in the mixed region but are somewhat shifted toward the expert model dimension.

Student Models of Classical and Quantum Probability In the research on student understanding of probability in learning introductory quantum mechanics

(Bao & Redish, 2002), we developed a tutorial to help students understand several basic issues of probability such as probability density function and normalization in a classical environment. After instruction using this tutorial, we gave two MCMR questions in the final exam (see the Appendix). The total number of students is 118. In the following discussion, I will show a detailed analysis of the data from the two questions using the methods introduced in the previous sections.

The first question (question 1.1) asks the students to consider a classical system described by the potential energy diagram and to work out the classical probability. This context setting is very similar to the one used in the tutorial in which the students use the energy diagram to get the velocity of the moving object and then the classical probability of finding the object in certain areas. Among the choices of the question, (a) and (c) give correct classical description of the motion of the particle; (e) gives the correct probability for both classical and quantum situations, therefore, using this choice alone cannot determine if a student is applying a classical model or a quantum model (or even both); choice (b) is an incorrect description of the classical motion of the particle; choice (d) reflects an common incorrect model where students consider that the “deep” portion of the potential well will attract the particle there making it more likely to be found in that area.

Based on these choices, we can define five common models, M1, …, M5, shown below:

M1: the correct model of classical probability – the probability density of finding a classically moving object at a position is inversely proportional to the speed of the object.

M2: the correct model of quantum probability – the probability density of finding a quantum object at a position is given by the square of the amplitude of the wavefunction.

M3: an incorrect “attraction” model – a “deeper” potential well can attract a particle more strongly and make it stay there longer resulting a larger probability in the corresponding area.

M4: an incorrect interpretation of quantum wavefunction – the amplitude of a wavefunction represents the speed of an object.

M5: the null model – students fail to use any of the models identified with M1, …, M4.

Table 3. Modeling scheme for the two MCMR questions on classical and quantum probability

M1 M2 M3 M4 M5 Question 1.1 ea, ec, eac e but no c d b none of e,b,d.

M1 M2 M3 M4 Question 1.2 d, da, db da, db, dab c others

From the design of the question, it is easy to see that choices (b) and (d) match well with M4 and M3 respectively. Therefore, based on a student’s selection of the two choices, we can infer if the student has used the corresponding models. To determine whether a student used M1 or M2 isn’t as simple since both models produce the same answer – choice (e). A careful look at the question suggests that if a student uses a consistent quantum model, he/she shouldn’t pick choice (c). On the other hand, if a student uses a


consistent classical model, he/she is very much likely to pick choices (a) and (c). As a result, if a student picks (e) but not (c), we consider that the quantum model has been used. If a student picks (e) but also (c) and/or (a), we consider that the classical probability model has been used. Finally, if a student doesn’t pick any of the choices (b), (d) and (e), it is considered that the student doesn’t have a model that gives reasoning for probability and is then given the null model. These modeling schemes are summarized in Table 3 and are used to convert students’ responses on question 1.1 into model state vectors. For example, if a student responded with (a) and (e), the result is then a mixed model state with equal probabilities between M1 and M2. The corresponding model state vector is ( )T

21 00011⋅ , where the symbol

“T” represents the transpose of a row vector, which gives a column vector (model state vectors are all column vectors).

The second question (question 1.2) probes student understanding of the quantum probability. For this question, we can define four common models:

M1: the correct model of quantum probability.

M2: the correct model of classical probability.

M3: the “attraction” model.

M4: the null model.

The modeling schemes that map the choices of this questions into the four different models are also given in Table 3. Again, the measurement of M1 and M2 is a little complicated since both models will result in the same choice (d). Since choices (a) and (b) both describe classical behaviors of the system, if a student picks (d) but none of (a) and (b), we consider the student having a consistent model of quantum probability. If a student picks (d) and also both of (a) and (b), we consider the student having a consistent model of classical probability. When a student picks (d) plus one of (a) and (b), a mixed model state is then assigned to this student.

Model States Measured within Individual Questions

Using the method discussed in equations (14 – 19, 32), the student data on the two questions are processed to obtain the explicit-state model density matrices and the eigenvalues and eigenvectors (see Table 4). The results indicate that with the classical system (question 1.1), about half of the students were able to apply the model of classical probability; about 15% of the students applied the quantum model, which is inappropriate in this case. There is non-trivial mixing between the first two models showing that some students used both models simultaneously. From the density matrix, it is easy to see that the mixing involving M3, M4, and M5 is much less (especially, students using M5 don’t mix at all with any other models). In this case, the eigenvectors and eigenvalues obtained from the model density matrix reflects in general three groups of students (each holding a rather consistent model) with sizes of approximately 18%, 4% and 15% corresponding to the last three models respectively. Here, the nontrivial null model population (15%) reflects a group of students who were able to give certain correct descriptions of the classical system (e.g. choices (a) or (c)), but fail to make appropriate connections to the concept of probability.

Table 4. Explicit-state model density matrix for question 1.1

Density Matrix λ1 λ2 λ3 λ4 λ5 0.512 0.116 0.177 0.042 0.153

DDDDE of question

1.1 (5-D)

0.480 0.107 0.010 0.004 0.0000.107 0.148 0.008 0.007 0.0000.010 0.008 0.174 0.020 0.0000.004 0.007 0.020 0.045 0.0000.000 0.000 0.000 0.000 0.153

0.958 0.278 -0.062 -0.010 0.000 0.283 -0.954 0.078 0.064 0.000 0.036 0.101 0.983 0.146 0.000 0.014 -0.050 0.151 -0.987 0.000 0.000 0.000 0.000 0.000 -1.000


In the second question (question 1.2), the results indicate that about 75% of population used the appropriate model for quantum probability, much higher than the number of students who used the appropriate model for the classical probability (Table 5). A small fraction of the population (about 6%) tried the inappropriate classical model. Still about 16% of the students used the “attraction” model (it will be shown later whether these students were the same group that applied the attraction model in question 1.1). There is only a very small fraction (~3%) on the null model dimension, indicating that most students applied a model relevant to probability on this question. The mixing between the different models is again very small (there is still some mixing between M1 and M2 but is smaller than in question 1.1 due to the smaller number of students used M2). Therefore, in this case, we can consider the population in general consists of four groups of students that each holds a rather consistent model.

Table 5. Explicit-state and general model density matrices for questions 1.1 and 1.2

Density Matrix λ1 λ2 λ3 λ4 0.512 0.116 0.163 0.210 DDDDE of

question 1.1

(4-D)

0.480 0.107 0.010 0.004 0.107 0.148 0.008 0.007 0.010 0.008 0.174 0.020 0.004 0.007 0.020 0.198

0.958 0.279 -0.030 -0.056 0.283 -0.957 0.025 0.064 0.036 0.066 0.870 0.488 0.021 0.052 -0.492 0.869 0.750 0.061 0.163 0.025

DDDDE of question

1.2

0.744 0.058 0.028 0.000 0.058 0.066 0.003 0.000 0.028 0.003 0.164 0.000 0.000 0.000 0.000 0.025

0.995 -0.084 -0.048 0.000 0.085 0.996 0.002 0.000 0.048 -0.006 0.999 0.000 0.000 0.000 0.000 -1.000 0.626 0.093 0.170 0.111

DDDDE of both

questions

0.612 0.083 0.019 0.002 0.083 0.107 0.006 0.004 0.019 0.006 0.169 0.010 0.002 0.004 0.010 0.112

0.986 -0.154 0.050 0.026 0.158 0.971 -0.038 -0.173 0.043 -0.014 -0.983 0.176 0.006 -0.181 -0.170 -0.969 0.672 0.074 0.158 0.096

DDDD of both

questions

0.612 0.129 0.078 0.087 0.129 0.107 0.029 0.019 0.078 0.029 0.169 0.025 0.087 0.019 0.025 0.112

0.945 -0.232 -0.213 -0.090 0.230 0.951 0.066 -0.195 0.168 -0.139 0.963 -0.156 0.162 0.148 0.150 0.964

Model States and Changes of States Measured Across Different Questions

The results from the explicit-state model density matrix suggest that within a single context scenario, most students applied one model (small mixing). It is interesting to analyze if a single student may change to use different models when the context scenarios are changed. To do this, we need to use several questions (>1) to measure the same set of models and use the general-state model density matrix in the data analysis.

The two questions in this example have some differences in the setup of the common models; however, if we emphasize the information on students’ use of appropriate models of probability in different contexts, the first three models measured by both questions become identical. Then, since the 4th and 5th models of question 1.1 are not really functional models for probability, we can collapse these two into a single null model dimension. With this treatment, we can restructure a 4-dimension model space for both of the questions:


M1: an appropriate model of probability based on classical or quantum reasoning.

M2: an inappropriate model of probability based on classical or quantum reasoning.

M3: the “attraction” model.

M4: the null model.

The modeling schemes to process students’ responses remain largely the same except that the last two model elements for question 1.1 are combined into the null model. Then applying the method discussed in equations (32) – (37), the students’ responses on both questions are processed to give the general-state model density matrix and its eigenvalues and eigenvectors (see Table 5). The explicit-state model density matrix for the two questions is also included (it basically equals to the average of the two explicit-state model density matrices of the individual questions).

As discussed earlier, the general-state model density matrix reflects the information of model mixing when context changes. From Table 5, we can see that DDDD shows more mixing between the first two models than that of DDDDE , indicating that under changing contexts students are more likely to use different models (details on comparison of model mixing are discussed in the Appendix). Similar results are also reflected in the third model (the attraction model), which implies that when a student uses the attraction model, it is likely that this student will use only this model within each question (very small mixing in DDDDE), but has a larger probability for this student to use another model on a different question (larger mixing in DDDD). However, the overall small mixing in both DDDD and DDDDE indicates that many students who used the attraction model held on to this model in different questions. Based on the analysis shown in the Appendix, we can conclude that when the context changes (from 1.1 to 1.2 and vise versa), many students changed to use a different model: 3% went from an appropriate model to an inappropriate model (M1 � M2); 9% from M1 to M3, and 14% from M1 to M4. These are all captured as implicit mixed states and can be extracted by comparing the differences between DDDD and DDDDE (see the Appendix).

Model States Statistics

When the primary eigenvalues aren’t very large (<0.8), it indicates that the students are behave rather differently in the class. To find the details of different individual model states and the changes of the states across questions, we can perform cluster analysis or model state statistics. Since there are four common models, the state-space has a dimension of 24 − 1 = 15 (the empty state is again ignored). To reduce the dimensions, M3 and M4 are combined together into a single dimension. Since the majority of the students used M1 and/or M2, this treatment won’t significantly affect the results. Now the total dimension of the state-space is reduced to 23 − 1 = 7. It is then interesting to see how each question may cue the students into the seven different model states and how students may change to a different state when the question is changed. To represent the different states, a binary coding scheme is used: e.g., 100 represents a pure state of consistent use of M1, whereas 110 stands for an explicit mixed state of simultaneous use of both M1 and M2. Table 6 shows a matrix representing the shift of individual students’ model states between the quantum context (QM – question 1.2) and the classical context (CM – question 1.1). All the possible model states cued by the quantum context are listed horizontally in different columns, whereas the model states cued by the classical context are listed vertically in different rows. The sequence of these states follow the significance and the popularity of the corresponding states. Each cell of the matrix represents the percentage of the population who were measured to have the “column” state in the quantum context and the “row” state in the classical context. Based on this matrix, one can inspect in detail the number of students who may be cued into one model state in one context and changed to a different state with a different context. By summation along the rows or columns, we can get a set of numbers representing the percentage of the students who were cued into the different states in the quantum context or the classical context respectively. A lot of information can be extracted from the results in Table 6 – examples are shown in Table 7.


Table 6. Shift of individuals’ model states on questions 1.1 and 1.2 (in %)

CM\QM 100 010 110 001 011 101 111 CM 100 33 0 1 3 0 0 0 36 010 3 0 0 0 0 1 0 3 110 11 1 4 3 0 2 0 20 001 17 0 6 11 0 1 0 35 011 1 0 0 0 0 1 0 2 101 1 0 0 0 0 1 0 2 111 1 0 0 0 0 0 1 2

QM 66 1 11 16 0 5 1

Table 7. Interpretations of the shift of individuals’ model states

Summary of the General Methodology

The different methods discussed above all have their unique features and advantages. Situations encountered in research can be very complicated and it is often necessary to combine multiple methods in an analysis to extract different types of information needed in the analysis. For example, model density matrix allows people to quickly identify the possible entanglement between different common models. The results can be obtained with many questions so that they reflect a more complete measurement of the interactive relation between the contexts and the students’ use of their knowledge. The model state statistics and cluster analysis can provide better resolution on the features of individuals’ model states, but the use of these methods needs careful balance between the level of detail and the dimensions of the state-space.

In general, it is found useful to first compute and compare DDDD and DDDDE to identify general features on mixed use of models and the types of mixing. When appropriate (large primary eigenvalues), the eigenvalues analysis can be applied. When details about specific individual model states are needed, one can establish a state-space based on the significance and popularity (as reflected by the model density matrices) of the different possible model states, and perform model state statistics or cluster analysis.

From this example, we can see that a variety of information can be extracted by doing the different calculations. The questions in the examples are by no means good ones and are used to show how one can implement the numerical methods in practice. Even with the crude instruments, the analysis still provides some interesting information about student understanding, which are useful in assessment and further development of the research instruments.

Use only classical model

Use only quantum model

Use both quantum and classical model

Question 1.1 (CM) 36% 3% 20%

Question 1.2 (QM) 1% 66% 11%

33% use the appropriate model on both questions.

0% uses an inappropriate model on

both questions.

4% use both M1 and M2 on both questions On both

questions 3% use quantum model through out.

11% use correct model in QM but both M1 and

M2 in CM.

17% use correct model in QM but other naïve models (M3) in CM.


V. Measurement Concerns and Implications Explicit Mixed Model State vs. the Use of a Hybrid Model

In the introduction part of this paper, it is discussed that a hybrid models represents a unique model and is intrinsically different from a mixed model state which describes the ways of using different models. However, since a hybrid model contains components from several models, it is likely that a student using a hybrid model or being in an explicit mixed model state can produce the same type of responses on a question. If in an interview situation, the researcher can inspect in detail the student reasoning on whether it reflects the use of multiple independent models (a mixed state) or a single locally coherent model (a hybrid model) that were put together with pieces from multiple models. With other instruments such as MCMR questions, it is then difficult to distinguish the two situations if they both create the same type of responses. A treatment to address this issue is that any measurement instruments (especially MC questions) need to be developed based on systematic quantitative research, with which the possible common models including all the possible forms of hybrid models should be identified. Then in the design of the questions, there should be special context settings and question formats (such as unique choices of MC questions) that allow the measurement of individual hybrid models, although in practice, this may be difficult to realize but it reflects the basic methodology of measurement. In summary, the hybrid models should be included in the model space whereas the measurement instruments such as MC questions are mostly used to measure the ways that students use the different models including the hybrid models in changing contexts. The identifications of certain types of hybrid models need to be completed in the stage of the qualitative research.

Measurement of Student’s Awareness of Conflicts With MCMR questions, we can measure if a student is cued into using multiple models

simultaneously. However, measuring the existence of multiple models doesn’t necessary mean that we know if a student is aware of the conflicts between the different models. Depending on the specific designs of the questions, a question may or may not bring a student into explicit awareness of the conflicts among the different models cued by the question. Sometimes the settings of a question just don’t show (or guide the students to consider) any apparent contradictory. For instance, consider two types of models and responses in the question used by the example of mechanical waves: the response from applying the “particle” model – “hit the string harder to make the pulse move faster”; and the response from applying the wave model – “increase the tension of the string to make pulse move faster”. To a student who hasn’t achieved a complete understanding of the underlying physics, both types of responses don’t show apparent contradictions between them; and therefore can be selected at the same time without causing any conflicts in the student’s mind.

It is also possible that the choices of a question have apparent logical contradictions but a student may fail to recognize (or simply ignore) the logical relations. For example, in the first question used in the probability example, there are students who picked both (b) and (e), and even both (d) and (e), which by simple logic should be excluded. The specific reasons for students to “ignore” such contradictions are yet to be studied and are beyond the scope of this paper. The purpose of this example is to show some extreme cases of how students may use different models at the same time without realizing any conflicts.

When students can recognize the conflicts and try to reconcile, the measured results become complicated. In such case, when pure states are measured, it is then difficult to determine if such states represent simple activation of one model or the outcome of a reconciliation process with several models. When mixed states are measured, it is also insufficient to determine if such mixing came from simple cueing of several models or the outcome of an unsuccessful reconciliation process. These difficulties can be helped with systematic qualitative research, where researchers can investigate in detail if a particular question can bring a student (from a particular population) into explicit awareness of conflicts and if so, how likely a student may attempt to reconcile and how successful such reconciliation can be. In addition, it is also possible to implement unique methods in designing questions to help extract certain information


on if a student has gone through a reconciliation process. For example, one can design and use questions that are less likely to trigger explicit conflicts and also questions that are more likely to trigger conflicts. By comparing a student’s responses on the two types of questions, we can extract certain information about the student’s reconciliation process. One may also design questions to explicitly request students’ reflection on the confidence level (or how much guessing). MC questions can also be designed to allow responses to include the most possible alternative answers, to use multiple-answers with different confidence levels, or to list alternative answers before giving final choice (and if a judgment process is taken). However, both the question design and the interpretation of the data should take into account the context dependence factor and should always be based on extensive qualitative research.

As we can see, there are many issues that one needs to consider in developing the measurement instrument. By careful controls of the variables in the measurement, and by rigorous research with interviews in the development and validation of the instrument, it is possible to design effective multiple-choice questions to measure the various aspects of student understanding.

Empirical Observations of Students’ Conceptual Development Process

Based on the literature and our own investigations on student models of physics (Thornton, 1994; Maloney & Siegler, 1993; Bao and Redish 2001b, c; Bao, et al, 2002), we can put together a picture to characterize the different stages of conceptual development in terms of the situations of students’ use of models. It is suggested that in learning a particular concept topic, a student’s learning can be described with five typical stages shown in Figure 10. The process represents a general progressive learning path and a student can begin and end with any one of the stages.

The first is the pre-naïve stage, which represents the situation where students by themselves won’t even be able to develop/posses a logically consistent model of the concept topic. Many examples of this situation have been observed in our interviews, where students fail to provide any logically-sound reasoning for a physical phenomenon. Many students just can’t provide reasoning of any kind. Some students attempt to put together reasoning but they usually fail to recognize the relevant variables from the context and don’t have appropriate causal relations in their reasoning – e.g. a result can also be the cause of itself. Basically, these students have a very poor ability of logical manipulation. Students in this stage can also be measured with model-based multiple-choice instrument, which will show an increased probability on null model dimension (an example of this is shown in Bao, et al, 2002).

The second is the naïve model stage, in which case students hold consistently one or several incorrect models of the concept topic. These models are often developed by students based on their real-life experience and previous school learning. In terms of manipulation of variables and causal relations, these naïve models are usually in a similar level comparing to the expert model. The problems of the naïve models are often due to the inappropriate use of logical rules and context variables.

The third and forth stages correspond to the two mixed model states – the implicit and explicit mixed states. The fifth stage is the expert model state in which case the students are able to apply the expert models of the concept topic consistently in different context settings. Starting from the third stage, it is possible to bring students into explicit awareness of conflicts of the different models with carefully prepared context settings. Also starting from the third stage, students can develop hybrid models as a result of instruction.

From the literature, it is suggested that an average undergraduate population in an introductory physics course often starts with the naïve model state and ends up somewhere in the mixed stages. Our

Pre-Naive

Consistent Use of Naïve Models

Implicit Mixed Model States

Explicit Mixed Model State

Consistent Use of Expert Models

Hybrid Models

Figure 10. Stages of conceptual development


previous research indicates that with traditional instruction, students often move to the middle of the mixed stages, and with research-based instructions (e.g. Tutorials), students can move very close to the expert state.

Possible Applications in Instruction

As indicated by many studies on student conceptual change process, when students hold naïve models, effective teaching strategies often require bringing students into explicit awareness of the conflicts of their existing naïve models (Posner, Strike, Hewson & Gertzog, 1982; McDermott & Shaffer, 1998). Therefore, the measurement of what types of context settings can cue multiple models and the explicit awareness of conflicts can play a valuable role in instruction. It provides useful information on both the students’ states of understanding and on if (or which ones of) the context scenarios of the questions are appropriate to be incorporated in teaching to help students. The possible instructional applications of the results of such measurements are summarized below:

(1) When students are measured as in the pre-naïve stage, the first task of the instruction is often to help students recognize the relevant variables of the contexts and understand the basic causal relations that the students can use to manipulate the variables to put together some reasoning, i.e., to help them understand the basic characteristics of the physical phenomenon. At this stage, since many students’ reasoning capabilities are quite “fragile”, encouragement and facilitation on students’ development of the ability to do logical reasoning (not necessarily the correct types) is often the major theme of the instruction. Activities to confront students’ incorrect ideas should be carefully controlled to avoid causing frustrations among students.

(2) If pure naïve model is detected, it indicates that the students don’t understand the expert model at all regarding the contexts provided. The instructor can begin with helping students develop some basic understanding of the expert model and/or confronting students’ naïve models. In this stage, strong confrontation of students’ existing models can cue students’ desire to change their current understandings; however, since the students don’t have much of the correct model yet, it is often difficult for them to quickly get into the correct direction and thus can cause significant frustrations. Therefore, it might works better if the instruction first helps students learn some new ideas in reasoning without being troubled by any contradiction. Later, the instruction can guide the students to re-interpret their existing examples (including the ones used in the questions) and see the conflicts of using their old models.

(3) If implicit mixed model state is detected, it shows that the students have started to understand the expert model but such understandings are often strongly tied to limited contexts and students may treat different expert-equivalent contexts as unrelated situations (context dependent fragmentation of knowledge). It also implies that the context scenarios of the questions won’t spontaneously activate explicit awareness of multiple models and the conflicts between these models. To use these context scenarios in instruction, it is often necessary to use several questions that each may cue a different model and guide the students to realize the conflicts by comparing their reasoning on the different questions.

(4) If explicit mixed model state is measured, it reveals that the students have further developed understanding of the concept topic and start to recognize the relevance and applicability of the expert models in a wider range context settings (usually they start to recognize the commonalities among the set of expert-equivalent context scenarios and may actively seeking consistency over a range of contexts); however, they still keep their naïve models at the same time. These students are the most readily prepared population to explicitly work out the conflicts between the different models. Since the context scenarios used in the measurement can simultaneously cue multiple models, they can be used directly as examples in teaching. As instructional tools, these context scenarios may need some modification so that they not only bring up multiple models but also activate students into explicit awareness of the conflicts among those models.


(5) As a related issue, we may found students developing hybrid models during instruction. In this situation, it is important to identify the specific contexts with which the hybrid models are developed. Since a hybrid model is often locally consistent within the contexts in which it is developed, to create cognitive conflicts, it is then necessary to prepare different context settings that the students can recognize as equivalent and that when applying the hybrid model, contradictory results can be produced.

In the 3rd, 4th and 5th cases, since the students already know the correct model (or parts of the correct model), the instruction can more directly confront the students’ naïve models and guide them to apply the correct model under different contexts. Regardless what states the students are in, the instruction should always involve as many real-life examples as possible (since most of the students have rich experience and also incorrect interpretations with such examples), and guide the students to re-interpret these examples using the correct models developed in learning. Without a rigorous re-interpretation process, it is often difficult for students to develop a robust understanding of the concept (rather students can develop fragmented knowledge strongly tied to specific contexts). That is, the features of the contexts that may cue a student into using naïve models need to be systematically addressed so that they would consistently cue a student into using the correct model. Similarly, the instruments used in the measurement should also involve diverse context settings to ensure a more complete evaluation of students’ understanding.

Appendix • A MCMR Question on Mechanical Waves

A long, taut string is attached to a distant wall (see figure). A demonstrator moves her hand up and down exactly once and creates a very small amplitude pulse which reaches the wall in a time t0. How, if at all, can the demonstrator repeat the original experiment to produce a pulse that takes a less time to reach the wall.

Pick any correct statements from the following list. a) Move her hand more quickly (but still only up and down once and still by the same amount). b) Move her hand more slowly (but still only up and down once and still by the same amount). c) Move her hand a larger distance but up and down in the same amount of time. d) Move her hand a smaller distance but up and down in the same amount of time. e) Use a heavier string of the same length, under the same tension f) Use a lighter string of the same length, under the same tension g) Use a string of the same density, but decrease the tension. h) Use a string of the same density, but increase the tension. i) Put more force into the wave. j) Put less force into the wave. k) None of the above.

• Two MCMR Questions on Classical and Quantum Probability In the figure below is shown a plot of a one-dimensional potential energy function U[x] (lower plot) and the

wavefunction (upper plot) of an eigenstate for an electron in the influence of that potential. The energy, E, associated with that state is also shown on the energy plot. Distances are measured in nanometers and energies in eV.


I II III IV V

Binding Energy = 9.0215

-3 -2 -1 0 1 2 3

E

The potential energy U[x] is defined so that it has four values in five regions of the x axis:

Region I: x < -1 U[x] = 0 Region II: -1 < x < -0.5 U[x] = -30 Region III: -0.5 < x < +0.5 U[x] = -15 Region IV: +0.5 < x < +1 U[x] = -6 Region V: 1 < x U[x] = 0

1.1 If the particle were moving classically (i.e., its motion were described by Newton's laws) in the potential U[x], and it had an energy E, which of the following statements would be true? List all that apply.

(a) It would move the fastest when it was in region II. (b) It would move the fastest when it was in region III. (c) If we took a photograph of the particle at a random time, we would never find it in region IV. (d) If we took a photograph of the particle at a random time, we would be most likely to find it in region II. (e) If we took a photograph of the particle at a random time, we would be most likely to find it in region III.

1.2 If the particle were moving quantum mechanically (i.e., its motion were described by the Schrödinger equation) in the potential U[x], and it had an energy E, which of the following statements would be true? List all that apply.

(a) If we measured the position of the particle at a random time, we would never find it in region I. (b) If we measured the position of the particle at a random time, we would never find it in region IV. (c) If we measured the position of the particle at a random time, we would most likely find it in region II. (d) If we measured the position of the particle at a random time, we would most likely find it in region III. (e) The state shown represents the lowest energy state that can be found in this well.

• Compare Relatively the Level of Model Mixing from a Model Density Matrix

Consider a model density matrix (e.g. DDDD) obtained by averaging over population and/or over different questions:

∑=

=

=N

kkk

w

N 1

1

1 uuoo

ooro

omo

mmr

ηη

ηµµµ

ρ

ρρρ

DDDD (37)


The relative model mixing rate between model η and physical mode µ is defined with γηµ, where

µµηη

ηµηµ ρρ

ργ =

(38)

When all the individual ku has the same coefficients on the ηth and µth model dimensions, which

are then constrained to equal ηηρ and µµρ respectively, γηµ has the maximum value equal to 1. If

the individual students have no mixing between the ηth and µth models, ku will have at least one zero coefficient on one of two model dimensions. In this case, γηµ has the minimum value that equals 0. All other situations will result in a value between 0 and 1 (Bao, 1999).

• Estimation of the Fractions of Students with Mixed States from a Model Density Matrix In the case of a MCMR question, we found in most situations, students may have explicit mixed

states involving two common models, which give model states such as T)0,21,21( . The related two pure states are (1,0,0)T and (0,1,0)T. Assume that α1 of the population have the state of (1,0,0)T, α2 of the population have the state of (0,1,0)T and α3 of the population have the state of T)0,21,21( . Then the diagonal components of the model density matrix corresponding to the first two models are:

23

111ααρ +=

; (39)

2

3222

ααρ += . (40)

The off-diagonal component reflecting the mixing between the two models is:

23

12αρ =

(41)

Therefore, based on the off-diagonal components, we can obtain an approximate estimation of the fraction of the students who is in a perfectly mixed state between the two models.

Using this method, the estimated fraction of mixed states are calculated with the model density matrices shown in Table 5. Only the mixed states between M1 and other models are obtained and listed in Table 8. There are two rows of data: the top row gives the estimated fraction based on the model density matrix whereas the lower rows (italicized) are obtained from model state statistics (basically from counting the different states). As we can see, the estimated results with explicit-state model density matrices match well with the statistic results (error < 0.01). With the general-state model density matrix, the estimated results are smaller than the results from the model state statistics. The reason is that there are two questions used in the measurement making it likely for the components of a model state vector to have a range of different values. Therefore, the assumptions made for using Equation (41) are not satisfied anymore. With multiple questions, the normalization used in calculating the model state vector, shown in Equation (15), can make the components much smaller than 21 causing the estimated results smaller than the fraction of the students who used multiple models on a single or multiple questions. However, the model state vector and the model density matrix are normalized to produce probabilities (in a predictive sense) that a student would use certain models in a single measurement with a single output. The model state statistics only counts the number of students who used multiple models which are not normalized but can be converted to the same set of probabilities represented by the model density matrix.


Table 8. Estimation of the fractions of mixed model states

Density Matrix Methods Μ1 ↔Μ2 Μ1 ↔Μ3 Μ1 ↔Μ4

From DDDDE 0.215 0.020 0.008 DDDDE of

question 1.1

(4-D)

0.480 0.107 0.010 0.004 0.107 0.148 0.008 0.007 0.010 0.008 0.174 0.020 0.004 0.007 0.020 0.198

From Statistics 0.220 0.025 0.008

From DDDDE 0.116 0.056 0.000 DDDDE of question

1.2

0.744 0.058 0.028 0.000 0.058 0.066 0.003 0.000 0.028 0.003 0.164 0.000 0.000 0.000 0.000 0.025


From DDDDE 0.165 0.038 0.004 DDDDE of both

questions

0.612 0.083 0.019 0.002 0.083 0.107 0.006 0.004 0.019 0.006 0.169 0.010 0.002 0.004 0.010 0.112


From DDDD 0.259 0.156 0.175 DDDD of both

questions

0.612 0.129 0.078 0.087 0.129 0.107 0.029 0.019 0.078 0.029 0.169 0.025 0.087 0.019 0.025 0.112


Although has limitations, this estimation method allows a quick and quantitative inspection about the features of students’ mixed use of different models. As we can see from Table 8, within each question, there are some mixing between M1 and M2 (22% in 1.1 and 12% in1.2). There are much less mixing between M1 and other models. With both questions are considered, the general-state model density matrix shows more mixing between all the models listed, which indicates that when contexts are changed (from 1.1 to 1.2 and vise versa) a significant number of students have changed to use a different models (about 10% more between M1 and M2, 12% more between M1 and M3 and 17% more between M1 and M4). A more careful inspection would show that 3% of the students consistently used quantum model on both questions and are measured as if changed from an inappropriate model to an appropriate model (see Table 6). This doesn’t happen with the model on classical probability – none of the students used the classical model on probability on both questions. This phenomenon is unique to the two questions used in the measurement and should in general be avoided in designing questions.

Acknowledgement The author would like to thank Dr. Edward F. Redish for many constructive discussions. The research

is supported in part by the NSF grants REC-0087788 and REC-0126070.

Reference:

Bao, L. (1999). “Dynamics of Student Modeling: A Theory, Algorithms, and Application to Quantum Mechanics,” Ph.D. dissertation, University of Maryland.

Bao, L. (2001a). “Context-Explicit Modeling of Conceptual Learning Process: Theory, Assessment,

and Instruction,” Presented at AAPT, January 2001. Bao, L., Hogg, K. & Zollman, D. (2002). “Model Analysis of Fine Structures of Student Models: An

Example with Newton's Third Law,” Am. J. Phys. 70 (7), 766-778 (2002).


Bao, L. and Redish, E. F. (2001a). “Concentration Analysis: A Quantitative Assessment of Student States,” PERS of Am. J. Phys. 69 (7), S45-53, (2001).

Bao, L. and Redish, E. F. (2001b). “Model Analysis: Assessing the Dynamics of Student Learning,”

submitted to Cognition and Instruction. Bao, L. and E. F. Redish, “Understanding probabilistic interpretations of physical systems: A pre-

requisite to learning quantum physics”, Am. J. Phys. 70 (3), 210-217, (2002) Bao, L. & Redish, E. F. (1998) “Student Misunderstandings of the Quantum Wavefunction,” AAPT

Announcer 28 (2), 92.

Clancey, W.J. (1993) Situated action: A neuropsychological interpretation response to Vera and Simon. Cognitive Science, 17, 87-116.

Clough, E. and Driver, R. (1986). A study of consistency in the use of students’ conceptual

frameworks across different task contexts, Science Education, 70 (4), 473-496. diSessa, A. (1993) “Towards an epistemology of physics,” Cognition and Instruction. 10, 105-225. Fischbein, E., Stavy, R. and Ma⋅Naim, H. (1989) The psychological structure of naïve impetus

conceptions, Int. J. Sci. Educ., 14(1), 63-81. Greeno, J.G. (1992). The situation in cognitive theory: Some methodological implications of

situativity. Presented at the American Psychological Society, San Diego, CA. Halloun, I. A. & Hestenes, D. (1985), Common sense concepts about motion. Am. J. Phys. 53 (11). Kandel, E.R, Schwartz, J.H. & Jessell, T.M. (2000) Principles of Neural Science, McGraw-Hill. Lave, J. (1988). Cognition in practice: Mind mathematics and culture in everyday life. Cambridge

University Press: Cambridge. Minstrell, J. (1992) “Facets of students’ knowledge and relevant instruction”, In: Research in Physics

Learning: Theoretical Issues and Empirical Studies, Proceedings of an International Workshop, Bremen, Germany, March 4-8, 1991, edited by R. Duit, F. Goldberg, and H. Niedderer (IPN, Kiel Germany, 1992) 110-128.

Maloney, D. P. and Siegler, R. S. (1993) “Conceptual competition in physics learning,” Int. J. Sci.

Educ., 15(3), 283-295. McDermott,L. C. and Shaffer, P. S. (1998). Tutorials in Introductory Physics, Prentice Hall, New

York NY.

Palmer, D. (1997) “The effect of context on students’ reasoning about forces,” Int. J. Sci. Educ. 19(6), 681-696.

Posner, G.J., Strike, K.A., Hewson, P.W., & Gertzog, W.A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education, 66, 211-227.


Redish, E. F. and Bao, L. (1998) “Student mental models of conductivity,” AAPT Announcer 28 (2), 92

Resnick, L.B. (1991). Shared cognition: Thinking as social practice. In L.B. Resnick, J.M. Levine, &

S.D. Teasley (Ed.), Perspectives on socially shared cognition. Washington, DC: American Psychological Association.

Rolls, E.T. & Treves, A. (1998) Neural Networks and Brain Function. Oxford University Press: New

York. Squire, L.R. & Zola, S.M. (1996), Hippocampus.

Thornton, R. K. (1994) “Conceptual Dynamics: Changing Student Views of Force and Motion,”

Proceedings of the International Conference on Thinking Science for Teaching: the Case of Physics, Rome, Sept. 1994

Treves, A. & Rolls, E.T. (1992) Computational constraints suggest the need for two distinct input

systems to the hippocampal CA3 network. Hippocampus 2: 189-199. Vosniadou, S. (1994) “Capturing and modeling the process of conceptual change,” Learning &

Instruction, (4), 45-69. Vosniadou and Brewer (1992) Mental models of the earth: A study of conceptual change in

childhood. Cognitive Psychology, 24, 535-585. Whitelock, D. (1991) Investigating a model of commonsense thinking about causes of motion with 7

to 16-year-old pupils, Int. J. Sci. Educ., 12(5), 563-537. Wittmann, M.C., Steinberg, R.N., Redish, E.F. (1999) “Making Sense of How Students Make Sense

of Mechanical Waves,” Phys. Teach. 37, 15-21. Yerrick, R. K., Pedersen, J. E., & Arnason, J. (1998). “We're just spectators”: a case study of science

teaching, epistemology, and classroom management. Science Education. 82, 619-648.

Endnotes 1 One of our current projects is to develop assessment tools using Model Analysis with multiple-choice

questions and rapid responses systems (such as web-surveys or in-class real time response systems), which allow the instructor to probe students’ conceptual understanding during a class session.

2 Techniques on question design and instrumental error analysis were discussed in detail in references (Bao, 1999; Bao & Redish, 2001a; Bao & Redish, 2001b).

Explicit and Implicit States of Model Mixing in …lbao/archive/papers/Model...Model Mixing (Bao,...

Documents

Transcript of Explicit and Implicit States of Model Mixing in …lbao/archive/papers/Model...Model Mixing (Bao,...