Session F3C

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference F3C-1

Work in Progress - Item Retrieval System in Distributed Repositories to Generate Adaptive

Assessments Supported in Items Response Theory (.IRTT)

Yehiry Pulido, Gloria Fernández, Silvia Baldiris and Juan Guevara

ylpulidov@correo.udistrital.edu.co, gmfernandezn@correo.udistrital.edu.co, baldiris@silver.udg.edu, jcguevarab@udistrital.edu.co

Abstract– The assessment process is one of the most important issues in the learning process and in many case it is the process that defines the instruction sequence because it measures the performance of the student in the educational process. During recent decades the inclusion of TICs in the teaching-learning process have facilitated addressing the diversity of student and teacher features. Technologies to enhance learning have allowed the different ways of learning and teaching that coexist in the educational context to be adapted through user modeling and adaptation processes. Our purpose in this paper is to introduce “. IRTT” (item response theory tool), an adaptive tool based on item response theory to generate assessment according to the student competence level, in the context of a learning management system (LMS), in particular .LRN [2]. Index Terms - E-learning, LMS, Items Response Theory, Distributed Repositories.

INTRODUCTION

Previous works have dealt with the problem of measuring student competence levels based on the classical test theory (CTT). Assessment structures in the IMS Learning Design (IMS LD) [1] have been automatically generated using planning techniques [2]. These structures are organized in different levels of competences according to a particular scale of measure based on Bloom’s taxonomy [3]. The structures also define the course sequence, i.e., the student growth in his/her competence level through the different Bloom knowledge levels. In this paper, we are focusing on the automatic generation of item response theory (IRT) based on adaptive tests in the contexts of a unit of learning (UoL). IRT makes it possible to alleviate the workload for designers of adaptive courses on the complex task of authoring adaptive tests that take into account the demonstrated performance of students in the virtual environment. The item retrieval system is supported by distributed item repositories of which the most suitable items are selected according to the temporal student model. The student

temporal user model is based on competence level and it is updated each time the student responds to an item in the system. This document is structured as follows: in section 2, the ITR is introduced, section 3 describes the method used for IRTT development and, finally, in section 4 some conclusions are drawn and future work is recommended.

ITEM RESPONSE THEORY (IRT)

Test theories offer validated frameworks to conceptually link unobservable variables with observable variables. Item response theory is a general statistical theory about examinee items and test performance and how performance relates to the abilities or traits (in our case, competence levels) that are measured by the item in the test. Within the general IRT framework, many models have been formulated and applied to real test data [4]. This article will consider the models that apply dichotomously scored data. The most general and accepted IRT model is shown in equation 1: (1) This model defines the probability that a particular item has to be correctly answered given a proficiency level (θ) using the logistic function Probability (θ). The model assumes the relation between item performance and the competence is given by a logistic function described by one, two, or three parameters. Parameter a tells us how the item can discriminate between highly proficient students and less proficient students; parameter b indicates the difficulty level of the item; and parameter c indicates how likely the examinees are to obtain the correct answer by guessing [5]. In our case, the unobservable variable (θ) is referred to the student competence level. Although, there are many formulas associated with the different ITR models, in this section we only introduce the ones used in our proposal.

Session T1A

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference T1A-2

I. Item Characteristic Function Birnbaum's three-parameter model [1] states that the probability of a correct answer to an item, given a value of the knowledge level (θ), is defined by (2): . (2)

II. Student Proficiency Level Calculation Method

One of the most common methods for calculating the student proficiency is the maximum likelihood method [3]. It consists of finding the value of θ that maximizes the likelihood function, as shown in (3). | ∏ (3)

Where u = (u1, ..., un) is the vector of the examinee’s answers, that is, for i = 1,...,n; u is 1 if the answer to the ith item is right and 0 otherwise.

III. Item Selection Methods

For selecting the items, the test information function is used. It consists of selecting the item that maximizes the item information for the provisional estimate skill level so far (θ). It is given by the following function (4) [1]: . . . (4)

DESCRIPTION OF .IRTT FUNTIONALITY

I. General Description

As mentioned before, .IRTT was conceived as a tool to support teachers in the difficult task of delivering adaptive test to students. In this order of ideas, the teacher’s role is, actually, to control the definition of assessment process. In particular, teachers 1) provide the system with an initial sample of items for the item difficulty definition; 2) define the parameters for the initial sample; and 3) provide new items and their initial parameters. Teachers also provide an initial level of competences for their set of students. The students are the objects of assessment. They login into dotLRN learning management system to present the adaptive test. They are interested in verifying their performance in the system and the system offers them the possibility of following their advances through an open learner model. Figure 1 shows a general vision of .IRTT.

FIGURE 1.

GENERAL FRAMEWORK OF .IRTT

.IRTT consists of two important components. The first component, the .IRRT retrieval engine, offers a web service client to connect with some web service-based items repositories and also an intelligent mechanism to select the most suitable items according to the student model. The second component, .IRTT package upon .LRN integrates in the learning platform the necessary user interfaces to support the teacher and student actions. At execution time, when a student requests a particular test, the .IRTT package verifies the achieved and desired level of competence of this particular student and selects the most adequate repository of items (according to the available number of items or topic features). Then it recovers the most adjusted item and returns it to the interface of the student to begin her test. According to her response, IRTT infers the new competence level of the student and taking into account the desired competence level and the finalization criteria, it considers delivering a new item or finishing the test. This process is repeated until the application decides to terminate the test. Teachers could define how many times students could take the test and what the finalization criterion is. While items are stored in distributed repositories a traffic analysis is needed to optimize the process of selecting the most suitable items. This process includes the study of the busiest times and the number of service requests, i.e., the number of users attempting to access the system simultaneously. II. .IRTT Methodology of development

In order to construct .IRTT we identify some necessary steps to follow: 1. Conduct an assessment test for a group of students as a

statistical sample for the b parameter inference. 2. Identify a level of proficiency for each student, based on

his or her correct answers. 3. Identify a level of difficulty for each item. 4. Perform a calibration of the items from an iterative

process involving the student attempted competition and the difficulty of the item.

5. After having the item parameters estimated, they are used to model the response pattern of an item, with the mathematical function CCI (item characteristic curve).

6. To implement adaptive testing, initially you must select a method of estimating the level of competence (θ), which is not directly observable and must be estimated for each student from her response to items.

7. You should choose a method of item selection. 8. From her response, the system calculates a new level of

competences, which is taken into account to apply again the method of item selection.

9. Carry out this last step until the stopping criterion is implemented to establish the final competence level.

Session T1A

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference T1A-3

III. Parameter Estimation

As mentioned in the first section, the .IRTT model assumes the relation between item performance and the student competence is given by a logistic function described by one, two, or three parameters. The difficulty ( ) parameter is defined considering an average value of the results of students to an item. Teachers must use a previous set of data for the inference. Considering the experience of different teachers we asked, the discrimination parameter ( ) was defined depending on the type of question:

TABLE I DISCRIMINATION

Classification of item Value Definition Questions Intermediate Questions Application Questions

0.2 1.2 2

Finally, and taking into account [1], the guess ( ) parameter was defined. was based on the number of alternatives available to the question as follows:

TABLE II GUESS

Classification of item Value 2 alternatives 3 alternatives considering the options yes / no 3 alternatives 4 alternatives 5 alternatives 6 alternatives

0.50 0.33 0.33 0.25 0.20 0.16

IV. How are the items selected and how is the student

competence level updated? From the results of this initial test, we identified the probability of a correct answer to each item in the sample. That means we generate each item characteristic curve using the equation 01. The a, b, c values correspond to the parameters of item I defined as explained in last section. For the item characteristic curve generation, the value of θ was based on a scale from -3 to 3, but it is not a mandatory scale. Based on the probability of a correct answer for each particular item and the student’s previous response, the system obtains the most optimal student competence level (θ), using the maximum likelihood method, according to Equation 02. To select the next question, we used the method of maximum information in equation 03, where all the values are defined as explained above.

CONCLUSIONS

In this paper we deal with the problem of relieving teachers' design work when they have to create adaptive tests in the context of a virtual learning environment using a three-parameter model of item response theory as a strong and highly utilized tool for adaptive testing. At the design time, teachers need to provide only a small amount of information while the system provides a transparent interface with item distributed repositories to reuse. At the execution time, adaptive tasks are delivered to

users in the context of a virtual learning environment using .IRTT. Although the production of item repositories and their semantic characterization require an initial extra effort for the teachers, this effort decreases as the possibility of reutilization grows. One of the main issues of our proposal is the integration task. Actually, the integration task is progressing and a layered evaluation has been defined for the validation process of the proposal. Studies are being developed to evaluate the validity of our approach. Comparisons among traditional assessment process and adaptive assessment based on .IRTT are also being conducted. Improvement of .IRTT based on user modeling process considering intrinsic user features are under studied at the moment.

BIBLIOGRAPHY

[1] Salcedo, P. “Adaptive Multiplatform UML Course Based on Item Response Theory”. Engineering Magazine. Number 10. August 2004.

[2] Hambleton R.K. and Swaminathan H. Item Response Theory: Principles and applications. Boston: Kluwer. 1985.

[3] Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-de-laCruz, L. and Rios, A. “SIETTE: A web-based tool for adaptive testing”. International Journal of Artificial Intelligence in Education. 2004. Vol. 14, pp. 29-61.

[4] Ronald, K., Hambleton and Russell W. J. Comparison of Classical test theory and item response theory and their applications to test development. University of Massachusetts at Amherst.

[5] Chong Ho, Yu.. “A Simple Guide to the Item Response Theory (IRT) and Rasch Modeling”. 2010.

[6] Morales, E, Gil, A. and Garcia, F. “Architecture for Recovery of quality learning objects in distributed repositories”.

[7] Leguizamón, M. “Design and development of educational computer (MEC's): a possibility to integrate information technology with other curricular areas”.

[8] Prieto, M. “METHADIS: Methodology for the design of adaptive hypermedia systems for learning based on learning styles and cognitive styles”.

[9] I.G.L. Consortium, “IMS Learning Design”. Version 1.0. Final Specification., 2003.

[10] Hernández, J., Baldiris, S., Santos, O.,Fabregat, R. and Boticario, J. “Conditional IMS LD Generation using user modeling and planning techniques,” In 8th IEEE International Conference on Advanced Learning Technologies, Riga: 2009.

[11] Bloom, B. “Taxonomy of Educational Objectives, New York: David Mckay”, 1956.

[12] Funahashi, K. “Multilayer Neural Networks and Bayes Decision Theory”. Neural Networks Vol11. 1998.pp 209-213.

AUTHOR INFORMATION

Pulido Yehiry, Researcher, Universidad Distrital Francisco José de Caldas, ylpulidov@correo.udistrital.edu.co. Fernández Gloria, Researcher, Universidad Distrital Francisco José de Caldas, gmfernandezn@correo.udistrital.edu.co.

Session T1A

978-1-61284-469-5/11/$26.00 ©2011 IEEE October 12 - 15, 2011, Rapid City, SD 41st ASEE/IEEE Frontiers in Education Conference T1A-4

Baldiris Silvia, Researcher, Universidad de Girona, baldiris@silver.udg.edu. Guevara Juan, Researcher, Universidad Distrital Francisco José de Caldas, jcguevarab@udistrital.edu.co.