A cluster-based analysis to diagnose students’

learning achievements

Miguel Rodríguez Artacho

Comp. Science School

Universidad Nacional de Educación a Distancia (UNED)

(Spain)

miguel@lsi.uned.es

@martacho

EADTU Webinar 2016 1

Content

1. General Objectives

2. Background and Motivation

3. Proposed Diagnostic Test Methodology

4. Conclusions

5. Future Work

EADTU Webinar 2016 2

GENERAL OBJECTIVES

EADTU Webinar 2016

Scope

Recognizing Learning Flaws trough Testing: Clustering Based Methodology and Reliability

General Objective

The design and implementation of a methodology for learning weakness diagnosis and

assessment based on:

Adaptive feedback to the students in order to individually identify learning

weaknesses and misconceptions about a topic right after assessment through testing.

Classification of the students via clustering of the detected learning disabilities, as a

support for the design of feedback strategies and activities for improving their academic

performance.

3

BACKGROUND AND MOTIVATION

EADTU Webinar 2016

Problems with prior knowledge diagnostic assessment using standardized tests with

manual scoring: Type I ICFES multiple choice questions with only one correct answer.

This kind of questions are used for: Midterm exams, SABER 5th, 9th, 11th and SABER PRO

mandatory state tests in Colombia.

The traditional education system uses pass/fail scoring scale based written exams for

assessment The score does not provide enough information about learning that

can be used for performance improving.

The recognition of learning disabilities and misconceptions is key and complex process

that has to be manually performed.

4

BACKGROUND AND MOTIVATION: TESTS

EADTU Webinar 2016

Disadvantages of traditional tests : The same test, with a fixed

number of items, is given to all test takers. They have limited answer

choices. The test is long in order to make it more accurate.

The assessment uses traditional methodologies which do not allow :

− Identification of systematic misconceptions and weak

understanding of concepts in order to plan strategies to improve

their academic performance.

− The classification and grouping of the students to undertake a re-

orientation of the reinforcement activities.

− The individual recognition of the level of learning disabilities and

misconceptions.

5

BACKGROUND AND MOTIVATION: FEEDBACK

EADTU Webinar 2016

A diagnostic assessment methodology that provides a classification score,

identifies learning disabilities, misconceptions and weak understanding of

concepts, allowing to group the students with similar problems in clusters, is

required.

Structure of the proposed diagnostic assessment methodology:

Item Response Theory (IRT) is used as the method to obtain the skill level of

each concept.

The use of a system of interrelated concepts and dependences to identify

cognitive disabilities (misconceptions and weak understanding of concepts)

The use of Clustering to classify the students in groups with similar disabilities

6

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016

Traditional Test ITR

Lack of invariance in the properties of the

tests with respect to the test subjects.The

characteristics of the items depend on the

group of persons.

Different tests can be comparable, as

the skill level trend to be the same

between different item sets

Asumes the same error level for all

subjects, or the test liability is the same

for all the participants (as a property of the

test)

Similar level of assessment accuracy

for all different participants.

Item Response Theory (ITR) [Thurstone, 1925], [Lord, 1952, 1968]

ITR allows invariant measured variables that are independent with respect to the

examinees and the used test instruments.

7

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016

ITR Models

1, 2 and 3 parameters unidimensional logistic models

Dichotomous answer format (only one answer)

Performance and skills assessment

ITR – Model proofing

The test instrument, with the items containing the object variable, is applied to

Validate the ITR assumptions

Select the optimum models based on statistical analysis

ITR – Once the model is selected …

Estimate the parameters of the selected model

Calculate the skill or proficiency level of the test subjects

Identify learning disabilities in the test subjects

8

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016

Diagnostic Methodology : Item selection

At least one assessment item assigned to each node of the framework.

The knowledge domain to be evaluated, categorized into sub-topics and pre-

requisites.

The dependences between the items and the concepts (concepts for the assessment

in each item).

The weight of the concepts in each item.

9

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016

An inference example (probability and statistics)

10

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016 11

PROPOSED DIAGNOSTIC ASSESMENT METHODOLOGY

EADTU Webinar 2016

Diagnostic Methodology

Tool used: R

http://www.r-project.org/

12

PROPOSED DIAGNOSTIC: LEARNING PATHS

EADTU Webinar 2016

Diagnostic Methodology

13

CLUSTERING

EADTU Webinar 2016

Cluster Generation

14

CLUSTERING

EADTU Webinar 2016

Cluster Generation

List of weakly-understood concepts per each examinee

Total weight of each weakly-understood concept in the test (TP CI d)

Calculate the total weight of the weakly-understood concepts in the test (PTcd) per

each examinee, as in :

15

CLUSTERING

EADTU Webinar 2016

Cluster Generation

16

CLUSTERING

EADTU Webinar 2016

Cluster Generation

17

CONCLUSIONS

EADTU Webinar 2016

Psychometric aspects

The Item Response Theory (IRT) was selected for this work after a proper understanding of

its advantages with respect to the Classical Test Theory (CTT).

An statistical procedure was proposed to select and validate the optimum model to use with

the obtained data from the tests used in this work. A computer program was designed on the R

language for analysis purposes .

A comparative studied was performed between the score for the skills level of a group of

examinees obtained with the classical test theory (TCT, average score) and that obtained with

the IRT model (unidimensional 3 parameters logistic model 3PL)

18

CONCLUSIONS

EADTU Webinar 2016

Regarding the Diagnostic Methodology

A software for diagnostic was implemented:

• Process answers of the examinees ( Deficient and Minimum) to generate the weakly-

understood concepts per student

• Represent the suggested leaning paths for each examinee.

• An index representing the total weight (or total sum of weigths) of the weakly-understood

concepts in the test per examinee is generated.

Regarding the Cluster

A computer program was implemented in R in order to generate a list classifying the

examinees in groups with similar misconceptions or learning disabilities.

Data mining tools can be as useful as Intelligent Systems in certain domains to diagnose

student models.

19

CONCLUSIONS

EADTU Webinar 2016

This work is useful for public education institutions in Colombia because it serves as a

solution for the efficient diagnostic of the learning disabilities in students by using a test.

The design and implementation of the diagnostic procedure, suppported with IRT and

clustering procedures, allow to perform a comprehensive diagnostic of the learning

disabilities, misconceptions and weak understanding of concepts in students.

The work provides the students with a tool for the easy identification of their learning and

cognitive disabilities, and the suggested self-learning path to improve their academic

performance

Provide feedback

20

ACKNOWLEDGEMENTS

EADTU Webinar 2016

Luz Stella Robles Pedrozo and Miguel Rodríguez-Artacho, "A cluster-based

analisys to diagnose students' learning achievements,” Global Engineering

Education Conference (EDUCON), 2013 IEEE, Berlin, 2013, pp. 1118-1123. doi:

10.1109/EduCon.2013.6530248

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6530248&is

number=6530074

21

A cluster-based analysis to diagnose students’

learning achievements

THANKS!

Miguel Rodríguez Artacho

Comp. Science School

http://ltcs.uned.es

Universidad Nacional de Educación a Distancia (UNED)

(Spain)

miguel@lsi.uned.es

@martacho

EADTU Webinar 2016 22