Using Bayesian Networks to Predict Test Scores
-
Upload
luke-schwartz -
Category
Documents
-
view
34 -
download
0
description
Transcript of Using Bayesian Networks to Predict Test Scores
![Page 1: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/1.jpg)
04/19/23 ASSISTment 1
Using Bayesian Networks to Predict Test Scores
by Zach PardosNeil Heffernan, Advisor
![Page 2: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/2.jpg)
04/19/23 ASSISTment 2
Introduction Overview
• ASSISTment tutoring system
• The Task
• Bayesian networks
• Platform selection
![Page 3: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/3.jpg)
04/19/23 ASSISTment 3
ASSISTment Tutoring System
• Online tutoring system developed at WPI- Assess student knowledge/learning– Assists and prepares students for the MCAS– 2nd year of operation
• Participation includes over…– 2,000 students– With 20 teachers/classes– At 6 schools
![Page 4: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/4.jpg)
04/19/23 ASSISTment 4
ASSISTment Tutoring System
• Students attempt to answer top level questions based on previous MCAS test questions
• If the student answers incorrectly or asks for a “hint” they are given supporting questions, called scaffolds, or hint text messages
• All answers and actions are logged on the server
![Page 5: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/5.jpg)
04/19/23 ASSISTment 5
The Task
• To use Bayesian networks to assess students’ knowledge levels in the ASSISTment system and predict their performance on the MCAS test.
• Research topic: Compare predictive performance of fine-grain vs. coarse-grain skill models.
![Page 6: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/6.jpg)
04/19/23 ASSISTment 6
Bayesian Networks
• "The essence of the Bayesian approach is to provide a mathematical rule explaining how you should change your existing beliefs in the light of new evidence. In other words, it allows scientists to combine new data with their existing knowledge or expertise.”
- The Economist (9/30/00)
![Page 7: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/7.jpg)
04/19/23 ASSISTment 7
Bayesian Networks
• “New data”– 2,000 students answering questions online– MCAS test results
• “Existing knowledge or expertise”– Various grain skill models– Prof. Neil Heffernan
• Bayes Rule:
Where ‘R’ is a random variable with value ‘r’ and evidence ‘e’
P(e)
r)r)P(RR|P(ee)|rP(R
![Page 8: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/8.jpg)
04/19/23 ASSISTment 8
Platform Selection
• Bayesian network software choices:– GeNIe– MSBNx– BayesiaLab– Netica– MATLAB with BNT (Bayes Net Toolkit)– Java Bayes
![Page 9: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/9.jpg)
04/19/23 ASSISTment 9
Platform selection
Choice: MATLAB with BNT• Pros:
– Provides wide selection of inference engines– MATLAB’s robust programming environment– Automation– Runs on GNU/Linux
• Existing Perl interface for the many scripts that will perform data mining tasks.
• Cons– Little Slow
![Page 10: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/10.jpg)
04/19/23 ASSISTment 10
Project Overview
• The datasets
• Skill models
• Parameters
• Implementation
• Results
![Page 11: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/11.jpg)
04/19/23 ASSISTment 11
The Datasets
• Student online response data– 600 students from 2004-2005– Student selection criteria:
• Completed at least 100 items online• Completed the 2005 MCAS test
– 2,568 question items
• Student state MCAS test scores for ’05– Used for calculating prediction accuracy – No test data used for training/parameter learning
![Page 12: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/12.jpg)
04/19/23 ASSISTment 12
Skill Models
• Skill models describe the skills which are related to the online and MCAS questions.
• Skill models used:– MCAS1– MCAS5– MCAS39– WPI106
![Page 13: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/13.jpg)
04/19/23 ASSISTment 13
Skill Models
• Skill models used for the MCAS test consisting of 29 multiple choice questions
• MCAS1
• MCAS5
![Page 14: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/14.jpg)
04/19/23 ASSISTment 14
Skill Models
• MCAS39
• WPI106
• The MCAS1 is a two layer network with skill nodes mapped to question nodes. The other 3 networks have a third, intermediary layer of ‘AND’ nodes. This allows all question nodes to have the same number of parameters (slip/guess). The ‘AND’ nodes also reflect the notion that a student must know all tagged skills to answer the item correct.
![Page 15: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/15.jpg)
04/19/23 ASSISTment 15
Skill Models
Inequality-solving Equation-Solving
X-Y-Graph
Congruence
WPI-106 WPI-39 WPI-5 WPI-1
Equation-concept
setting-up-and-solving-equations
Patterns-Relations-Algebra
The skill of “math”
Plot Graph modeling-covariation
Slope understanding-line-slope-concept
Similar Triangles understanding-and-applying-congruence-and-similarity
Geometry
PerimeterCircumferenceArea
using-measurement-formulas-and-techniques
Perimeter
Transfer table for skill models
![Page 16: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/16.jpg)
04/19/23 ASSISTment 16
Parameters
• Parameters were set as a best guess starting point.
• Test model guess parameter is 0.25 because questions are multiple choice (out of four)
Original Parameters
Online Model
Test Model
Skills 0.50 Imported
Guess 0.10 0.25
Slip 0.05 0.05
• Preliminary learning of parameters using EMon the MCAS1 network indicates a guess of 0.30, slip of 0.38 and prior of 0.44 on the skills. These numbers were calculated recently andare not used in our prediction results thus far.
Learned Parameters
Online Model
Skills 0.44
Guess 0.30
Slip 0.38
![Page 17: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/17.jpg)
04/19/23 ASSISTment 17
Implementation
• The main routine ‘bn_eval()’ takes in:– Name of skill model– StudentID– BNT object of the skill model bayes net
• ‘bn_eval()’ outputs:– Status messages– Predicted score/Actual score/Accuracy– Logs prediction and skill assessment data
![Page 18: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/18.jpg)
04/19/23 ASSISTment 18
Implementation
• The evaluation is a 2 stage process• Stage 1
– Bayes skill model for the online data is loaded– Student’s online results are compiled and sequenced
for the network– Student is given credit for all scaffold questions
relating to a top level item answered correctly– Results are entered into the network as evidence– Marginals on the skill nodes are calculated using
liklihood_weighting approximate inference .
![Page 19: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/19.jpg)
04/19/23 ASSISTment 19
Implementation
• Stage 2 of evaluation– Bayes skill model for the MCAS test is loaded– Skill marginals calculated from stage 1 are entered
into the test model as soft evidence– Marginals on the question nodes are calculated using
jtree (join-tree) exact inference.– Test score points are summed by multiplying each
marginal by 1 and then taking the ceiling of the total score.
– Predicted test score is compared to actual student test score.
![Page 20: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/20.jpg)
04/19/23 ASSISTment 20
Implementation
• Example student run using MCAS1 model
![Page 21: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/21.jpg)
04/19/23 ASSISTment 21
Implementation
• Assessed skill marginals using MCAS1
![Page 22: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/22.jpg)
04/19/23 ASSISTment 22
Implementation
• Example student run using MCAS5 model
![Page 23: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/23.jpg)
04/19/23 ASSISTment 23
Implementation
• Assessed skill marginals using MCAS5
![Page 24: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/24.jpg)
04/19/23 ASSISTment 24
Implementation
• Example student run using MCAS39 model
![Page 25: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/25.jpg)
04/19/23 ASSISTment 25
Implementation
• Assessed skill marginals using MCAS39
![Page 26: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/26.jpg)
04/19/23 ASSISTment 26
Implementation
• Example student run using WPI106 model
![Page 27: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/27.jpg)
04/19/23 ASSISTment 27
Implementation
• Assessed skill marginals using WPI106
![Page 28: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/28.jpg)
04/19/23 ASSISTment 28
Results
• Model performance/accuracy results:– MAD is Mean Average Difference. The test is out of 29 points
so a MAD score of 4.5 indicates that the model on average predicts a score that is 4.5 points from the actual score.
MODEL MAD (RAW) % ERROR
WPI-39 4.500 15.00 %
WPI-106 4.970 16.57 %
WPI-5 5.295 17.65 %
WPI-1 7.700 25.67 %
74.33
82.3583.43 85
65
70
75
80
85
MCAS1 MCAS5 WPI106 MCAS39
Model Accuracy (%)
![Page 29: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/29.jpg)
04/19/23 ASSISTment 29
Future Work
• Reduce runtime– Optimize the number of samples used with
liklihood_weighting inference for each model.
• Increase accuracy– Learn full parameters in all models– Use analysis to improve skill model tagging
• Experiment with alternative models– Combine skill models into a hierarchy– Introduce time as a variable (DBNs)
![Page 30: Using Bayesian Networks to Predict Test Scores](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812e1a550346895d93830d/html5/thumbnails/30.jpg)
04/19/23 ASSISTment 30
References
A copy of this presentation as well as our initial paper submitted to ITS2006 entitled “Using Fine-Grained Skill Models to Fit Student Performance with Bayesian Networks” can be found online at:
http://users.wpi.edu/~zpardos/bayes.html
Thanks to the WPI-CS department, Neil Heffernan, contributors at CMU and the ASSISTment developers.