Intelligent Tutoring(Student-Problem Table)

Timothy K. Shih

Outline

• Purpose of ITS (Intelligent Tutoring System) and ATS (Adaptive Testing System)

• Evaluation Methods• Student-Problem Chart• Item Respond Theory

Purpose• ITS (Intelligent Tutoring System)

– Students learns in different pace– Can not provide individualized lecture– Intelligent Tutoring: students learn in different

topology– Tutorial Generation: remedial lectures after tests

• ATS (Adaptive Testing System)– Traditional test methods are not flexible– Item Response Theory: a relatively modern test

method– Different students use different number of questions

and different questions– Instance hints to students during test

Evaluation Methods

• Quality of Distance Education (timely course contents, presentation quality, etc.)

• Type of Evaluations– Student to instructor: teaching evaluation– Student/instructor to content: content evaluation– Instructor to student: performance evaluation– Administrator to program: DL program evaluation

InstructionObjective

ReadinessEvaluation

InstructionActivity

InstructionEvaluation

Feedback

• Is the objective satisfied?• Considering the background of students, is the

course material too difficult?• Which part of instruction was not explained clearly

by the instructor?• Which student is not working hard?

The General Model of Instruction

• Placement evaluation: The evaluation assesses how a new student or a new instructor is adapted to the new environment (to test both prerequisite skill and course objective).

• Formative evaluation: During the process of instruction, informal quizzes are given to students. The outcome of evaluation are feedback to both the instructor and the students.

• Diagnostic evaluation: The identification of persistent or recurring learning difficulties that are left unresolved by the standard correction perspective of formative evaluation.

• Summative evaluation: Grades are given to the students to certify the students’ global level of knowledge on the topic taught.

• Self-evaluation: A mix of formative and diagnostic evaluations, which is taken by the students to evaluate whether to face summative evaluation.

• Self-regulation

Evaluation Methods

• Norm-referenced evaluation: The credit of an individual is compared to the average credit of a group to justify the performance of the individual.

• Criterion-referenced evaluation: A standard terminal behavior is established. The credit of an individual is compared to the standard.

Evaluation Strategies

Mastery Learning

Formative Assessment

Repeated Training

Course Unit A

FormativeAssessment A

FormativeAssessment B

Course Unit B

Pass ?

Pass ?

Training Preparation

Yes

Yes

No

No

Student-Problem Chart

• The Student-Problem Chart Analysis Theory (S-P Table) [Sato 1975]

• Caution Indices for Students and Items [Sato 1975]

• The Student-Navigation Chart (S-N Table)

• Caution Indices for Students and Course Units

• Cross References of S-P Table and S-N Table

An Example of Student-Problem Chart

S

P

2 3 7 4 9 1 6 5 10 8 grades percentage Student

caution index

Determine

classification

7 1 1 1 1 1 1 1 1 1 1 10 100% 0.00 A

5 1 1 1 1 1 1 1 1 1 0 9 90% 0.00 A

9 1 1 1 1 1 0 1 1 0 1 8 80% 0.57 A’

4 1 1 1 1 1 1 1 0 0 0 7 70% 0.00 B

10 1 1 1 0 1 0 1 0 1 0 6 60% 0.50 B’

2 1 1 0 1 1 0 1 0 0 0 6 60% 0.50 B’

14 1 1 1 0 1 0 1 0 0 0 5 50% 0.00 B

1 1 1 1 0 1 0 0 0 1 0 5 50% 0.40 B

13 1 1 0 1 0 0 1 0 0 1 5 50% 0.70 B’

6 1 0 0 1 0 1 0 1 1 0 5 50% 1.00 B’

15 1 1 1 1 0 1 0 0 0 0 4 40% 0.40 C

11 1 0 0 0 0 1 0 1 0 1 4 40% 1.20 C’

3 1 0 1 0 1 0 0 0 0 0 3 30% 0.33 C

8 0 0 1 0 0 1 0 0 0 0 2 20% 0.71 C’

12 0 1 0 0 0 0 0 0 0 0 1 10% 0.25 C

numbers 12 11 10 9 8 8 7 6 5 4 80

Correct

percentage

80 73 67 60 53 53 47 40 33 27

Problem

Caution index

0.10 0.35 0.55 0.15 0.15 0.75 0.00 0.29 0.38 0.55

Determine

classification

A A A’ A A A’ B B B B’

Problem Caution Index Diagnosis

Problem Caution Index

Percentage of students who answer questions correctly

A A’

B B’

0.50 1.000

50%

100%

Student Caution Index Diagnosis

Student Caution Index

Percentage of student score

A A’

B B’

0.50 1.000

50%

100%

C C’

75%

Student-Navigation: The Original Table

網頁課程

學習者 1 2 3 4 5 6 7 8 9 10

精熟

學習分數 學習

總分

1 1 2 1 2 2 2 2 2 2 3 8 19

2 2 2 3 3 2 3 3 3 3 3 10 27

3 2 1 2 2 1 1 3 1 1 3 5 17

4 1 1 3 1 1 1 1 3 3 2 4 17

5 1 3 1 3 1 1 1 2 2 3 5 18

6 1 2 1 2 1 2 1 2 1 3 5 16

7 1 1 2 2 2 2 1 2 3 2 7 18

8 2 2 2 3 2 3 1 2 2 2 9 21

9 2 2 3 3 3 3 3 2 3 2 10 26

10 1 1 3 3 2 1 3 2 3 2 7 21

11 1 2 3 3 2 3 2 2 2 3 9 23

12 1 3 1 2 1 2 2 2 1 3 6 18

13 1 1 1 1 2 2 2 2 2 2 6 16

14 1 1 1 1 1 1 1 1 3 2 2 13

15 1 1 1 1 1 1 1 1 1 1 0 10

16 1 1 2 1 2 2 1 1 3 1 4 15

17 1 3 2 2 2 1 1 2 3 2 7 19

18 1 1 2 3 2 2 3 2 2 3 8 21

19 1 1 1 1 1 1 1 3 1 1 1 12

20 1 1 3 1 1 1 3 1 1 2 3 15

精熟

瀏覽分數 4 9 12 13 11 11 10 15 14 17 18.1

瀏覽總分 24 32 38 40 32 35 36 38 42 45 36.2

Sorted by Mastery Learning Score網頁課程

學習者 1 2 3 4 5 6 7 8 9 10

精熟

學習分數 學習

總分

2 2 2 3 3 2 3 3 3 3 3 10 27

9 2 2 3 3 3 3 3 2 3 2 10 26

11 1 2 3 3 2 3 2 2 2 3 9 23

8 2 2 2 3 2 3 1 2 2 2 9 21

18 1 1 2 3 2 2 3 2 2 3 8 21

1 1 2 1 2 2 2 2 2 2 3 8 19

10 1 1 3 3 2 1 3 2 3 2 7 21

17 1 3 2 2 2 1 1 2 3 2 7 19

7 1 1 2 2 2 2 1 2 3 2 7 18

12 1 3 1 2 1 2 2 2 1 3 6 18

13 1 1 1 1 2 2 2 2 2 2 6 16

5 1 3 1 3 1 1 1 2 2 3 5 18

3 2 1 2 2 1 1 3 1 1 3 5 17

6 1 2 1 2 1 2 1 2 1 3 5 16

4 1 1 3 1 1 1 1 3 3 2 4 17

16 1 1 2 1 2 2 1 1 3 1 4 15

20 1 1 3 1 1 1 3 1 1 2 3 15

14 1 1 1 1 1 1 1 1 3 2 2 13

19 1 1 1 1 1 1 1 3 1 1 1 12

15 1 1 1 1 1 1 1 1 1 1 0 10

精熟

瀏覽分數 4 9 12 13 11 1 10 15 14 17 18.1

瀏覽總分 24 32 38 40 32 35 36 38 41 45 36.2

Sorted by Mastery Navigation Score網頁課程

學習者 10 8 9 4 3 6 5 7 2 1

精熟

學習分數 學習

總分

2 3 3 3 3 3 3 2 3 2 2 10 27

9 2 2 3 3 3 3 3 3 2 2 10 26

11 3 2 2 3 3 3 2 2 2 1 9 23

8 2 2 2 3 2 3 2 1 2 2 9 21

18 3 2 2 3 2 2 2 3 1 1 8 21

1 3 2 2 2 1 2 2 2 2 1 8 19

10 2 2 3 3 3 1 2 3 1 1 7 21

17 2 2 3 2 2 1 2 1 3 1 7 19

7 2 2 3 2 2 2 2 1 1 1 7 18

12 3 2 1 2 1 2 1 2 3 1 6 18

13 2 2 2 1 1 2 2 2 1 1 6 16

5 3 2 2 3 1 1 1 1 3 1 5 18

3 3 1 1 2 2 1 1 3 1 2 5 17

6 3 2 1 2 1 2 1 1 2 1 5 16

4 2 3 3 1 3 1 1 1 1 1 4 17

16 1 1 3 1 2 2 2 1 1 1 4 15

20 2 1 1 1 3 1 1 3 1 1 3 15

14 2 1 3 1 1 1 1 1 1 1 2 13

19 1 3 1 1 1 1 1 1 1 1 1 12

15 1 1 1 1 1 1 1 1 1 1 0 10

精熟

瀏覽分數 17 15 14 13 12 11 11 10 9 4 18.1

瀏覽總分 44 37 40 39 37 34 31 35 31 23 36.2

Student-Navigation Table網頁課程

學習者 10 8 9 4 3 6 5 7 2 1

精熟

學習分數 學習

總分

2 3 3 3 3 3 3 2 3 2 2 10 27

9 2 2 3 3 3 3 3 3 2 2 10 26

11 3 2 2 3 3 3 2 2 2 1 9 23

8 2 2 2 3 2 3 2 1 2 2 9 21

18 3 2 2 3 2 2 2 3 1 1 8 21

1 3 2 2 2 1 2 2 2 2 1 8 19

10 2 2 3 3 3 1 2 3 1 1 7 21

17 2 2 3 2 2 1 2 1 3 1 7 19

7 2 2 3 2 2 2 2 1 1 1 7 18

12 3 2 1 2 1 2 1 2 3 1 6 18

13 2 2 2 1 1 2 2 2 1 1 6 16

5 3 2 2 3 1 1 1 1 3 1 5 18

3 3 1 1 2 2 1 1 3 1 2 5 17

6 3 2 1 2 1 2 1 1 2 1 5 16

4 2 3 3 1 3 1 1 1 1 1 4 17

16 1 1 2 1 2 2 2 1 1 1 4 15

20 2 1 1 1 3 1 1 3 1 1 3 15

14 2 1 3 1 1 1 1 1 1 1 2 13

19 1 3 1 1 1 1 1 1 1 1 1 12

15 1 1 1 1 1 1 1 1 1 1 0 10

精熟

瀏覽分數 17 15 14 13 12 11 11 10 9 4 18.1

瀏覽總分 45 38 42 40 38 35 32 36 32 24 36.1

S Curve

N Curve

Student-Navigation Table with Indices網頁課程

學習者 10 8 9 4 3 6 5 7 2 1

精熟

學習分數

學習

總分

平均

學習次數

學習者

注意係數

判定

類別

2 3 3 3 3 3 3 2 3 2 2 10 27 2.7 0 A

9 2 2 3 3 3 3 3 3 2 2 10 26 2.6 0.73 A’

11 3 2 2 3 3 3 2 2 2 1 9 23 2.3 0.22 B

8 2 2 2 3 2 3 2 1 2 2 9 21 2.1 0.88 B’

18 3 2 2 3 2 2 2 3 1 1 8 21 2.1 0.22 B

1 3 2 2 2 1 2 2 2 2 1 8 19 1.9 0.24 C

10 2 2 3 3 3 1 2 3 1 1 7 21 2.1 0.22 B

17 2 2 3 2 2 1 2 1 3 1 7 19 1.9 0.56 C’

7 2 2 3 2 2 2 2 1 1 1 7 18 1.8 0.03 C

12 3 2 1 2 1 2 1 2 3 1 6 18 1.8 0.64 C’

13 2 2 2 1 1 2 2 2 1 1 6 16 1.6 0.20 C

5 3 2 2 3 1 1 1 1 3 1 5 18 1.8 0.32 C

3 3 1 1 2 2 1 1 3 1 2 5 17 1.7 0.56 C’

6 3 2 1 2 1 2 1 1 2 1 5 16 1.6 0.25 C

4 2 3 3 1 3 1 1 1 1 1 4 17 1.7 0.22 C

16 1 1 2 1 2 2 2 1 1 1 4 15 1.5 0.58 C’

20 2 1 1 1 3 1 1 3 1 1 3 15 1.5 0.45 C

14 2 1 3 1 1 1 1 1 1 1 2 13 1.3 0.36 C

19 1 3 1 1 1 1 1 1 1 1 1 12 1.2 0.46 C

15 1 1 1 1 1 1 1 1 1 1 0 10 1 0 C

精熟瀏覽分數 17 15 14 13 12 11 11 10 9 4 18.1

瀏覽總分 45 38 42 40 38 35 32 36 32 24 36.1

平均瀏覽次數 2.25 1.9 2.1 2 1.9 1.75 1.6 1.8 1.6 1.2

網頁注意係數 0.39 0.53 0.52 0.02 0.26 0.11 0.34 0.17 0.54 0.25

判定類別 A B’ A’ A B B B B B’ B

Course Node Caution Index Diagnosis

B

0

A A’

B’

Course Node Caution Index

0.50 1.00

2

3Average number of navigation to course nodes

1

Student Navigation Caution Index Diagnosis

0

A A’

B B’

Student Navigation Caution Index

0.50 1.00

2

2.5Average of student navigation

1

C C’

Cross Reference of S-P and S-N Tables

S-N S-P Observation

A A Excellent students, with stable learning progress

A A’ Good students, but careless from time to time

A B Good students, but do not work hard enough

A B’ Good students, but careless and do not work hard enough

A C Good students, but due to some reasons, do not perform well. Need special attention

A C’

• How do we develop Web courses with an easy to access style? – Course Patterns

• How do we evaluate student learning performance? – Navigation, Tests

• How can we help them from the evaluation? – SPC Table

• How can computer help them? – Tutorials

Timothy K. Shih, Nigel Lin, Jianhua Ma, and Runhe Huang (U. of Aizu, Japan)

A Course Development and Student Assessment System

• Objectives – Mastery Learning– Formative Evaluation– Norm-referenced Evaluation

• Methods and Systems– Course Development with Patterns– Instruction and Assessment Tools– Individualized Course Content Generation

Research Motivation

• Students lost in a complicated navigation graph

• A text book has its fixed structure• Course Pattern = Structure Pattern +

Content Pattern• Only one or two course patterns are used in

a virtual university

Course Development with Patterns

• Pattern definition language• Limited number of sections in a course unit• Small section size• Fixed basic structure for Structure Pattern

(i.e., tree, linked list, set)• Fixed basic structure for Content Pattern

(i.e., text, picture, Pop-up Quiz, Question)

Course Pattern Construction

Reusable Course Unit

Reusable Session

Course Unit

Session

A Graphical User Interface is used to construct structure patterns.

Example: Tree and Linked Lists

Web Course Structure Pattern

Example: Tree, Linked Lists, and Sets

Web Course Structure Pattern (Cont.)

Formal Definition of Course PatternsStructurePattern := RCU StructurePRCU := Null | CourseUnitName @CourseUnitName := ASCIIStructureP := Session | TreePattern | ListPattern | SetPattern TreePattern := Session #< TreeP >TreeP := Null | StructurePattern TreePListPattern := #[ ListP ]ListP := Null | StructurePattern ListPSetPattern := #{ SetP }SetP := Null | StructurePattern SetP

Session := SessionName SessionContent Pop-upQuizzes SessionQuestionsSessionName := ASCIISessionContent := WebDocumentPop-upQuizzes := Null | Pop-upQuiz Pop-upQuizzesPop-upQuiz := YesNoQuestion Constraints | MultipleChoiceQuestion Constraints | FillInBlankQuestion ConstraintsSessionQuestions := Null | SessionQuestion SessionQuestionsSessionQuestion := YesNoQuestion | MultipleChoiceQuestion | FillInBlankQuestion

Constraints := Trigger | not Constraints | Constraints and Constraints | Constraints or Constraints | Constraints xor ConstraintsTrigger := TimeTrigger | ObjectNumberMetTriger | SpecificObjectMetTrigerTimeTrigger := TimeVisited >= IntegerObjectNumberMetTriger := NoOfObjectsVisited >= IntegerSpecificObjectMetTriger := CurrentMObject = MObjectYesNoQuestion := WebDocumentMultipleChoiceQuestion := WebDocumentFillInBlankQuestion := WebDocument

Note: Null is a null object @ represents the structure pattern is a reusable course unit # is an integer which represents the number of internal objectsSpecial Variables: TimeVisited CurrentMObject NoOfObjectsVisitedDefinitions omitted are: Integer is an integer ASCII is an ASCII string MObject is a multimedia object WebDocument is an HTML document

Database Schema Generation

• The Structure Pattern Table: representation of tree, list and set

• The Content Pattern Table: representation of object types and object coordinates

• The Quiz Table: representation of test content and constraints

• The Question Table: representation of assignment content

• Purpose: Easy to use and trace objects• Components

– Fixed Patterns– Text, Picture, Audio, and Video for Course

Materials– Pop-Up Quizzes (real-time interaction)– Assignments (as homework and references of

exams)• A Graphical User Interface is used to

construct content patterns

Web Course Content Pattern

• The Web Course Pattern Tools: Pattern tools are used to help the construction of Web course patterns.

• The Web Navigation Patrol: The patrol is a mobile agent program which run on client sites. The agent collects navigation messages of Web interaction, which are used in the analysis model of student behavior.

• The Performance Analysis Model: This analysis model is based on some instruction theory from the educational literature, such as the SP-table mechanism with the computation of student caution indices.

• The Progressive Learning State Machine Constructor: The constructor generates intelligent tutorials from the course material provided by the instructor.

Instruction and Assessment Tools

Pattern DDL Program

Pattern GUI

Pattern Specification

Web Course Design Editor

Web CoursePattern

Database

Curriculum Development Committee

Web CourseContentDatabase

Web Browser with Web Navigation Patrol

WebNavigationDatabase

Progressive LearningState Machine

PerformanceAnalysis Model

Pattern Formal Syntax

Students Instructors

Pattern Design

MessageSpecification File

DDL Statements

Course Pattern

Course Content

Course Content

Course Pattern

Web Navigation Records

PerformanceRecords

NavigationRecords

BNF Grammar

IntelligentTutorial

InteractionCourse Pattern

Course Design

Analysis

An Instruction Design & Delivery System

• The Student-Problem Chart Analysis Theory (S-P Table) [Sato]

• Caution Indices for Students and Items [Sato]• The Student-Navigation Chart (S-N Table) [Shih]• Caution Indices for Students and Course Units

[Shih]• Cross References of S-P Table and S-N Table

[Shih]• The Student-Problem-Course Table (SPC Table)

[Shih]

Models of Learning Assessment

S

P

2 3 7 4 9 1 6 5 10 8 grades percentage Student

caution index

Determine

classification

7 1 1 1 1 1 1 1 1 1 1 10 100% 0.00 A

5 1 1 1 1 1 1 1 1 1 0 9 90% 0.00 A

9 1 1 1 1 1 0 1 1 0 1 8 80% 0.57 A’

4 1 1 1 1 1 1 1 0 0 0 7 70% 0.00 B

10 1 1 1 0 1 0 1 0 1 0 6 60% 0.50 B’

2 1 1 0 1 1 0 1 0 0 0 6 60% 0.50 B’

14 1 1 1 0 1 0 1 0 0 0 5 50% 0.00 B

1 1 1 1 0 1 0 0 0 1 0 5 50% 0.40 B

13 1 1 0 1 0 0 1 0 0 1 5 50% 0.70 B’

6 1 0 0 1 0 1 0 1 1 0 5 50% 1.00 B’

15 1 1 1 1 0 1 0 0 0 0 4 40% 0.40 C

11 1 0 0 0 0 1 0 1 0 1 4 40% 1.20 C’

3 1 0 1 0 1 0 0 0 0 0 3 30% 0.33 C

8 0 0 1 0 0 1 0 0 0 0 2 20% 0.71 C’

12 0 1 0 0 0 0 0 0 0 0 1 10% 0.25 C

numbers 12 11 10 9 8 8 7 6 5 4 80

Correct

percentage

80 73 67 60 53 53 47 40 33 27

Problem

Caution index

0.10 0.35 0.55 0.15 0.15 0.75 0.00 0.29 0.38 0.55

Determine

classification

A A A’ A A A’ B B B B’

The Student-Problem Chart

The Student-Problem-Course (SPC) Table

View A: Score View B: Navigation View C: Test

Each view has two caution indices

The Revised Model – the SPC Table

The CSa index (caution index of student w.r.t. problem): indicates the outcome of exam. Students of a high index need special attention. On-line tutorial can be generated for these students.

The CPa index (caution index of problem w.r.t. student): indicates the quality of problems. Problems of a high index value can be re-designed.

The CSb index (caution index of student w.r.t. course unit navigation): indicates the degree of student navigation. High value means the student either works too hard (too much navigation), or too lazy (low navigation). On-line tests can be generated for those lazy students.

The CCb index (caution index of course unit w.r.t. student navigation): indicates which course unit is less visited and less effective. Course content can be revised or access paths can be re-constructed.

The CPc index (caution index of problem w.r.t. course unit): each problem is selected for an exam from a unique course unit.

The CCc index (caution index of course unit w.r.t. problem): indicate which course unit does not have a problem chosen in an exam. Problems of this course unit can be selected in the next exam of a higher priority.

The Six Caution Indices

The Student-Course Table

Course Unit Number

Stu

den

t Nu

mb

er

1 2 3 4 5 6 7 8 910 11 12 13 14 TCN CSb

1 NF1,1 NF1,2       NF1,j               NF1,n TCN1 CSb1

2 NF2,1 NF2,2                       NF2,n TCN2 CSb2

3 NF3,1                         NF3,n TCN3 CSb3

4                                

5                                

6           NFi,j               NFi,n TCNi CSbi

7                                

8                                

9                                

10 NFm,1                         NFm,n TCNm CSbm

TSN TSN1 TSN2       TSNj              TSNn    

CCb CCb1 CCb2       CCbj              CCbn    

Caution Indices Computation

Average of Course Navigation = (Σi = 1 m TCN i ) / m =

Average of Student Navigation = (Σj = 1 n TSN j ) / n =

where TCN is the Total Course Navigation

and TSN is the Total Student Navigation

TCN i = Σj = 1 n NF i,j

TSN j = Σi = 1 m NF i,j

CSb i = 1 – (Σj = 1 n (NF i, j) (TSN j) )– (TCN i) / (Σj = 1

TCN i TSN j - (TCN i) )

CCb j = 1 – (Σi = 1 m (NF i, j) (TCN i) ) – (TSN j) / (Σi = 1

TSN j TCN i - (TSN j) )

Caution Indices Computation

• The Assessment Criteria of Mastery Learning Navigation– The frequency of the mouse movement on

course objects– The duration of course navigation– The activation on op-up quizzes

• Navigation Factor: to evaluate how hard a student is working on the Web course material

Learning Effect Function

If the degree of frequency or duration is less than the minimal threshold, the elementary message does not count.

If the degree of frequency or duration is greater the maximal threshold, it is counted as the maximal value.

Effective navigation frequency =

Σ Web object Effective Frequency / Σ Web object Actual Frequency

Effective navigation duration =

Effective Duration / Actual Duration

Navigation Frequency and Duration

• Fixed Time Activation (assigned by the instructor to enforce the student)

• Criterion-based Activation (no time limitation, activated on buttons to decide pass/no pass, Mastery Learning)

• Popup Quizzes are brought up by the system at the navigation

Effectiveness of quiz interaction =

Number of quizzes answered correctly / Total number of quizzes

Activation on Pop-up Quizzes

Navigation Factor = Effective navigation frequency * W1+

Effective navigation duration * W2 +

Effectiveness of quiz interaction * W3

where W1 + W2 + W3 = 1.0, and W1 > 0.0, W2 > 0.0, W3 > 0.0

Thus, 0.0 <= Navigation Factor <= 1.0

Mastery Learning Navigation Factor

• Web document structure (physical links)

• Web knowledge structure (logical links in a traversal)

• Knowledge Structure can represent the dependency relations among course units, from the perspective of an instructor

Web Document and Knowledge Structures

• Based on Test Results• Based on Student Navigation Factors• Based on Test Problem Used• Based on Web Knowledge Structure, or

Dependency Relations among course units

• The generated lectures are state machines which include new physical links

• Automatic lecture generation can be proceeded dynamically

Automatic Lecture Generation

• Tutorial on Missing Course Content

• Based on Test Score• Based on Navigation Factor

Let alpha_1 be the max threshold of caution index CSa

Let beta_1 be the min threshold of navigation factor Fb

TutorialSetInitiation

Input: SPC table

Output: TutorialInitSet for each student

Algorithm:

For each student S with CSa > alpha_1

For each factor Fa in View A of S, where Fa = 0

Find the factor Fc in View C w.r.t. Fa, where Fc = 'Y'

Find the Factor Fb in View B w.r.t. Fc

If Fb < beta_1 then

Find course C w.r.t. Fb

Assert course C into the TutorialInitSet for student S

Automatic Tutorial Generation

• On-Line Test for Lazy Student• Based on Student Caution Index• Based on Navigation Factor

Let alpha_2 be the max threshold of caution index CSb

TestSetInitiation

Input: SPC Table

Output: TestInitSet for each student

Algorithm:

For each student S with CSb > alpha_2

For each factor Fb in View B, where Fb < beta_1

Find course C w.r.t. Fb

Select problems of course C

Assert problems to the TestInitSet for student S

Automatic Test Generation

• Find Inappropriate Test Problem• Based on Test Problem

Let alpha_3 be the max threshold of caution index CPa

ProblemRefinement

Input: SPC Table

Output: problem update

Algorithm:

For each problem P with CPa > alpha_3

Find the factor Fc in View C w.r.t. problem P, Fc = 'Y'

Find course C w.r.t. Fc

Refine problem P in course C

Automatic Problem Refinement Indication

• Find Misleading Course Content

• Based on Course Content

Let alpha_4 be the max threshold of caution index CCb

Let alpha_5 be the min threshold of caution index CCc

CourseRefinement

Input: SPC Table

Output: course update

Algorithm:

For each course C with CCb > alpha_4

Refine content of course C and its associated Web structure

Find the corresponding caution index CCc of course C

If CCc < alpha_5

suggest to use problems in course C for the test

Automatic Course Refinement Indication

Implementation Technology

• Windows NT Server, IIS (Internet Information Server)

• ASP (DHTML, VB Script, ActiveX, CSS Component)

• MS SQL Server

• Student Management Tool– Manages

General Information of Students

– Shows Study Records of Students

System Implementation

• Course Structure and Tutorial Management Tool– Show Course

Structure– Manage Tutorial

Designed by Instructors

System Implementation

• Course Unit Design Tool– Manage General

Information of Course Units

– Allow Teacher to Design Course Unit with 5 Types of Course Unit Objects

• Text• Picture• Button• Sound• Video

System Implementation

• Problem Management Tool– Manage Problem

Database– 2 Types of Problems

• Assignments• Pop-up Quizzes

– Allow Teachers to Design 3 Types of Problems

• Multiple Choice• True or False• Fill-in-Blank

System Implementation

• Exam Tool– Allow Teacher to

Hold On-Line Exam in 3 Steps• Set up Exam

Time• Select Course

Units as an Exam Scope

• Select Problems in the Exam Scope

System Implementation

• Assessment Tool– Create SPC Table

Based on Exam– 3 Views of SPC

Table• View A: Score• View B: Navigation• View C: Exam

System Implementation

• On-Line Learning Tool– Allow Students to

Study Course Unit via Web Browser

– 3 Agents Running under the System• Course Unit

Object Navigation Tracking Agent

• Course Unit Study Record Agent

• Pop-up Quiz Agent

System Implementation

• On-Line Exam Tool– Allow

Student to Attend On-Line Exam

– Also Works on PDA

System Implementation

• On-Line Tutorial Tool– Instructor’s Tutorial

• Allow student to view pre-designed tutorials

– Student’s Tutorial• Each student has

his own tutorial generated according to SPC table

System Implementation

DemoSystem Demo

Summary of ITS

• SP Table was designed before Web technology

• Navigation factors can be tracked• Pop-up Quizzes can be used• Learning Topology can be pre-defined

• Automatically generate individualized tutorial (and Web site)

Other Intelligent Technologies

• Automatic FAQ reply• Automatic grading of chat room

participation• Analysis of collaborative learning

behavior• Semantic Web for content searching

Summary

• Automatic tutorial generation• Toward automatic test generation

• Semantics• Educational Professionals