MHCI PSLC Data Shop Project Spring Progress Report.

MHCI PSLC Data Shop Project

Spring Progress Report

MHCI PSLC Team | Spring 2005 2

The Team

Jason HumDesign Lead

Sandi LoweProject Manager

Meghan MyersClient Liaison

Sam ZaissDocuments Manager

Jeff WongTech Lead


Outline

I. What is the PSLC?II. What is the Data Shop?III. Analyzing DataIV. Exporting DataV. Project ThemesVI. Preliminary DesignVII. Project PlanVIII.Final Takeaways


What is the PSLC?

Collect Process Access

Pre-Defined

Free-Form

PSLCLearnLab

Data Shop

Export


Background on PSLC

• Further current education research• Enable new education research• Support collaboration• Support LearnLab courses


Background on LearnLab Courses

• 7 course areas under PSLC– 3 Language (Chinese, French, ESL)– 2 Math (Algebra, Geometry)– 2 Science (Physics, Chemistry)

• Various stages of development


What is the Data Shop?


Pre-Defined

Free-Form

LearnLab

Data Shop

Export

PSLC


Collecting the Data

• Allows researchers to set up an in-vivo LearnLab experiment

• Facilitates logging of data in these experiments


Processing the Data

• Strip away student information• Perform computations on data to

streamline analysis• Create supporting data tables


Accessing the Data

• Built as a web application• Selected data analysis techniques• Data filtering and exportation


Analyzing Data: Pre-Defined Reports


Pre-Defined

Free-Form

LearnLab

Data Shop

Export

PSLC


Learning Curves

• Displays students’ performance over time on a particular knowledge component

• Log Files must include:– Knowledge Components– Student performance on individual

opportunities– Match Knowledge Components to

Problems / Sub-Goals

• Supporting Data:– Requirements Solicitation

(Koedinger, Aleven, Ritter)– General Research Contextual

Inquiries (U2)– LearnLab Research Contextual

Inquiries (U13)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Courtesy of Data Shop


Help Effectiveness Report

• The probability of the Section getting task or sub-goal x incorrect after a particular help message

• Log Files must include:– Help / Hints– Whether a student received help,

and if the answer that followed was correct or incorrect

– Measure of student’s knowledge before help message

• Supporting Data:– Requirements Solicitation

(VanLehn, Ritter)– General Research Contextual

Inquiries (U1, U10)– LearnLab Research Contextual

Inquiries (U11, U12)


Error Report

• Determine common mistakes that students make on tasks • Log Files must include:

– Erroneous Input & Tutor Response to Error– Error Classifications

• Supporting Data:– Requirements Solicitation (VanLehn, Koedinger, Ritter)– General Research Contextual Inquiries (U3, U8, U10)– LearnLab Research Contextual Inquiries (U11, U12)


Problem Profile

• Context and student performance on a particular problem • Log Files must include:

– Problem Text– Record of students’ answers on the problem and whether the answers

were correct or incorrect

• Supporting Data:– General Research Contextual Inquiries (U1, U3, U7, U8, U9, U10)– LearnLab Research Contextual Inquiries (U11, U12)– Course Committee Survey (Chem)– Think Aloud Pilot (U2)


Analyzing Data: Free-Form Analysis


Pre-Defined

Free-Form

LearnLab

Data Shop

Export

PSLC


Timeline Visualization

• Visualizes the pattern of each student’s actions

• Log Files must include:– Time measurements (ms)

• Supporting Data:– Users want to drill down and tell

stories– General Research Contextual

Inquiries (U1, U5, U8, U9, U10)

Courtesy of Andrea Knight


Session Browser

• Visualizes the pattern and context of each student’s actions • Log Files must include:

– Time measurements (ms)– Event Hierarchy

• Supporting Data:– Users want to drill down and tell stories– Requirements Solicitation (VanLehn)– General Research Contextual Inquiries (U1, U3, U4, U5, U7, U8, U9, U10)

Courtesy of Jack Mostow


Event Filtering (aka Rollup Table)

• Allows researcher to specify their variables and calculations over those variables

• Supporting Data:– Requirements Solicitation (Koedinger, VanLehn)– General Research Contextual Inquiries (U5, U8)– U5: “By the time I get the data I need out of the system, I’ve

forgotten the question I wanted to ask”


Exporting Data


Pre-Defined

Free-Form

LearnLab

Data Shop

Export

PSLC


Project Themes


Pre-Defined

Free-Form

LearnLab

Data Shop

Export

PSLC


Project Themes

1. Support All LearnLab Tutors

Problem SubgoalKnowledge Component Right Answers Wrong Answers Unanticipated Errors Wrong Answer Feedback Hint Error Clusters

Algebra ? ? ? ? ? ? ? ? ?Chemistry

Chinese ? ? ? ? ? ? ? ? ?ESL ? ? ? ? ? ? ? ? ?

French ? ? ? ? ? ? ? ? ?Geometry –

Physics / -

Concept in Tutor

– Concept not in Tutor

? Not enough information


Project Themes

2. Facilitate Inter-Report Navigation


Project Themes

3. Let Researchers See Only the Data They Want to See, When They Want to See It.– Reports show only what’s relevant– Find out how researchers want to narrow

down their data in reports– Minimize interface elements for each

report


Preliminary Design


Design for Theme 2

• Large buttons to move between reports • Key Considerations:

– Data transfer from report to report– Saving reports with interesting data along the way– Issues of scale


Design for Theme 3

• Specialized filtering for each report • Key Considerations:

– Condensing report specification & modification into one set of controls

– Distinguishing between singular & multiple selection

– Enabling fast selection

• Design Support:– List boxes for long lists of variables (StatTransfer)– Clear representation of curriculum hierarchy

(Froogle)

• Design Challenges:– Minimizing space while maximizing speed– Providing sufficient context to the user


User Testing Findings

• 2 users doing an analysis with learning curves– 1st year HCI researcher PhD, familiarity with tutor

studies– 4th year education researcher PhD, no tutor studies

• Needs discovered:– Compare curves against pre- and post-tests– Show p-value and regression equation– Describe sample– Click on a Learning Curve spike to understand

context– Show text of the problem


Project Plan

Pre-Summer Workshop• Add breadth to current structure

– Error Reports– Help Effectiveness– Problem Profiles

• Develop Filtering Designs (Mid-Fi)• Prototype Session Browser & Timeline

Visualization (Low-Fi)• Heuristic Evaluation• User Testing

Pre-Workshop: 6 wks Post-Workshop: 5 wks1 wk


Project Plan

Summer Workshop (Week of June 27th)• High-Intensity User Testing, Iteration

Post-Summer Workshop• Database Connections for Pre-Defined Reports• Session Browser & Timeline Visualization (Mid-Fi)• User Testing• Revisions

Pre-Workshop: 6 wks Post-Workshop: 5 wks1 wk


Final Takeaways

1. Rules for Data Sharing– Sharing outside research group

• Competitive Analysis revealed need for sharing regulations

• U5: “I’m happy to share my data if the researchers follow PSLC’s guidelines”

• For now, no open sharing

– Sharing inside research group• Seen as a need in General Research CIs (U1 -

U8)• Designed & tested by end of summer


Final Takeaways

2. Context Matters– Understanding data is hard without context– Issues discovered to date:

• Complex naming => complex design (CA) • Tutor writers use intimate knowledge of the

tutor frequently (U5, U7, U9, U11, U12)• Researchers unfamiliar to the tutor need

that intimate knowledge (TA Pilot Test)

– TA Pilot Test U2: “So you’d have… a master book that would tell you what all the problems are? Because I’d never remember.”


Final Takeaways

3. Courses in Development Should Log Necessary Data

– Survey course committees as needed– Proposed reports will function with

current PSLC tutors– Can’t predict what future tutors may and

may not log


Fin

Feedback?

MHCI PSLC Project [email protected]


Behavior Graph

• Illustrates the strategies students are using to solve problems

• Log Files must include:– Sub-goals

• Supporting Data:– Requirements Solicitation (Koedinger)– General Research Contextual Inquiries

(U8, U10)– Course Committee Survey (Chem)– Think Aloud Pilot (U2)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Courtesy of Data Shop


Physics Survey

Tutor Element ANDES WhyAtlas

Problem Problem Task

Sub goal Entries Expectations / Responses

Knowledge Comp Principles Principles

Right Answers Right Answers Correct Features

Wrong Answers Recognized Errors Recognized Errors

Unanticipated Errors Unrec. Errors Unrec. Errors

Wrong Ans Feedback

Red Incorrect Error State

Hint Next Step Help / What’s Wrong Help

N/A

Error Cluster Error Classes Misconceptions


Chinese/French Survey

Tutor Element Chinese/French

Problem Activities

Sub goal N/A

Knowledge Comp Skills

Right Answers Correct Answers

Wrong Answers Incorrect Answers

Unanticipated Errors Unpredicted Errors

Wrong Ans Feedback

Error Feedback

Hint Funtional / Contextual Help

Error Cluster (e.g., Orthography, Meaning, Form)

MHCI PSLC Data Shop Project Spring Progress Report.

Documents

Transcript of MHCI PSLC Data Shop Project Spring Progress Report.