Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Thinking Hard Together: Thinking Hard Together: the Long and Short of the Long and Short of
Collaborative Idea Generation in Collaborative Idea Generation in Scientific InquiryScientific Inquiry
Hao-Chuan Wang, Carolyn P. Rosé, Yue CuiCarnegie Mellon University
Chun-Yen ChangNational Taiwan Normal UniversityChun-Chieh Huang, Tsai-Yen Li
National Chengchi University
Funded through the National Science Foundation, The Pittsburgh Science of Funded through the National Science Foundation, The Pittsburgh Science of Learning Center, and the Office of Naval ResearchLearning Center, and the Office of Naval Research
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Why Collaborative Idea Generation?Why Collaborative Idea Generation?
Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266
• Supported by common CSCL environments
• Idea generation (problem finding) is recognized as preparation for problem solving (e.g., Hmelo-Silver, 2004)
• Its value as a learning task is not well established
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Problem:Problem: Process Losses in Group Process Losses in Group Idea GenerationIdea Generation
• Well known problem (Connelly, 1993; Diehl & Stroebe, 1987; Kraut, 2003)
• Non-interacting groups may produce more and better ideas (Hill, 1982; Diehl & Stroebe, 1987)*How many people does it How many people does it
take to screw in a light bulb?take to screw in a light bulb?
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Research QuestionsResearch Questions
• Can we design support for collaborative idea generation that mitigates process losses?
• Are there long term learning benefits of collaborative idea generation that outweigh the short term productivity losses?
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
OutlineOutline• Vision: Using language technologies to
offer context sensitive support for collaborative idea generation
• Experimental Study (2X2 factorial design)– Pairs versus Individuals– With versus without support from an
automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
OutlineOutline• Vision: Using language technologies
to offer context sensitive support for collaborative idea generation
• Experimental Study (2X2 factorial design)– Pairs versus Individuals– With versus without support from an
automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
The Debris-FlowThe Debris-FlowHazard Task Hazard Task (Chang & Tsai, 2005)(Chang & Tsai, 2005)
Task1: “What are the possible factors that might cause a debris-flow hazard to happen?”
Task 2: “How could we prevent it from happening?”
Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
The Constructive Process of The Constructive Process of Idea Generation Idea Generation (Brown & Paulus, 2002)(Brown & Paulus, 2002)
• Idea Generation Prompt– What are the possible factors that might cause a debris-flow hazard to
happen?
• Domain Knowledge– Debris flow refers to the mass movement of rocks and sedimentary
materials in a fluid like manner– There are many typhoons (i.e. hurricanes) in Taiwan during the
summer• Bridging Inferences
– Heavy rain implies the presence of massive amounts of water– The presence of massive amounts of precipitation is likely to cause the
fluid movement of rocks • Idea
– Typhoons could be a factor causing a DFH to happen
IdeasDomainKnowledge
BridgingInferences
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
http://www.cs.cmu.edu/~cprose/TagHelper.html
Automatic Support for Automatic Support for Collaborative Idea GenerationCollaborative Idea Generation
• Trigger dialogue agents with an automatic analysis of a collaborative learning interaction
TagHelper
Labeled Texts
Unlabeled Texts
Labeled Texts
A Model that can Label More Texts
Time
Idea
Lab
els
Student 1 People stole sand and stones to use for construction.
Agent Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?
Student 2 Development of mountain areas often causes problems.
Student 1 It is okay to develop, but there must be some constraints.
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
2 Part Feedback Generation2 Part Feedback GenerationStudent 1People stole sand and stones to use for construction.
Agent Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable.
Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?
Comment: acknowledges the idea the student just contributed
Tutorial: points the student in a direction for moving on with brainstorming
Designed based on “category labels” shown to be effective in previous studies of idea generation (Dugosh et al., 2000; Nijstad & Stroebe, 2006)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
OutlineOutline• Vision: Using language technologies to
offer context sensitive support for collaborative idea generation
• Experimental Study (2X2 factorial design)
– Pairs versus Individuals– With versus without support from an
automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
MethodMethod• Experimental Design: 2X2 Factorial design
– Working in Pairs vs Working Individually– Feedback from VIBRANT vs No Feedback
• Participants: 42 10th grade students from a central Taiwan high school were randomly assigned to 4 conditions
– Individual+NoFeedback (7 students)– Individual+Feedback (7 students)– Pair+NoFeedback (7 pairs)– Pair+Feedback (7 pairs)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Collaborative Idea Generation Support Collaborative Idea Generation Support Using Dialogue AgentsUsing Dialogue Agents
Conferencing Mode:Student 1, Student 2 &Dialogue Agent
Task Description
Student 1’sContribution
Student 2’sContribution
Agent’sFeedback
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Experimental ProcedureExperimental Procedure
1. Background reading (10 min)2. Pretest (15 minutes)3. Brainstorming 1 (30 minutes)
• Experimental manipulation takes place during this phase
4. Brainstorming 2 (10 minutes)• Serves as a “transfer task”
5. Post test (15 minutes)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
What counts as a good idea?What counts as a good idea?• The Debris-Flow Hazard task was designed by a
panel of science educators as an assessment of creative problem solving ability
• It has been used in a series of classroom studies in Taiwan
• The student responses from these assessment studies have been evaluated by the panel
• Based on this data, the panel decided on a set of “valuable ideas” for each task, which are used to measure idea generation success
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Analysis of Verbal Data: First Analysis of Verbal Data: First Brainstorming TaskBrainstorming Task
• Conversation logs segmented into idea units– Typically at contribution boundaries– Contributions containing more than one idea broken up– Agreement of 2 coders on 10% of data was satisfactory
(Kappa .7)• Idea units coded for one of 19 specific idea
labels, other ideas related to correct solutions, and other ideas
– Agreement of 2 coders on 10% of data was good (Kappa .82)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Analysis of Verbal Data (Second Analysis of Verbal Data (Second Brainstorming Task)Brainstorming Task)
• Idea units from second brainstorming task already segmented
– Coded with 15 pre-defined “valuable” ideas defined by experts or other
– Agreement of 2 coders on 10% of the data was acceptable (Kappa .74)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
OutlineOutline• Vision: Using language technologies to
offer context sensitive support for collaborative idea generation
• Experimental Study (2X2 factorial design)– Pairs versus Individuals– With versus without support from an
automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Expected Productivity loss in pairsExpected Productivity loss in pairs
• D.V.: Number of unique ideas• Results of ANOVA:
- Nominal Dyad > Real Dyad(p<.001, d=1.51)
• Productivity loss is evidenced
RealDyad
Nominal Real
Conditions
Num
. Uni
que
Idea
(Bra
inst
orm
ing
1)
02
46
810
1214
NominalDyad
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Students in Pairs Learned Less Students in Pairs Learned Less
IndividualsPairs
• D.V.: Post test score• Covariate: Pre test score• Results of ANCOVA:
Individuals > Pairs(p<.01, Cohen’s d=.1.68)*
• Effect mediated by process loss
– R2=.48, p< .005, N=42
*Based on conversion formula by Cohen(1988): d = 2 * f for k=2
Ind Pair
Conditions
Adj
uste
d P
ostte
st S
core
02
46
810
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
SystemSystem Feedback as Learning Support during Feedback as Learning Support during BrainstormingBrainstorming
• D.V.: Post test score• Covariate: Pre-test
score• Evaluating the influence
of system effect on domain learning (pre to post learning increases)
• Result: System Feedback > No Feedback (p<.05, d=.70)
NoF F
Conditions
Adj
uste
d P
oste
st S
core
02
46
810
No Feedback
System Feedback
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Effect of Working in Pairs on Task 2 Effect of Working in Pairs on Task 2 SuccessSuccess
Individuals
Pairs • D.V.: Number of unique ideas in the subsequent idea generation task
– Note: All students worked alone on this task
• Results: Pairs > Individuals(p<.01, d=.92)*
– No main effect of System Feedback
Ind Pair
Conditions
Num
. Uni
que
Idea
s (B
rain
stor
min
g 2)
01
23
45
67
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
• Sig. Interaction Effect(p<.05, d=.78)*
– Best: Pairs with Feedback
– Worst: Individuals with Feedback
Ind+NoF Ind+F Pair+NoF Pair+F
Condition
Num
. Uni
que
Idea
s (B
rain
stor
min
g 2)
01
23
45
67
IndividualsPairs
Effect of Working in Pairs on Task 2 Effect of Working in Pairs on Task 2 SuccessSuccess
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
• Main effect on Pair/Ind– F(1,38)=4.19, p<.05, d=.6– Pair > Individual
• No interaction
• Evidence of mediating success on Task 2– Significant correlation with
success on task 2– R2=.49, p < .0001, N=42
Ind Pair
Condition
P(ta
sk2
| rel
ated
task
1)
0.0
0.1
0.2
0.3
0.4
0.5
Connection between Task 1 and Connection between Task 1 and Task 2 in Pairs ConditionsTask 2 in Pairs Conditions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
““Off-task Ideas”Off-task Ideas”• Solutions were mentioned
more frequently in the pairs conditions and in the feedback conditions
– Note: Only marginal• Mention of solutions
during the first task correlated with success during the second task
– R2=.13, p< .05, N=42
NoF FNo-Feedback/Feedback
Mea
n (v
alua
ble
solu
tions
)0.
00.
51.
01.
52.
02.
5
NoF FNo-Feedback/Feedback
Mea
n (v
alua
ble
solu
tions
)0.
00.
51.
01.
52.
02.
5
Nominal Real
Type of Dyad
Mea
n (v
alua
ble
solu
tions
)0.
00.
51.
01.
52.
02.
5M
ean
(val
uabl
e so
lutio
ns)
0.0
0.5
1.0
1.5
2.0
2.5
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Connection between Task 1 and Connection between Task 1 and Task 2 in Pairs ConditionsTask 2 in Pairs Conditions
Speaker Text
Student 1 People stole sand and stones to use for construction.
VIBRANT
Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?
Student 2 Development of mountain areas often causes problems.
Student 1 It is okay to develop, but there must be some constraints.
Dialogue from Task1Dialogue from Task1
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Pair+NoF Pair+F
Condition
LSA
-bas
ed S
imila
rity
0.0
0.1
0.2
0.3
0.4
0.5
• t(38)=2.57, p<.05, d=.76• Individuals in the Pair+F
condition tended to stay in the same topic more
• t(24)=1.994, p<.05, d= .77• Pair+F > Pair+NoF• Peers in the Pair+F condition
‘talked’ more similarly
Greater Cohesiveness in Pairs Greater Cohesiveness in Pairs with Feedbackwith Feedback
Pair+NoF Pair+F
Condition
Pro
potio
n of
Sta
ying
in th
e S
ame
0.0
0.1
0.2
0.3
0.4
0.5
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Pairs versus IndividualsPairs versus Individuals• Pairs approached the problem from a broader
perspective– Better preparation for problem solving
• Feedback increased the intensity of Task 1 performance
– Trend of effect on Task 2 consistent with the overall positive effect of working in pairs
– Negative effect on Task 2 only in the Individual condition where focus was narrowly on Task 1
• But how can we balance success at learning/Task 1 with success at Task 2?
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Process AnalysisProcess Analysis• Conjecture: Process losses occurred at early
stage of group idea generation (Diehl&Stroebe, 1991)– Never tested directly– Folk wisdom: brainstorm alone before group
brainstorming– Note: Opposite has been proven effectively
previously (Brown & Paulus, 2002)• Finding: about half of unique ideas contributed
in first 5 minutes• Cognitive interference strongest during that time• Feedback will show an effect after 5 minutes
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Process AnalysisProcess Analysis
Process loss Pairs vs Individuals:F(1,24)=12.22, p<.005, 1 sigma
Process loss Pairs vs Individuals: F(1,24)=4.61, p<.05, .61 sigma
Negative effect of Feedback:F(1,24)= 7.23, p<.05, -1.03 sigma
Positive effect of feedback:F(1,24)=16.43, p<.0005, 1.37 sigma
0 5 10 15 20 25 30
02
46
810
12
Time Stamp
#Uni
que
Idea
s
Unique Ideas
Nom+NNom+FReal+NReal+F Pairs+Feedback
Individuals+NoFeedback
Pairs+NoFeedback
Individuals+Feedback
Pairs+Feedback
Individuals+NoFeedback
Pairs+NoFeedback
Individuals+Feedback
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
OutlineOutline• Vision: Using language technologies to
offer context sensitive support for collaborative idea generation
• Experimental Study (2X2 factorial design)– Pairs versus Individuals– With versus without support from an
automatic brainstorming support agent• Main results and process analysis• Conclusions and current directions
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
What did we learn?What did we learn?• Good News
– Interaction with dialogue agents during brainstorming increases learning
– Feedback increases idea generation in pairs and individuals after 5 minutes
• Bad News– Process losses in group brainstorming may
hinder learning– Feedback decreases idea generation in the
first five minutes
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Current DirectionsCurrent Directions• Follow-up study
– Students brainstorm alone for five minutes without feedback
– Then work together for 25 minutes with feedback• Continuing to investigate automatic forms of
collaborative learning support– Thermodynamics (Kumar et al., 2007)– Middle school math (Kumar et al, to appear)
• Continued work on automatic collaborative learning process analysis (Rosé et al, Under Review)
Language Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon UniversityLanguage Technologies Institute & Human-Computer Interaction Institute, Carnegie Mellon University
Any Questions?Any Questions?
Htpp://www.cs.cmu.edu/~cprose/TagHelper.htmlHtpp://www.cs.cmu.edu/~cprose/TagHelper.html
Top Related