Susan Finger and Sharad OberoiCarnegie Mellon University

ArtifactWebs: Navigable Product Structures

Collaborative learning in design

• Goal– Develop tools that encourage process

competence, constructive skills, and reflective practice• Web–based collaboration tool• Meeting capture and summarization• Navigable artifact webs

Collaborative learning in design

• Goal– Develop tools that encourage process

competence, constructive skills, and reflective practice• Web–based collaboration tool• Meeting capture and summarization•Navigable artifact webs

Collaborative learning in design

• Assertions– Most learning in design classes takes place in

team meetings and in individual activities undertaken to help meet team goals

– Argumentation, co-construction, and reflection are important elements of collaborative learning

Outline

• Setting– Engineering design capstone course– Ongoing project to understand collaborative

learning by student design teams

Engineering design capstone course

• Required for all accredited engineering programs in US

• Commonly stated goal: Students should synthesize all the engineering knowledge they have acquired as undergraduates

Engineering design course projects• The projects are usually:

– Team-based– Company-sponsored (or client-driven)– Non-competing (each team has an independent

project) – Often taught by academics with little project

experience and even less design experience

• The grade is usually based on– The quality of the final product– The self-reported quality of the team interactions

Engineering design course projects• Students

– are novices in their domain knowledge– are novices in their knowledge of the design

process– often judge their success by the grade they earn or

by the artifacts they produce

• Teacher

– rarely plans to use the team’s design directly– usually does not attend group meetings – often does not know if a feasible solution exists to

the design problem as stated

Engineering design course projects• Team membership can change over time,

so it is difficult to keep track of the progress as well as the options explored

• Inherent temptation to start the work over from scratch, wasting time and resources

• These problems exist for both industry and student teams, but are usually more severe for student teams

Engineering design course projects

assessmentactivities

learning goals

Collaborative learning research group

• Our focus is to develop tools that encourage process competence, constructive skills, and reflective practice– Need to capture process to understand student

learning– Collaboration tools designed for industry rarely

work well for student teams– Sequence of two National Science Foundation

grants on collaborative learning in design

NSF Grant: Collaborative Learning across Time and Space

• Goal: To take advantage of advances in mobile computing to create collaboration tools for student design teams

• Means: Create an environment that – facilitates group collaboration for students– enables faculty to peer into the collaborative

learning process

• Hook: Students design the tools they need for their own collaboration

Kiva collaboration tool

• Takes advantage of students willingness to send email, use IM, post on newsgroups, send text messages

• Design goal: Create an interface that students perceive to be equivalent to their preferred communication modes; that is:

make it feel like chat

Design education testbed

• RPCS: Rapid prototyping of computer systems– Interdisciplinary, capstone design course– Ambitious projects, e.g.

• GM companion car-driver interface• Context aware cell-phone• Wireless classroom on the Voyager science

boat

Capturing in-process data

• For 4 years, RPCS has used the Kiva for team collaboration– Light-weight collaboration tool– Combines functions of e-mail and bboards– Widely accepted and liked by student teams; it

feels likes chat and meets their needs– Each year’s Kiva has hundreds of threads and

thousands of posts and filesWe have 4 years of data of all the team

conversations and files that would normally go through email or chat

Alan  I was asking which formula you wanted to use. which comes down to which regression line we are using to map from the fuel values to RPM. I used : y = 327.89x3 - 2194.6x2 a 5087.4x - 2719.1. R2 = 0.9997

Kim Ah I see.. To be quite honest, I was planning on discussing this issue during class for tomorrow. For now we can just use the one that you wrote above and we will talk more about it during tomorrow's lecture. Thanks!!

Chris 

It also might be good to do something as a special case in the formula so that we don't return a negative number for low values of fuel consumption. It looks really weird in the dashboard. ;)

Alan hehe.. Right. Thanks for testing that. :

Alan Kim: I switched formulas. Now I am using this one. y = -1812.5x4 + 6744.9x3 - 8322.2x2 + 4325.9x + 4.1796. R2 = 0.9993. This is because I had to re-center the data to 0, and this new formula works much better then the alternatives. This is engine 1 I think. (the first set of numbers)

Alan Sigh.. another new formula. Forgot the upper bound. y = -81.054x3 + 311.21x2 + 396.24x - 10.003. R2 = 0.9859

Kiva usage

• How do students use the Kiva? – Group coordination (18%) – Knowledge and work exchange (33%)– Preparation of deliverables (24%)– Other (25%)

ADEPT - Assessing Design Engineering Projects Classes with Multi-disciplinary Teams

• Develop a physical infrastructure that enables the capture of synchronous and asynchronous interactions of student design teams– The (complete) up-to-date record of all of a team’s

interactions will enable us to create ArtifactWebs that integrate and summarize team communications

– The ArtifactWebs will provide traceability and accountability for individual contributions to shared knowledge

– The ArtifactWebs will enable facilitated improvement of engineering design courses (i.e. the instructor will know when to intervene)

Capturing in-process data

• This year, we collected audio files of meetings– Individual speaker– Automated speech to text transcripts– Observation and coding of all team meetings

We have 1 year of data of team conversations (with many gaps)

Objectives

• To create ArtifactWebs that– represent the state of the project based on the

artifacts described in the project documents– enable designers to search and navigate to

find relevant information quickly and efficiently– evolve as the artifact, and the documents

about it, evolve.

Design documentation

• Design project documents are generated by different team members at different times during a project, so no one is aware of everything that is in all the documents

• Locating the right information among evolving documents or reference documents can be time consuming

• Even for teams with well-structured document management systems, finding the correct paragraph or document fragment for a given topic can be difficult

Visionary Scenario

A student in the wearable computer class is working on developing a text to speech module for a mobile device. Someone tells her that last year’s class developed an OCR (optical character recognition) module for the Trinetra project. She accesses the Trinetra DesignWeb through the class web space.

Visionary Scenario

She quickly searches (using standard search) to find the subweb for the OCR module.

She then browses within the OCR module exploring various aspects of the OCR design from the previous team.

Visionary Scenario

Finally she focuses on the modules on the mobile device. She reads the segment of the final report on the OCR mobile module as well as some of the supporting documents that led to the final decisions in the OCR design.

Challenges

• Levels of abstraction• Alternate views for different users• Credibility of source (transcripts of

meetings vs. final reports)• Identifying the structure of created

knowledge, especially for different versions of the same document

• Identifying the design intent

Strategy overview

• Divide documents into topic segments• Cluster segments by semantic similarity

(e.g. revisions of same paragraph or similar paragraphs from different sources)

• Summarize each cluster• Create a diagram that connects the key

words in the document summaries• Develop graphical display algorithms that

enable users to search and navigate the graphs to access the underlying documents

Segmentation

• Divide documents into topic segments– use the explicit structure of the documents

(table of contents and internal headings) – use existing text segmentation algorithms such

as TextTiling, which performs semantic clustering of terms and topic identification based on clustering

• Issue: Size of segments (big or little chunks)

Clustering

• Cluster segments by semantic similarity (e.g. revisions of same paragraph or similar paragraphs from different sources)– InfoMagnets, created by Rosé, uses Latent

Semantic Analysis and document clustering to automatically generate a bubble diagram, which a user can then incrementally adjust through the interface.

• Issue: Non-standard vocabulary across disciplines

Summarizing

• Summarize each cluster– Summarization is widely used in web searches – Many potential summarization algorithms exist

• Issues: What types of summaries are useful for designers and what types are useful for creating the graphs

Graphing

• Create a diagram that connects the key words in the document summaries– Use co-word analysis to find relationships

among the key words in the document summaries

• Issues: Level of granularity and strength of relationships

Visualizing

• Develop graphical display algorithms that enable designers to search and navigate the graphs to access the underlying documents

• Issues: Algorithm and interface design

Design teams

documents Auto-summarization

Summarized fragmentsDocument

fragments

Network of versioned fragments

Collocation analysis

Version matching Credibility mapping Document structure

and associated metadata

Design teams

documents Auto-summarization

Summarized fragmentsDocument

fragments

Network of versioned fragments

Collocation analysis

Version matching Credibility mapping Document structure

and associated metadata

Design teams

documents Auto-summarization

Summarized fragmentsDocument

fragments

Network of versioned fragments

Collocation analysis

Version matching Credibility mapping Document structure

and associated metadata

Conclusions

• Creating ArtifactWebs automatically from student design documents is useful for organizing the information into product structures.

• These product structures can be used for developing computational environments that support systematic modeling and also for characterizing design problems.

• ArtifactWebs can help us understand the content and nature of information related to various aspects of the artifact and how designers generate and refine it.

Questions?

Prior work

• Previous work on automatic topic segmentation has focused on segmentation of expository text written by professionals– technical articles, such as journal papers– non-technical articles (e.g. blogs) – multi-party dialogues in a synchronous (e.g. chat) or

asynchronous environment (e.g. discussion-boards)• Student project reports do not come under any of

these categories• Nobody has evaluated student design reports that

are often characterized by their authors’ lack of experience in technical writing

Proposed Solution

• Navigable ArtifactWebs that will:– Aid instructors and students alike by giving them a

bird’s eye view of the evolving design.– Enable team members to explore the ideas that

have been generated during the design process, the connections between the ideas, and the evolution of the ideas.

– Direct the users to the relevant fragment of a document that contains the detailed discussion of an idea, in addition to searching the relevant topics using a query-based approach.

Challenges

• Levels of abstraction• Alternate views for different users• Credibility of source (transcripts of

meetings vs. final reports)• Identifying the structure of created

knowledge, especially for different versions of the same document

• Identifying the design intent

Background

• Two broad categories of previous work in topic segmentation:

1. Lexical Cohesion Models: based on the central idea that the segmentation of text is guided primarily by distribution of terms used in it, in contrast to using cue words for the purpose. Examples: TextTiling (Hearst, 1997) and Latent Semantic Analysis (Landauer and Dumais, 1997)

2. Content-oriented Models: based on the evaluation of reoccurrence of topic patterns over multiple thematically similar discourses. Examples: Approaches based on Hidden Markov models (Barzilay et al, 2004).

TextTiling (Hearst, 1997)

1. Block comparison approach: Adjacent pairs of text blocks are compared for overall lexical similarity. The sentences are grouped into blocks of size N/2 each, where the more the terms are similar to each other in the two blocks, the higher the lexical score we get at the gap between them.

2. Vocabulary introduction approach: Adjacent pairs of text blocks are compared for overall lexical dissimilarity. The sentences are grouped into blocks of size N/2 each, where the more thematically unrelated terms are introduced, the higher the lexical score we get at the gap between them.

TextTiling (Contd)

3. Lexical chain-based approach: Adjacent pairs of text blocks are compared for identifying the number of active chains, or terms that repeat within threshold sentences and span the sentence gap. This approach is based on the assumption that when a term is repeated in a more or less short distance (called a hiatus), a lexical chain is created between these two occurrences. Thematic boundaries are set in the text at places where the number of chains is minimal. This approach attempts to segment texts at places where the local cohesion is the lowest.

Museli (Arguello et al,2006)

• Used for evaluating dialogues. • It combined evidence of topic shifts from lexical

cohesion with linguistic evidence such as syntactically distinct features.

• It used unigrams, bigrams, POS-tagging and lexical scores as the features to solve the segmentation problem as a binary classification problem where each contribution is classified as NEW_TOPIC if the contribution introduces a new topic and SAME_TOPIC otherwise.

Three degenerative approaches

a) Classifying all contributions as NEW_TOPIC (ALL), b) Classifying no contributions as NEW_TOPIC (NONE), c) Classifying contributions as NEW_TOPIC at uniform

intervals (EVEN), corresponding to the average reference topic length

Experiments

• Data Source: Documents created by students in the Rapid Prototyping of Computer Systems classes at Carnegie Mellon as our data-set.

Experiments

• Evaluation Metrics: a) Pk measure determines the probability of

misclassifying two contributions a distance of k contributions apart from each other by determining if they constitute the same topic segment or not. Lower Pk values are preferred over higher ones.

b) F-measure refers to the weighted harmonic mean of precision and recall.

Experiments (Contd)

• Gold Standard: We use the section and sub-section headings for student documents as tags for different student document fragments and the boundaries between them as the correct segmentation locations.

Experiments (Contd)

• Methodology:1. TextTiling: Block comparison approach2. Museli: Naïve Bayes classifier with an attribute

selection wrapper and the Chi-square test for ranking the attributes using 10-fold cross-validation. [All along we were careful not to include instances from the same document in both the training and test sets on any fold so that the results would not be biased.]We trained a model with the top 1000 features, and applied that trained model to the test data.

3. Three degenerative approaches

Results

TextTiling works best, while Museli worked worst.

Conclusions

• TextTiling was designed to partition texts into contiguous, non-overlapping subtopic segments. So it works best for segmenting student design documents.

• Naïve Bayes algorithm works best when there is much more training data available than what we provided, and where the documents are more stratified, so there is less chance of overlapping words in each category.

• The degenerative approaches give us a baseline for what happens with regular segmentation regardless of the content.

Possible weak points in our approach

• Stoplist could be customized for each document.• Lemmatization removed words with the same

root, but this may have implications on engineering design documents.

Summary

• We evaluated the approaches considered successful for automated topic segmentation of monolithic text written by professional authors and multi-party dialogues to the documents written by students as part of their project-based courses.

Future/Ongoing Work

Future/Ongoing Work

• We plan to evaluate the implementation of content-oriented models used successfully for a series of thematically similar discourses (such as a number of newspaper articles about similar events) for different versions of student design documents. Since these approaches greatly rely on the structure of the document for successfully implementing the topic segmentation, having student documents characterized by a lack of such explicitly defined structures would be interesting.

• We believe that such evaluations will slowly but steadily help us move towards achieving the vision of ArtifactWebs.

Thanks!

2002 NSF grant

• Team-Based Design: Collaboration Across Time and Space – Daniel P. Siewiorek, Susan Finger and Asim

Smailagic – Combined Research and Curriculum

Development Grant

Kiva vision

• Rapid Prototyping of Computer Systems– 25 students primarily from engineering, and

also from design, English, HCI designed, developed, integrated, and tested an environment for student team work

– In their visionary scenario, the Kiva is an interactive physical and digital workspace that addresses the requirements of interdisciplinary teams. It is the digital equivalent of a dedicated project room.

Kiva vision

First implementation

Second iteration

Assessment strategy

• Pre and post essays on design process• Pre and post domain knowledge tests• Focus groups • Coding and analysis of the posts

Proposed Solution

• Navigable ArtifactWebs that will:– Aid instructors and students alike by giving them a

bird’s eye view of the evolving design.– Enable team members to explore the ideas that

have been generated during the design process, the connections between the ideas, and the evolution of the ideas.

– Direct the users to the relevant fragment of a document that contains the detailed discussion of an idea, in addition to searching the relevant topics using a query-based approach.

Potential Benefits to Members

• Ability to find information in documents from prior designs

• Ability to track a design-in-progress through the completeness of its DesignWeb