Post on 01-Sep-2014
description
Lu Xiao Facul ty of Informat ion & Media Studies
Depar tment of Computer Science The Univers i ty of Western Ontar io
Ht tp: / /h i i . f ims.uwo.ca
14-03-04
1
Shared Rationales in Group Activities
Shared Rationales in Group Activities
14-03-04
2
Related Studies: Explanations in Knowledge-based Systems (KBS) Types of explanations, Content of explanations, Effects of KBS Explanations (explanation use behavior, learning, perceptions, and judgmental decision making) Shared Information in Group Activities � How and why group members share information � The factors of information sharing, information pooling phenomenon � The information practices and cultures that members develop � The effects of shared information and aspects of the shared information (e.g.,
representation strategy, the use of language) ¡ Influence the change of people’s attitudes ¡ Shared reflections
Tools to support information sharing in group activities � Group decision support systems for hidden profiles � Tools for capturing, archiving, and reusing design rationales � Tools to foster reflective thinking in group learning activities
Rationale - the information that justifies one’s ideas, approach, and solution in group activities.
Research Gaps
14-03-04
3
1). the effects of shared rationales in group activities 2). Design requirements to promote the processes of articulating, sharing, and managing rationales in group activities
The Role of Shared Rationales in Group Ideation and Deliberation Activities
14-03-04
4
The effects of rationale awareness in � small group ideation activities � Large online crowdsourcing ideation activities
The effects of shared rationales in � Large online deliberation activities
Rationale Awareness, as part of Activity Awareness (Carroll et al., 2003, 2005, 2011; Carroll, Rosson, Farooq, & Xiao, 2009), refers to one’s awareness of the other group members’ rationales in a group activity
14-03-04
5
The Effects of Rationale Awareness in Small Group Ideation Activities
The effects of rationale sharing (Xiao, 2011a, 2012; Xiao & Carroll, 2013)
• Rationale awareness can contribute to one’s • awareness of others’ knowledge and intellectual
contribution; can affect the development of his/her reflection skills
• Explicit rationale sharing has potential downsides such as groupthink
14-03-04
6
Related Work: � Quality measure of different means
¡ Providing real-time assessment ¡ Collecting multiple assessment ¡ Analyzing workers’ behavior ¡ Parallel vs. iterative approach
The Effects of Shared Rationales in Online Crowdsourcing Ideation Activities
14-03-04
7
In an ideation task performed through online crowdsourcing processes, whether and how sharing previous workers’ good rationales of their generated ideas affects the ideas’ quality in the task? Pros: awareness of the others’ knowledge and intellectual contributions Cons: problems with explicit rationale sharing
The Effects of Shared Rationales in Large Online Ideation Activities
Research Design
In an idea evaluation task performed through online crowdsourcing processes, whether and how showing the idea’s rationale affect its evaluation?
Hypothesis: Making the ideas’ rationales available to all of its evaluators reduce the variation between evaluations by multiple raters
Research Design
� Two iterative conditions in the idea generation task: presence vs. absence of previous workers’ rationales
� Two idea evaluation conditions in the idea evaluation task: presence vs. absence of the idea’s rationale
� Manipulation of the rationale’s quality: Experiment 1, Experiment 2, and Experiment 3 ¡ The quality of the ideas and rationales was checked after all the
iterations were completed (Experiment 1) ¡ The quality of the ideas and rationales was checked at the end of each
iteration (Experiment 2 and 3) – stricter and better quality control of ideas and rationales
Research Design (Little et al.,2010)
Brainstorming/Idea Generation Task
• Six company descriptions • Five names for a company description in each iteration • Six iterations for each company
Rating/Idea Evaluation Task
Each name had 10 ratings
Turkit
• Open source software: Java/JavaScript API for running iterative tasks on Mechanical Turk.
Manipulation Check
14-03-04
12
Results – Average Quality of the Ideas
14-03-04
13
Results – Best Quality of the Ideas
14-03-04
14
Results: Rationale Awareness in Idea Evaluation
Findings: the Shared Rationales in Online Crowdsourcing Ideation Task
� In an idea generation task, the awareness of previous workers’ rationales may slightly improve the average quality but NOT the best quality of the generated ideas in iterative approach
� In an idea evaluation task, the awareness of an idea’s rationale can affect the evaluation outcome and the quality of the rationale may play a significant role on the evaluation
(Xiao, CSCW, 2012; Xiao, CI, 2012; Xiao, JASIST, to appear)
The Role of Shared Rationales in Group Ideation and Deliberation Activities
14-03-04
17
The effects of rationale awareness in � small group ideation activities � Large online crowdsourcing ideation activities
The effects of shared rationales in � Large online deliberation activities
Shared Rationales in Large Online Deliberation
14-03-04
18
� Deliberation concept ¡ Habermas (1989) – public sphere ¡ Halpen and Gibbs (2013)
a communication process that involves at least two individuals; that focuses on a social or political issue where the solutions are identifiable by participants; and that values equality among participation and emphasizes rational thinking and logic instead of a power struggle.
14-03-04
19
Wikipedia’s Article for Deletion (AfD) discussions
Step 1 : Types of rationales; factors of deliberation outcome
What are the types of rationales used in the deliberation? Are there any relationships among the kinds of votes, the article’s topic, the discussion situation (unanimous or non-unanimous), and the final decision?
Shared Rationales in Large Online Deliberation
14-03-04
20
Possible outcomes of deletion discussions Coded as…
Outcome Explanation Keep Article is kept. Changes may or may not be suggested as part of debate. Keep Withdrawn Nomination is withdrawn. No consensus Debate is inconclusive. This might result from disagreement, lack of
participation, or other factors. Article defaults to keep. Procedural close Debate is closed because of problems with the nomination. Delete Article is deleted. Delete Speedy delete Article is deleted under the “speedy” criteria outlined at WP:CSD. Userfy Article is deleted but a copy is given to a user to work on as a draft, and may
be recreated as an article later. Other
Incubate As with userfy, but in a communal space rather than related to a single user. Merge Article is deleted but some or all of its content is added into one or more
existing articles. Rename or Move Article’s title is changed. Its scope may or may not be amended. Convert Article is converted into another type of page, usually one with a structural
function. For example, a list might be changed into a category to be added to the list entries.
Transwiki Article is deleted from English Wikipedia but moved to another Wikimedia project as appropriate – for example, a French article to French Wikipedia or an image gallery to Wikimedia Commons.
Redirect Article’s content is replaced with a pointer to another page. Split Article is divided into one or more new pages, or part of the article is moved
to another page.
Research Methodology
� Qualitative Analysis ¡ Open coding process to classify rationales used in deletion
debates on three selected dates
� Quantitative Analysis ¡ Chi Square Tests ¡ Relationships among articles’ topics and deliberation
outcomes, discussion situations ¡ Relationship between the SOPA act event and the deliberation
outcome
SOPA (Stop Online Piracy Act) act event: On January 18, 2012, the English Wikipedia, Google, and an estimate of 7,000 other smaller websites coordinated a service blackout, to raise awareness.
Sample
� Date selection for qualitative analysis
� Date selection for quantitative analysis ¡ Previous sample ¡ 20 dates for chi-square tests that require larger sample size (a priori power
analysis) ÷ Before the SOPA act event: Jan. 1 - 10, 2012 and Nov. 1 - 10, 2011 (N = 1453) ÷ After the SOPA act event: Jan. 20 - 29, 2012 and March 20- 29, 2012 (N = 1202)
Day # of articles
Total votes for “keep”
Total votes for “delete”
Total “other” votes (merge, userfy, etc)
1 Jun. 2010 89 127 280 37
1 Jun. 2011 73 119 212 23
15 Jan. 2012 67 109 200 63
Findings - Types of rationales; factors of deliberation outcome
14-03-04
23
� Rationales are mainly about the articles’ notability (50%) and credibility (12%); Wikipedia policies are often referred to as well (10%)
� Relationship between the deliberation outcome ÷ and the type of votes: in the case that the decision is delete, there
tend to have more delete votes than keep votes, whereas in the case that the decision is keep, the delete votes are not more than keep votes; the votes other than keep and delete significantly affect those decisions that would change the articles’ status.
÷ and the topic of article : articles about people, for-profit organizations, and definitions are slightly more likely to be deleted than expected; articles about locations or events are more likely to be kept than expected; and articles about non-profit organizations and media are more likely to be suggested for other options (e.g., merge, redirect, etc) than expected
(Xiao & Askin, JASIST, to appear)
Findings - Types of rationales; factors of deliberation outcome
14-03-04
24
� Relationship between the discussion situation ¡ and the type of rationale: more agrees in non-unanimous
situations ¡ and the deliberation outcome: in non-unanimous
situations, it is more likely to have final decisions as keep or other solutions (e.g., merge, redirect, etc)
¡ and the community participation: ÷ more unique Wikipedia IDs in non-unanimous discussions ÷ More participants in a non-unanimous discussion; the most
involved participant was more likely to be recognized in the discussion
(Xiao & Askin, JASIST, to appear)
Findings – the Impact of SOPA blackout event on the Deliberation
14-03-04
25
Before the blackout of the site in response to the proposed Stop Online Piracy Act (SOPA) law, there were slightly less keep cases than expected and after the event there were slightly more keep cases than expected. The effect was more significant on the decisions which would change the articles’ status. These articles were more likely to be deleted before the Act, whereas after that Act (or during the discussions about it) they were more likely to be offered suggestions for other options
14-03-04
26
Wikipedia’s Article for Deletion (AfD) discussions Step 1 : Types of rationales; factors of deliberation outcome
Shared Rationales in Large Online Deliberation
Step 2: Computational linguistic approaches to extract the rationales
Rationale Extractions for Knowledge Management
14-03-04
27
� Motivation – to benefit new editors
� Approach - extraction of rationales that reflect the needed knowledge on Wikipedia policies in AfD discussions
� Technique –Illocutionary Act (Searle, 1976) ¡ Representatives ¡ Directives ¡ Commissives ¡ Expressives ¡ Declarations
Rationale Extractions for Knowledge Management
14-03-04
28
� Directives (Ervin-Tripp, 1976) ¡ Need statements ¡ Imperatives ¡ Imbedded imperatives ¡ Permission directives ¡ Question directives ¡ Hints
Detect Imperatives
14-03-04
29
1. A verb (in its base form) as the root in the phrase structure and this particular verb has no subject child in the dependency structure.
(ROOT (S (INTJ (VB please)) (VP (VB refrain) (PP (IN from) (S (VP (VBG making) (NP (JJ personal) (NNS attacks)))))) (. .)))
2. A personal pronoun or noun (e.g., you, they, username) followed by a modal verb (e.g., should, must) "You must discuss the matter there, and you need to be specific”
Rationale Extractions for Decision-Making Support
14-03-04
30
� Motivation – to facilitate efficient final decision-making
� Approach – elimination of redundancy by identifying representative rationales in the discussion
� Technique – text similarity and sentiment analysis
classify by sentiment polarity
group similar rationales
Discussion
Group A
positive neutral negative
Group B
positive neutral negative
Rationale Extractions for Decision-Making Support
Select representative rationales – similarity score, number of users,
policies
• SEMILAR, a semantic similarity toolkit, was used to compute text similarity
� Compared the performance of similarity measure among algorithms and human evaluation: ¡ Weighted Latent Semantic Analysis (LSA) ¡ Latent Dirichlet Allocation (LDA)
Text Similarity
� Determine the sentiment polarity of a rationale in our language context (“notable”) ¡ MPQA Subjectivity Lexicon + additional words
Sentiment Analysis
Data Input Stanford Parser
Part-of-speech tagged text
Dependency relations
Check modified MPQA subjectivity lexicon to obtain the prior polarity
(if not in MPQA, marked as ‘non-sentiment’)
MPQA format: type=strongsubj len=1 word1=aberration pos1=adj stemmed1=n priorpolarity=negative
• Local negation: A not usually modifies the sentiment word. – “The place is not notable.”
• Predicate negation: using verbs with negative polarity. – “I disagree that the place is notable.”
• Subject negation: a subject leads to the negation of its predicate. – “Neither one of us agrees that the place is notable.”
Sentiment Analysis
� Preposition negation: the polarity of the object following the preposition “of” can be changed by the word modified by the preposition. ¡ “It is a viola&on of notability.”
� Modifier negation: some sentiment word’s polarity can be negated by its modifier. ¡ “The place is of indeterminable notability.”
Sentiment Analysis
• Modifier negation – Phrase in the following combination:
Noun modified by adjective Noun modified by noun Adjective modified by adverb Adverb modified by adverb Verb modified by verb
Sentiment Analysis
� Using machine learning methods to determine the polarity of a phrase that has a modifier and a word
– Features: • First word token • Second word token • First word polarity • Second word polarity • First word part-of-speech • Second word part-of-speech
l Performance of Naïve Bayes, k-nearest neighbor (KNN) and decision tree:
÷ Data: 961 instances (phrases)
÷ Evaluation:10 folds cross validation
Sentiment Analysis
Naïve Bayes K-nearest neighbor Decision Tree
Accuracy (%) 77.94 83.77 80.65
� Bottom-up (recursive) algorithm ¡ Based on dependency structure
Sentiment Analysis
� Evaluation ¡ Data: 236 sentences from discussions in AfD ¡ 3 classes: positive, negative, neutral ¡ Accuracy: 58.47%
Sentiment Analysis
The Role of Shared Rationales in Group Ideation and Deliberation Activities
14-03-04
40
The effects of rationale awareness in � small group ideation activities � Large online crowdsourcing ideation activities
The effects of shared rationales in � Large online deliberation activities
Current research plan To automatic detect rationales from online ideation activities and deliberation activities
Current Research Plan
Rhetorical Structure Theory has been recently used to identify justifications in the social Web (Biran, and Rambow, 2011), where the existence of certain discourse structures has been considered argument indicators. Justification is defined as: 1. Recommendation for action, and motivation for proposed action. 2. Statement of like or dislike or of desires and longing, and subjective reason for this like
or dislike or desire or longing 3. Statement of like or dislike or of desires and longing, and claimed objective reason for
this like or dislike or desire or longing 4. Statement of subjectively perceived fact, with a proposed objective explanation 5. A claimed general objective statement and a more specific objective statement that
justifies the more general one
Presentational relations from RST Treebank were primarily considered
14-03-04
42
The Effects of Rationale Awareness in Small Group Ideation Activities
Discourse relations in shared rationales in the small group ideation activities (Xiao, 2013):
Most used strategies in justifying one’s ideas in the activities were: providing contextual information (circumstance), additional information (elaboration), and evaluation of the information (evaluation)
We are extending Biran and Rambow’s (2011) approach by conducting further analysis on these discourse relations and their potential connections to different types of reasoning.
Acknowledgement
14-03-04
43
John M. Carroll, Mary Beth Rosson, Craig Ganoe, Shawn Clark, Steven High, Anabel Quan-Haase, Yan Luo, Tatiana Vashchilko, Trina Joyce Sajo, William Klie, Becky Ellis, Mengshuo Chen, Nicole Askin, Jill Kavanaugh, Lindsay Baker, Achchana Nadarajah, Yumo Yin, Vadim Mazalov, Wanting Mao, Taraneh Khazaei
Funding support: NSERC Discovery, NSERC Engage, MITACS, SSHRC, FIMS Internal Funding, SSHRC 4A assistance