By: Robin S. Poston University of Memphis – U.S.A Cheri Speier Michigan State University –...
description
Transcript of By: Robin S. Poston University of Memphis – U.S.A Cheri Speier Michigan State University –...
Effective Use of Knowledge Effective Use of Knowledge Management Systems:Management Systems:
A Process Model of Content Ratings And A Process Model of Content Ratings And Credibility IndicatorsCredibility Indicators
By: Robin S. Poston University of Memphis – U.S.A
Cheri SpeierMichigan State University – U.S.A
Presenter: Maged Younan
Knowledge Management SystemsKnowledge Management Systems
KMSs should facilitate the efficient and effective use of firm’s intellectual resources
KMSs Store a huge amount of information
Corporations investment in KMSs is expected to reach $13 Billion by 2007
KMS user should be therefore able to locate relevant and high quality content easily and quickly
What is the problem with KMSs?What is the problem with KMSs?
Have you ever tried to search for information on the internet (or intranet)?
What were the results? How many links / documents did you get?
How many of these were relevant and satisfied your need?
How many included low quality or even incorrect information?
To solve the problem of KMSs, content ratings were introduced
Content ratings are simply the feedback of the previous visitors to the same document / link...etc
Content Ratings -if valid- should help future knowledge workers (searchers) to evaluate and select proper content quickly and accurately
Content RatingsContent Ratings
Are content ratings always valid?Are content ratings always valid?
Do content ratings usually reflect the actual content quality? How accurate are they?
Content ratings may not be always valid due to the following reasons:
Lack of user experience (inappropriate context) Delegation of search tasks to juniors Subjectivity of rating – bias Intentional manipulation of rating
Credibility IndicatorsCredibility Indicators
Credibility indicators are used to assess the content and/or the rating validity.
Credibility indicators will take may depend on:
No. of raters Rater expertise Collaborative filtering
What do we want to study?What do we want to study?
Experiment 1: Examined the relationship between rating validity and KMS search and evaluation process
Experiments 2,3 and 4: Examined the moderating effect of credibility indicators
Experiment 1Experiment 1
““Relationship between rating validity Relationship between rating validity and KMS search and evaluation and KMS search and evaluation
process”process”
Experiment 1 - BackgroundExperiment 1 - Background
On a complex task people usually anchor on inappropriate content
Knowledge workers usually begin with the assumption that available ratings are valid
If rating is not valid, searchers will mislabel high rated content as being of high quality and vice versa
Experiment 1 - AssumptionsExperiment 1 - Assumptions
Searchers will follow any of the following search and evaluation processes depending on the rating validity:
Anchoring on the content and making no adjustment as the content rating is valid
Anchoring on the content and making no adjustment while the content rating is low in validity
Anchoring on the content and adjusting away as the content rating is low in validity
Experiment 1 - HypothesesExperiment 1 - Hypotheses The following hypotheses are thus generated:
H1: Knowledge workers will implement different search and evaluation processes depending on the validity of content rating
H2a: Anchoring on low quality content but adjusting away from that anchor results in higher decision than not adjusting away from the anchor
H2b: Anchoring on high quality content (and not adjusting away) results in higher decision quality than anchoring on low quality content and adjusting
H2c: Anchoring on low quality content and adjusting results in longer decision time than anchoring on high or low quality content and not adjusting away
Setting The ExperimentSetting The Experiment
14 different work plans were created and added to the KMS
3 quality measures were introduced: Clarity Project steps Assigning consultant levels to each project step Availability of senior consultant assignment to special tasks
Setting The ExperimentSetting The Experiment
The 14 work plans varied in quality such that:
1 plan met all 3 quality criteria 6 plans met 2 quality criteria 6 plans met 1 quality criterion 1 plan did not meet any of the 3 quality criteria
Subjects of the experiment had prior but limited experience with the task domain
Setting The ExperimentSetting The Experiment
A pilot test before the experiment was conducted to ensure that subjects had the ability to differentiate between low and high quality work plans
Work plans were given content ratings as follows
No. of Q. Criteria Valid rating Invalid Rating3 5 12 4 21 2 40 1 5
Experiment 1 – Dependant variablesExperiment 1 – Dependant variables
The response of the experiment done was:
The decision quality ( No. of lines matching with the lines of the work plan with best quality)
The decision time (measured in minutes)
Chi square tests were conducted to ensure that no significant effect exists for age , gender, experience and years in school of the candidates (Subject pool is homogeneous)
Experiment 1 – Interpreting the resultsExperiment 1 – Interpreting the results
After running the experiment, candidates were divided into three main clusters
Anchoring on high quality content and making no adjustment
Anchoring on low quality content and making no adjustment
Anchoring on low quality content and adjusting away from the anchor
Experiment 1 – ResultsExperiment 1 – Results
Strong relationship was proven between the validity of the rating and whether the subject adjusts away from an initial anchor or not. (This supports Hypothesis 1)
Significant correlation between the time spent and the decision quality was discovered
Hypotheses 2a and 2b were strongly supported
Hypothesis 2C was not supported
Experiment 1 - ResultsExperiment 1 - Results
H1: Knowledge workers will implement different search and evaluation processes depending on the validity of content rating - Supported
H2a: Anchoring on low quality content but adjusting away from that anchor results in higher decision than not adjusting away from the anchor - Supported
H2b: Anchoring on high quality content (and not adjusting away) results in higher decision quality than anchoring on low quality content and adjusting – Supported
H2c: Anchoring on low quality content and adjusting results in longer decision time than anchoring on high or low quality content and not adjusting- Not Supported
Other ExperimentsOther Experiments
3 more experiments were conducted to assess the moderating effect of adding credibility indicators.
No. of raters Rater expertise Collaborative filtering (Recommending similar
content or identifying content that has been used by others having the same context)
Experiment 2,3 and 4 - HypothesesExperiment 2,3 and 4 - Hypotheses
The following hypotheses were generated and assessed:
H3a: Given Low validity ratings, knowledge workers will adjust away -from an anchor on low quality content- more when the number of raters is low than when the number is high – NOT Supported
H3b: Given Low validity ratings, knowledge workers will adjust away -from an anchor on low quality content- more when the expertise of rater is low than when the expertise is high – NOT Supported
H3c: Given Low validity ratings, knowledge workers will adjust away -from an anchor on low quality content- more when the collaborative filtering sophistication is low than when it is high – Supported
ConclusionsConclusions
Results suggest that ratings influence the quality of the decisions taken by knowledge workers (KMS users)
The paper also provides other useful data for KMS designers and knowledge workers; for example the fact that collaborative filtering has a powerful moderating effect than the number of raters and the raters expertise is a new important point.
Future studies assisting individuals in overcoming invalid ratings should be conducted
Thank YouThank You