Knowledge Extraction from the Web Monika Henzinger Steve Lawrence.
Implicit feedback: Good may be better than best Steve Lawrence.
-
Upload
mary-munoz -
Category
Documents
-
view
215 -
download
3
Transcript of Implicit feedback: Good may be better than best Steve Lawrence.
Implicit feedback: Good may be better than best
Steve Lawrence
Limitations of the web
• Dead links• Lack of support for author royalties• Poor indexing and navigation support• Better system?
– Enforce link consistency– Allow authors to collect royalties– Support for better navigation and indexing
Web
• Xanadu (1960)– “Improved” design, fixes all of these limitations– Essentially unused
• The web– Widely used
• Disadvantages of the “improved” design– Extra effort imposed on users– Added complexity in the system– Extended development time
• e.g., if link consistency is enforced, no longer can anyone make information available simply by putting a file in a specific directory
• The web has become very popular in part due to its limitations
• “Good may be better than best”
Web vs. Xanadu
• Ted Nelson– Much credit: “hypertext”, inspiration for the web, Lotus
notes, HyperCard– More to Xanadu not covered here (transclusion,
bidirectional links, version management)
• According to Nelson: – “On both the desktop and world-wide scale, culturally
and commercially, we are poorer for these bad tools [the web]”
– “The World Wide Web is precisely what we were trying to prevent”
CiteSeer
• CiteSeer– Metadata not required for submission– Specific citation formats not required
• More optimal system?– Require manual submission which specifies title, author, etc.
(CORR)– Require citations to be submitted in a specific form (Cameron)
• CiteSeer is likely to contain more errors• Error rate on articles not processed is 100%
– Value of explicit feedback not obtained is 0
• Much lower overhead and complexity for users
Implicit vs. explicit feedback
• Explicit feedback– Overhead for the user
• Implicit feedback– No overhead for the user
• Implicit feedback may be better than explicit feedback because you may not be able to get sufficient explicit feedback
• Other issues - accuracy of feedback
“Good may be better than best”
• Not a binary choice– Often many possible systems
• Also– “Worse is better”– “Best is the worst enemy of good”– “MIT approach” vs. “New Jersey approach” for design (Gabriel)
• The increased overhead, complexity and/or cost (for the system and/or the users), and extended development times of more optimal systems may make them far less successful than alternatives
Convenience of access
119,924 conference articles (bibliographical data from DBLP)
Explicit metadata usage
• Only 34% of sites use description or keywords tags on their homepage– Analyzed 2,500 random servers
• 0.3% of sites contained Dublin Core tags
• “Attention is the scarce resource.” Herb Simon (1967)
• Difficult to obtain explicit feedback
Implicit vs. explicit feedback
• Limitations of implicit feedback– Hard to determine the meaning of a click. If the best
link is not displayed, users will still click on something– Click duration may be misleading
• People leave machines unattended• Opening multiple windows quickly, then reading them all
slowly• Multitasking
• Limitations of explicit feedback– Spam– Inconsistent ratings
CiteSeer
CiteSeer
• Scientific literature digital library• Over 600,000 documents indexed
– Earth’s largest free full-text index of scientific literature– (Los Alamos arXiv about 200,000 papers)
• Over 20,000 hosts accessing the site daily• Accesses from over 150 countries per month• Over 10 requests per second at peak times
Improving implicit feedback
• Have to go to details page before getting link to article– Have seen abstract before downloading– Shown context of citations before downloading
No download link
Document information page
Citation context
CiteSeer: explicit feedback
• Document ratings and comments
CiteSeer: explicit feedback
• Allow users to correct errors• Authors may be motivated to correct errors
relating to their own work• How many explicit corrections? (About 600,000
papers)• How many explicit ratings? (percentage of
document accesses)
Explicit feedback
• Over 300,000 explicit corrections/updates– How many bogus updates?– (We require a validated email address)
• Explicit ratings: 0.17% of document accesses
Explicit corrections
• Over 100 bogus correction attempts
Comparison of feedback types
• How well do document access, document downloads, and explicit ratings predict high-citation papers?
• Low citation papers (<= 5 citations)• High citation papers (> 5 citations)• Ratio of downloads/accesses/ratings for high to
low-citation papers– Accesses ?– Downloads ?– Ratings ?
Comparison of feedback types
• Low citation papers (<= 5 citations)• High citation papers (> 5 citations)
• Ratio of downloads/accesses/ratings for high to low-citation papers
• Accesses 2.5• Downloads 3.1
• Ratings 0.96 (low 2.3 high 2.2)
CiteSeer: user profiling
• Profiling system not currently active (scale)• Profile contains documents, citations, keywords,
etc. of interest• User notified of new related documents or
citations by email or via the web interface• Both implicit and explicit feedback• Record the actions of a user for
recommendations– View – Download– Ignore
CiteSeer: user profiling
• Implicit feedback should be more successful in CiteSeer due to citation context, query-sensitive summaries, document details pages, and the expense of document downloads– Users can better determine the relevance of
documents before they request details or download articles
• Analyze co-viewed/downloaded documents to recommend documents related to a given document– Similar to one of Amazon’s book recommenders
Profile creation
• (Pseudo)-documents added to user’s profile whenever a user performs an action in the profile editor or on a real document when browsing
• Action “interestingness” a(.)Explicitly added to profile Very high positiveDownloaded High positiveDetails viewed Moderate positiveRecommendation ignored Low negativeRemoved from profile Set to zero
Paper recommendations
• New papers recommended periodically via email or the web interface
• New paper d* recommended if it has a sufficiently high “interestingness”
• Threshold initially set at a small positive value
Dd
ddD ddRwdI *),(*)(
Profile adaption
• Adaption occurs via manual adjustment and machine learning
• User can explicitly modify a profile by adjusting the weight of pseudo-documents
• Browsing actions implicitly modify the weight of corresponding pseudo-documents
• User response to recommendation of a paper d* is used to update weights that contributed to the recommendation
• where is the learning rate
*),(*)( ddRdaww ddd
Weight update rule properties
• Weights modified according to their contribution to recommendations
• Overall precision/recall threshold automatically adapted. Ignoring recommendations raises the threshold for recommending a paper. Explicitly adding papers lowers the threshold
• The influence of different relatedness measures is adapted separately
REFEREE
• Recommender framework where outside groups can test recommendation systems live on CiteSeer
• Implemented a version of Pennock’s Personality Diagnosis recommender for initial testing
ResearchIndex
RFD
Events,recommendation
requests
Recs
Broadcast events, request
recommendations from specific engines
Logs
Recs
Events
RFD Client
Recommender Engine
Requests to join,recommendations when requested
REFEREE
• Statistics on recommender performance available quickly
• For evaluation we focus on measuring impact on user behavior
• Implicit feedback more effective because users see a lot of information about documents before they can download them
• Which recommenders best?– Users who viewed x also viewed?– Exact sentence overlap?– Papers that cite this paper?– Citation similarity?
Recommendations followed
Recommendation type Recommendations followed
Sentence overlap 8.2%
Cited by 5.1%
CCIDF (bibliographic coupling) 3.1%
PD-1 2.1%
Users who viewed 2.0%
PD-2 2.0%
Co-citation 1.9%
NewsSeer
NewsSeer
• Primarily a single page with implicit feedback only
• Also supports explicit feedback but this is optional
NewsSeer statistics
• About 1 million pageviews• About 10,000 users (>= 5 requests)
– 5,000 users (>= 10 requests)
• How many users rated an article?• What percentage of requests were ratings on the
homepage?• What percentage of requests were for the source
ratings page?
NewsSeer statistics
• 1,000 users rated an article from the 10,000 with >= 5 requests– About 10%– About 20% of the top 2,500 users– About 30% of the top 1,000 users– 20 of 56 users that did >1,000 requests– 10 of 21 users that did >2,000 requests
• Homepage 51% (auto-reloaded)• View article 40%• Keyword query 4% (was not available initially)• Ratings on homepage 5%• Source rating page views 0.2%
MusicSeer
Music similarity
Music similarity
• Music similarity survey
• Erdös game
• Erdös game
Music similarity
• Simple survey
Music similarity
MusicSeer
• Survey– 713 users, 10,997 judgments
• Game– 680 users, 11,313 judgments
Summary
• Implicit feedback may be better because there is much lower overhead
• Much greater participation may more than compensate for the less accurate information received
• Can structure system to maximize implicit feedback gained
• Can obtain explicit feedback if enough incentive, or easy enough