Digitalization of Scientific Information and Its Dissemination
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
-
Upload
aliaksandr-birukou -
Category
Technology
-
view
1.154 -
download
1
description
Transcript of Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web
Maurizio Marchese, Aliaksandr Birukou, Fabio Casati and the LiquidPub team
Department of Information Engineering and Computer Science - DISIUniversity of Trento, Italy
The Tuesday Conversation, January 12, 2010 – DG-INFSO-Brussels
Challenge: doing science in the 21st
• The Web has changed many fields:▫ News (blogs, RSS feeds, ...)▫ Music (p2p networks, iTunes, lastFM, …)▫ Travel (Orbiz, Google maps,…) ▫ Photos (Flikr, …)▫ …
• Has it changed also scientific knowledge production and dissemination processes ?
Challenge: doing science in the 21st
• Yes ! But - so far - mainly▫ new and faster dissemination access channels▫ distributed working environment▫ …
• Scientific knowledge processes are still based on the traditional notion of “paper” publication and on peer review as quality assessment method▫ Philosophical Transactions of the Royal Society of London—
founded in 1665▫ Journal des scavans— 1665▫ Royal Society of Edinburgh’s Medical Essays and Observations,
- 1731, introduces peer review as we would recognize it today
We have a dream
Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen
project.liquidpub.org•Publish and perish: why the current publication and review model is killing research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and•Liquid Publications: Scientific Publications meet the Web
Fabio Casati, Fausto Giunchiglia and Maurizio Marchese.
From. www.52en.com/img/dream_01.jpg
ObjectivesUnderstand Improve
▫ Peer review and innovation▫ Evaluation processes, and
quality/impact of research (people, papers, projects)
▫ Dissemination models and overhead
▫ Scientific communities
▫ Better ways to do the same things
▫ Better ways to do different and new things
▫ Principles▫ Models▫ IT services
5
�
�
PR: Initial Goals• Understand how well peer
review works• Metrics + Analysis• Understand how to improve
the process
• Gatekeeping aspects (in/out)• Quality improvements
“Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine”Goodman S.N., Berlin J., Fletcher S.W., Fletcher R.H.
“Not everything that can be counted counts,and not everything that counts can be counted.” -- Albert Einstein
Metric DimensionsQuality
FairnessEfficiency
Statistics
BiasesUnbiasing Effort invariant alternatives
Effort vs. quality
Min. criteria
Kendall DistanceDivergence
DisagreementRobustness
Quality-related Metrics: real vs. ideal
• Real peer review ranking vs. ideal ranking▫ Ideal ?
Subjective vs. ObjectiveBut each process could/should define approximate indicators of quality like: citations, downloads, community voting, success in a second phase, publication, citations, patents…
• IF an approximate ideal ranking is available we can measure the difference in various ways, e.g▫ Kendall τ distance / Kendall τ rank correlation▫ Divergence metric
Comparing rankings
Comparing rankings
Divergence Metric
©√
Nt = 1/nNormalized t
Nt = n/n
N-D
iver
genc
eNdiv(1,n, C) = n-1/n
©√
NDivρiρa(t,n, C ) = pt (i)wi
i=0
t
∑e.g.
NDivρiρa(1,n, C ) = p1(0)
11
⎛ ⎝
⎞ ⎠
+ p1(1)01
⎛ ⎝
⎞ ⎠
=n −1
n11
⎛ ⎝
⎞ ⎠
....
pt (i) =Ci
tCt−in− t
Ctn ;wi =
t − it
When the second ranking is random, we have:
indipendent
correlated
inv. correlated
prior vs. after discussion
NDiv(53,206,206) = 0,36 ca. 74 (36%) contributions have been effected by the discussion phase
Results: peer review ranking vs. citation count
14
Div
Normalized t
Fairness
• Definition: A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it
• The key is in the assignment of a paper to reviewers: a paper assignment is unfair if the specific assignment influences (makes more predictable) the fate of the paper.
Computed Normalized Rating Biases
C1 C2 C3 C4
top accepting 2,66 3,44 1,52 1,17
top rejecting -1,74 -2,78 -2,06 -1,17
> + |min bias| 13% 5% 9% 7%
< - |min bias| 12% 4% 8% 7%
C1 C2 C3 C4
Unbiasing effect (divergence) 13% 9% 11% 14%
Unbiasing effect (reviewers affected) 10 16 5 4
Disagreement metric
• Through this metric we compute the similarity between the marks given by the reviewers on the same contribution.
• The rationale behind this metric is that in a review process we expect some kind of agreement between reviewers.
18/13
Normalized Disagreement
19/13
Disagreement vs number of reviews
Estimation of the "optimized”number of proposals per reviewer
• There are different groups of papers, for example: bad, moderate and good papers
• We want to distribute the papers among reviewers in a way such that each reviewer will have statistically at least one paper from each group▫ In this way the reviewer will have a better view of
the overall quality of the papers
Estimation of the "optimized”number of papers per reviewer
The road aheadReal-Time accuracy estimation
Speed Ranking
Ranking vs. marking
PrinciplesModelsIT Services
24
Principles and objectives• Everything counts!! (not just papers, not just
“innovations”)• Minimal dissemination overhead• Early sharing• Early feedback/interaction• Find diversity• Interestingness and sharing as measures of reputation• No gatekeeping. Use the filtering power of the
community▫ We are not necessarily right!!
Let the community select the principles and models
A large number of technologies are out there
Blogs
Wikis
Collaborative tagging and social bookmarking
Scientific Search Services
Journals with collaborative peer review processs
More Complex Systems
But how they can be effectively used ? • Let’s explore some dimensions of the issue
▫ Agile, Collaborative, Open Source “scientific”processes LiquidBook
▫ New models for dissemination, sharing, interactions, evaluation LiquidJournal
29
Liquid Books Initial Definition
A Scenario: Text books
LiquidBook Structure
LiquidBook Lifeycycle
Liquid Book: State of the art and what’s newo WikiBooks: open‐content textbooks ‐ community for creating a free library of
educational textbooks that anyone can edit
o Differently from WikiBook you can have different roles in the community:o Authorso Contributorso People who just rate, write comments/reviews
o We want to offer a (legal) framework to authors to easier collaboration
o Tailored material for different needs (classes, professionals). Several PersonalizedEditions which stay up‐to‐date with the current state of the art.
o Multi‐Faceted Content (presentations, excercises are available too)
o Sharing and reusing of content among a trusted network of authors, who guarantee the quality
Examples of collaboratively written booksoHow to Think Like a Computer Scientist series of publications by Green Tea Press, where the same core programming text has been adapted to several different programming languages
o97 Things Every Programmer Should Know example of collaborative written book, with hundreds of contributors
oBusiness Model Generation example of collaborative written book, with 470 co‐authors and without a publisher
36
Journals today
• Based on traditional notion of paper
• Traditional peer-review• Solid in nature• Established reputation
The Web EraThe Web has changed the way we get, share, produce and consume
scientific content
Internet
ReadersAuthors
How do I get interesting content!
How do I make my work visible!
Journals: revisited
• Original reasons for the current model are gone• Back to the roots: How to provide interesting
content?
datasetsdatasetspaperspapersblogsblogs
Liquid journals: Proposal
Scientific contributions• Different types, maturity and certification levels
Filling a LJ• Editor decides what to put
Editor
liquid
Consuming a LJ
Readers
Demo at http://project.liquidpub.org/research-areas/liquid-journal
General InfoCreated on 11 Apr 9.00amOwner AlexSubscribers 0
Editors
Issues[no issues yet]
Latests activities
Journal Created11 Apr 9.00am
Latests subscribers
Create Issue
HomeContentIssuesStatsSettings
We are still learning from you. Soon you’ll get our suggestions :)
No suggestions from your friends
Settings
Settings
DisableDisable
DisableDisable
Peer-review journal
Peer review, an overviewPeter, Pablo, PedroThe world of scientific publications has …
Rethinking peer reviewJoeExploring new ideas for peer review in …
Anonymized review dataJuan, AlejandroReview data from 10 conferences in …
(0) LJ references(4) citations
Peer review, scientific publications,
Peer review, open access
Springer
Arxiv(1) LJ references(3) citations
Peer review,
Arxiv(1) LJ references(1) citations
[none]
[Carl’s LJ]
[Liquidpub LJ]
Alex
Joe
[no subscribers yet]
Update Now
Hi Alex!
Liquid journals: Characteristics
• Filled Semi-automatically (query based)• Multi-faceted content• Separation of knowledge production from
publication• Use the editing power of the community• Subscribe to the editors you trust• Use the wisdom of the community
Liquid journals: Benefits• Everybody becomes a journal editor (selfishly)• No gatekeeping, quality enforced by the
community (gatekeeping is the noise)• You get things you want to read (or you want
your group to read)• You get diversity• You leverage the selection work of your friends• Measure interestingness and quality by sharing• Reduce (optimize) dissemination overhead and
encourage early sharing
Related work• Reference management tools (Mendeley/CiteULike
collections)▫ A way to share set of papers among a group of peers ▫ LJ are not just manually edited collections
We provide automatic feed of new content using “liquidity” of queriesEditor can organize contributions in issues, just like in traditional journalsFlexible workflow (maintained by Gelee)
• Overlaid journals (RIOJA)▫ Provide an interface on top of public (open access)
repositoriesLJ does not only focus on open access articles, but includes also articles from different digital libraries and preliminary ideas from blogs
▫ Whole idea is even older – see (Smith 2000)
48
Engineer a system that• Accesses heterogeneous src w/no API• Is modular so that functions can be reused▫ In a world where modularity had little success
• Facilitates the creation of arbitrary dissemination and evaluation models (by non-programmers)
• Provides commonly needed research services• Implements liquid journals (agile – our main
macro-story)• Simple and lightweight
AccessAccess
Cached metadataCached metadata
Liquid Journals dashboardLiquid Journals dashboard
DiversityDiversity CachingCaching
Search computingSearch
computing
CrawlingCrawling
ReputationReputation
SharingSharing
Search by topic
Search by topic
Personalized tagging
Personalized tagging
Disambiguation
Disambiguation
Liquid journals APILiquid journals API
Fabio Casati - ECOWS 2009 52
Research servicesAPI
API
Karaku client‐codeKaraku client‐code
KarakuKarakuREST APIREST API
RE
ST
AP
IR
EST
A
PI
RE
ST
AP
IR
EST
A
PI
ResManResMan
REST APIREST API
AdaptersAdapters
UI
Cached SKOsCached SKOs
CrawlingCache mgmtSubscriptions/streamDisambiguationPersonalized taggingLiquidityRendering
[Insert header here] [Insert footer]
Vertical services
Community discovery
Search computingSearch computing
ResevalTopicSearch
Gelee
Mashart
Liquid journalsLiquid journals Liquid booksLiquid books ……
Collabo
rative
programming
Gelee
[Insert header here]
[Insert footer]
Paper at ICDE 09. Video avail online and demo at icsoc
Mashart
�
Paper at ER’09, demo at icsoc ‘09
project.liquidpub.org
RESEVAL http://demo.liquidpub.org/reseval/ complex’09
Group Management
TELETEACHING/HUM_INT (chi,hicss)
AI/DB (icai,aaai)
ROBOTIC/M.MEDIA (icra,icpr)
TELECOM (icc,globecom)
APPLIED COMPUTING/CRYPTO(sac,compsac)
SOFTWARE ENGINEERING(kbse,icse)
DIST. SYSTEM/COMPILER(ipps,iccS)
GENETIC AND EVO ALG(cec,gecco) HUMMAN –COMP INTER(icchp,hci)
Overview of the Community Network
Take-Home Message• Flaws of current practices (or, lack of evidence that
they work as expected)• Research services for novel dissemination model• Principles, models, composable IT services in a
restricted domain• Use cases:▫ LiquidBook: Sharing and reusing of content among a
trusted network of authors, who guarantee the quality▫ LiquidJournals: Interestingness, reward innovation,
sharing, everything contributes, use the filtering power of the community
Collabiration with ICST.org• The Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering:an international society, focuses on ICT in its broadest sense
• Supports research, innovation and technology transfer in IT
• Collaboration on:▫ LiquidJournals will be fed with data from ICST’s
eScripts (electronic journals) and distributed via PeerNet (social network)
▫ Courseware platform – integration with LiquidBooks▫ Review analysis – using Assyst (conf management
system) as a data source
Collaboration with other projectshttp://project.liquidpub.org/collaboration/collaboration
To know more
• Google liquidpub-announce and subscribe if you are interested (VERY low traffic)
• First release at the end of the month
63
Thank you