Reshaping Scientific Knowledge Dissemination and Evaluation in the Age of the Web

Maurizio Marchese, Aliaksandr Birukou, Fabio Casati and the LiquidPub team

Department of Information Engineering and Computer Science - DISIUniversity of Trento, Italy

The Tuesday Conversation, January 12, 2010 – DG-INFSO-Brussels

Challenge: doing science in the 21st

• The Web has changed many fields:▫ News (blogs, RSS feeds, ...)▫ Music (p2p networks, iTunes, lastFM, …)▫ Travel (Orbiz, Google maps,…) ▫ Photos (Flikr, …)▫ …

• Has it changed also scientific knowledge production and dissemination processes ?

Challenge: doing science in the 21st

• Yes ! But - so far - mainly▫ new and faster dissemination access channels▫ distributed working environment▫ …

• Scientific knowledge processes are still based on the traditional notion of “paper” publication and on peer review as quality assessment method▫ Philosophical Transactions of the Royal Society of London—

founded in 1665▫ Journal des scavans— 1665▫ Royal Society of Edinburgh’s Medical Essays and Observations,

- 1731, introduces peer review as we would recognize it today

We have a dream

Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen

project.liquidpub.org•Publish and perish: why the current publication and review model is killing research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and•Liquid Publications: Scientific Publications meet the Web

Fabio Casati, Fausto Giunchiglia and Maurizio Marchese.

From. www.52en.com/img/dream_01.jpg

ObjectivesUnderstand Improve

▫ Peer review and innovation▫ Evaluation processes, and

quality/impact of research (people, papers, projects)

▫ Dissemination models and overhead

▫ Scientific communities

▫ Better ways to do the same things

▫ Better ways to do different and new things

▫ Principles▫ Models▫ IT services

5

�

�

PR: Initial Goals• Understand how well peer

review works• Metrics + Analysis• Understand how to improve

the process

• Gatekeeping aspects (in/out)• Quality improvements

“Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine”Goodman S.N., Berlin J., Fletcher S.W., Fletcher R.H.

“Not everything that can be counted counts,and not everything that counts can be counted.” -- Albert Einstein

Metric DimensionsQuality

FairnessEfficiency

Statistics

BiasesUnbiasing Effort invariant alternatives

Effort vs. quality

Min. criteria

Kendall DistanceDivergence

DisagreementRobustness

Quality-related Metrics: real vs. ideal

• Real peer review ranking vs. ideal ranking▫ Ideal ?

Subjective vs. ObjectiveBut each process could/should define approximate indicators of quality like: citations, downloads, community voting, success in a second phase, publication, citations, patents…

• IF an approximate ideal ranking is available we can measure the difference in various ways, e.g▫ Kendall τ distance / Kendall τ rank correlation▫ Divergence metric

Comparing rankings

Comparing rankings

Divergence Metric

©√

Nt = 1/nNormalized t

Nt = n/n

N-D

iver

genc

eNdiv(1,n, C) = n-1/n

©√

NDivρiρa(t,n, C ) = pt (i)wi

i=0

t

∑e.g.

NDivρiρa(1,n, C ) = p1(0)

11

⎛ ⎝

⎞ ⎠

+ p1(1)01

⎛ ⎝

⎞ ⎠

=n −1

n11

⎛ ⎝

⎞ ⎠

....

pt (i) =Ci

tCt−in− t

Ctn ;wi =

t − it

When the second ranking is random, we have:

indipendent

correlated

inv. correlated

prior vs. after discussion

NDiv(53,206,206) = 0,36 ca. 74 (36%) contributions have been effected by the discussion phase

Results: peer review ranking vs. citation count

14

Div

Normalized t

Fairness

• Definition: A review process is fair if and only of the acceptance of a contribution does not depend on the particular set of PC members that reviews it

• The key is in the assignment of a paper to reviewers: a paper assignment is unfair if the specific assignment influences (makes more predictable) the fate of the paper.

Computed Normalized Rating Biases

C1 C2 C3 C4

top accepting 2,66 3,44 1,52 1,17

top rejecting -1,74 -2,78 -2,06 -1,17

> + |min bias| 13% 5% 9% 7%

< - |min bias| 12% 4% 8% 7%

C1 C2 C3 C4

Unbiasing effect (divergence) 13% 9% 11% 14%

Unbiasing effect (reviewers affected) 10 16 5 4

Disagreement metric

• Through this metric we compute the similarity between the marks given by the reviewers on the same contribution.

• The rationale behind this metric is that in a review process we expect some kind of agreement between reviewers.

18/13

Normalized Disagreement

19/13

Disagreement vs number of reviews

Estimation of the "optimized”number of proposals per reviewer

• There are different groups of papers, for example: bad, moderate and good papers

• We want to distribute the papers among reviewers in a way such that each reviewer will have statistically at least one paper from each group▫ In this way the reviewer will have a better view of

the overall quality of the papers

Estimation of the "optimized”number of papers per reviewer

The road aheadReal-Time accuracy estimation

Speed Ranking

Ranking vs. marking

PrinciplesModelsIT Services

24

Principles and objectives• Everything counts!! (not just papers, not just

“innovations”)• Minimal dissemination overhead• Early sharing• Early feedback/interaction• Find diversity• Interestingness and sharing as measures of reputation• No gatekeeping. Use the filtering power of the

community▫ We are not necessarily right!!

Let the community select the principles and models

A large number of technologies are out there

Blogs

Wikis

Collaborative tagging and social bookmarking

Scientific Search Services

Journals with collaborative peer review processs

More Complex Systems

But how they can be effectively used ? • Let’s explore some dimensions of the issue

▫ Agile, Collaborative, Open Source “scientific”processes LiquidBook

▫ New models for dissemination, sharing, interactions, evaluation LiquidJournal

29

Liquid Books Initial Definition

A Scenario: Text books

LiquidBook Structure

LiquidBook Lifeycycle

Liquid Book: State of the art and what’s newo WikiBooks: open‐content textbooks ‐ community for creating a free library of 

educational textbooks that anyone can edit

o Differently from WikiBook you can have different roles in the community:o Authorso Contributorso People who just rate, write comments/reviews

o We want to offer a (legal) framework to authors to easier collaboration

o Tailored material for different needs (classes, professionals). Several PersonalizedEditions which stay up‐to‐date with the current state of the art.

o Multi‐Faceted Content (presentations, excercises are available too)

o Sharing and reusing of content among a trusted network of authors, who guarantee the quality

Examples of collaboratively written booksoHow to Think Like a Computer Scientist series of publications by Green Tea Press, where the same core programming text has been adapted to several different programming languages

o97 Things Every Programmer Should Know example of collaborative written book, with hundreds of contributors

oBusiness Model Generation example of collaborative written book, with 470 co‐authors and without a publisher

36

Journals today

• Based on traditional notion of paper

• Traditional peer-review• Solid in nature• Established reputation

The Web EraThe Web has changed the way we get, share, produce and consume

scientific content

Internet

ReadersAuthors

How do I get interesting content!

How do I make my work visible!

Journals: revisited

• Original reasons for the current model are gone• Back to the roots: How to provide interesting

content?

datasetsdatasetspaperspapersblogsblogs

Liquid journals: Proposal

Scientific contributions• Different types, maturity and certification levels

Filling a LJ• Editor decides what to put

Editor

liquid

Consuming a LJ

Readers

Demo at http://project.liquidpub.org/research-areas/liquid-journal

General InfoCreated on 11 Apr 9.00amOwner AlexSubscribers 0

Editors

Issues[no issues yet]

Latests activities

Journal Created11 Apr 9.00am

Latests subscribers

Create Issue

HomeContentIssuesStatsSettings

We are still learning from you. Soon you’ll get our suggestions :)

No suggestions from your friends

Settings

Settings

DisableDisable

DisableDisable

Peer-review journal

Peer review, an overviewPeter, Pablo, PedroThe world of scientific publications has …

Rethinking peer reviewJoeExploring new ideas for peer review in …

Anonymized review dataJuan, AlejandroReview data from 10 conferences in …

(0) LJ references(4) citations

Peer review, scientific publications,

Peer review, open access

Springer

Arxiv(1) LJ references(3) citations

Peer review,

Arxiv(1) LJ references(1) citations

[none]

[Carl’s LJ]

[Liquidpub LJ]

Alex

Joe

[no subscribers yet]

Update Now

Hi Alex!

Liquid journals: Characteristics

• Filled Semi-automatically (query based)• Multi-faceted content• Separation of knowledge production from

publication• Use the editing power of the community• Subscribe to the editors you trust• Use the wisdom of the community

Liquid journals: Benefits• Everybody becomes a journal editor (selfishly)• No gatekeeping, quality enforced by the

community (gatekeeping is the noise)• You get things you want to read (or you want

your group to read)• You get diversity• You leverage the selection work of your friends• Measure interestingness and quality by sharing• Reduce (optimize) dissemination overhead and

encourage early sharing

Related work• Reference management tools (Mendeley/CiteULike

collections)▫ A way to share set of papers among a group of peers ▫ LJ are not just manually edited collections

We provide automatic feed of new content using “liquidity” of queriesEditor can organize contributions in issues, just like in traditional journalsFlexible workflow (maintained by Gelee)

• Overlaid journals (RIOJA)▫ Provide an interface on top of public (open access)

repositoriesLJ does not only focus on open access articles, but includes also articles from different digital libraries and preliminary ideas from blogs

▫ Whole idea is even older – see (Smith 2000)

48

Engineer a system that• Accesses heterogeneous src w/no API• Is modular so that functions can be reused▫ In a world where modularity had little success

• Facilitates the creation of arbitrary dissemination and evaluation models (by non-programmers)

• Provides commonly needed research services• Implements liquid journals (agile – our main

macro-story)• Simple and lightweight

AccessAccess

Cached metadataCached metadata

Liquid Journals dashboardLiquid Journals dashboard

DiversityDiversity CachingCaching

Search computingSearch 

computing

CrawlingCrawling

ReputationReputation

SharingSharing

Search by topic

Search by topic

Personalized tagging

Personalized tagging

Disambiguation

Disambiguation

Liquid journals APILiquid journals API

Fabio Casati - ECOWS 2009 52

Research servicesAPI

API

Karaku client‐codeKaraku client‐code

KarakuKarakuREST APIREST API

RE

ST

AP

IR

EST

A

PI

RE

ST

AP

IR

EST

A

PI

ResManResMan

REST APIREST API

AdaptersAdapters

UI

Cached SKOsCached SKOs

CrawlingCache mgmtSubscriptions/streamDisambiguationPersonalized taggingLiquidityRendering

[Insert header here] [Insert footer]

Vertical services

Community discovery

Search computingSearch computing

ResevalTopicSearch

Gelee

Mashart

Liquid journalsLiquid journals Liquid booksLiquid books ……

Collabo

rative 

programming

Gelee

[Insert header here]

[Insert footer]

Paper at ICDE 09. Video avail online and demo at icsoc

Mashart

�

Paper at ER’09, demo at icsoc ‘09

project.liquidpub.org

RESEVAL    http://demo.liquidpub.org/reseval/ complex’09

Group Management

TELETEACHING/HUM_INT (chi,hicss)

AI/DB (icai,aaai)

ROBOTIC/M.MEDIA (icra,icpr)

TELECOM (icc,globecom)

APPLIED COMPUTING/CRYPTO(sac,compsac)

SOFTWARE ENGINEERING(kbse,icse)

DIST. SYSTEM/COMPILER(ipps,iccS)

GENETIC AND EVO ALG(cec,gecco) HUMMAN –COMP INTER(icchp,hci)

Overview of the Community Network

Take-Home Message• Flaws of current practices (or, lack of evidence that

they work as expected)• Research services for novel dissemination model• Principles, models, composable IT services in a

restricted domain• Use cases:▫ LiquidBook: Sharing and reusing of content among a

trusted network of authors, who guarantee the quality▫ LiquidJournals: Interestingness, reward innovation,

sharing, everything contributes, use the filtering power of the community

Collabiration with ICST.org• The Institute for Computer Sciences, Social-

Informatics and Telecommunications Engineering:an international society, focuses on ICT in its broadest sense

• Supports research, innovation and technology transfer in IT

• Collaboration on:▫ LiquidJournals will be fed with data from ICST’s

eScripts (electronic journals) and distributed via PeerNet (social network)

▫ Courseware platform – integration with LiquidBooks▫ Review analysis – using Assyst (conf management

system) as a data source

Collaboration with other projectshttp://project.liquidpub.org/collaboration/collaboration

To know more

• Google liquidpub-announce and subscribe if you are interested (VERY low traffic)

• First release at the end of the month

63

Thank you