reviewing papers

methodologies for scientific researchPRODEI

carlos soares & augusto de sousa[based on materials kindly provided

by prof. Eugénio Oliveira & Serge Demeyer (U. Antwerp)]

plan

• the big picture• reviews: what, to, who, when, why• the process of reviewing

carlos soares / MSR 2

my research

carlos soares / MSR 3

MIC- Metodologias de Investigação Científica Eugénio Oliveira- 2017 September, 2005 2

MSR- Methodologies for Scientific Research

Eugénio Oliveira- 2017 MSR- Methodologies for Scientific Research

Source: GETA and Graduate School of Electrical and Communications Engineering course: Introduction to Research Methodology, Helsinki University of Technology,

this should be easy compared to the work done so far, right?...

publishing: the roadmap

carlos soares / MSR 4

workshops and conferences

carlos soares / MSR 5

journals

Peer Review Process

324311

Revisions required

Author submits revised manuscript

Author submits article

Article assessed by editor

Sent to reviewers

Reviews assessed by editor

Accepted

Publication

Production

Further review needed?

Rejected

Rejected

carlos soares / MSR 6https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/the-peer-review-process.html

why do I need to understand the reviewing process?!

• … I’m just starting my phd!• what I really need is to learn how to publish…

the best way to get a paper accepted is to understand how it

will be reviewed(but remember: it is still a stochastic process, so there are no guarantees…)

carlos soares / MSR 7

gps

• the big picture• reviews: what, to, who, when, why• the process of reviewing

carlos soares / MSR 8

what?

• answer the question– “does the paper make a sufficient contribution to

the field”?

• … more specifically– correct– new– significant– well-presented

carlos soares / MSR 9

to?

• research results– most common

• survey– increasingly common

• tutorial• synthesis of existing results• confirmation of previous results– not very common in CS&E

carlos soares / MSR 10

who?

• senior researcher– busy

• member of the community– shares virtues– … and vices

• voluntary duty– enthusiastic– … or maybe not

• … or a phd student– 2nd opinion– … or just distributing

workload

carlos soares / MSR 11

when?

• train, bus, plane– (very busy, remember?)

• …very late at night

• effort– 8-10h/journal– 4-5h/conference

carlos soares / MSR 12

why?

• public service• professional obligation• self-interest– “I scratch your back, you scratch mine”

• dangerous attitude, if taken too literaly

• + if phd student– understanding the review process to write better

papers– entry in the résumé

carlos soares / MSR 13

gps

• the big picture• reviews: what, to, who, when, why• the process of reviewing– reading– writing– deciding

carlos soares / MSR 14

reading

• similar to grading– in contrast to literature review

• which assumes that paper was already reviewed

• attitude– careful reading– critical analysis– … but open mind

• unless a fatal flaw is found– not necessary to read in detail

carlos soares / MSR 15

writing

• recommendation and the reasons for it• paragraph summarizing of the point of the paper

– for the use of the editor– to clarify that the referee actually understood the point

• evaluation– validity – significance

• quality of the work– methodology– techniques– accuracy– errors– presentation

carlos soares / MSR 16

… in a civilized tone

• reviews are sent to authors

• anonymity is an ongoing discussion– single blind• reviewer knows who the author

– double blind– open review• growing trend

• … very interesting and important!

carlos soares / MSR 17

writing(in more detail)

• purpose?• appropriateness? • significance?

– assuming results are correct, why should we care?– contextualized in related work?

• validity of the approach?• … and is it correctly executed?• correctness of conclusions?• presentation satisfactory?

– incomprehensibility is a reason for rejection– typos, grammar, punctuation and wording

• what did you learn ?

carlos soares / MSR 18

deciding

• typical levels1. major results; very significant

• fewer than 1% of all papers2. good, solid, interesting work

• a definite contribution (≤ 10 %)3. minor, but positive, contribution to knowledge

• perhaps 10-30 %4. elegant and technically correct but useless

• e.g. sophisticated analyses of flying pig5. neither elegant nor useful, but not actually wrong6. wrong and misleading7. so badly written that technical evaluation is impossible

• or simply not suitable for journal/conference

carlos soares / MSR 19

… in context

• what are the standards of the journal/conference?– compare to average paper– … not best or worst

• acceptance rate does not necessarily represent selectivity!– although often used as such

carlos soares / MSR 20

review form: example (conference)

• Overall evaluation– 3: strong accept– 2: accept– 1: weak accept– 0: borderline paper– -1: weak reject– -2: reject– -3: strong reject

• Please provide a detailed review, including a justification for your scores. Summarise both the reasons to accept and reject the paper.

• Reviewer's confidence.– 5: (expert)– 4: (high)– 3: (medium)– 2: (low)– 1: (none)

• Confidential remarks for the program committee. If you wish to add any remarks intended only for PC members please write them below. These remarks will only be seen by the PC members having access to reviews for this submission. They will not be sent to the authors. This field is optional.

carlos soares / MSR 21

review form: example (journal)

• SUMMARY: BRIEFLY STATE WHAT THE PAPER IS REALLY ABOUT AND BRIEFLY SUMMARIZE REVIEW.

• CONTRIBUTIONS: WHAT IS (ARE) THE CLAIMED CONTRIBUTION(S) OF THE PAPER? HOW SIGNIFICANT/IMPORTANT IS (ARE) THE CLAIMED CONTRIBUTION(S)?

• SUPPORT: HOW ARE THESE CONTRIBUTIONS SUPPORTED? IS THE SUPPORT ADEQUATE? CORRECT? IS THE EVALUATION ADEQUATE? WELL-DESIGNED? COMMENT ON THE SUPPORT FOR EACH CLAIMED CONTRIBUTION AND, IF APPROPRIATE, ON HOW THE SUPPORT COULD BE IMPROVED.

• TECHNICAL QUALITY: IS THE PAPER TECHNICALLY SOUND? DOES IT ADEQUATELY ADDRESS THE LIMITATIONS OF THE RESULTS? COMMENT ON HOW TECHNICAL QUALITY COULD BE IMPROVED. ARE THE STRENGTHS/WEAKNESSES AND REASONS FOR OBSERVED RESULTS EXPLORED SYSTEMATICALLY?

• ORIGINALITY: ARE THE CLAIMED CONTRIBUTIONS NOVEL? DOES THE PAPER PLACE THE WORK APPROPRIATELY IN THE CONTEXT OF RELATED RESEARCH?

• CLARITY: COMMENT ON THE WRITING, ORGANIZATION & EXPOSITION. IS TECHNICAL MATERIAL (ALGORITHMS, EVALUATION METHODOLOGY, ETC.) DESCRIBED IN SUFFICIENT DETAIL? COULD A READER REPLICATE THE WORK? ARE THE TITLE AND ABSTRACT APPROPRIATE? ARE THE FIGURES/TABLES SUFFICIENT?

• DETAILED COMMENTS:

• REQUIRED CHANGES: DISTINGUISH BETWEEN MINOR AND MAJOR CHANGES THAT NEED TO BE MADE FOR THE PAPER TO BE ACCEPTABLE.

carlos soares / MSR 22