1INFO5993 Research Methods, s1 2009 INFO5993 Research Methods Kalina Yacef info5993/ Lecture based...

1INFO5993 Research Methods, s1 2009

INFO5993 Research MethodsINFO5993 Research Methods

Kalina YacefKalina Yacefhttp://www.cs.usyd.edu.au/~info5993/http://www.cs.usyd.edu.au/~info5993/

Lecture based in part on materials by Alan Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Fekete, Mary Lou Maher, Joseph Davis, Irena

Koprinska and othersKoprinska and others

Research – Components and Research – Components and ProcessProcess

Research Publications – Types Research Publications – Types and Quality Metricsand Quality Metrics


OutlineOutline Administrative mattersAdministrative matters Research Research

Definition, key components, Definition, key components, processprocess Finding a research questionFinding a research question

Guide to research literature Guide to research literature types of publications and how are they types of publications and how are they

producedproduced Quality metrics – how to measure Quality metrics – how to measure

research impact?research impact?


AdministraviaAdministravia Course web page: Course web page:

http://www.cs.usyd.edu.au/~info5993/http://www.cs.usyd.edu.au/~info5993/ 2 hours lectures, 3-5pm on Mondays2 hours lectures, 3-5pm on Mondays

A couple of exceptions in the semester due to guest A couple of exceptions in the semester due to guest lecturer availabilitylecturer availability

Coordinator: Kalina YacefCoordinator: Kalina Yacef Lectures given by the coordinator and invited lecturers Lectures given by the coordinator and invited lecturers

(IT academics, learning centre staff, librarians)(IT academics, learning centre staff, librarians) No textbook, online resources – check the web pageNo textbook, online resources – check the web page AssignmentsAssignments

1 – search results, w51 – search results, w5 2 - literature review and outline of research (25%), w72 - literature review and outline of research (25%), w7 3 - presentation (15%) +feedback on other presentations 3 - presentation (15%) +feedback on other presentations

(10%) w10-13(10%) w10-13 seminar attendance required – max penalty 5%seminar attendance required – max penalty 5%


Topics OverviewTopics Overview Introduction to research – definition, components, Introduction to research – definition, components,

process, how to find a research questionprocess, how to find a research question Types of research publications, quality metricsTypes of research publications, quality metrics Literature review, how to search for relevant Literature review, how to search for relevant

publications publications Writing a literature review and research proposalWriting a literature review and research proposal Oral presentation skillsOral presentation skills Research methods in IT (statistical analysis, Research methods in IT (statistical analysis,

mathematical analysis, algorithm analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, etc.) simulation, qualitative analysis, etc.)

Ethics. Avoiding plagiarism. Intellectual Property.Ethics. Avoiding plagiarism. Intellectual Property.


Definition of ResearchDefinition of Research1) From the Merriam-Webster dictionary: 1) From the Merriam-Webster dictionary: 11:: careful or diligent search careful or diligent search 22:: studious inquiry or examination; studious inquiry or examination; especiallyespecially:: investigation or experimentation aimed at the investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical theories or laws in the light of new facts, or practical application of such new or revised theories or lawsapplication of such new or revised theories or laws

33:: the collecting of information about a particular subject the collecting of information about a particular subject

2) Booth, Columb & Williams, “The Craft of Research”: 2) Booth, Columb & Williams, “The Craft of Research”: ““Research is gathering information that answers a Research is gathering information that answers a

question and so solves a problem.” question and so solves a problem.”


Is This Research?Is This Research?

To understand political decisions, a To understand political decisions, a journalist finds out who contributed to journalist finds out who contributed to election campaign fundelection campaign fund

To buy a laptop, a student compares To buy a laptop, a student compares various brands, configurations and pricesvarious brands, configurations and prices

To help companies stay competitive, a To help companies stay competitive, a market researcher collects and interprets market researcher collects and interprets informationinformation

To fix a computer, a technician finds out To fix a computer, a technician finds out what procedure to usewhat procedure to use


Academic ResearchAcademic Research In academic research, you must not only answer a In academic research, you must not only answer a

question, but you must find something question, but you must find something new new and and interestinginteresting

You join a community of researchersYou join a community of researchers You must You must advanceadvance the collective understanding of this the collective understanding of this

communitycommunity Each community has a cumulative tradition with a Each community has a cumulative tradition with a

set of interesting questions, tools and methods, set of interesting questions, tools and methods, practices, a style and language for writing up the practices, a style and language for writing up the researchresearch Research is a conversation and ongoing social activity!Research is a conversation and ongoing social activity!

You need critical and careful reading of published You need critical and careful reading of published researchresearch to learn what the community already knowsto learn what the community already knows to fit your work into the communityto fit your work into the community to be prepared for your own work to be evaluatedto be prepared for your own work to be evaluated


Key Components of Key Components of ResearchResearch A question of interest (research question)A question of interest (research question)

A claim (contribution)A claim (contribution) EvidenceEvidence Argument (links evidence to claim) Argument (links evidence to claim)


A Research QuestionA Research Question Every piece of research should address a question of Every piece of research should address a question of

interest to the communityinterest to the community Each community has traditional questions:Each community has traditional questions:

What happens? Why does it happen? How should one do What happens? Why does it happen? How should one do something? What something should one do?something? What something should one do?

Many questions fit into an on-going agenda, e.g.Many questions fit into an on-going agenda, e.g. Data mining foundations – mining sequential data; high-Data mining foundations – mining sequential data; high-

performance implementations of data mining algorithms, etc.performance implementations of data mining algorithms, etc. Mining emerging data - e-commerce , web search data, Mining emerging data - e-commerce , web search data,

moving object data, data from sensor networksmoving object data, data from sensor networks …… See a recent Conference Call for PapersSee a recent Conference Call for Papers


A Claim (Contribution)A Claim (Contribution) Every piece of research makes a claim (the Every piece of research makes a claim (the

“contribution”) answering a research question“contribution”) answering a research question Claims can be very diverse among fields and Claims can be very diverse among fields and

within fieldswithin fields Ex. for a “what happens” question - when using weak Ex. for a “what happens” question - when using weak

concurrency control, how often is the data corruptedconcurrency control, how often is the data corrupted Ex. for a “ why something happens” - what factors Ex. for a “ why something happens” - what factors

lead to project success in open-source developmentlead to project success in open-source development Ex. for a “better way to do something” - modifying Ex. for a “better way to do something” - modifying

algorithm X in a particular way improves its algorithm X in a particular way improves its performance (speed, accuracy, etc)performance (speed, accuracy, etc)

Ex. for a “better something to do” - our system Ex. for a “better something to do” - our system allows users to see the model of their skills kept in a allows users to see the model of their skills kept in a teaching system teaching system

Be explicit about the meaning of “better”


EvidenceEvidence You must back up the claim with evidence, e.g.You must back up the claim with evidence, e.g.

Empirical evaluation of a machine learning algorithm to evaluate its Empirical evaluation of a machine learning algorithm to evaluate its accuracyaccuracy

Analysis of the computational complexity of an algorithmAnalysis of the computational complexity of an algorithm A mathematical proof to show that some process/algorithm has A mathematical proof to show that some process/algorithm has

desired propertiesdesired properties A prototype implementation to show that a system can be built to A prototype implementation to show that a system can be built to

achieve the claimed functionalityachieve the claimed functionality A simulation model which is executed and analysed to show certain A simulation model which is executed and analysed to show certain

propertiesproperties Measurements of a running system to show it has good performanceMeasurements of a running system to show it has good performance Observations of behaviour in an organisation to show what is Observations of behaviour in an organisation to show what is

happeninghappening Various research methods, each defined by the sort of Various research methods, each defined by the sort of

evidence that it can produceevidence that it can produce each community has its own standards of quality and reasonablenesseach community has its own standards of quality and reasonableness


ArgumentArgument You should show that the evidence you offer You should show that the evidence you offer

supports the claim you makesupports the claim you make It is essential that you deal with natural or It is essential that you deal with natural or

obvious objections to the correctness or obvious objections to the correctness or importance of the workimportance of the work

that is, you must think like your readers, and that is, you must think like your readers, and anticipate their reactionsanticipate their reactions

In systems work, this is often called an In systems work, this is often called an “evaluation” of the design“evaluation” of the design


Research Paper - Research Paper - ExampleExample

Identify the Identify the Research questionResearch question ClaimClaim EvidenceEvidence ArgumentArgument


Claim and Argument - Claim and Argument - ExamplesExamples

This system design leads to better This system design leads to better performance on some metricperformance on some metric make sure you limit how much worse this make sure you limit how much worse this

makes other metrics (such as cost!)makes other metrics (such as cost!) make sure your measurements are fair (don’t make sure your measurements are fair (don’t

compare with “strawman” design but with compare with “strawman” design but with state-of-the-art)state-of-the-art)

This system design offers better This system design offers better functionality for some usesfunctionality for some uses make sure you show it can be implemented make sure you show it can be implemented

with adequate performancewith adequate performance


Claim and Argument – Claim and Argument – Examples (2)Examples (2)

This behaviour can be explained by This behaviour can be explained by this theorythis theory make sure you don’t have confounding factors make sure you don’t have confounding factors

such as level of experience, or method novelty, such as level of experience, or method novelty, or subject expectations (“placebo effect”)or subject expectations (“placebo effect”)

This is what happensThis is what happens make sure you don’t interfere too much with make sure you don’t interfere too much with

what happens when you gather data, or what happens when you gather data, or misinterpret it due to observer expectations misinterpret it due to observer expectations


Common Mistakes 1Common Mistakes 1

Gather lots of data without a focussed Gather lots of data without a focussed question or methodquestion or method

A collection of facts is not a contribution!A collection of facts is not a contribution! it must reveal some pattern or understanding it must reveal some pattern or understanding

that you make explicitthat you make explicit


Common Mistakes 2Common Mistakes 2 Build a system without a focused question Build a system without a focused question

or planned evaluationor planned evaluation E.g. let’s see how to use aspect-oriented E.g. let’s see how to use aspect-oriented

programming in a sensor networkprogramming in a sensor network An innovative system is not a An innovative system is not a

contribution!contribution! it must be a worthwhile innovation in a sense it must be a worthwhile innovation in a sense

you make explicityou make explicit E.g. better performanceE.g. better performance E.g. new functionalityE.g. new functionality


Negative ResultsNegative Results Sometimes, you don’t get the result you hoped Sometimes, you don’t get the result you hoped

forfor You gather data that does not reveal any pattern or You gather data that does not reveal any pattern or

understandingunderstanding E.g. no factor seems to correlate well with project successE.g. no factor seems to correlate well with project success

You design a system that turns out to be worse than You design a system that turns out to be worse than the state-of-the-artthe state-of-the-art

E.g. your machine learning algorithm runs slower than E.g. your machine learning algorithm runs slower than expectedexpected

You can still salvage a thesisYou can still salvage a thesis Try to find some way to contribute to our Try to find some way to contribute to our

understanding, or suggest fruitful directions for understanding, or suggest fruitful directions for further workfurther work

E.g. what features of the algorithm make it slowE.g. what features of the algorithm make it slow Make sure the problem is intrinsic, not just your bad Make sure the problem is intrinsic, not just your bad

coding/experiment design/etccoding/experiment design/etc


Ground-Breaking WorkGround-Breaking Work Very rarely, a piece of research will establish a Very rarely, a piece of research will establish a

whole new agenda for a field, or even a new fieldwhole new agenda for a field, or even a new field the contribution can be as much in the possibilities for the contribution can be as much in the possibilities for

further work, as in the result itself!further work, as in the result itself! In some sense, this is work that asks a new type In some sense, this is work that asks a new type

of question, or introduces a new methodof question, or introduces a new method

We don’t recommend this for We don’t recommend this for Hons/MIT/MSc/PhDHons/MIT/MSc/PhD save the idea till you have time enough, and flexibility save the idea till you have time enough, and flexibility

enough to deal with inevitable digressions/difficultiesenough to deal with inevitable digressions/difficulties

INFO5993 Research Methods, s1 2009 20

Great scholars do not Great scholars do not solve problems; they solve problems; they

create them.create them.

-Albert Einstein-Albert Einstein


Idealised Research Idealised Research ProcessProcess

Find a question to seek an answer forFind a question to seek an answer for Method: Choose an appropriate research Method: Choose an appropriate research

method and make flexible plansmethod and make flexible plans Evidence: Gather the data, do the Evidence: Gather the data, do the

experiment, build the prototype etc.experiment, build the prototype etc. Contribution: Analyse, interpret, and Contribution: Analyse, interpret, and

concludeconclude Argument: Write the reportArgument: Write the report

Importance of “writing” (aided by thinking Importance of “writing” (aided by thinking from the point of view of your readers)from the point of view of your readers)


Actual Research ProcessActual Research Process Research explores new areas and the results are Research explores new areas and the results are

not predictable!not predictable! The research plan is iterativeThe research plan is iterative

Gathering evidence leads to changes to the claimGathering evidence leads to changes to the claim sometimes one refines the claimsometimes one refines the claim

E.g. limit the scope E.g. limit the scope from “algorithm X outperforms Y” to “algorithm X outperforms Y from “algorithm X outperforms Y” to “algorithm X outperforms Y

when the independence assumption is violated” when the independence assumption is violated” From “X has higher throughput” to “X has higher throughput if the From “X has higher throughput” to “X has higher throughput if the

contention rate is low”contention rate is low” sometimes one must change the claim entirelysometimes one must change the claim entirely sometimes while gathering evidence, one finds new questions sometimes while gathering evidence, one finds new questions

which look worth answering!which look worth answering!

New claims or questions need further evidence, New claims or questions need further evidence, revised plans, maybe even different methodsrevised plans, maybe even different methods


The “Great Expedition The “Great Expedition into Unknown Terrain” into Unknown Terrain”

metaphormetaphor Imke TammenImke Tammen

http://www.itl.usyd.edu.au/supervision/http://www.itl.usyd.edu.au/supervision/casestudies/casestudy.cfm?id=8casestudies/casestudy.cfm?id=8 students and supervisors as co-explorersstudents and supervisors as co-explorers


Finding a QuestionFinding a Question Especially when you are learning to do research, it Especially when you are learning to do research, it

may be already chosen for you by supervisormay be already chosen for you by supervisor or supervisor may suggest an area, and leave you to find or supervisor may suggest an area, and leave you to find

the questionthe question A question may arise from some previous researchA question may arise from some previous research

Further work, issues not addressed, holes in the Further work, issues not addressed, holes in the evidence collectedevidence collected

A question may come from the combination of A question may come from the combination of previous researchprevious research Bring two areas together, use a technique from one area Bring two areas together, use a technique from one area

in anotherin another A question may arise due to new technologyA question may arise due to new technology

new hardware or technique may require new models, new hardware or technique may require new models, new hardware may influence use or performance or new hardware may influence use or performance or feasibilityfeasibility


Suitable Research Suitable Research QuestionsQuestions

Answerability – can the questions be Answerability – can the questions be answered through research?answered through research?

Scale: Consider available resources Scale: Consider available resources (equipment, time, skills)(equipment, time, skills)

Scope: Often start with broad topic Scope: Often start with broad topic space/ bigger question, then narrow space/ bigger question, then narrow in to a specific questionin to a specific question


Tips for Finding Research Tips for Finding Research QuestionsQuestions

Try the research topic generator Try the research topic generator http://www.cs.purdue.edu/homes/dec/essay.topic.gehttp://www.cs.purdue.edu/homes/dec/essay.topic.generator.htmlnerator.html


Tips for Finding Research Tips for Finding Research Questions (2)Questions (2)

Read the papers your supervisor gave youRead the papers your supervisor gave you follow the references, check the web pages of the follow the references, check the web pages of the

authorsauthors read carefully the “Future research” sectionsread carefully the “Future research” sections write down your ideas!!write down your ideas!!

Find the top conferences in your field Find the top conferences in your field scan the call for papers and associated workshops for scan the call for papers and associated workshops for

hot topicshot topics scan the conference proceedings to identify important scan the conference proceedings to identify important

topics, key people and research groups. Check their topics, key people and research groups. Check their web pages.web pages.

Find review (survey) articlesFind review (survey) articles


Tips for Finding Research Tips for Finding Research Questions (2)Questions (2)

Callahan, 2001Callahan, 2001

1. Pick a rough area that interests you and your supervisor 2. Do a literature review on the topic and find 5 problems that

interest you. These are easy to find since problems are what the papers are trying to solve. In addition, the conclusions of many papers will indicate open problems.

3. Write them up in a short paragraph with references, as if you were writing a short 2-page dissertation.

4. Take those to your supervisor and discuss them. Out of the 5, you should be able to agree on 1 as your area of research.

5. Now, take this topic and repeat steps 2 to 4 until the topic is sufficiently narrowed.


Describing Your Research Describing Your Research ProblemProblem

You need several clear, concise and You need several clear, concise and succinct statements of the research succinct statements of the research problem problem of different lengthsof different lengths

e.g. one minute (elevator) pitche.g. one minute (elevator) pitch e.g. ten minutes introduction to full seminare.g. ten minutes introduction to full seminar

Issues you must deal with:Issues you must deal with: Can it be understood by others without too Can it be understood by others without too

much background?much background? Does it demonstrate a good understanding of Does it demonstrate a good understanding of

the research community?the research community?


Guide to Research Guide to Research LiteratureLiterature

Types of publicationsTypes of publications conference and workshop papersconference and workshop papers journal papersjournal papers technical reports technical reports monographsmonographs


Conference PapersConference Papers Call for papers - ~1 year before meetingCall for papers - ~1 year before meeting Paper submission - ~4-8 months before meetingPaper submission - ~4-8 months before meeting

Page limit e.g. 8 pagesPage limit e.g. 8 pages Details often omitted (proofs, design technicalities)Details often omitted (proofs, design technicalities)

Program Committee reviews the papersProgram Committee reviews the papers Criteria: significance, originality, soundness, Criteria: significance, originality, soundness,

readabilityreadability Final version for proceedings due ~3 months Final version for proceedings due ~3 months

before meetingbefore meeting revise by author in light of reviewsrevise by author in light of reviews but not checked againbut not checked again

Annual or bi-annual conferencesAnnual or bi-annual conferences


Selection ProcessSelection Process Typically 3 reviewersTypically 3 reviewers Acceptance rate – variesAcceptance rate – varies

Some 10-15%, others 50%Some 10-15%, others 50% Some review “blind” (author details not Some review “blind” (author details not

shown to reviewers), others do notshown to reviewers), others do not

- Example – a reviewer’s form- Example – a reviewer’s form- Ask your supervisor for guidance about - Ask your supervisor for guidance about which are the reliable and important which are the reliable and important conferences in your field!conferences in your field!


Standard of conferencesStandard of conferences

Identify the top conferences in your Identify the top conferences in your areaarea Ask your supervisorAsk your supervisor

CORE ranking (sometimes CORE ranking (sometimes inaccurate)inaccurate)

Affiliation (ACM, IEEE, etc..)Affiliation (ACM, IEEE, etc..) Acceptance rate and review processAcceptance rate and review process


““I regret to inform you…”I regret to inform you…” When a submission is not accepted by a conferenceWhen a submission is not accepted by a conference The author should use the reviewer’s comments to The author should use the reviewer’s comments to

revise and improve the paper, e.g.revise and improve the paper, e.g. if reviewer misunderstood something, author explains it if reviewer misunderstood something, author explains it

more clearlymore clearly if reviewer points to missing citations, author adds themif reviewer points to missing citations, author adds them If reviewer is not convinced, author can do more If reviewer is not convinced, author can do more

experimentsexperiments Then submit revised paper to another conference Then submit revised paper to another conference

in the same communityin the same community Often the resubmission is to a lower prestige conferenceOften the resubmission is to a lower prestige conference Submit to the same conference next year? Not often – IT Submit to the same conference next year? Not often – IT

changes rapidlychanges rapidly


Workshop PapersWorkshop Papers A workshop is typically a smaller meeting A workshop is typically a smaller meeting

than a conference than a conference Sometimes workshop papers are just like Sometimes workshop papers are just like

conference papersconference papers Other workshops are more preliminaryOther workshops are more preliminary

can publish a position paper (draft of an idea can publish a position paper (draft of an idea without evidence, or proposal for future work)without evidence, or proposal for future work)

less rigorously reviewed, the goal is mainly to less rigorously reviewed, the goal is mainly to allow the community to meetallow the community to meet


Journal ArticleJournal Article Typically longer than a conference paperTypically longer than a conference paper Often based on a conference paper with additions, Often based on a conference paper with additions,

corrections and improvementscorrections and improvements Refereed by Refereed by

at least 3 reviewers, experts in the fieldat least 3 reviewers, experts in the field they spend months on the paper checking details, etc. they spend months on the paper checking details, etc.

Decisions: accepted, accepted with minor revisions, Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejectedmajor revisions and resubmission, rejected

Revisions, refereed againRevisions, refereed again Accepted, published after several months (journal issues Accepted, published after several months (journal issues

have limited capacity)have limited capacity) Time from submission to publication varies, typically Time from submission to publication varies, typically

1-1.5 years but may be 3-4 years1-1.5 years but may be 3-4 years


Standard of JournalsStandard of Journals Many journals in each area with different Many journals in each area with different

standardsstandards Typically IEEE Transactions and ACM Typically IEEE Transactions and ACM

Communications are some of the top-ranked Communications are some of the top-ranked journalsjournals

Not all IEEE Trans. and ACM Comm. are top Not all IEEE Trans. and ACM Comm. are top journalsjournals

Check CORE ranking (not always accurate Check CORE ranking (not always accurate though- check with your supervisor)though- check with your supervisor)

Ask you supervisor which journals are the top-Ask you supervisor which journals are the top-ranked and most important in your area!ranked and most important in your area!


Technical ReportTechnical Report Issued by the author’s department, with a Issued by the author’s department, with a

number and datenumber and date May be based on a conference paperMay be based on a conference paper

Longer, includes all the boring details that are Longer, includes all the boring details that are omitted from the conference paper due to space omitted from the conference paper due to space limitationslimitations

Used to establish priority Used to establish priority E.g. produce TR before submitting to E.g. produce TR before submitting to

conference or journal + conference and conference or journal + conference and journal papers may get rejectedjournal papers may get rejected

- - Find the School of IT’s TRs!Find the School of IT’s TRs!


PhD or MSc ThesisPhD or MSc Thesis Very extensive accountVery extensive account

Show much of the research processShow much of the research process Extensive survey of the literatureExtensive survey of the literature Very complete evaluation of the workVery complete evaluation of the work

The goal is to establish that the author is The goal is to establish that the author is ready to become independent researcherready to become independent researcher

i.e. PhD and MSc provide research trainingi.e. PhD and MSc provide research training Typically checked by 2 or 3 reviewersTypically checked by 2 or 3 reviewers


MonographMonograph A collection of selected papers from a A collection of selected papers from a

conference or workshopconference or workshop A bit more checking than for the A bit more checking than for the

conference/workshopconference/workshop An author can offer a coherent and unified An author can offer a coherent and unified

account of a whole research topicaccount of a whole research topic often combines their own results with other often combines their own results with other

people’speople’s Revisits several papers using unified notation, Revisits several papers using unified notation,

better exposition, better literature review, etc.better exposition, better literature review, etc. Publisher may get reviewers but their focus is Publisher may get reviewers but their focus is

“will it sell” not “is it correct”“will it sell” not “is it correct”


WarningsWarnings Quality of conferences and journals varies, Quality of conferences and journals varies,

and this is reflected in the checking of the and this is reflected in the checking of the paperspapers

Read papers with a critical eye!Read papers with a critical eye! Some communities are very clique-Some communities are very clique-

dominateddominated Unpopular opinions are not welcomeUnpopular opinions are not welcome Clique leaders can publish anything, even half-Clique leaders can publish anything, even half-

baked ideas without evidencebaked ideas without evidence


Fake Conferences and Fake Conferences and Random PapersRandom Papers http://pdos.csail.mit.edu/scigen/http://pdos.csail.mit.edu/scigen/

A random paper accepted to a journal?A random paper accepted to a journal?


The Research CommunityThe Research Community A community has conferences and journals A community has conferences and journals

of high prestige which they read and of high prestige which they read and publish inpublish in

They meet often, and each knows (more or less) They meet often, and each knows (more or less) what others are doingwhat others are doing

You must place your work in the context of a You must place your work in the context of a communitycommunity

Divided geographicallyDivided geographically Europe vs America vs AsiaEurope vs America vs Asia


Quality MetricsQuality Metrics How important is an article? How influential is an How important is an article? How influential is an

author?author? Based on citation analysis - number of times a paper or Based on citation analysis - number of times a paper or

author is citedauthor is cited How to calculate citations – Google Scholar + other softwareHow to calculate citations – Google Scholar + other software

Assumption: important authors and articles are cited Assumption: important authors and articles are cited more often than the others more often than the others

Increasingly used by governments, funding bodies, promotion Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s workcommittees to evaluate the quality of author’s work

Some drawbacksSome drawbacks Citing errors – authors with the same names are not separatedCiting errors – authors with the same names are not separated Cliques (friends, colleagues) cite each other in turn to build Cliques (friends, colleagues) cite each other in turn to build

their citation indextheir citation index Negative citations are included (citations to incorrect results)Negative citations are included (citations to incorrect results)


ISI Citation DatabaseISI Citation Database Very popular, established in 1960, contains Very popular, established in 1960, contains

>40million records, contains>40million records, contains Arts and Humanities Citation Index (A&HCI)Arts and Humanities Citation Index (A&HCI) Science Citation Index (SCI)Science Citation Index (SCI) Social Sciences Citation Index (SSCI)Social Sciences Citation Index (SSCI)

HoweverHowever it doesn’t index a large number of journalsit doesn’t index a large number of journals ignores open-access journalsignores open-access journals doesn’t index conferencesdoesn’t index conferences

““Read the Rise and Rise of Citation Analysis” by L. Read the Rise and Rise of Citation Analysis” by L. Meho!Meho!


Journal’s Impact FactorJournal’s Impact Factor Journal impact factorsJournal impact factors

Used to determine the importance of a Used to determine the importance of a journaljournal

E.g. journal impact factor for 2007 =E.g. journal impact factor for 2007 =

# citations in 2007 to articles published in the journal in # citations in 2007 to articles published in the journal in 2005-62005-6

= -------------------------------------------------------------------------------= -------------------------------------------------------------------------------

# articles published in the journal in 2005-6# articles published in the journal in 2005-6

Check CS journal impact factors on ISI Web Check CS journal impact factors on ISI Web of Knowledge! (electronic journals -> J-of Knowledge! (electronic journals -> J->Journal Citation Report)>Journal Citation Report)


CORE’s ratingsCORE’s ratings Computing Research and Education Computing Research and Education

Association of Australasia (CORE)Association of Australasia (CORE) Australia and New ZealandAustralia and New Zealand

Ranking of journals and conferences in CS Ranking of journals and conferences in CS – not finalised– not finalised

http://www.core.edu.au/http://www.core.edu.au/


Author’s Citation Indexes for Author’s Citation Indexes for Measuring ImpactMeasuring Impact

total number of citations total number of citations h-indexh-index

proposed by J.E. Hirsh in 2005:proposed by J.E. Hirsh in 2005:““A scientist has index h if h of his/her Np papers A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-have at least h citations each, and the other (Np-h) papers have no more than h citations each. “h) papers have no more than h citations each. “

What is the h-index?What is the h-index?Paper 1: 20 citationsPaper 1: 20 citationsPaper 2: 15 citationsPaper 2: 15 citationsPaper 3: 8 citationsPaper 3: 8 citationsPaper 4: 4 citationsPaper 4: 4 citationsPaper 5: 3 citationsPaper 5: 3 citations……

An h-index of 4 means that there are at least 4 papers cited at least 4 times each.


Author’s Citation Indexes for Author’s Citation Indexes for Measuring Impact (2)Measuring Impact (2)

g-indexg-index Proposed by L. Egghe 2006: Proposed by L. Egghe 2006:

““Given a set of articles ranked in decreasing order of Given a set of articles ranked in decreasing order of the number of citations that they received, the g-index the number of citations that they received, the g-index is the (unique) largest number such that the top g is the (unique) largest number such that the top g articles received (together) at least g*g citations.”articles received (together) at least g*g citations.”

improves h-index by giving more weight to highly improves h-index by giving more weight to highly cited articles cited articles

Several variants of h-index and g-indexSeveral variants of h-index and g-index

Calculate the g-index for the example from the Calculate the g-index for the example from the previous slide!previous slide!


h vs g indexh vs g index

RanRankk

# # cit.cit.

11 5522 5533 5544 5555 5566 5577 5588 5599 551010 55

RanRankk

# # cit.cit.

11 10010022 505033 505044 505055 5566 5577 5588 5599 551010 55

RR22 ∑∑citcit11 5544 101099 15151616 20202525 25253636 30304949 35356464 40408181 4545100100 5050

RR22 ∑∑citcit11 10010044 15015099 2002001616 2502502525 2552553636 2602604949 2652656464 2702708181 275275100100 280280

h-index?

A B

g index?


h vs g indexh vs g index

RanRankk

# # cit.cit.

1111 551212 551313 551414 551515 551616 551717 551818 551919 552020 55

RR22 ∑∑citcit121121 285285144144 295295169169 300300196196 305305225225 310310256256 315315289289 320320324324 325325361361 330330400400 335335


Publish or PerishPublish or Perish http://www.harzing.com/resources.htm#/pophttp://www.harzing.com/resources.htm#/pop

.htm.htm

Perform a citation analysis of your supervisor’s Perform a citation analysis of your supervisor’s publications! What are the limitations of publications! What are the limitations of

citation analysis in general? citation analysis in general? g- and h-indexes as citation metrics?g- and h-indexes as citation metrics?

1INFO5993 Research Methods, s1 2009 INFO5993 Research Methods Kalina Yacef info5993/ Lecture based...

Documents

Transcript of 1INFO5993 Research Methods, s1 2009 INFO5993 Research Methods Kalina Yacef info5993/ Lecture based...