INFO4990 Research Methods

1INFO4990 Research Methods, s2 2008

INFO4990 Research MethodsINFO4990 Research Methods

Bing Bing ZhouBing Bing Zhouhttp://www.cs.usyd.edu.au/~info4990/http://www.cs.usyd.edu.au/~info4990/

Lecture based in part on materials by Alan Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Fekete, Mary Lou Maher, Joseph Davis, Irena

Koprinska and othersKoprinska and others

Research – Components and Research – Components and ProcessProcess


OutlineOutline Administrative mattersAdministrative matters Research Research

Definition, key components, Definition, key components, processprocess Finding a research questionFinding a research question


AdministrativiaAdministrativia Course web page: Course web page:

http://www.cs.usyd.edu.au/~info499http://www.cs.usyd.edu.au/~info4990/0/

2 hours lectures/workshops, 4-6pm 2 hours lectures/workshops, 4-6pm on Mondayson Mondays

Coordinator: Bing Bing ZhouCoordinator: Bing Bing Zhou Lectures given by the coordinator and Lectures given by the coordinator and

invited lecturers (IT academics, invited lecturers (IT academics, learning centre staff, librarians)learning centre staff, librarians)

Consultation time: Monday 3-4pmConsultation time: Monday 3-4pm


AdministrativiaAdministrativia No textbook, on-line resources – No textbook, on-line resources –

check the web pagecheck the web page AssignmentsAssignments

1 – search results (15%), w41 – search results (15%), w4 2 - literature review and outline of 2 - literature review and outline of

research (45%), w8research (45%), w8 3 - presentation (30%) +feedback on 3 - presentation (30%) +feedback on

other presentations (10%) w12-13other presentations (10%) w12-13 Basser seminar attendance required – Basser seminar attendance required –

max penalty 5%max penalty 5%


Topics OverviewTopics Overview Introduction to research – definition, Introduction to research – definition,

components, process, how to find a components, process, how to find a research questionresearch question

Literature reviewLiterature review how to search for relevant publications how to search for relevant publications

Writing a literature review and research Writing a literature review and research proposalproposal

Oral presentation skillsOral presentation skills Research methods in IT (statistical analysis, Research methods in IT (statistical analysis,

mathematical analysis, algorithm analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, ethics, etc.) simulation, qualitative analysis, ethics, etc.)


Definition of ResearchDefinition of Research1) From the Merriam-Webster dictionary: 1) From the Merriam-Webster dictionary: 11:: careful or diligent search careful or diligent search 22:: studious inquiry or examination; studious inquiry or examination; especiallyespecially:: investigation or experimentation aimed at the investigation or experimentation aimed at the discovery and interpretation of facts, revision of discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or accepted theories or laws in the light of new facts, or practical application of such new or revised theories or practical application of such new or revised theories or lawslaws

33:: the collecting of information about a particular subject the collecting of information about a particular subject

2) Booth, Columb & Williams, “The Craft of Research”: 2) Booth, Columb & Williams, “The Craft of Research”: ““Research is gathering Research is gathering informationinformation that answers a that answers a questionquestion and so and so solvessolves a problem.” a problem.”


Is This Research?Is This Research?

To understand political decisions, a To understand political decisions, a journalist finds out who contributed to journalist finds out who contributed to election campaign fundelection campaign fund

To buy a laptop, a student compares To buy a laptop, a student compares various brands, configurations and pricesvarious brands, configurations and prices

To help companies stay competitive, a To help companies stay competitive, a market researcher collects and interprets market researcher collects and interprets informationinformation

To fix a computer, a technician finds out To fix a computer, a technician finds out what procedure to usewhat procedure to use


Academic ResearchAcademic Research In academic research, you must not only answer a In academic research, you must not only answer a

question, question, but you must find something but you must find something newnew and and interesting interesting

andand you must you must advanceadvance the collective understanding of the the collective understanding of the

communitycommunity Each community has a cumulative tradition with a Each community has a cumulative tradition with a

set of interesting questions, tools and methods, set of interesting questions, tools and methods, practices, a style and language for writing up the practices, a style and language for writing up the researchresearch Research is also a conversation and ongoing social activity!Research is also a conversation and ongoing social activity!

You need critical and careful reading of published You need critical and careful reading of published researchresearch to learn what the community already knowsto learn what the community already knows to fit your work into the communityto fit your work into the community to be prepared for your own work to be evaluatedto be prepared for your own work to be evaluated


Key Components of Key Components of ResearchResearch A question of interest (research question)A question of interest (research question)

A claim (contribution)A claim (contribution) EvidenceEvidence Argument (links evidence to claim) Argument (links evidence to claim)


A Research QuestionA Research Question Every piece of research should address a question of Every piece of research should address a question of

interest to the communityinterest to the community Each community has traditional questions:Each community has traditional questions:

What happens? Why does it happen? How should one do What happens? Why does it happen? How should one do something? What something should one do?something? What something should one do?

Many questions fit into an on-going agenda, e.g.Many questions fit into an on-going agenda, e.g. Data mining foundations – mining sequential data; high-Data mining foundations – mining sequential data; high-

performance implementations of data mining algorithms, etc.performance implementations of data mining algorithms, etc. Mining emerging data - e-commerce , web search data, Mining emerging data - e-commerce , web search data,

moving object data, data from sensor networksmoving object data, data from sensor networks …… See a Conference Call for PapersSee a Conference Call for Papers


A Claim (Contribution)A Claim (Contribution) Every piece of research makes a claim Every piece of research makes a claim

(the “contribution”) answering a (the “contribution”) answering a research questionresearch question

Claims can be very diverse among Claims can be very diverse among fields and within fieldsfields and within fields


EvidenceEvidence You must back up the claim with evidence, e.g.You must back up the claim with evidence, e.g.

A mathematical proof to show that some process/algorithm has A mathematical proof to show that some process/algorithm has desired propertiesdesired properties

Analysis of the computational complexity of an algorithmAnalysis of the computational complexity of an algorithm Empirical evaluation of a machine learning algorithm to evaluate its Empirical evaluation of a machine learning algorithm to evaluate its

accuracyaccuracy A simulation model which is executed and analysed to show certain A simulation model which is executed and analysed to show certain

propertiesproperties A prototype implementation to show that a system can be built to A prototype implementation to show that a system can be built to

achieve the claimed functionalityachieve the claimed functionality Measurements of a running system to show it has good Measurements of a running system to show it has good

performanceperformance Observations of behaviour in an organisation to show what is Observations of behaviour in an organisation to show what is

happeninghappening Various research methods, each defined by the sort of Various research methods, each defined by the sort of

evidence that it can produceevidence that it can produce each community has its own standards of quality and each community has its own standards of quality and

reasonablenessreasonableness


ArgumentArgument You should show that the evidence you offer You should show that the evidence you offer

supports the claim you makesupports the claim you make It’s essential that you deal with natural or obvious It’s essential that you deal with natural or obvious

objections to the correctness or importance of the objections to the correctness or importance of the workwork

that is, you must think like your readers, and that is, you must think like your readers, and anticipate their reactionsanticipate their reactions

In systems work, this is often called an In systems work, this is often called an “evaluation” of the design“evaluation” of the design


Claim and Argument - Claim and Argument - ExamplesExamples

This system design leads to better This system design leads to better performance on some metricperformance on some metric make sure you limit how much worse this make sure you limit how much worse this

makes other metrics (such as cost!)makes other metrics (such as cost!) make sure your measurements are fair (don’t make sure your measurements are fair (don’t

compare with “strawman” design but with compare with “strawman” design but with state-of-the-art)state-of-the-art)

This system design offers better This system design offers better functionality for some usesfunctionality for some uses make sure you show it can be implemented make sure you show it can be implemented

with adequate performancewith adequate performance


Claim and Argument – Claim and Argument – Examples (2)Examples (2)

This behaviour can be explained by This behaviour can be explained by this theorythis theory make sure you don’t have confounding factors make sure you don’t have confounding factors

such as level of experience, or method novelty, such as level of experience, or method novelty, or subject expectationsor subject expectations

This is what happensThis is what happens make sure you don’t interfere too much with make sure you don’t interfere too much with

what happens when you gather data, or what happens when you gather data, or misinterpret it due to observer expectations misinterpret it due to observer expectations


Common Mistakes 1Common Mistakes 1

Gather lots of data without a focussed Gather lots of data without a focussed question or methodquestion or method

A collection of facts is not a contribution!A collection of facts is not a contribution! it must reveal some pattern or understanding it must reveal some pattern or understanding

that you make explicitthat you make explicit


Common Mistakes 2Common Mistakes 2 Build a system without a focused question Build a system without a focused question

or planned evaluationor planned evaluation E.g. let’s see how to use aspect-oriented E.g. let’s see how to use aspect-oriented

programming in a sensor networkprogramming in a sensor network An innovative system is not a An innovative system is not a

contribution!contribution! it must be a worthwhile innovation in a sense it must be a worthwhile innovation in a sense

you make explicityou make explicit E.g. better performanceE.g. better performance E.g. new functionalityE.g. new functionality


Negative ResultsNegative Results Sometimes, you don’t get the result you hoped Sometimes, you don’t get the result you hoped

forfor You gather data that does not reveal any pattern or You gather data that does not reveal any pattern or

understandingunderstanding E.g. no factor seems to correlate well with project successE.g. no factor seems to correlate well with project success

You design a system that turns out to be worse than You design a system that turns out to be worse than the state-of-the-artthe state-of-the-art

E.g. your machine learning algorithm runs slower than E.g. your machine learning algorithm runs slower than expectedexpected

You can still salvage a thesisYou can still salvage a thesis Try to find some way to contribute to our Try to find some way to contribute to our

understanding, or suggest fruitful directions for understanding, or suggest fruitful directions for further workfurther work

E.g. what features of the algorithm make it slowE.g. what features of the algorithm make it slow Make sure the problem is intrinsic, not just your bad Make sure the problem is intrinsic, not just your bad

coding/experiment design/etccoding/experiment design/etc


Ground-Breaking WorkGround-Breaking Work Very rarely, a piece of research will establish a Very rarely, a piece of research will establish a

whole new agenda for a field, or even a new fieldwhole new agenda for a field, or even a new field the contribution can be as much in the possibilities for the contribution can be as much in the possibilities for

further work, as in the result itself!further work, as in the result itself! In some sense, this is work that asks a new type In some sense, this is work that asks a new type

of question, or introduces a new methodof question, or introduces a new method

We don’t recommend this for We don’t recommend this for Hons/MIT/MSc/MEngHons/MIT/MSc/MEng save the idea till you have time enough, and flexibility save the idea till you have time enough, and flexibility

enough to deal with inevitable digressions/difficultiesenough to deal with inevitable digressions/difficulties

INFO4990 Research Methods, s2 2008 20

Great scholars do not Great scholars do not solve problems; they solve problems; they

create them.create them.

-Albert Einstein-Albert Einstein


Idealised Research Idealised Research ProcessProcess

Find a question to seek an answer forFind a question to seek an answer for Method: Choose an appropriate research Method: Choose an appropriate research

method and make flexible plansmethod and make flexible plans Evidence: Gather the data, do the Evidence: Gather the data, do the

experiment, build the prototype etc.experiment, build the prototype etc. Contribution: Analyse, interpret, and Contribution: Analyse, interpret, and

concludeconclude Argument: Write the reportArgument: Write the report

Importance of “writing” (aided by thinking Importance of “writing” (aided by thinking from the point of view of your readers)from the point of view of your readers)


Actual Research ProcessActual Research Process Research explores new areas and the results are Research explores new areas and the results are

not predictable!not predictable! The research plan is iterativeThe research plan is iterative

Gathering evidence leads to changes to the claimGathering evidence leads to changes to the claim sometimes one refines the claimsometimes one refines the claim

E.g. limit the scope E.g. limit the scope from “algorithm X outperforms Y” to “algorithm X outperforms Y from “algorithm X outperforms Y” to “algorithm X outperforms Y

when the independence assumption is violated” when the independence assumption is violated” From “X has higher throughput” to “X has higher throughput if the From “X has higher throughput” to “X has higher throughput if the

contention rate is low”contention rate is low” sometimes one must change the claim entirelysometimes one must change the claim entirely sometimes while gathering evidence, one finds new questions sometimes while gathering evidence, one finds new questions

which look worth answering!which look worth answering!

New claims or questions need further evidence, New claims or questions need further evidence, revised plans, maybe even different methodsrevised plans, maybe even different methods


Finding a QuestionFinding a Question Especially when you are learning to do research, it Especially when you are learning to do research, it

may be already chosen for you by supervisormay be already chosen for you by supervisor or supervisor may suggest an area, and leave you to find or supervisor may suggest an area, and leave you to find

the questionthe question A question may arise from some previous researchA question may arise from some previous research

Further work, issues not addressed, holes in the Further work, issues not addressed, holes in the evidence collectedevidence collected

A question may come from the combination of A question may come from the combination of previous researchprevious research Bring two areas together, use a technique from one area Bring two areas together, use a technique from one area

in anotherin another A question may arise due to new technologyA question may arise due to new technology

new hardware or technique may require new models, new hardware or technique may require new models, new hardware may influence use or performance or new hardware may influence use or performance or feasibilityfeasibility


Suitable Research Suitable Research QuestionsQuestions

Answerability – can the questions be Answerability – can the questions be answered through research?answered through research?

Scale: Consider available resources Scale: Consider available resources (equipment, time, skills)(equipment, time, skills)

Scope: Often start with broad topic Scope: Often start with broad topic space/ bigger question, then narrow space/ bigger question, then narrow in to a specific questionin to a specific question


Tips for Finding Research Tips for Finding Research QuestionsQuestions

Read the papers you supervisor gave youRead the papers you supervisor gave you follow the references, check the web pages of the follow the references, check the web pages of the

authorsauthors read carefully the “Future research” sectionsread carefully the “Future research” sections write down your ideas!!write down your ideas!!

Find the top conferences/journals in your field Find the top conferences/journals in your field scan the call for papers and associated workshops for scan the call for papers and associated workshops for

hot topicshot topics scan the conference proceedings/journals to identify scan the conference proceedings/journals to identify

important topics, key people and research groups. important topics, key people and research groups. Check their web pages.Check their web pages.

Find review (survey) articlesFind review (survey) articles


Tips for Finding Research Tips for Finding Research QuestionsQuestions

Callahan, 2001Callahan, 2001

1. Pick a rough area that interests you and your supervisor 2. Do a literature review on the topic and find 5 problems that

interest you. These are easy to find since problems are what the papers are trying to solve. In addition, the conclusions of many papers will indicate open problems.

3. Write them up in a short paragraph with references, as if you were writing a short 2-page dissertation.

4. Take those to your supervisor and discuss them. Out of the 5, you should be able to agree on 1 as your area of research.

5. Now, take this topic and repeat steps 2 to 4 until the topic is sufficiently narrowed.


Describing Your Research Describing Your Research ProblemProblem

You need several clear, concise and You need several clear, concise and succinct statements of the research succinct statements of the research problem problem of different lengthsof different lengths

e.g. one minute (elevator) pitche.g. one minute (elevator) pitch e.g. ten minutes introduction to full seminare.g. ten minutes introduction to full seminar

Issues you must deal with:Issues you must deal with: Can it be understood by others without too Can it be understood by others without too

much background?much background? Does it demonstrate a good understanding of Does it demonstrate a good understanding of

the research community?the research community?

INFO4990 Research Methods

Documents

Transcript of INFO4990 Research Methods