Critical Evaluation

of Research Information

James Bisset (james.bisset@durham.ac.uk ) Academic Liaison Librarian (Research Support)

Session outline- Importance of evaluation- Forms of value - Group activity

- Evaluating Research Information

Session outline- Importance of evaluation- Forms of value - Group activity

- Evaluating Research Information

Self awareness –

what do you bring

to the table

Session outline- Importance of evaluation- Forms of value - Group activity- Evaluating Research Information

Self awareness –

what do you bring

to the table

Some practical

tips and

evaluation criteria

Part 1 Importance

of evaluation

• Much training is about directing you to the right information = searching and retrieval

• As postgraduate researchers you have to be critical and reflect on what you find.

• Be aware of your impact on your own research, and the research of others.

• What defines your evaluative criteria?

The need to evaluate information

• Resources are interconnected and they evolve

• Information resources are transformed into knowledge

• Knowledge becomes a resource

• Therefore prior knowledge shapes what we go on to create

Ecology of Resources

Role of the researcher

• In theory we can select almost any information to complete a task

• In practice we filter it by selecting resources we think most appropriate

• Motivation - affected by the learning we have already done

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP

Other factors

• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP

• You need to be critical and reflect on all of the sources you find and use.

• You have a professional responsibility, as your research will impact on others.

• You are creating knowledge…… which evolves, and will shape what others create…… similarly, the information you discover will shape the knowledge you create.

• You need to be aware of the filters already impacting upon the information you use.

Part 1 Summary

Part 2 Forms of value

• Filtering process = value judgement– By researcher–Made on their behalf

• What forms of value are there and how do they work together to create information literate researcher?

Forms of Value

Objective form of value

• Scientific measures of validity or reliability

• Exists so that personal values don’t unduly influence work

• Omit this scheme of value and we risk information (and knowledge formed from it) becoming counterknowledge (Thompson, 2008).

Intersubjective form of value

• Based on the shared values of a community e.g. morals, ethics, laws, economics

• Allows for discussion of scientific method as it can’t explain everything

• Acceptance in a community• If we omit this then values are

relativist

Subjective form of value

• Decisions you make– Is this what I want, do I need this, is it

relevant?

• Privileges you as the researcher in the decision making process

• If we omit it we get groupthink (Janis, 1972) or battery cognition (Blaug, 2007)

• Importance therefore of asserting individual criticality

Groupthink

Cognitive biases• Subjectivity is vulnerable to bias &

hunches• Concept of cognitive bias was

developed in 1970s by Tversky and Kahneman

• Four main groups- Social - Probability/belief - Memory - Decision making

Social biases• Ascribe positive or negative traits to

self, individuals or groups• Loading values or anticipating action

based on prior experience or a bias against self, individuals or groups

• Academic impact: need to verify information and not rely on own views; important to remember when analysing human subjects

Memory biases• How you perceive past events• False memory, positive memory,

imbalanced memory• E.g. A Photo, a Suggestion, a False

Memory • Academic impact: importance of

accurate record keeping and note taking

Memory biases

Memory biases

Probability and belief

• To disregard or to pay too much attention to probability

• Academic impact: need to treat each research finding as distinct and to judge it in its own right

Decision-making biases

• Influences on your decisions by own biases or those of a group

• Academic impact: need to be objective and consider all possible routes of enquiry and treat all research findings as valid until proved otherwise e.g. Semmelweis reflex

Cognitive biases• On your table, group the

forty cards into four piles of ten• Social • Memory • Probability• Decision

Three forms of valueSubjectiv

e

Inter-subjectiveObjective

• Different concepts of ‘value’ and the need to evaluate in terms of each of these. - need to be objective and look at the measurable facts - need to be inter-subjective and apply to the wider context - need to be subjective, and assess the value based upon our own needs

• Explored some of the key bias which may subconsciously be impacting upon how your (subjective) measure of ‘value’ might effect how you filter information and your evaluative criteria.#

Part 2 Summary

Part 3 Evaluation of Research Information

Evaluating information In a literature review you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Method of production and methodology• Currency

Evaluating information In a literature review you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Method of production and methodology• Currency

• Read the abstract, introduction or summary.• Scan the bibliographic information which may

highlight key subject areas not specifically alluded to.

• Emphasis may not be clear until you read in full.

Relevance to the topic

• Read the abstract, introduction or summary.• Scan the bibliographic information which may

highlight key subject areas not specifically alluded to.

• Emphasis may not be clear until you read in full.

Relevance to the topic

Be aware of what is filtering your choices… - Vocabulary and broadness of interpretation. Are you under-estimating the value of a source because it doesn’t match your choice of keywords precisely?

• Are the authors acknowledged experts in the field? - frequently cited? - have you or colleagues heard of them? - do they have an h-index? - can you find any profile information where they work?

• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige)

Authority

• Are the authors acknowledged experts in the field? - frequently cited? - have you or colleagues heard of them? - do they have an h-index? - can you find any profile information where they work?

• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige)

Authority

Be aware of what is filtering your choices y - Is the prestige of the author impacting on how you evaluate the content?

• Is the subject controversial?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

• Is the subject controversial?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

Be aware of what is filtering your choices - Does the author demonstrate any hidden bias on the topic? - Evaluate yourself? Are you subconsciously over-valuing the resource because it confirms your own prejudices? Are you being objective?

• Is the article peer-reviewed?• Can you identify the editor/editorial

board for the publication?

Method of production

• Is the article peer-reviewed?• Can you identify the editor/editorial

board for the publication?

Method of production

Be aware of what is filtering your choices… - Be aware of editorial policy which may decide what is published.

2.23.39 – 2.25.56http://

news.bbc.co.uk/1/hi/help/3681938.stm

• Various criteria you can assess a resource by.

- a lot more ‘citation’ tools available for journal literature.

• How much time do you realistically have?

Part 3 Summary

Bibliography• Blaug, R. (2007) ‘Cognition in a hierarchy’, Contemporary Political Theory. 6: 24–44

• Goldacre, B. (2008) Bad science. London: Harper Press.

• Janis, I. (1972) Victims of groupthink: a psychological study of foreign-policy decisions and fiascoes. Boston: Houghton Mifflin.

• Kahneman, D. and Amos, A. (1972) ‘Subjective probability: a judgment of representativeness’, Cognitive Psychology. 3(3): 430-454.

• Luckin, R. (2010) Redesigning learning contexts: technology-rich, learner-centred ecologies. Abingdon: Routledge.

• Strange, D., Hayne, H. and Garry, M. (2008) ’A photo, a suggestion, a false memory’, Applied Cognitive Psychology. 22: 587–603.

• Thompson, D. (2008) Counterknowledge: How We Surrendered to Conspiracy Theories, Quack Medicine, Bogus Science and Fake History. London: Atlantic Books.

• Whitworth, A. (2009) Information Obesity. Oxford, UK: Chandos. In particular chapter 2.

• Whitworth, D (2010) “The three domains of value: Why IL practitioners must take a holistic approach” Available at: http://prezi.com/rxqnzpoooolb/the-three-domains-of-value-why-il-practitioners-must-take-a-holistic-approach/

• http://www.informationliteracy.ie/

Image Credits[Slide 5, 38] Via Flickr Creative Commons, by Martin LaBar. Original available at http://www.flickr.com/photos/32454422@N00/163107859/

[Slide 3] Via Flickr Creative Commons, by Kevin Dooley. Original available at http://www.flickr.com/photos/12836528@N00/2577006675

[Slide 10] Via Flickr Creative Commons, by shellorz. Original available at http://www.flickr.com/photos/59198719@N00/2192821345

[Slide 13] Via Flickr Creative Commons, by Richard Cocks. Original available at http://www.flickr.com/photos/richardland/3999234316/

[Slide 12] Via Flickr Creative Commons, by Photo Extremist. Original available at http://www.flickr.com/photos/thevlue/4839060646/

[Slide 11] Via Flickr Creative Commons, by What Dave Sees. Original available at http://www.flickr.com/photos/whatdavesees/2487875504/

Image Credits[Slide 14] Via Flickr Creative Commons, by vl8189. Original available at http://www.flickr.com/photos/27630470@N03/

[Slide 15] Via Flickr Creative Commons, by opensourceway. Original available at http://www.flickr.com/photos/47691521@N07/4371001458/

[Slide 17] Via Flickr Creative Commons, by otherthings. Original available at http://www.flickr.com/photos/18619970@N00/3057937540

[Slide 30-31] Photo provided by colleague

[Slide 54] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf

Measuring Researche

r Developm

ent

Vitae Researcher Development Framework [see image credits]