Critical EvaluationCritical ReadingCritical Thinking

james.bisset@durham.ac.uk

Session outline- What is Critical Reading / Critical

Thinking?- Definitions, Three types of reading a text

- Approaching a process for critical reading- Scanning/Skimming, Critical Reading, Critical Thinking, SQ3R

- Evaluation of Research Information- What to look for as a critical reader when evaluating a text

- Recognising your evaluative criteria in your role as a researcher- What you bring to the table, self awareness and cognitive bias

Exercise 1• Spend 5 minutes to read the short extract on your desks, and

make some brief notes which you would find useful to return to later to re-appraise yourself of the text.

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Part 1What do we mean by

critical reading & thinking?

The non-critical reader

- Reads a text as a source for...- memorising facts & statements

- repeating facts & statements- building a narrative around facts & statements without analysing validity, reliability or applicability

The critical reader- Reads a text as...

- One interpretation of facts

- Recognises the importance of...- what a text says- how the text evidences and portrays the subject matter

Critical Reading“ Critical Reading involves understanding the content of a text as well as how the subject matter is developed. Critical reading takes in the facts, but goes further. “

http://www.rimt.edu.au/studyandlearningcentre/

Exercise 1• Do the notes you made share more similarities with the first or second example on the handout?

(Note, the example notes provided represent only one interpretation of the text of the extract. You may have identified additional points, concerns, facts and questions).

The critical thinker- Reads a text as...

- One interpretation of facts

- Recognises what a text says and does…- applies own knowledge & values- to evaluate and interpret a text’s overall meaning.

Critical Thinking“Critical thinking involves reflecting on the validity of what you have read in light of our prior knowledge and understanding of the world.“

http://www.criticalreading.com/critical_reading_thinking.htm

What a text says…(Restatement)

What a text says…(Restatement)

What a text does…(Description)

What a text means…(Interpretation)

What a text says…(Restatement)

What a text does…(Description)

(how it says what it says)

What a text means…(Interpretation)

(what it means to you, as the reader)

What a text says…(Restatement)

Restate the same topics and facts.

What a text does…(Description)

Discuss the topics & facts within the context of how the original

argument was made.

What a text means…(Interpretation)

Interprets an overall meaning within the wider context of the readers

prior knowledge and values.

Exercise 2• Spend 5 minutes to read the following short [edited] extract and

think about: - what the text says - what the text does - what the text means

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Goals of critical reading

- recognise author's purpose - what is within the scope of their writing, and what isn’t - what they are trying to do; does it match what you are looking for

Goals of critical reading

- recognise author's purpose - what is within the scope of their writing, and what isn’t - what they are trying to do; does it match what you are looking for

- understand tone & persuasive elements of the argument - in contrast to the objective data and evidence - what are they trying to sell you; what are you actually being sold

Goals of critical reading

- recognise author's purpose - what is within the scope of their writing, and what isn’t - what they are trying to do; does it match what you are looking for

- understand tone & persuasive elements of the argument - in contrast to the objective data and evidence - what are they trying to sell you; what are you actually being sold

- recognise bias - identifying patterns of choice of content and language (eg negative vs positive language, repeated omission or discounting)

Critical Reading & Thinking

- Is not about:- being negative or finding fault.

- It is about:- assessing the strength of the evidence and the argument presented

Part 2 An efficient approach to critical reading:

Critical Reading & Thinking

- "If we sense that assertations are ridiculous or irresponsible (critical thinking), we examine the text more closely to test our understanding (critical reading)“

https://www.york.ac.uk/media/biology/documents/careers/critical_reading_handout.pdf

Critical Reading & Thinking

- "If we sense that assertations are ridiculous or irresponsible (critical thinking), we examine the text more closely to test our understanding (critical reading)“

https://www.york.ac.uk/media/biology/documents/careers/

critical_reading_handout.pdf

- Conversely, you can only think about a text critically if you have understood it (critical reading)- to understand why we agree or disagree with an alternative opinion, statement or conclusion. - to understand which issues we agree and/or disagree with in an argument.

Critical Reading: Myth busting

- You do not have the time to read everything.

- You do not have the time to read everything critically.

- You must be selective.- Stay focussed: get the info you need.

Adopt an efficient approach

- Start with some basic principles- Quickly scan/skim the material- [Critical Reading] Read more

thoroughly and make notes- [Critical Thinking]

Consider/Review against your prior knowledge and understanding of the topic

Some basics

Some basics- Most readers have an attention

span of 15-20 minutes.

Some basics- Most readers have an attention span of 15-

20 minutes.

- Be clear about why you are reading the text.

Some basics- Most readers have an attention span of 15-20

minutes.- Be clear about why you are reading the text.

- Pause for thought

Some basics- Most readers have an attention span of 15-20

minutes.- Be clear about why you are reading the text.- Pause for thought

- Don’t (always) read in isolation. Discuss authors, publications, ideas and arguments with colleagues.

Some basics- Most readers have an attention span of 15-20

minutes.- Be clear about why you are reading the text.- Pause for thought- Don’t (always) read in isolation.

Discuss authors, publications, ideas and arguments with colleagues.

- Usually, reading the text once is not enough.

SQ3R

SQRRR

SQRRR- Survey- Question- Read- Recall- Review

Scan / Skim text to provide an overall

impression of usefulness, scope,

structure and argument.

SQRRR- Survey- Question- Read- Recall- Review

Formulate questions you want to answer / points

you want to confirm through more in depth

reading.

SQRRR- Survey- Question- Read- Recall- Review

Read the text in a more focussed way, aiming to

answer the questions

formulated.

SQRRR- Survey- Question- Read- Recall- Review

Pause.

Then test your understanding and

memory of the text, and if you feel

you have addressed all your questions raised.

SQRRR- Survey- Question- Read- Recall- Review

You could also ‘test’ your

understanding by discussing with a

colleague.

SQRRR- Survey- Question- Read- Recall- Review

Return to the text. Read in more

detail, taking notes and identifying any further questions

raised or left unanswered.

Survey

Scanning / Skimming the

text

Scanning text- Before reading in any detail- “Scan” your eye over the text

quickly - to identify specific words or phrases - to get a feel for structure• headings & subheadings• figures, data, images• contents pages

• index (for keywords)• reference list• abstract / body text

Scanning text – why?

- Note – requires you to have thought about why you are reading / what you are looking for.

Scanning text – why?

- Evaluate the relevance and usefulness

Scanning text – why?

- Evaluate the relevance and usefulness

- Make a judgment on whether you should read further

Scanning text – why?

- Evaluate the relevance and usefulness

- Make a judgment on whether you should read further

- Can help you to decide what parts of a document you want to focus time on.

Skimming text- Note – scanning the text first can help

you decide where to concentrate your time.

Skimming text- Speed read to get an overview

of - structure of text- scope and content of the text• Note key points in summaries / abstract• Read 1st and last paragraph / section to get main points• look at 1st sentence of each paragraph to get a feel of content and thrust of argument.

Exercise 3• From a 30 second view, skim the text (focus on first and last lines

of each paragraph) to try and get an impression of the scope and content of the text.

• Holmes, P., Fay, R., Andrews, J., Attia, M., (2013) Researching multilingually: New theoretical and methodological directions. International Journal of Applied Linguistics 23, 285–299.

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

The findings from this study document how researchers understand, and make choices about [using] data when more than one language is involved – and the complex negotiated relationships between researcher and researched as they engage with one another in multilingual sites. The findings challenge the status quo regarding institutional practices and the limitations imposed by interpreters and translators as touched on in current literature. They also permit us to propose a framework that attempts to theorise researching multilingually praxis.

The research has implications for policy, most notably in universities, and particularly where English is the dominant language and where policy reflects this. Questions concerning languages in theses, publications, and examinations point to the need for the decolonisation of the linguistic imperialism of English. Policy also needs to privilege multiple languages in the writing up of research so that researchers, researched, and the communities with which the research is concerned can all access the research.

Pedagogically, our study highlights the need for researcher training programmes – case studies, research methodology papers/books, pedagogic materials – in higher education and among community researcher/practitioners to describe and provide insights into the complexities and possibilities of researching multilingually.

First Impressions?

The findings from this study document how researchers understand, and make choices about [using] data when more than one language is involved– and the complex negotiated relationships between researcher and researched as they engage with one another in multilingual sites. The findings challenge the status quo regarding institutional practices and the limitations imposed by interpreters and translators as touched on in current literature. They also permit us to propose a framework that attempts to theorise researching multilingually praxis.

.How researchers deal with research where knowledge of more than one language is required;

The findings from this study document how researchers understand, and make choices about [using] data when more than one language is involved – and the complex negotiated relationships between researcher and researched as they engage with one another in multilingual sites. The findings challenge the status quo regarding institutional practices and the limitations imposed by interpreters and translators as touched on in current literature. They also permit us to propose a framework that attempts to theorise researching multilingually praxis.

.How researchers deal with research where knowledge of more than one language is required;

Proposes a framework for what is required in supporting such research;

The research has implications for policy, most notably in universities, and particularly where English is the dominant language and where policy reflects this. Questions concerning languages in theses, publications, and examinations point to the need for the decolonisation of the linguistic imperialism of English. Policy also needs to privilege multiple languages in the writing up of research so that researchers, researched, and the communities with which the research is concerned can all access the research.

Addresses implications for policy in HEIs; specifically...

The research has implications for policy, most notably in universities, and particularly where English is the dominant language and where policy reflects this. Questions concerning languages in theses, publications, and examinations point to the need for the decolonisation of the linguistic imperialism of English. Policy also needs to privilege multiple languages in the writing up of research so that researchers, researched, and the communities with which the research is concerned can all access the research.

Addresses implications for policy in HEIs; specifically...

- (language) accessibility of research outputs needs to be addressed;

.

Pedagogically, our study highlights the need for researcher training programmes – case studies, research methodology papers/books, pedagogic materials – in higher education and among community researcher/practitioners to describe and provide insights into the complexities and possibilities of researching multilingually.

- researcher training needs to address the issues raised.

Addresses implications for policy in HEIs; specifically...

Question

Why are you reading the text?

What are you looking for?

- background info?- latest developments?- seek evidence to support/refute an

idea? - to reinforce your own prejudices?

- to examine a methodology or identify how a result was reached?

- because you have to?

Read

Tips whilst reading- Identify core arguments

- link evidence to any conclusions drawn - identify arguments you feel are under-evidenced / purpose behind arguments + interpretation of author. - identify alternative conclusions which could have been drawn.

Tips whilst reading- Look for repetitions of

argument, phrases or words to give clues to authors intentions. - what do they consider crucial? - does this match what you think is crucial?

Recall & Review

Tips for making notes

- Make notes as you read... - throw away your highlighter - annotate margins: key issues / questions raised - develop your own symbols...

(AP) ** // !!

Tips for making notes

Tips for making notes

An argument should:- explain why the authors considered what they are doing is worthwhile- explain the approach and methodology chosen- explain why the data collected/material selected was most appropriate- how conclusions drawn link to wider context

Tips for making notesBroad definition of

evidence:- encompasses what you read, not just the data collected and presented   - choice of methodology   - context of data collection / creation (eg sources used, scope of study/experiment)   - rationale for interpretations and conclusions drawn   - relevance of theory underpinning argument

Part 3 Evaluation of Research Information

Evaluating information When reading critically you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Purpose / Objectivity• Presentation• Methodology• Currency

Evaluating information When reading critically you need to

evaluate:• Relevance to the topic• Authority of the author, publisher etc• Purpose / Objectivity• Presentation• Methodology• Currency

• Before reading the text…• Read the abstract, introduction or

summary.• Scan the bibliographic information

which may highlight key subject areas not specifically alluded to.

• Emphasis may not be clear until you read in full.

Relevance to the topic

• Upon reading the text…• What level is the information at?• Does it contain, and discuss in enough

detail the information you are seeking?• Is the research relevant to the subject

domain / geographical area / demographic / time period you are interested in?

Relevance to the topic

“In the course of a series of investigations into various aspects of mental inheritance an intensive study has been made of so-called ‘identical’ twins. The cases examined fall into two main groups: first, those reared together in their parents' homes; secondly, those separated in early infancy, and brought up apart. With the latter, despite wide differences in environmental conditions, the correlations for intelligence, unlike those for school attainments, prove to be surprisingly high. It is argued that this implies that ‘intelligence’, when adequately assessed, is largely dependent on genetic constitution.” http://10.1111/j.2044-8295.1966.tb01014.x

“ Burt’s study of monozygotic twins reared apart … involved the largest number of separated twin pairs at the time and produced the highest estimate of heritability for IQ”

Relevance to the topic Exampl

e

Restatement: Burke identified a link between IQ and inherited genes.

Description: This article compares the evidence for IQ being determined by inherited genes as opposed to IQ being affected by external environmental factors.

Interpretation: Evidence for IQ being an inherited trait rather than affected by external factors has potential implications for the development of social and education policy.

Relevance to the topic Exampl

e

• Key topics and ideas.• Level of information presented.• Relevance in terms of location/subject/scope.• Does the content and level match your

needs.

Relevance to the topic

Be aware of what is filtering your choices… - Vocabulary and broadness of interpretation. Are you under-estimating the value of a source because it doesn’t match your choice of keywords precisely?

“The trouble with quotes from the internet is that

you never know if they are genuine.”

Abraham Lincoln(1809-1865, President of the United States of

America).

Authority

• Is it clear where the information comes from?

• Are the authors acknowledged experts in the field? - frequently cited? do they have an h-index? - have you or colleagues heard of them? - can you find any profile information where they work? - how well respected is the author, and their work, in their related field of research?

• Where is it published? - impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige) - is it peer reviewed?

Authority

• Sir Cyril Burt• Fellow of British Academy• Author of over 350 articles and a number of books.

• “pioneer research on the inheritance of mental ability”

AuthorityExampl

e

Authority – Citations Exampl

e

””the most satisfactory attempt” to estimate hereditability of IQ” and

“”the most valuable” of all the separated twin research.”

”the largest of its kind and the only one where “the distribution of children into

foster homes was random ” “

Richard J Herrnstein

Arthur R Jenson Exampl

e

””the most satisfactory attempt” to estimate hereditability of IQ” and

“”the most valuable” of all the separated twin research.”

• Professor at the University of California, Berkeley. • Author of 400+ peer reviewed papers. • In 2002, was listed in the Review of General Psychology’s top 50 “most eminent psychologists of the 20th century”

”the largest of its kind and the only one where “the distribution of children into

foster homes was random ” “

• Edge Pierce Professor of Psychology at Harvard University.• Author of several peer reviewed papers and books. • Credited with discovering and developing several models and theories as one of the founding researchers in the field of quantitative analysis of behaviour.

Richard J Herrnstein

Arthur R Jenson

”the only one of its kind in which the calculation of heritability had any

meaning.”

””the best data“ on separated twins.”

William B Shockley

Hans Eysenck Exampl

e

”the only one of its kind in which the calculation of heritability had any

meaning.”

• Professor of Psychology at the Institute of Psychiatry, King’s College, London. • In 2002, was listed in the Review of General Psychology’s top 100 “most eminent psychologists of the 20th century” as the most cited living psychologist at the time of his death.

””the best data“ on separated twins.”

• Alexander M. Poniatoff Professor of Engineering and Applied Science at Stanford University.• Joint awarded Nobel Prize in Physics in 1956.• 1960’s and 1970’s moved also into area of hereditary behaviour.

William B Shockley

Hans Eysenck

- British Journal of Mathematical and Statistical Psychology - Impact Factor 1.258 - 5th of 13 journals in category “Psychology, Mathematical”

- British Journal of Educational Psychology - Impact Factor 2.093 - 11th of 50 journals in category “Psychology, Educational”

- British Journal of Psychology - Impact Factor 2.103 - 26th of 126 journals in category “Psychology, Multi-disciplinary”

Authority – Impact Factors

Exampl

e

• Are the authors acknowledged experts in the field? - frequently cited? do they have an h-index? - have you or colleagues heard of them? - can you find any profile information where they work? - how well respected is the author, and their work, in their related field of research?

• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige) and is it peer reviewed?

Authority

Be aware of what is filtering your choices y - Is the prestige of the author or the publication impacting on how you evaluate the content?

• Is the subject controversial?

• Does the author use emotive language?

• What is the authors purpose in writing the paper?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

** This is an over-simplification…. **

Objectivity

IQ is inherited IQ is affected by external

factorsBurt, C (1943) “Ability and Income” British Journal of Educational Psychology

Burt, C.L. (1957) “Heredity and Intelligence; A reply to criticisms” British Journal of Statistical Psychology

Burt, C.L. (1958). "The inheritance of mental ability", American Psychologist,

Burt, C.L. (1972). "Inheritance of general intelligence", American Psychologist,

Burt C (1966) “The Genetic Determination of Differences in Intelligence: A Study of Monozygotic Twins Reared Apart and Together.” British Journal of Psychology

Exampl

e

• Is the subject controversial?

• Does the author use emotive language?

• What is the authors purpose in writing the paper?

• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?

Objectivity

Be aware of what is filtering your choices - Does the author demonstrate any hidden bias on the topic? - Evaluate yourself? Are you subconsciously over-valuing the resource because it confirms your own prejudices? Are you being objective?

• are the methodology and sources of data used clearly identified or explained?

• what evidence is presented to support the ideas and conclusions expressed?

• is the methodology direct or indirect?• are the conclusions and assumptions made

by the author consistent, logical and justified?

• does the research raise any unanswered questions?

Methodology

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy

than the judgments of the pupil’s own teachers.”

Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less trustworthy in the long run that the

results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy

than the judgments of the pupil’s own teachers.”

Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less trustworthy in the long run that the

results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy

than the judgments of the pupil’s own teachers.”

Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less trustworthy in the long run that the

results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

“ tests of the usual type… as a means of estimating genotypic differences, even the most carefully constructed tests are highly fallible instruments, and … their verdicts are far less trustworthy

than the judgments of the pupil’s own teachers.”

Burt, C. (1957)British Journal of Statistical Psychology

“the unaided judgments even of the most experienced teachers … are nevertheless far less trustworthy in the long run that the

results obtained by properly applied intelligence tests.”

Burt, C. (1943)British Journal of Educational Psychology

Methodology Exampl

e

• Various criteria you can assess a resource by.

- a lot more ‘citation’ tools available for journal literature.

• How much time do you realistically have?

Part 3 Summary

• Some quotes and opinions were taken from one article:

Tucker, W.H.(1994) “Fact and fiction in the discovery of Sir Cyril Burt’s Flaws” Journal of the History of the Behavioral Sciences (30).

• Does that change your opinion on some of the previous slides assertions and emphasis?

Part 3 Summary

Part 4Recognisin

g your evaluative

criteria

Critical Thinking“Critical thinking involves reflecting on the validity of what you have read in light of our prior knowledge and understanding of the world.“

http://www.criticalreading.com/critical_reading_thinking.htm

• Resources are interconnected and they evolve

• Information resources are transformed into knowledge

• Knowledge becomes a resource

• Therefore prior knowledge shapes what we go on to create

Ecology of Resources

• Resources are interconnected and they evolve

• Information resources are transformed into knowledge

• Knowledge becomes a resource

• Therefore prior knowledge shapes what we go on to create

Ecology of Resources

Cognitive biases• Subjectivity is vulnerable to bias &

hunches• Concept of cognitive bias was

developed in 1970s by Tversky and Kahneman

• Four main groups- Social - Probability/belief - Memory - Decision making

1) “Hwang faked all research on human stem cells”

“Hwang faked all research on human stem cells”

Halo effect“The tendency for a person's

positive or negative traits to "spill over" from one personality area to another in others' perceptions of

them”

2) Strange, Hayne & Garry (2008) ‘A photo, a suggestion, a false memory’ Applied Cognitive Psychology. 22, 587–603.

Strange, Hayne & Garry (2008) ‘A photo, a suggestion, a false memory’ Applied

Cognitive Psychology. 22, 587–603.

False memory bias

“A form of misattribution where imagination, or incorrect recall, is

mistaken for a memory.”

3) “Once people become aware of an event or

conclusion, they will usually express a good deal of confidence about the predictability of the

outcome.”

Fischhoff, B., and Beyth, R. (1975) ‘"I knew it would happen" Remembered probabilities of once-future things’

Organizational Behaviour and Human Performance. 13, 1-16.

Hindsight bias“The inclination to see past events as

being more predictable than they actually were”

Cognitive biases• Shape how we read, interpret and

evaluate the information we receive.

• Shape how others read, interpret and evaluate the information they receive.

• Any published research is just one interpretation of the facts available

Further Reading• Tucker, W.H.(1994) “Fact and fiction in the discovery of Sir Cyril Burt’s Flaws”

Journal of the History of the Behavioral Sciences (30).

• University of Leicester: What is Critical Reading http://www2.le.ac.uk/offices/ld/resources/writing/writing-resources/critical-reading

• University of Newcastle: Think Criticallyhttp://www.ncl.ac.uk/library/research-support/informed-researcher/think-critically/

• Open University: Critical Reading Techniqueshttp://www.open.ac.uk/skillsforstudy/critical-reading-techniques.php/

• Dan Kurland’s Critical Reading Websitehttp://www.criticalreading.com/

• The SQ3R methodhttp://www.ic.arizona.edu/ic/wrightr/other/sq3r.html

Image Credits[96,97] Via Flickr Creative Commons, by Martin LaBar. Original available at http://www.flickr.com/photos/32454422@N00/163107859/

[4,65,194] Via Flickr Creative Commons, by Kevin Dooley. Available at http://www.flickr.com/photos/12836528@N00/2577006675

[104] Via Flickr Creative Commons, and by Brian Yap: Original available at http://www.flickr.com/photos/30265340@N00/465957804

[20] Via Flickr Creative Commons, and by FutUndBeidl: Available at http://www.flickr.com/photos/61423903@N06/7369580478

[Slides 111] Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf

Measuring Researche

r Developm

ent

Vitae Researcher Development Framework [see image credits]