Detecting Bias in Research Papers - NCSL
Transcript of Detecting Bias in Research Papers - NCSL
![Page 1: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/1.jpg)
Detecting Bias in Research Papers
2007 LSSS, RACSS & LRL Joint SeminarSanta Fe, New Mexico September 11, 2007
![Page 2: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/2.jpg)
Goal:
Identify the implicit theories underlying policy and program proposals
Show how those theories lead to specific policy options
Demonstrate a tool for uncovering and examining the implied theories supporting these options
![Page 3: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/3.jpg)
Research Context
Conducting secondary research
Analyzing papers, reports, and studies proposing new policies
![Page 4: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/4.jpg)
Approach
Describe “systems thinking”
Apply this conceptual tool to the analysis supporting a major policy proposal
![Page 5: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/5.jpg)
What is bias?
An inclination, leaning, tendency or bent
A prepondering disposition or propensity
A predisposition towards, predilection, or prejudice
![Page 6: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/6.jpg)
Thomas Jefferson on theoretical bias:
“The moment a person forms a theory his imagination sees in every object only the tracts that favor that theory.”
![Page 7: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/7.jpg)
Einstein put it this way:
“Our theories determine what we measure.”
![Page 8: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/8.jpg)
Paul Simon on bias
“All lies and jest, still a man hears what he wants to hear and disregards the rest.” (“The Boxer”)
![Page 9: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/9.jpg)
Theoretical Bias
Implied, unstated
Revealed in the dots one chooses to study (i.e., the variables)
And how one connects those dots (cause and effect relationships)
![Page 10: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/10.jpg)
Mental Models
Internal images about how the world works and why
Some are tacit and hard to see
Shape how we think and act
Determine policy options
![Page 11: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/11.jpg)
Fifth Discipline, Peter Senge
“Systems Thinking” and “Working with Mental Models” are two of Senge’s five disciplines
Five disciplines help organizations overcome learning disabilities
![Page 12: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/12.jpg)
Systems Thinking
Helps identify and assess mental models and hidden bias
Identifies structures and patterns underlying complex situations
Turn faucet
Filled Glass
Emptyglass Drink
![Page 13: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/13.jpg)
Systems ThinkingEnglish language favor linear thinking
Subject Verb Object
Learning styles and brain hemispheric preferences
Left brain tends to process information step-by-step
Right brain tends to see the whole picture at once (Ricki Linksman, How to Learn Anything Quickly (1996))
![Page 14: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/14.jpg)
Example: Cold War Arms Race
USSR Arms
US Threatened
US Needs Arms
US Arms
USSR Threatened
USSR NeedsArms
![Page 15: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/15.jpg)
Systems ThinkingSystems thinking sees circles of causes and effects
It connects separate chains of events
USSR Builds Arms
Threat To US
US NeedsArms
US Arms
Threatto
USSR
USSR NeedsArms
Arms Spiral
![Page 16: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/16.jpg)
Using Systems Thinking to Identify Mental Model: HOMEConnecticut
Ad Hoc Statewide Group
Sponsored by Partnership for Strong Communities (http://www.ctpartnershiphousing.com/)
Planners, economists, home builders, realtors, advocates, and other stakeholders
![Page 17: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/17.jpg)
2007 HOMEConnecticut Legislative Proposal
Diagnosed state’s affordable housing problem (i.e., cause and effect)
Proposed a program based on that diagnosis (i.e., intervention)
![Page 18: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/18.jpg)
Root Cause: New housing drives up school costs and property taxes…
HousingTypes
School Budgets
Property Taxes
Zoning
![Page 19: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/19.jpg)
Reaction: Towns increase lot sizes, and fewer homes are built
Increased lot sizes
Less need for new teachers
and classrooms
Stable school
budgets
![Page 20: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/20.jpg)
But the towns’ solution creates problems for the state
Large LotZoning
Increased HousingPrices
Young Workers & BusinessesLeave State
Economic Decline
![Page 21: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/21.jpg)
HOMEConnecticut removes need for fiscal zoning
HousingProduction
School Budgets
Zoning
StateIncentivePayments
![Page 22: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/22.jpg)
HOMEConnecticut Strategy
Zoning incentive payments
Building incentive payments
Project-based rent subsides
School cost reimbursements
Infrastructure improvement funds
![Page 23: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/23.jpg)
Detecting Bias—key questions
What is HOMEConnecticut’sperspective?
Does it jump from observation to generalization? (i.e., “leaps of abstractions”)
Does it leave anything out? (i.e., “exposing the left-hand column”)
![Page 24: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/24.jpg)
What’s the perspective?
The HOMEConnecticut campaign was “launched in 2005 to preserve the quality of life and strong, competitive economy that has distinguished Connecticut, but which is now threatened by a dramatic lack of housing affordable to workers, families, and young professionals.”
![Page 25: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/25.jpg)
Watch out for Leaps of Abstraction
Avoid processing information quickly
Identify and reflect on cause and effect statements
Do legislative environments allow reflection?
“Leaps of abstraction impede learning because they become axiomatic” (Senge).
![Page 26: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/26.jpg)
Leap of Abstraction?
Connection between (1) increasing housing costs and (2) 25- to 34-year-old cohort declining by 50,000 between 2000 and 2005
Large minimum lot sizes drive up housing costs
Towns set large minimum lot sizes to control education spending
![Page 27: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/27.jpg)
Exposing the left-hand column
Forces your “imagination” to see those “tracts” that don’t support your theory
Surface hidden assumptions and consider alternative theories
![Page 28: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/28.jpg)
Hidden Assumption: Purpose of Setting Lot Sizes
Statement: “Reacting to fiscal and other concerns, most communities have restricted most types of housing development with zoning that limits the number of housing units per acre.”
Assumption: Zoning commissions set lot sized to rein in education spending and discourage poorly designed projects
![Page 29: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/29.jpg)
Hidden Assumption: Housing Prices and Workers Leaving State
Statement: “Over the last five years, Connecticut housing prices have increased 63.6%, while wages have risen 18.5%.”
Statement: “Since 2000, Connecticut has lost a higher percentage of 25-34-year-old population than any state in the nation.”
Assumption: Housing prices are driving young workers out of the state.
![Page 30: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/30.jpg)
Hidden Assumption: Out migration stunting economic growth
Statement: “The shortage of younger workers hurts our business’ ability to hire employees right now, but this age bracket also represents our future economic growth.”
Assumptions:Young workers are “leaving” the state
They’re leaving because they can’t afford housing here
![Page 31: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/31.jpg)
PA 07-4, June Special Session Zoning adoption grants
$2,000 for each unit that can be built in a locally adopted, state-approve zone
Building permit grants
$2,000 grants for each multifamily, duplex, or townhouse unit
$5,000 grants for each single-family detached unit
![Page 32: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/32.jpg)
Mental Models and Hidden Bias
Robert S. McNamara’s 8th
Lesson: “Be Prepared to Reexamine Your Reasoning”
![Page 33: Detecting Bias in Research Papers - NCSL](https://reader031.fdocuments.us/reader031/viewer/2022020912/62033db4a4349246ab53e0ca/html5/thumbnails/33.jpg)
Detecting Bias in Research Papers
John RappaOffice of Legislative ResearchConnecticut General [email protected]