Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows...
-
Upload
jonas-cramton -
Category
Documents
-
view
215 -
download
0
Transcript of Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows...
![Page 1: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/1.jpg)
Concrete Approaches to Quality Assessment:
Moving Beyond Peer ReviewAugust 6, 2003
Howard BurrowsAutonomous Systems Institute
Lee, NH
![Page 2: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/2.jpg)
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
![Page 3: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/3.jpg)
Hardcore Underpinnings
There is a dialectic between ontology and epistemology.
John Burke Grammar of Motives
![Page 4: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/4.jpg)
Ontology, Epistemology, Dialectic
• Ontology – what is taken to exist (what needs a name)
• Epistemology – what do you know about it
• Dialectic – thesis, antithesis, synthesis
![Page 5: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/5.jpg)
Dialectic betweenOntology and Epistemology
• What counts as data when you form your beliefs
• The relation between evidence and justice.
• Are your beliefs “sensible” and “reasonable”
• The relation between metrics and evaluation
![Page 6: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/6.jpg)
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
![Page 7: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/7.jpg)
Evaluating Peer Review
• Selecting reviewers; providing incentives
• Informed objectivity; conflict of interest
• Review criteria; comparing across panels
Ronald N. KostoffUS Office of Naval Research
![Page 8: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/8.jpg)
Selecting Reviewers: Peer vs Non-Peer
Balancing interests:
• Economic- government, academia, industry, society
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
![Page 9: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/9.jpg)
Informed objectivity
• “Good old boy” consensus
• Natural bias often subtle, needs diversity
• With paradigm shifts, experience becomes a liability
![Page 10: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/10.jpg)
US National Science FoundationReview Criteria
• Intellectual merit – creative and original concepts
• Broader impacts
– benefits to society
![Page 11: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/11.jpg)
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
![Page 12: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/12.jpg)
The Age of Unreason
“Changes are not what they used to be.”from book “The Age of Unreason”
Charles Handy, 1989
a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution
![Page 13: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/13.jpg)
NASA’s Earth Science Information Partners
Courtesy Don Collins of the DAAC Alliance
![Page 14: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/14.jpg)
Federalism
• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)
• Heterogeneous, diverse communities Data centers, academics, government, and industry
• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.
![Page 15: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/15.jpg)
Outline
1. Hardcore philosophical underpinnings
2. Evaluating peer review
3. Alternatives to peer review
4. Justifying decisions about quality
![Page 16: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/16.jpg)
Justifying Decisions about Quality
• Concept spaces and mapping
• Fact-value continuum?
• Fate of Knowledge (social construction)
![Page 17: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/17.jpg)
Concept spaces and mapping
![Page 18: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/18.jpg)
Fact-value continuum?
• Can we distinguish fact claims from value judgments?
• Are there really objective as opposed to subjective distinctions?
• Amartya Sen introduces value judgments in economics
![Page 19: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/19.jpg)
Fate of knowledge
• Knowledge is social
• Cognitive processes are social (reasonable)
• Actions based on knowledge are justified through social processes.
![Page 20: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/20.jpg)
Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
![Page 21: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/21.jpg)
Promising developments
• Semantic web not words; rather “meaningful” data, concepts, and ideas.
• Science draws meaning from data; and has changed the way it justifies this.
• The semantic web offers to improve or supplant “peer” review (and education).
• The semantic web provides a “marketplace” for learning.
![Page 22: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/22.jpg)
Profiles for discussion
• People
• Content
• Technology
• Values and priorities
• Economic Environment
![Page 23: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/23.jpg)
Evolution to a Semantic Web
• HTML – Generic display of hypertext
• XML – Generic display of data
• RDF – Triples (subject verb object)
• OWL – Taxonomy, inference rules, and proofs
![Page 24: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/24.jpg)
Changes in Science
• 1930 –Reason and Sense (logical positivism)
• 1960 – Beyond Reason (linguistic turn)
• 1990 – Social Construction (peer review)
• 2020 – Back to Reason and Sense
![Page 25: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/25.jpg)
“Peers” and “Educators”
• Ontology – Locally coherent naming structures
• Realism – “Reifying” the structures
• Under standing – Logical foundations
• Knowledge – Justifying decisions (personal beliefs)
![Page 26: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/26.jpg)
Business Plan and “Marketplace”
• Sales agents – Know-bots negotiate deals
• Value chains – Multiple entities in assembly line
• Public choice – Market forces and “fair” voting rules
• Learning Economy – From food chain to ecosystem
![Page 27: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/27.jpg)
Back to the Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
![Page 28: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/28.jpg)
Problems in Public Funding
National Archives -
GPRA -
Curriculum -
2% GNP -
What data is valuable?
Is science making progress?
What is “educated”?
How much to spend?
![Page 29: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/29.jpg)
The Age of Unreason
“Changes are not what they used to be.”from book “The Age of Unreason”
Charles Handy, 1989
a. Change by discontinuous leapsb. Learning from the past dangerousc. Evolution yes, but allow for revolution
![Page 30: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/30.jpg)
Category I Earth Science Information Partners
Courtesy Don Collins of the DAAC Alliance
![Page 31: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/31.jpg)
MPEG-7: Metadata for Content Description
DataSignal
Features Model Semanticsstructure
Objects
Events
Actions
People
Labels
Relationship
Clusters
Classes
Collections
Probabilities
Confidences
Color
Texture
Shape
Motion
Camera motion
Regions
Segments
Mosaics
Relationship(Spatio-
temporal)
Images
Video
Audio
Multimedia
Formats
Layout
SemanticModelsFeaturesStructureData
Courtesy John Smith, IBM
![Page 32: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/32.jpg)
Data, Community, Public Use
Farmers USDA
$$$ on pest control
Yes,treat your crops today
Johns Hopkins
$$ onfire antsstudies
NCDC
GHRC
USGS
$ onreal timedata
IBM
UsersApplication solutionprovider
Science dataprovider
Backbone datacenter
Technologyprovider
Real-timeriskmap
Historicaldata
Software
Real-timedata
Real-timedata
Courtesy Yuan-Chi Chang, IBM
![Page 33: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/33.jpg)
Other Cross Discipline ESIP’s
Science
• Oceanography (2)
• Terrestrial Studies (4)
• Climate (3)
• Technology (3)
Public Use
• Education (3)
• Regional Policy (4)
• Public Health
• Media
• Legal
![Page 34: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/34.jpg)
![Page 35: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/35.jpg)
Community I
• “A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.”
E. O. WilsonConsilience: The Unity of Knowledge
![Page 36: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/36.jpg)
Community II
• “It is the disorder of the scientific community—the laminated, finite, partially independent strata supporting one another; it is the disunification of science—the intercalation of different patterns of argument—that is responsible for its strength and coherence..”
Peter GalisonImage and Logic, 1997
![Page 37: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/37.jpg)
Federalism
• Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly)
• Heterogeneous, diverse communities Data centers, academics, government, and industry
• Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.
![Page 38: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/38.jpg)
Federations Preserve Heterogeneity
• Economic- government, academia, industry
• Intellectual- academic disciplines, policy agencies
• Practical- problem based, market driven
![Page 39: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/39.jpg)
Economic Model
• Public funding caps out at less than 3% of the GNP
• Dynamic “pricing” is needed to demonstrate value to end-users
• “Market forces” or other methods of public choice provide robust mechanism to rank options
![Page 40: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/40.jpg)
Trading Zones
• Economic- government, academia, industry
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
![Page 41: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/41.jpg)
Peer Review
Nurturing cooperation vs competition
• Economic- government, academia, industry
• Intellectual- engineers, scientists, policy agencies
• Practical- curiosity, problem based, market driven
![Page 42: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/42.jpg)
Back to the Vision
• Concept-based science communication
• Personalized Learning Environment
• Economic Environment supports learning
![Page 43: Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649caf5503460f94973d3c/html5/thumbnails/43.jpg)
Business Plan and “Marketplace”
• Sales agents – Know-bots negotiate deals
• Value chains – Multiple entities in assembly line
• Public choice – Market forces and “fair” voting rules
• Learning Economy – From food chain to ecosystem