Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... ·...
Transcript of Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... ·...
![Page 1: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/1.jpg)
Critical Assessment Practices
A Discussion on When and How to Use Student Learning Data Without Doing Harm
Zoe Fisher, Instructional Designer, Pierce College
Ebony Magnus, Library Manager, Southern Alberta Institute of Technology
Nicole Branch, Associate University Librarian for Learning & Engagement, Santa Clara University
#ACRLInstruction
![Page 2: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/2.jpg)
Today's Outcomes
Attendees will...
● Consider critical approaches to library assessment practices● Examine trends and implications of libraries using student data through various
modalities● Explore practical approaches and methodologies for implementing critical
assessment of student learning
![Page 3: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/3.jpg)
Our Thoughts
Zoe will talk about the current context for library assessment work, emerging trends, and the distinctions between student learning assessment and student surveillance.
Ebony will use a critical assessment frame to consider questions of motivation, accountability, and agency in relation to collection of student data.
Nicole will share her experiences using transformative mixed methods to model developing student learning outcomes for the ACRL Framework for Information Literacy.
![Page 4: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/4.jpg)
Zoe FisherInstructional Designer, Pierce College
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 5: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/5.jpg)
What kinds of student learning data do we collect in the library, and why?
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 6: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/6.jpg)
We collect data in order to...
● Provide services (e.g., collecting identifying information in order to establish borrowing privileges).
● Measure existing library services/spaces/collections or anticipate needs for new services/spaces/collections.
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 7: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/7.jpg)
We also collect data in order to...
● Be accountable and meet reporting requirements for internal and external stakeholders (e.g., accreditation).
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 8: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/8.jpg)
What are some trends in library use of student learning data?
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 9: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/9.jpg)
2010 VAL Report
2013-2016Assessment in Action cohorts
2017Academic Library Impact
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 10: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/10.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 11: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/11.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 12: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/12.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 13: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/13.jpg)
From the AiA Executive Summary (published April 2017)
Library use increases student success. Several AiA studies point to increased academic success when students use the library. The analysis of multiple data points (e.g., circulation, library instruction session attendance, online database access, study room use, interlibrary loan) shows that students who used the library in some way achieved higher levels of academic success (e.g., GPA, course grades, retention) than students who did not use the library.
![Page 14: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/14.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 15: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/15.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
Suggested Actions (from Academic Library Impact, p.5)● Know how other academic stakeholders are using learning analytics.● Research the safeguards needed to ensure student privacy or confidentiality.● Strategically collect data that can be integrated into learning analytics software.● Advocate for the inclusion of library data in the volumes of information collected from multiple systems
within the academic institution.● Integrate library data into campus analytics components.● Work with stakeholders to statistically analyze and predict student learning and success based on
shared analytics.
![Page 16: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/16.jpg)
Zoe Fisher@zoh_zoh #ACRLInstruction
Suggested Actions (from Academic Library Impact, p.5)● Know how other academic stakeholders are using learning analytics.● Research the safeguards needed to ensure student privacy or confidentiality.● Strategically collect data that can be integrated into learning analytics software.● Advocate for the inclusion of library data in the volumes of information collected from multiple
systems within the academic institution.● Integrate library data into campus analytics components.● Work with stakeholders to statistically analyze and predict student learning and success based on
shared analytics.
![Page 17: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/17.jpg)
Library Assessment Conference 2018Assessing Student Learning in Library Instruction: A Faculty Perspective Doreen Bradley and Jo Angela Oehrli (University of Michigan)
Information Literacy Assessment for Instruction Improvement and Demonstration of Library Value: Comparing Locally-Grown and Commercially-Created TestsKathy Clarke (James Madison University) and Carolyn Radcliff (Carrick Enterprises)
Developing Library Learning Outcomes: Reflecting on Instruction across the LibraryAshley McMullin, Jennifer Schwarz, and Janice Scurio (DePaul University)
What Could We Do, If Only We Knew? Libraries, Learning Analytics, & Student SuccessMegan Oakleaf (Syracuse University), Malcolm Brown (Educause), Scott Walter (DePaul University), Dean Hendrix (University of Texas at San Antonio), and Joe Lucia (Temple University)
Library Participation in Learning Analytics Initiatives: Library and Student PerspectivesMichael Perry (Northwestern University); M. Brooke Robertshaw (Oregon State University); Andrew Asher and Kyle Jones (Indiana University); Kristin Briney (University of Wisconsin-Milwaukee); Abigail Goben (University of Illinois at Chicago) and Dorothea Salo (University of Wisconsin at Madison)
![Page 18: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/18.jpg)
Library Assessment Conference 2018Assessing Student Learning in Library Instruction: A Faculty Perspective Doreen Bradley and Jo Angela Oehrli (University of Michigan)
Information Literacy Assessment for Instruction Improvement and Demonstration of Library Value: Comparing Locally-Grown and Commercially-Created TestsKathy Clarke (James Madison University) and Carolyn Radcliff (Carrick Enterprises)
Developing Library Learning Outcomes: Reflecting on Instruction across the LibraryAshley McMullin, Jennifer Schwarz, and Janice Scurio (DePaul University)
What Could We Do, If Only We Knew? Libraries, Learning Analytics, & Student SuccessMegan Oakleaf (Syracuse University), Malcolm Brown (Educause), Scott Walter (DePaul University), Dean Hendrix (University of Texas at San Antonio), and Joe Lucia (Temple University)
Library Participation in Learning Analytics Initiatives: Library and Student PerspectivesMichael Perry (Northwestern University); M. Brooke Robertshaw (Oregon State University); Andrew Asher and Kyle Jones (Indiana University); Kristin Briney (University of Wisconsin-Milwaukee); Abigail Goben (University of Illinois at Chicago) and Dorothea Salo (University of Wisconsin at Madison)
![Page 19: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/19.jpg)
Are we interested in measuring what students do, or what they learn?
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 20: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/20.jpg)
Surveillance of student behavior in library assessment looks like...● EZ proxy logins● ID card swipes● Article downloads● Circulation records● Tracking the amount of time students spend in different spaces (physical and
online)
... And then connecting these individual actions with individual student performance and achievement.
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 21: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/21.jpg)
Surveillance of student behavior in library assessment looks like...● Correlating library use at 13 different access points with retention and GPA
(Soria, Fransen, and Nackerud, 2013). Correlating EZ proxy logins and circulation of physical materials with retention (Murray, Ireland, and Hackathorn, 2016).
● Correlating the use of online library resources with GPA (Eastern Kentucky University from Assessment in Action, 2016).
● Correlating attendance at information literacy one-shot sessions with final course grades and retention (Northwest Arkansas Community College from Assessment in Action, 2016).
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 22: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/22.jpg)
In contrast, student learning assessment of information literacy looks like...● Writing and measuring student learning outcomes. ● Collecting and reviewing samples of student work.● Reflective discussions among teachers about their pedagogy. ● Revising assignments to improve student learning outcomes.● Revising student learning outcomes to improve student learning.
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 23: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/23.jpg)
In contrast, student learning assessment looks like...
● Writing and measuring student learning outcomes. ● Collecting and reviewing samples of student work.● Reflective discussions among teachers about their pedagogy. ● Revising assignments to improve student learning outcomes.● Revising student learning outcomes to improve student learning.
This kind of work might not get published or fed into a dashboard, but it matters, and it has a huge impact on your students.
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 24: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/24.jpg)
I would like to see academic library assessment that is...● Qualitative.● Ethnographic.● Longitudinal.● Rooted in conversations around pedagogy, power, and barriers/obstacles to
learning faced by students.● Respectful of students and their experiences, honoring their voices and
expertise.
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 25: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/25.jpg)
Recommend Readings Worth Your Time● "A day in the life: Practical strategies for understanding student space-use
practices" (Asher et al, 2016)● "How do students get help with research assignments? Using drawings to
understand students' help seeking behavior" (Beisler & Medaille, 2016) ● "A phenomenographic study of English faculty's conceptions of information
literacy" (Boon, Johnston, and Webber, 2007)● "The phenomenon of information literacy" (Bruce, 1998)● "Learning outcomes, portfolios, and rubrics, oh my! Authentic assessment of an
information literacy program" (Diller & Phelps, 2008)● Academic Libraries for Commuter Students (Eds. Regalado & Smale, 2018) ● ERIAL Project (www.erialproject.org)
Zoe Fisher@zoh_zoh #ACRLInstruction
![Page 26: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/26.jpg)
Ebony MagnusLibrary Manager, Southern Alberta Institute of Technology
Ebony Magnus | @ebonymagnus #ACRLInstruction
![Page 27: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/27.jpg)
“Assessment is a social and political act, regardless of whether or not an agenda is made explicit.”
Ebony Magnus | @ebonymagnus #ACRLInstruction
DeLuca Fernandez, 2015, p. 5
![Page 28: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/28.jpg)
How do institutional assumptions and agendas shape the student data we collect?
Where does our accountability lie in the creation and use of student data?
Are we engaging students in the creation, analysis, and communication of data and insight?
Ebony Magnus | @ebonymagnus #ACRLInstruction
![Page 29: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/29.jpg)
"Are we critically analyzing and questioning the rationale of our own governments’ or institutions’ demands for assessment, and the implications on our own libraries, staff, and users?"
Ebony Magnus | @ebonymagnus #ACRLInstruction
Doucette, 2017, p. 293
![Page 30: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/30.jpg)
“Rarely are the people who participate in the research process as participants or “informants” considered to be the primary audience when it comes time to disseminate the research.”
Ebony Magnus | @ebonymagnus #ACRLInstruction
Gaudry, 2011, p. 113
![Page 31: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/31.jpg)
“This decision [to share or not share their data] belongs to the user and not to the library and the user being able to exercise full agency in this decision means fully informed consent with regard to what is collected and how it will be used and managed.”
Ebony Magnus | @ebonymagnus #ACRLInstruction
Hinchliffe, 2018
![Page 32: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/32.jpg)
“By raising consciousness of the ethical and value-based decisions implicit in any assessment context, the practice of assessment truly becomes a complex social practice rather than a collection of technical data gathering approaches that might unwittingly serve power interests unintended by well-meaning individuals.”
Ebony Magnus | @ebonymagnus #ACRLInstruction
Wall, Hursh, & Rodgers, 2014, p. 12
![Page 33: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/33.jpg)
Nicole BranchAssociate University Librarian for Learning & Engagement
Santa Clara University
Nicole Branch@benicolebe
![Page 34: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/34.jpg)
Applying Critical Assessment Practices
1Adopt a
theoretic
al stance
Apply cri
tical a
pproaches t
o the
research
/ass
essment p
rocess
2Rese
arch/ass
ess cr
itical is
sues
Focus o
n issu
es of p
ower, socia
l justi
ce,
critic
al consc
iousness
, and th
e common
good3
Align m
ethodologies
Select m
ethods that a
lign w
ith th
eoretical
stance 4
Apply fin
dings
Use fi
ndings to address
issu
es of p
ower,
socia
l justi
ce, a
nd the co
mmon good
Nicole Branch, @benicolebe
![Page 35: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/35.jpg)
Applying Critical Assessment Practices
1Adopt a
theoretic
al stance
Apply cri
tical a
pproaches t
o the
research
/ass
essment p
rocess
2Rese
arch/ass
ess cr
itical is
sues
Focus o
n issu
es of p
ower, socia
l justi
ce,
critic
al consc
iousness
, and th
e common
good3
Align m
ethodologies
Select m
ethods that a
lign w
ith th
eoretical
stance 4
Apply fin
dings
Use fi
ndings to address
issu
es of p
ower,
socia
l justi
ce, a
nd the co
mmon good
Nicole Branch, @benicolebe
![Page 36: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/36.jpg)
Transformative Research Paradigms
“recommend the adoption of an explicit goal for research to serve the ends of creating a more just and democratic society that permeates the entire research process” (2003, p. 159).
- Donna Mertens
Nicole Branch, @benicolebe
![Page 37: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/37.jpg)
Elements of Transformative Paradigms(a) Do the authors openly reference a problem in a community of concern?
(b) Do the authors openly declare a theoretical lens?
(c) Were the research questions (or purposes) written with an advocacy stance?
(d) Did the literature review include discussions of diversity and oppression?
(e) Did the authors discuss appropriate labeling of the participants?
(f) Did data collection and outcomes benefit the community?
(g) Did the participants initiate the research, and/or were they actively engaged in the project?
(h) Did the results elucidate power relationships?
(i) Did the results facilitate social change?
(j) Did the authors explicitly state their use of a transformative framework?
Sweetman, Badiee, & Creswell (2010)
![Page 38: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/38.jpg)
Elements of Transformative Paradigms(a) Do the authors openly reference a problem in a community of concern?
(b) Do the authors openly declare a theoretical lens?
(c) Were the research questions (or purposes) written with an advocacy stance?
(d) Did the literature review include discussions of diversity and oppression?
(e) Did the authors discuss appropriate labeling of the participants?
(f) Did data collection and outcomes benefit the community?
(g) Did the participants initiate the research, and/or were they actively engaged in the project?
(h) Did the results elucidate power relationships?
(i) Did the results facilitate social change?
(j) Did the authors explicitly state their use of a transformative framework?
![Page 39: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/39.jpg)
Applying Critical Assessment Practices
1Adopt a
theoretic
al stance
Apply cri
tical a
pproaches t
o the
research
/ass
essment p
rocess
2Rese
arch/ass
ess cr
itical is
sues
Focus o
n issu
es of p
ower, socia
l justi
ce,
critic
al consc
iousness
, and th
e common
good3
Align m
ethodologies
Select m
ethods that a
lign w
ith th
eoretical
stance 4
Apply fin
dings
Use fi
ndings to address
issu
es of p
ower,
socia
l justi
ce, a
nd the co
mmon good
Nicole Branch, @benicolebe
![Page 40: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/40.jpg)
Research InterestProblem/concern
● Local learning outcome development● Assessability● Critical practice● Outcome dissatisfaction● Learning outcomes as the foundation for assessment
Nicole Branch, @benicolebe
![Page 41: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/41.jpg)
Research InterestProblem/concern
● Local learning outcome development● Assessability● Critical practice● Outcome dissatisfaction● Learning outcomes as the foundation for assessment
How might we create local learning outcomes that are meaningful, important, and articulate social justice and critical thinking elements of the Framework?
Nicole Branch, @benicolebe
![Page 42: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/42.jpg)
Align Methodologies
1Adopt a
theoretic
al stance
Apply cri
tical a
pproaches t
o the
research
/ass
essment p
rocess
2Rese
arch/ass
ess cr
itical is
sues
Focus o
n issu
es of p
ower, socia
l justi
ce,
critic
al consc
iousness
, and th
e common
good3
Align m
ethodologies
Select m
ethods that a
lign w
ith th
eoretical
stance 4
Apply fin
dings
Use fi
ndings to address
issu
es of p
ower,
socia
l justi
ce, a
nd the co
mmon good
Nicole Branch, @benicolebe
![Page 43: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/43.jpg)
Concept Mapping
Nicole Branch, @benicolebe
![Page 44: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/44.jpg)
Concept Mapping
Kane, L. W., & McSweeney, Advancing Information Literacy in a Semester-Long Library Instruction Course: A Case Study, The Journal of Interactive Technology and Pedagogy, June 11, 2018.
![Page 45: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/45.jpg)
Concept Mapping
Kane, L. W., & McSweeney, Advancing Information Literacy in a Semester-Long Library Instruction Course: A Case Study, The Journal of Interactive Technology and Pedagogy, June 11, 2018.
![Page 46: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/46.jpg)
Concept Mapping Methodology● Developed by William Trochim● Integrated mixed methods● Collaborative engagement of stakeholder knowledge● Core elements include
○ Brainstorming○ Sorting○ Rating○ Mapping
Nicole Branch, @benicolebe
![Page 47: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/47.jpg)
Concept Mapping & Transformative Paradigm● Focus on social justice/issues of power● Engagement of participants as research partners● Could address an issue in a community● May be used to affect change/benefit community
Nicole Branch, @benicolebe
![Page 48: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/48.jpg)
Study● 11 Librarians from across the country● Subject areas (humanities, STEM, business, assessment)● Variety of institution types● Concept “Information has value”● Conducted entirely virtually
Nicole Branch, @benicolebe
![Page 49: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/49.jpg)
Concept Mapping Procedures
Define the Concept
“Information has value” as defined in the Framework for Information Literacy
Step 1
Brainstorm Focus Group
90 minutes, via Zoom
Generated 100 learning outcome statements
Step 2
Statement Synthesis
Removed redundancies
Adjusted language
Optional feedback from group
75 final statements
Step 3
Sorting & Rating
Individually sorted and rated
usabiliTEST card sorting software
Qualtrics survey software
Completed in one week
Step 4
Map Generation
Applied hierarchical cluster analysis to multi-dimensional scale
Named clusters and calculated the average cluster rating
Step 5
Final Reflection
Process reflection and feedback
Group analysis/reflection on map
Step 6
![Page 50: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/50.jpg)
Cluster Sample Learning Outcome
1. Value of Information Communities recognize the value of human networks in accessing information
2. Scholarly Conventions of Value critique traditional systems of assigning value to information in specific disciplines
3. Access & Navigation demonstrate facility in navigating systems of information to meet information needs
4. Formal Recognition of Value use citation to acknowledge the work of others, including those who have been traditionally marginalized
5. Access(ibility) is Power critique the concept of information neutrality
6. Information Privilege appraise their own and others' information privilege and marginalization
7. Information is Power(ful) recognize that the distribution of information can be politically motivated
8. Information Agency & Responsibility
navigate the ambiguity, complexity, and discomfort of finding and using information to arrive at greater understanding
9. Systems of Value recognize that the value of information can be context-specific
![Page 51: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/51.jpg)
Cluster Sample Learning Outcome
1. Value of Information Communities recognize the value of human networks in accessing information
2. Scholarly Conventions of Value critique traditional systems of assigning value to information in specific disciplines
3. Access & Navigation demonstrate facility in navigating systems of information to meet information needs
4. Formal Recognition of Value use citation to acknowledge the work of others, including those who have been traditionally marginalized
5. Access(ibility) is Power critique the concept of information neutrality
6. Information Privilege appraise their own and others' information privilege and marginalization
7. Information is Power(ful) recognize that the distribution of information can be politically motivated
8. Information Agency & Responsibility
navigate the ambiguity, complexity, and discomfort of finding and using information to arrive at greater understanding
9. Systems of Value recognize that the value of information can be context-specific
![Page 52: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/52.jpg)
Concept Map
![Page 53: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/53.jpg)
Apply Findings
1Adopt a
theoretic
al stance
Apply cri
tical a
pproaches t
o the
research
/ass
essment p
rocess
2Rese
arch/ass
ess cr
itical is
sues
Focus o
n issu
es of p
ower, socia
l justi
ce,
critic
al consc
iousness
, and th
e common
good3
Align m
ethodologies
Select m
ethods that a
lign w
ith th
eoretical
stance 4
Apply fin
dings
Use fi
ndings to address
issu
es of p
ower,
socia
l justi
ce, a
nd the co
mmon good
Nicole Branch, @benicolebe
![Page 54: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/54.jpg)
Applying FindingsAssessing student work
● Coding student work● Document-based interviews● Other approaches?
Illuminating Social Justice in the Framework, Communications in Information Literacy, Summer 2019
![Page 55: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/55.jpg)
Q&A/Discussion
More questions than we can answer today? Feel free to contact us.
Zoe Fisher - ztrope at gmail or www.quickaskzoe.com
Ebony Magnus - ebony[dot]magnus at sait[dot]ca or [at]ebonymagnus on twitter
Nicole Branch - nbranch at scu[dot]edu
![Page 56: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/56.jpg)
ReferencesDeLuca Fernández, S. (2015) Critical assessment. Webinar delivered for Student Affairs Assessment Leaders (SAAL) Structured
Conversations series. December 9, 2015. http://studentaffairsassessment.org/files/documents/SAAL-SC-Critical-Assessment-sdf-9-dec-2015-FINAL.pdf.
Doucette, L. (2017). Acknowledging the Political, Economic, and Values-Based Motivators of Assessment Work: An Analysis of Publications on Academic Library Assessment. In Baughman, S., Hiller, S., Monroe, K. & Pappalardo, A., Eds. Proceedings of the 2016 Library Assessment Conference. Washington, D.C.: Association of Research Libraries. http://old.libraryassessment.org/bm~doc/proceedings-2016.pdf.
Ekowo, M. & Palmer, I. (2016). The promise and peril of predictive analytics in higher education: A landscape analysis. New America. https://www.newamerica.org/education-policy/policy-papers/promise-and-peril-predictive-analytics-higher-education/
Gaudry, A.J.P. (2011). Insurgent research. Wicazo Sa Review, 26(1), 113-136. https://www.jstor.org/stable/10.5749/wicazosareview.26.1.0113
Hinchliffe, L.J. (2018). Privacy in user research: Can you? The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2018/09/05/privacy-in-user-research-can-you/
Jones, K.M.L & Salo, D. (2018). Learning analytics and the academic library: Professional ethics commitments at a crossroads, College & Research Libraries, 79(3), 304-323. https://doi.org/10.5860/crl.79.3.304
Magnus, E., Faber, M., Belanger, J. (2018). Towards a critical assessment practice. In the Library with the Lead Pipe. http://www.inthelibrarywiththeleadpipe.org/2018/towards-critical-assessment-practice/
Mertens, D. M. (2003). Mixed methods and the politics of human research: The transformative-emancipatory perspective. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 135-164). Thousand Oaks: Sage Publications.
![Page 57: Critical Assessment Practicesacrl.ala.org/IS/wp-content/uploads/Critical-Assessment... · 2019-03-18 · Critical Assessment Practices A Discussion on When and How to Use Student](https://reader033.fdocuments.us/reader033/viewer/2022041913/5e683ca0070ef17aa55e02a6/html5/thumbnails/57.jpg)
References, cont.Sutherland, S., & Katz, S. (2005). Concept mapping methodology: A catalyst for organizational learning. Evaluation and Program
Planning, 28(3), 257-269. https://doi.org/10.1016/j.evalprogplan.2005.04.017Sweetman, D., Badiee, M., & Creswell, J. W. (2010). Use of the transformative framework in mixed methods studies. Qualitative
Inquiry, 16(6), 441-454. https://doi.org/10.1177/1077800410364610Trochim, W. M. K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1),
1-16. https://doi.org/10.1016/0149-7189(89)90016-5Wall, A. F., Hursh, D., & Rodgers, J. W., III. (2014). Assessment for whom: Repositioning higher education assessment as an ethical
and value-focused social practice. Research & Practice in Assessment, 9(Summer), 5-17. https://www.rpajournal.com/assessment-for-whom-repositioning-higher-education-assessment-as-an-ethical-and-value-focused-social-practice/