Post on 17-Dec-2015
Magia G. KrausePh.D. Candidate
School of InformationUniversity of Michigan
Society of American Archivists Annual MeetingAugust 14, 2009
Undergraduates in the Archives:
Measuring the Impact of Archival Instruction Through Rubric-based Assessment
Instruction and Assessment in Academic Archives
Krause, Magia G.(2008). Learning in the archives: A report on instructional practices, Journal of Archival Organization, 6(4) 233-268.
Overview of Teaching ExperimentResearch Questions
Do undergraduate students learn from archival instruction?
What do undergraduate students learn from archival instruction?
Goals of studyIntroduction of a reliable assessment tool to
measure student learning Introduction of archival literacy scale that
might compliment a curriculum of undergraduate research education in archives
MethodologyQuasi-experiment in the fieldBackground
US history course in a large state university82 students total
Control and treatment groupsTo compare students that did not receive
archival instruction with students that didPre- and Post- Document Analysis Exercise
To provide a baseline comparison of the groups and look for an improvement in treatment group
Archival InstructionDescription of treatment (aka archival
instruction)Lecture/presentation to entire class by
archivist Hour-long session in the archives with 4
stationsBibliographic InstructionCritical ThinkingPhotograph AnalysisFootnote/Citation Analysis
Photographs from the Instruction Session
Document Analysis ExerciseDivided into 3 sections
Adapted from NARA’s Teaching with DocumentsWritten Document AnalysisPhotograph AnalysisFinding Aid Analysis
Difference between Pre- and Post- TestsAlternative primary sources and finding aid
Assessing Student LearningRubric
Basic definitionScoring guide that includes criteria and levels of
performanceTypes of rubrics
Holistic and analyticSteps in creating a rubric
Criteria : Archival Literacy SkillsObservationInterpretation/Historical ContextEvaluation/Critical ThinkingResearch Skills
Rubric for this study
Using the Rubric3 raters
Two professional archivists with at least five years of experience teaching undergraduates and me
Important for increasing the reliability of the scores
Achieving kappa
Final resultsOur Fleiss Kappa for the pretest was .80218 and .788 for
the post-test, both of which are considered excellent strengths of agreement.
Statistical TestsNon-parametric tests
Useful for analyzing ordinal and nominal dataGreat for small samplesTests used:
Mann Whitney U Test for Independent SamplesResults
The tests confirmed that the null hypothesis could be rejected with 95% confidence
Students in the treatment group did better on the post-test after participating in archival instruction. In fact, there is a statistically significant difference between the two groups’ scores for each of the criteria.
DiscussionThe Research Skills Problem
Disappointing resultsPossible factors
Students not actively engaged during instructionFollow-up assignment involved pre-selected
documents – no chance to practice searching for and identifying documents
Literature on undergraduate information-seeking behavior
Archival Literacy ScaleElements
4 categories of the rubric:ObservationInterpretationEvaluationResearch Skills
Testing for reliabilityResults
ConclusionsLimitations to the generalizability of results
– Need to replicate the study to increase the
reliability of the instrumentsWish for better instructional design for
instruction componentNeed for more communication and
collaboration between archivists to develop and share curricula and standardized tools to assist in instruction and assessment
Thank YouFor more information about the study
described in this presentation, please contact: Magia Krause mghetu@umich.edu.