Jill Singer, Buffalo StateBridget Zimmerman, Buffalo Stateand acknowledging the support and advice provided by Dan Weiler, Daniel Weiler Associates

LEVERAGING UNCERTAINTY: TOWARD A NEW GENERATION OF UNDERGRADUATE RESEARCH

CUR Conference 2012 atThe College of New Jersey

THE IMPACT OF UNDERGRADUATE RESEARCH: BUFFALO STATE’S EVALUATION PROGRAM

Outline

• Motivation for evaluation program and process we followed• Features and sequence of evaluation• Findings• Instruments and other resources• Possible next steps• Questions and comments

Motivation: Why Evaluate the Undergraduate Summer Research Program (USRF)• Campus requirement to implement

assessment program and measure outcomes• Summer research program is largest single

program administered by Buffalo State’s Office of Undergraduate Research

• Made sense to develop an assessment process that would apply both to science and non-science disciplines

• Multi-year effort aimed at development and field-testing a methodology for measuring student learning and related outcomes

• Currently in 5th year of full-scale implementation of evaluation

Background About the Summer Research Program

• Program, now in its 14th year, supports research, scholarly and creative activities in all academic disciplines

• Each award includes a student stipend of $2750, faculty stipend of $1000, and $500 for project-related travel and supplies

• Program length is minimum 8 weeks• Program enjoys high level of prestige and participants are

very supportive of our ongoing efforts to evaluate the impact of undergraduate researchProfessor Robert Wood and Jonathan Matecki,

Ceramics (2004), Rachel Wright, Painting (2010), Dan Naschke, Geology (2010), and Amelia Alessi, Biology (2008)

Purposes of the Evaluation

• Obtain reliable assessment of the program’s impact on participating students that goes beyond student survey data

• Obtain information from faculty mentors on how the program has influenced their teaching and view of student capabilities

• Provide information to participating students that helps them assess their academic strengths and weaknesses

• Creation of a longitudinal database that can provide impact data on a range of outcomes for different types of students and a variety of academic disciplines

Process: Retreat Pilot Full Scale

• A two day retreat in summer 2006 attended by faculty representing a range of disciplines (arts, humanities, social sciences, and physical sciences) was held to reach consensus on student learning outcomes to be measured•First day of the retreat devoted to identifying and confirming 11 outcome categories•Second day of retreat devoted to refining outcome categories and identifying outcome components•The retreat resulted in 11 outcome categories to be measured and a detailed rubric that describes the specific components for each outcome

Outcome Categories

CommunicationCreativityAutonomyAbility to deal with obstaclesPractice and process of inquiryNature of disciplinary knowledgeCritical thinking and problem solvingUnderstanding ethical conductIntellectual developmentCulture of scholarshipContent knowledge skills/methodology

…. and an option for the mentor and their student to identify and add outcomes for their particular project

Outcome Categories and Components

• Each outcome category includes several specific outcome components:

• For “creativity” components: brings new insights to the problem at hand; shows ability to approach problems from different perspectives; combines information in new ways and/or demonstrates intellectual resourcefulness; and effectively connects multiple ideas/approaches

• For “critical thinking and problem solving” components: trouble-shoots problems, searches for ways to do things more effectively and generates, evaluates and selects between alternatives; recognizes discipline-specific problems and challenges established thinking when appropriate; recognizes flaws, assumptions and missing elements in arguments

• For “ability to deal with obstacles” components: learns from and is not discouraged by set-backs and unforeseen events; shows flexibility and a willingness to take risks and try again

• This pattern is followed for all 11 outcome categories

Process: Retreat Pilot Full Scale

• Six student-mentor pairs participated in pilot implementation of the evaluation effort during summer 2007

• Focus groups composed of student researchers and faculty mentors led by Dan Weiler with Bridget Zimmerman taking notes

• Based on findings, minor modifications were made to the instruments and clarification of overall evaluation process

Process: Retreat Pilot Full Scale

• Full scale implementation of the evaluation program in 2008

• Evaluation data (qualitative and quantitative) compiled and analyzed by Bridget Zimmerman• Qualitative parts of the evaluation took advantage of the already existing mid-summer and final report forms and the quantitative portions used the newly developed assessment instruments

Sequence of Activities: Pre-to Early Research Mid-Research Late-to Post-Research

Features of the Evaluation

• Repeated assessments (pre-, mid- and end-of-research)

• Assessments in which faculty mentors and students both use the same outcome categories and components

• Scoring rubric used by faculty mentors and students that defines the meaning of each assessment score on a 5-point scale: always (5), usually (4), often (3), seldom (2) or never (1)

• A ‘confidence’ judgment in which mentors are asked to indicate their level of confidence in their score, using a five-point scale ranging from “very confident” to “not confident at all”

• Student self-assessments and mentors’ assessments of students, performed independently

• Student-mentor discussions to compare their independent assessments following each of the three assessments

Features of the Evaluation

• Designed to increase likelihood that faculty mentors have multiple opportunities to familiarize themselves with student work

• Assessments are conducted according to standards that are explicit and uniform across disciplines and across different student-faculty pairs

• Assessment scores include confidence rating to reflect the amount and quality of information underlying the scores

• Designed to provide opportunities for students to improve their understanding of their academic strengths and weaknesses

Reports

• In addition to completing the mid-research assessment survey, students and mentors complete mid-summer reports that include several questions about progress, changes from the original projects, and plans for the second half of the summer

• Student and mentors complete final reports at the end of the summer that include questions about research findings, places where students might present their work, mentoring process (for faculty), and the students upload their 3-4 page report

Analysis of Assessment Instruments• To determine if assessment instruments provided reliable and

valid impact measures, Cronbach’s alpha used to determine the strength of association for the 34 component questions for 2008-2010 • Mentor assessment had overall coefficient alpha of .96 and

student instrument had overall coefficient of .95• Coefficients support that there is a high level of internal consistency

• To determine if any items were redundant and non-discriminating, factor analysis/principal components analysis used for both mentors and students. The 34 items were highly correlated with each other• Analysis suggested minor reduction in number of questions,

but due to differences between mentor and student assessments, no questions could be eliminated

Note: In above analyses N=61, with 17 awards in 2008, 20 awards in 2009, and 24 awards in 2010

Findings (1)

• To understand impact of program, applied repeated measures analysis of variance (ANOVA) test to student and mentor pre-, mid-, and post-research assessments. Mentors’ confidence scores analyzed with a paired samples T-test (only pre-/post- confidence measures)• Mentors’ largest adjustments in appraisals of their students’

abilities occurred between the pre- and mid-research assessments; suggests that mentors’ understanding of students’ abilities takes place within first half of program as they began to interact with the students• Student self-ratings in pre-research assessment were higher

than mentor ratings; suggests that many students at first over-estimated their academic strengths. Confirmed by comments on post-research assessment that indicated they thought they knew a lot at the outset of the experience but later realized how much they did not know

Findings (2)

• Summarized in Table 3, CURQ Spring 2012 article:• For students, items with greatest change from pre-

to post- include:• Brings new insights into the problem at hand*• Shows ability to approach problems from different

perspectives• Demonstrates ability to properly identify and/or

generate reliable data*• Shows understanding of how knowledge is generated,

validated, and communicated within discipline• Displays thorough grasp of relevant research methods

and is clear about how these methods apply to the research project being undertaken*

Findings (3)

• Summarized in Table 3, CURQ Spring 2012 article:• For mentors, items with greatest change from pre-

to post-include items with * on previous slide and:• Effectively connects multiple ideas/approaches• Learns from and is not discouraged by set-backs and

unforeseen events• Shows flexibility and a willingness to take risks and

try again• Recognizes severity of creating, modifying,

misrepresenting, or misreporting data• Displays detailed and accurate knowledge of key

facts and concepts

Findings (4)

• Student findings:• Tended to initially rate themselves slightly higher than their mentor

rated them• Reported growth on every survey item• Because students often ‘inflated’ their pre-assessment scores and

later scored themselves more realistically, the actual differences between pre- and post-scores was less frequently significant

• Mentor findings:• Initially rated the students lower than the students rated themselves

but by the end of the program rated the students higher than the students’ self-reported scores (higher rate of significant differences)

• A few items with decreased scores indicated that the mentor initially overrated their student and later learned that the student was not as strong as originally scored (e.g., writing skills, contribution to discipline)

Findings (5)

• Student comments included remarks that highlighted:• Improvements in knowledge of discipline, how scholarly

activities take place in the discipline, new methods and tools for conducting research, and ability to interact and communicate on a scholarly level

• Contributions to the discipline and that they were making discoveries that had value as well as contributing resources to future students

• Value of their participation in the summer research program and how it would contribute to future endeavors (going to graduate school, competing in job market, networking, etc)

• Gained knowledge beyond the classroom, their peers that were not conducting research, and exceeded their own expectations for what they could accomplish

Findings (6)

• Mentor* comments included remarks that highlighted:• Balance between ‘teacher’ and ‘mentor’ allowing student to

explore, make mistakes, and grow without constant presence (reflects stages of mentoring)

• Value of working in a collaborative/collegial way with their student• Program’s influence on teaching practice. In particular, spoke of

desire to increase opportunities for students to engage in research within a course and also raised expectations about what students can accomplish

* While some mentors initially commented on the redundancy of completing a pre-, mid-, and post-assessment, at the end of the process many recognized the value of capturing the growth of their particular student. The reason for this in part stems from seeing how the pre-, mid-, and post-assessments and subsequent conversations with their student was invaluable to the student’s self-awareness of their growth, strengths, and weaknesses and that the student found the process very instructive.

Findings (7)

Findings (8)

Modifications Made Based on Findings• Orientation Session: Clearer instructions about evaluation process

provided to students and mentors along with summary of rationale for evaluation methodology and modifications to protocol

• Student Confidence Scores: Eliminated from surveys and replaced by single question at end asking how certain the student was about their skill level and responses

• Mentor Confident Scores: Eliminated from mid-summer report • Format of Survey: Simplified with optional outcomes identified at

end rather than after each outcome category• Show Prior Score: To make it easier for students and mentors to

remember how they rated each component in earlier surveys scores are listed after each component; A prompt has been added at the end of the survey that lists the items with a score change of 2 or more asking student/mentor to explain ‘why’ the score increased/decreased

Tracking Completion of Assessment

• To facilitate tracking and ensure that students and mentors complete forms in proper sequence and promptly, a page was created that allows me to view status of form (started, passed to mentor, completed) and to release forms only when research team is ready to complete them

Resources

• More information and static versions of instruments are available on the Office of Undergraduate Research website: www.buffalostate.edu/undergraduateresearch

• CUR Quarterly articles (Spring 2009 and Spring 2012) describe the approach and data/findings for years 2008-2010

Concluding Remarks

• Evaluation structured so perceptions explored in a systematic manner, lays out a range of student outcomes that all mentors and students must address, provides a rubric to guide their assessment scores• Unanticipated benefit to evaluation process is improved mentoring

(provided a mechanism for facilitating regular and semi-structured interactions between student and mentor)

• Evaluation program is recognized by students and mentors as a valuable part of participating in the summer research program.

• Results are reasonably consistent with prior research, except much more focused on specific outcomes

• Recognized by administration and has helped make a case for continued growth in the program funding

• Data from 2011 (24 summer research fellowships awarded) very consistent with findings from 2008-2010

Next Steps

• Continuation of evaluation program; currently implementing evaluation with the 24 student-mentor teams funded in 2012

• Considering ways of facilitating wider adoption and adaptation• While static versions of forms are available, code is

unique to Buffalo State’s server/computer technology support

• Analysis of results from the Alumni Survey (to date ~35% response rate from former students)

Questions and Comments?