FINALOACApproved2.2.18 1

Research & Decision Making Outcome Assessment Report Outcomes Assessment Committee Report Prepared by: Dr. Sarah Harris, Outcomes Assessment Coordinator College of the Sequoias, Fall 2017 Summary This report presents the results of the Outcomes and Assessment Committee’s 2016-2017 assessment of the Institutional Learning Outcomes “Research & Decision Making.” The committee assessed this outcome using a district-wide survey and rubric-based scoring of student-submitted research work. Survey results show students are confident in their ability to locate resources, but direct assessment indicates students would benefit from additional instruction in citation and source use. Introduction Each academic year, following an established five-year cycle, the Outcomes and Assessment Committee (OAC) conducts an assessment of one of the District’s five Institutional Learning Outcomes (ILO). In 2016 – 2017, the OAC designed and completed an assessment of the Research and Decision Making ILO: “Students will locate and evaluate information, including diverse perspectives, to make informed and ethical decisions.” This report summarizes the assessment design, results, and committee recommendations. Institutional Planning and Accreditation Assessment of ILOs is a key component of District planning for improvement. Assessment results are presented annually to faculty and students for dialogue and discussion, including the presentation of results to governance groups. Data from this report was presented to faculty and students during Fall convocation, to the Academic Senate in November, and to Instructional Council in December. Assessment is also a key component of accreditation. Assessment of student learning outcomes is a component of ACCJC Accreditation Standards for Assuring Academic Quality and Institutional Effectiveness (IB), Institutional Integrity (IC), and Instructional Programs (IIA). Standards related to this report include, but are not limited to:

I.B.1: The institution demonstrates a sustained, substantive and collegial dialog about student outcomes, student equity, academic quality, institutional effectiveness, and continuous improvement of student learning and achievement.

I.B.6: The institution disaggregates and analyzes learning outcomes and achievement for subpopulations of students. When the institution identifies performance gaps, it implements strategies, which may include allocation or reallocation of human, fiscal and other resources, to mitigate those gaps and evaluates the efficacy of those strategies. II.A.1: All instructional programs, regardless of location or means of delivery, including distance education and correspondence education, are offered in fields of study consistent with the institution’s mission, are appropriate to higher education, and culminate in

FINALOACApproved2.2.18 2

student attainment of identified student learning outcomes, and achievement of degrees, certificates, employment, or transfer to other higher education programs.

Methods Feedback from the 2015 – 2016 assessment of the Communication ILO included the recommendation that future ILO assessment include both direct and indirect measures, and that these measures be targeted at student learning rather than particular courses. The OAC committee therefore designed a two-part assessment process to assess the Research and Decision Making outcome, consisting of qualitative survey items and a rubric-based assessment of student-submitted research work. Indirect Assessment: Survey Items OAC Members composed two survey items for each of the five ILO’s, to be added to the District-wide Student Support Services Survey distributed to students every two years. Survey items were designed as Agree/Disagree statements on a 4-point Likert scale from “Strongly Agree” to “Strongly Disagree,” in alignment with other items on the survey. Questions were composed and revised by committee members with input and feedback from the Office of Research, Planning, and Institutional Effectiveness (RPIE). Questions items are listed in the table below with their corresponding ILO (See Table I). Table I: Student Support Services Survey ILO Question Items Q25: Please indicate your level of agreement with the following statements regarding your learning experience at COS: I listen to the positions of other people and incorporate them into my own arguments when it is appropriate to do so.

Communication

I am able to adjust my communication to a variety of audiences and purposes.

Communication

I can use information from the research resources available at COS to complete my assignments.

Research & Decision Making

I consider multiple perspectives when evaluating information. Research & Decision Making

My experiences at COS encouraged me to participate as a responsible citizen in my community (volunteering, political advocacy, and/or community-based activities).

Civic Engagement

I am aware of issues that affect my community. Civic Engagement I have successfully used mathematical methods to solve problems or interpret information.

Critical Thinking/ Problem Solving/ Analysis

I am able to analyze complex problems and recommend possible solutions.

Critical Thinking/ Problem Solving Analysis

I interact respectfully with people whose beliefs, values, and opinions are different from my own.

Life/ Interpersonal Skills

I regularly practice good time-management and study habits. Life/ Interpersonal Skills

FINALOACApproved2.2.18 3

Direct Assessment: Rubric Scoring To directly assess students’ ability to “locate and evaluate information, including diverse perspectives, to make informed and ethical decisions,” the OAC invited a selected sample of students to submit research work completed at COS, and scored that work using trained and normed faculty raters and a committee-designed rubric. Sample Selection The OAC, with the assistance of the Office of RPIE, identified students at the District who had completed a minimum of 30 units. The list was analyzed by division to determine a target percentage for sampling. Division of student enrollment was determined based on students’ declared majors. Using stratified sampling, the Outcomes Assessment Coordinator identified an initial, representative sample of 500 students to contact with an invitation to participate in the study. Following an initially low response rate, an additional 500 students were added to this list using the same sampling techniques. The initial list of students with 30+ units attempted was sorted by Division and by Student ID number, a representative percentage was identified for each division, and students were randomly selected for the sample using a random number generator to choose a start and end point from the larger list. This sample was then cross-checked against District-wide demographic data available from the District Factbook to ensure equity. This sample group of 1,000 students was contacted via campus email and Canvas message with an invitation to participate in the assessment. Participation was incentivized through the opportunity to win a Starbucks gift card via random drawing. Students were asked to “Please submit a sample of your work completed here at COS that shows your ability to do research. Ideally, the sample you submit should show your ability to complete research and make decisions based on that research.” Samples were collected via a Canvas Course site created for the assessment. Rubric Design and Scoring An initial draft of the scoring rubric was developed by the Outcomes Coordinator using the language of the ILO and the nationally validated AAC&U VALUE Rubrics for Information Literacy, Inquiry and Analysis. This rubric was reviewed and further refined by OAC committee members, then tested using initially submitted student samples by OAC members and revised for clarity (See Appendix A: Research and Decision Making Rubric). The rubric was designed with four items on a three-point scale; a score of 10 or higher was determined to be “meeting” the overall ILO. Faculty raters were solicited using the District’s hiring process for hourly work. Submitted work was scored by two full-time faculty members (in the Language Arts and Science Divisions), who were trained and normed in the use of the rubric by the Outcomes Assessment Coordinator. Student samples were double-blind scored in Canvas by both raters, with the Coordinator monitoring the scores for variance and requesting additional review and discussion on any items with more than one point of disagreement in the ratings. All final scores were within one point of

FINALOACApproved2.2.18 4

agreement; scores were combined and averaged on each rubric item for determination of final results. Results Indirect Assessment: Survey Items The Student Support Services Survey was distributed to District students in the Spring semester of 2017; ~1900 students responded to the ILO items specific to research (see Table II). Overall agreement on these items was very high, at over 95% for each item. Of the five ILO’s, Research & Decision Making had the highest overall agreement across the two items. Survey results were disaggregated using equity measures identified in the District’s equity plan, including race/ethnicity, age, gender. Results were also disaggregated by student status, first-generation status, and units attempted, with the goal of identifying any possible equity and/or system barriers to student learning and success. Where the number of respondents was less than ten, results were not reported to protect participants’ privacy. Results indicated high levels of agreement across all disaggregated survey results, with no apparent differences across demographic and other disaggregated groups. Table II: Research and Decision Making ILO Survey Results Demographic Group Characteristics % Agreement** By Enrollment Status Full-Time 97.2

Non-Credit * Part-Time 96.9

By Student Status Continuing 97.1 First-Time 99.2 First-Time Transfer 96.2 Returning 95.6

By Gender Female 97.2 Male 96.8 Other/Unknown 96

By Age 19 or less 96.9 20 – 24 96.6 25 – 29 97.7 30 – 49 97.6 50 + 97.6

By Race/Ethnicity African-American 97.3 American Indian/Alaska Native * Asian 96.6 Filipino 97.8 Hispanic 96.8 Multi-Ethnicity 96.5 Pacific Islander * Unknown 100 White 97.7

FINALOACApproved2.2.18 5

By First-Generation Status First-Generation 97.2 Not First-Generation 97.4 Unknown 95.1

By Units Completed 29 or less 96.9 30 – 59 96.5 60 – 89 97.2 90 + 97.8

All Respondents 97.5 *N < 10 respondents ** Percentage reflects an average of two survey items related to the indicated ILO. N ~ 1899 Direct Assessment: Rubric Scoring Of the students contacted, a total of 48 research samples were received from 44 students, for a response rate of 4.4%. For purposes of results reporting, individual student participants were used for demographic reporting, while all unique samples were used to report success on specific rubric items. In instances where a single student submitted more than one sample, the highest scoring sample was used for reporting.

Direct assessment results were disaggregated where possible using equity measures identified in the District’s equity plan, including race/ethnicity, age and gender. Results were also disaggregated by units attempted, with the goal of identifying any possible equity and/or system barriers to student learning and success. Where the number of participants was less than ten, results were not reported to protect participants’ privacy (see Table III).

AG3% BUS

9% CFS2%

FINA9%

I&T2%LANG2%

MATH9%

NURS18%

SCI7%

SOCS18%

STSV21%

DIRECTASSESSMENTPARTICIPANTSBYDIVISION

FINALOACApproved2.2.18 6

Table III: Research & Decision Making Rubric Scoring Demographic Group Characteristics % Meeting Outcome** By Gender Female 65.7%

Male * Other/Unknown *

By Age 19 or less * 20 – 24 57.9% 25 – 29 * 30 – 49 85.7% 50 +

By Race/Ethnicity African-American * American Indian/Alaska Native * Asian * Filipino * Hispanic 60.9% Multi-Ethnicity * Pacific Islander * Unknown * White 92.3%

By Units Completed 30 – 59 57.7% 60 – 89 92.9% 90 + *

All Respondents 68.2% *N < 10 respondents ** The Outcome is met by an overall rubric score of 10 or higher. N = 44 Responses were also reviewed for overall representation of Divisions, both in student enrollment and in selection of samples. Though divisions were evenly represented according to the sample selection procedures (see Direct Assessment Participants by Division), students submitted work primarily from courses in the Language Arts (45%) and Social Sciences (26%) divisions (see Direct Assessment Work Submitted by Division).

FINALOACApproved2.2.18 7

Scores were also disaggregated by rubric items (see Table IV). Results indicated that overall scores were higher on decision making items than on items more directly related to locating and evaluating information. Table IV: Research and Decision Making Rubric Scoring by Item Rubric Item % Meeting Outcome* Locate Information 68.8 Evaluate Information 62.5 Make Informed Decisions 72.9 Make Ethical Decisions 70.8

*N = 48 Discussion Students self-report high agreement that they can use information from the research resources available at COS to complete their assignments and that they consider multiple perspectives when evaluating information. This level of agreement shows little variance across demographic groups, and students have confidence in their ability to complete research tasks regardless of number of units completed, suggesting that accessing resources and considering multiple perspectives are skills that students may feel confident about early in their studies or prior to enrollment. Direct assessment of student-selected research work indicated close to 70% of submitted samples meet the outcome. The small number of participants in the sample means it is not possible to draw generalizable conclusions based on the sample. However, possible equity gaps in student success appear for Hispanic students, and these gaps would benefit from additional investigation.

BUS2%

CFS5% FINA

3%

LANG45%

MATH8%

PE3%

SCI8%

SOCS26%

DIRECTASSESSMENTWORKSUBMITTEDBYDIVISION(EXCLUDINGN/A)

FINALOACApproved2.2.18 8

Participants had slightly higher success on outcome measures related to decision making than locating and evaluating information. During scoring, raters reported that where samples were approaching success, they were most likely to have a clear purpose but less success in clearly identifying the source/s of their information, suggesting student may benefit from additional instruction in genre and audience-appropriate citation methods. The results of this assessment were shared with faulty and discussed during Fall Convocation and presented to the Academic Senate and the Board of Trustees. These discussions are reflected in the following section, which presents findings and recommendations. Findings and Recommendations 1. Equity gaps in research opportunities and/or instruction may exist for Hispanic

students, but more data is necessary.

Though the direct assessment sample was small, gaps in success for Hispanic students and for students with less than 60 units line up with existing District equity data on student success, basic skills placement, basic skills completion and units attempted. The high number of submitted samples from language arts/ English composition courses and Social Science courses such as Psychology and History (which carry an English 251/261 prerequisite requirement) suggest that student access to and completion of these courses is a key component of success for this outcome. However, a larger data set would be necessary to draw definitive conclusions.

15

13

11

1

0%

20%

40%

60%

80%

100%

30- 59 60- 89

DIRECTASSESSMENTRESULTSBYUNITSATTEMPTED

Meets DoesNotMeet

12

14

1

9

0%10%20%30%40%50%60%70%80%90%

100%

WhiteNon-Hispanic

Hispanic

DIRECTASSESSMENTRESULTSBYRACE/ETHNICITY

Meets DoesNotMeet

FINALOACApproved2.2.18 9

2. Research skills and instructions should be emphasized across the curriculum, specifically in courses outside of GE English Composition and Social Science courses.

Though students are doing research work in courses across multiple divisions according to outcomes mapping, students may not be connecting this work directly with the ILO. Instructors doing research work with students should make sure to emphasize the ILO and make the connection clear on related assignments. The District should also make sure research opportunities are available to students in courses that are accessible to students who place into the basic skills sequence, including instruction in resource location, usage, and citation.

3. Students are confident in their ability to do research, but would benefit from

additional instruction in source use and discipline-appropriate citation methods.

Survey results show students are confident in their ability to locate resources, but direct assessment indicates students would benefit from additional instruction in citation and source use. The O&A Committee will work with FEC to recommend faculty professional development on effective instructional methods to narrow this gap. The committee will also work with the library to create modules on discipline-specific citation methods that faculty can add to Canvas course shells, and will also compile and share additional available resources. Additionally, the O&A committee recommends, as a general practice, that O&A committee work closely with FEC to use assessment results to identify key professional development topics of benefit to faculty and students, and working with the Program Review committee to develop ways faculty can address assessment results, including relevant gaps identified through ILO assessment, in Program Review.

4. Future ILO assessments should continue to include a multiple methods approach,

and also allow multiple methods for student participation where possible.

Assessment results indicate that there can be gaps between students’ self-reported abilities and the direct assessment of student work, such that a multiple-measures approach provides stronger results.

Soliciting work directly from students provided additional, actionable assessment results, but the number of participants in the direct assessment was very low. Suggestions for improving participation include using multiple or additional methods to collect work, including the collection of work in hard copy or through offline methods (as Canvas participation may have been a barrier for some students). Focus groups including an activity or other direct collection of work may also be effective ways to increase student participation.

APPENDIX A: Research and Decision Making: ILO Assessment Rubric OAC Committee Spring 2017 Students will locate and evaluate information, including diverse perspectives, to make informed and ethical decisions. Criteria Meets: 3 Developing: 2 Evidence Not Present: 1 Not Addressed: 0 Locate Information Score:__________

The artifact includes information from a variety of sources appropriate to the relevant genre, discipline, and/or audience.

The artifact includes information from limited or similar research sources; sources are not always appropriate to the relevant genre, discipline, and/or audience.

The artifact includes information from few or no identifiable sources. Sources selected are inappropriate.

This artifact does not include any identifiable sources.

Evaluate Information Score:___________

Information from sources is accompanied by enough interpretation/ evaluation to develop a coherent analysis or synthesis. Viewpoints of experts are contextualized or questioned.

Information from sources is accompanied by some interpretation/ evaluation. Viewpoints of experts may be contextualized but are taken mostly as fact.

Information is presented with little to no evaluation or interpretation. Viewpoints of experts are accepted without question or context.

This artifact does not include any identifiable sources.

Use Information to Make Informed Decisions Score: ____________

Communicates, organizes, and synthesizes information to successfully achieve a clear purpose.

Communicates and organizes information in support of a purpose. Information may not be fully synthesized.

Communicates information, but information is fragmented and/or may be misquoted or misapplied. Purpose is unclear.

The artifact does not include any identifiable purpose.

Use Information to Make Ethical Decisions Score: ____________

Defines a clear purpose relevant to ethical decision making and appropriate to audience, genre, or discipline. Information is clearly and ethically referenced through citations or other discipline-appropriate methods.

Defines a purpose that is relevant to audience, genre or discipline. Information may lack some clear references or citations.

Defines a purpose that is unclear, unethical, inappropriate or not supported by evidence. Information presented lacks appropriate references or citations.

The artifact does not include any identifiable purpose.

Average Score: _____________