Assessment Report

Department/Program: Psychology Chair: Tom Kucharski

Report prepared by: Deryn Strange

Degree: Forensic Psychology Major

Time Period Covered for this Assessment Review: Fall 2011- Spring 2012

Assessment occurred in the following courses:

Course Semester and Year Psy 311 Spring 2012 All graduating Seniors Spring 2012

Direct Assessment of Learning Goals (Please attach to the report a copy of each rubric used.)

Appendix A contains the rubric we used for this assessment. Note that professors were asked to randomly choose 5 papers from their students and submit them to the assessment committee. The paper was a final report submitted in the last week of the semester.

Learning Goal(s) Assessed Course Number

Sample Size

Exceeds Expectations

Meets Expectations

Approaches Expectations

Does Not Meet

Expectations Choosing reputable and relevant research

311 37 0% 30% 54% 16%

Understanding of research methodology

311 37 0% 8% 35% 56%

Understanding of research design

311 37 0% 11% 40% 49%

Developing ethical research 311 37 0% 11% 38% 51%

2

Data Interpretation

311 19 0% 0% 28% 72%

Understanding strengths and weaknesses of research designs

311 37 0% 3% 54% 43%

Appropriate data analysis

311 19 0% 0% 61% 39%

Disciplinary Conventions

311 37 0% 8% 32% 59%

Indirect Assessment of Learning Goals

Appendix B contains the survey questions and responses to the survey.

Learning Goal(s)

Course or Program Based?

Sample Size, if Known

Instrument

Data

5: Ethics and Values

Graduating Seniors

30

Exit Survey

See Appendix B

9: Personal Development

Graduating Seniors

30

Exit Survey

See Appendix B

10: Career Planning and Development

Graduating Seniors

30

Exit Survey

See Appendix B

Conclusions What did you discover about student learning in your program?

311 Assessment: The majority of our students are either failing to meet or only approaching the learning goals we set for the Psy311 course (which cover basic research knowledge and skills, ethics and values, information technology and communication skills). For example, we found that many students performed literature searches, but those searches were not relevant and did not yield useful

3

information, suggesting that students did not really know what they were meant to be doing. It became clear that this issue was not well delineated in the rubric (the rubric specified relevant searches using PsycInfo). In addition, we found that there was little overlap between how professors graded a paper and whether the student demonstrated a good grasp of the learning goals that the assignment was designed to test. In a particularly egregious case, a professor awarded an A++ (a grade John Jay College does not even recognize) and the independent assessors agreed that the student had failed to meet expectations on any of the specific learning goals. Thus it is unclear what the professor was grading. Exit Survey: We were disappointed in the number of responses to the survey. Nevertheless, of the responses we did receive, students appeared to be comfortable with their training in the area of ethics and values with the majority of responses being either agree or strongly agree. They were equally happy with their personal development, knowledge of career options, and preparation for graduate school. Taken together then, students appear to be happy with the training they have received from our department and what they have learned and retained. While that is good news, currently their perceptions do not appear to match the reality (as we have seen with the 311 assessment this year). We are hopeful that once students begin to complete our new curriculum there will be no disconnect between how well they feel they have been trained and how well they have actually been trained.

Actions Taken

What action decisions did you make based on your data and conclusions? (Plan actions to take effect in the following semester or sooner if practical.)

Actions To Be Taken and By Whom

Timeframe for implementation and intermediate steps

Prof Strange to lead a discussion of the 311 Assessment results with professors who teach the course. The goal will be to discuss changes to the assessment structure of the course. For example, students who were able to submit sections of their assignment and get feedback early appeared to do more likely to “meet expectations.” Perhaps most students simply were not getting the feedback they needed throughout the course. It is expected that this meeting will result in a set of specific changes (such as a requirement that feedback on proposals be given early in the course). While we hope that some professors will integrate those changes into their classes for Fall 2012, it will not be required until Spring 2013.

Week 1, Fall 2012

4

Were last year’s actions implemented as planned? Please explain.

Last year’s actions were not particularly specific. We did discuss the results of our capstone assessment in a faculty meeting. However, the issue was referred back to the Curriculum Committee for further discussion and by the end of the academic year it was clear that we needed to alter the goals and expectations of the course. We found little uniformity of expectations across our capstone courses (in terms of work requirements, readings, presentations) in large part because these courses have been taught by adjuncts and there has been little oversight. The Curriculum Committee has been tasked with correcting these issues and presenting a plan to the full Faculty in early Fall 2012 for changes to take effect Spring 2013.

Assessment data and conclusions were discussed in a Department or Program meeting on __________________________. [date]

Appendix A Rubric for Psy311 Assessment of APA Areas 2, 5, 6 (and 7)

Note: this rubric is intended for use with a final paper asking students to develop a research design, proposal or a complete study. The paper must have a literature review section and also a section discussing the methodology for, or the enactment of, original research.

Evaluation Area (Learning Goal

#)

Criteria

4 Advanced

3 Proficient

2 Developing

1 Rudimentary

0 Poor

Choosing Reputable and

Relevant Research (#6)

Chooses completely reputable and relevant sources of research given the topic of the final paper. Clearly demonstrates the use of PsycInfo (i.e., no non-empirical sources).

Chooses largely reputable and relevant sources of research given the topic of the final paper. Appears to have used PsycInfo (i.e., some empirical papers)

Chooses some reputable and relevant sources of research given the topic of the final paper (i.e., 1 or 2 useful papers).

Chooses low quality sources (basic internet searches, i.e., wikipedia) that are minimally relevant given the topic of the final paper.

No attempt.

Understanding of Research

Methodology (#2)

Demonstrates a strong understanding of what research methodology or methodologies are being used (i.e., no errors).

Demonstrates moderate understanding of what research methodology or methodologies are being used (i.e., 1 or 2 errors).

Demonstrates minimal understanding of what research methodology or methodologies are being used.

Demonstrates little understanding of what research methodology or methodologies are being used (i.e., mainly errors).

No attempt.

Understanding of Research Design (#2)

Demonstrates a strong understanding of what research design or designs are being used (i.e., no errors).

Demonstrates moderate understanding of what research design or designs are being used (i.e., 1 or 2 errors).

Demonstrates minimal understanding of what research design or designs are being used

Demonstrates little understanding of what research design or designs are being used (i.e., mainly errors).

No attempt.

Developing Ethical

Research (#2 and #5).

Develops (and potentially conducts) an ethically thoughtful research design that fully accounts for participants’ rights.

Develops (and potentially conducts) an ethically thoughtful research design that largely accounts for participants’ rights (i.e., might miss one or two issues).

Develops (and potentially conducts) a research design that minimally accounts for participants’ rights (i.e., might mention ethics but not in an insightful way).

Develops (and potentially conducts) a research design that does little to account for participants’ rights.

No attempt.

Data Interpretation

(#2; If the study

Is able to accurately interpret all of the quantitative and/or qualitative results and generalize these interpretations appropriately

Is able to accurately interpret most of the quantitative and/or qualitative results and generalize these

Is able to accurately interpret some of the quantitative and/or qualitative results and generalize these

Is unable to accurately interpret quantitative and/or qualitative results and generalize these interpretations

No attempt.

6

is conducted)

(e.g. is able to interpret but not overstate while linking important implications)

interpretations appropriately

interpretations appropriately (i.e., makes a few errors).

appropriately (i.e., the majority of the section is errors).

Understanding Strengths and Weaknesses of

Research Designs (#2)

Accurately acknowledges the strengths and weaknesses of the study and demonstrates insights into possible corrections.

Accurately acknowledges the strengths and weaknesses of the studies.

Gives some sense that there are strengths and/or weakness to the studies.

Gives no sense that there are strengths and/or weakness to the studies.

No attempt.

Appropriate Data Analysis

(#7; if the study is conducted)

Is able to accurately use the appropriate statistical and/or thematic research analyses to address the research question at a professional-level.

Is able to accurately use the appropriate statistical and/or thematic research analyses to address the research question.

Is able to use some of the appropriate statistical and/or thematic research analyses to address the research question (i.e., makes a few errors).

Is unable to use the appropriate statistical and/or thematic research analyses to address the research question (i.e., the majority of the section is errors).

No attempt.

Disciplinary Conventions

(#7)

Demonstrates mastery of professional writing conventions in psychology, including appropriate language use and organization. Correctly implements APA style conventions.

Demonstrates basic proficiency in the professional writing conventions in psychology. Includes few major deviations from APA style conventions.

Demonstrates some knowledge of the professional writing conventions in psychology. Attempts to properly apply APA style conventions.

Demonstrates a limited awareness of professional writing conventions in psychology.

No attempt.

Key Exceeds expectations: “4” Meets expectations: “3” Approaches Expectations “2” Does not meet expectations: “1” and below

Appendix B

Graduating Senior Survey Responses - Spring 2012

Think back on the psychology courses you have taken. How strongly do you agree or disagree that your psychology classes explained the following: 1. The Institutional Review Board's (IRB) purpose and process.

Response Percent

Response Count

Strongly Disagree 3.6% 1 Disagree 7.1% 2 Neutral 21.4% 6 Agree 35.7% 10 Strongly Agree 32.1% 9 2. The ways that ethical principles are applied to psychological research.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 7.4% 2 Agree 40.7% 11 Strongly Agree 51.9% 14 3. How moral reasoning is used in our everyday lives

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 7.1% 2 Neutral 17.9% 5 Agree 46.4% 13 Strongly Agree 28.6% 8 4. Diversity as it is related to race, gender, class, religion and/or sexual orientation. Response Response

8

Percent Count Strongly Disagree 0.0% 0 Disagree 7.1% 2 Neutral 7.1% 2 Agree 35.7% 10 Strongly Agree 53.6% 15 5. Future careers in the filed of psychology.

Response Percent

Response Count

Strongly Disagree 3.6% 1 Disagree 10.7% 3 Neutral 10.7% 3 Agree 42.9% 12 Strongly Agree 32.1% 9 6. The steps you would need to take in order to pursue a graduate degree in psychology.

Response Percent

Response Count

Strongly Disagree 10.7% 3 Disagree 0.0% 0 Neutral 25.0% 7 Agree 28.6% 8 Strongly Agree 35.7% 10 Now that you have nearly completed the psychology curriculum, how strongly do you agree or disagree that your psychology classes have helped you to:Show this Page Only 1. Identify the values and assumptions made in statements (e.g., in the media, among colleagues or friends, in research).

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 3.7% 1

9

Neutral 7.4% 2 Agree 59.3% 16 Strongly Agree 33.3% 9 2. Respect the varying points of view held by others.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 3.7% 1 Neutral 3.7% 1 Agree 40.7% 11 Strongly Agree 51.9% 14 3. Tolerate diversity.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 11.1% 3 Agree 25.9% 7 Strongly Agree 63.0% 17 4. Respond constructively to criticism of your work.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 14.8% 4 Agree 44.4% 12 Strongly Agree 40.7% 11 5. Self reflect upon your experiences in order to learn from them.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 3.7% 1

10

Neutral 11.1% 3 Agree 29.6% 8 Strongly Agree 55.6% 15 6. Manage multiple tasks

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 11.1% 3 Neutral 11.1% 3 Agree 22.2% 6 Strongly Agree 55.6% 15 7. Become familiar with what career options are available to you in the filed of psychology.

Response Percent

Response Count

Strongly Disagree 7.7% 2 Disagree 7.7% 2 Neutral 11.5% 3 Agree 53.8% 14 Strongly Agree 23.1% 6 8. Apply the knowledge you learned in psychology to your everyday life.

Response Percent

Response Count

Strongly Disagree 3.7% 1 Disagree 7.4% 2 Neutral 3.7% 1 Agree 44.4% 12 Strongly Agree 44.4% 12 Tell us how strongly you agree or disagree that you have achieved the following:Show this Page Only 1. Awareness of APA ethical principles for psychologists and the code of conduct.

Response Percent

Response Count

11

Strongly Disagree 0.0% 0 Disagree 3.7% 1 Neutral 7.4% 2 Agree 59.3% 16 Strongly Agree 33.3% 9 2. Awareness of the need for ethics in research.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 7.4% 2 Agree 44.4% 12 Strongly Agree 51.9% 14 3. Awareness of the need for ethics in applied settings.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 7.4% 2 Agree 40.7% 11 Strongly Agree 51.9% 14 4. Respect for other points of view.

Response Percent

Response Count

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 14.8% 4 Agree 48.1% 13 Strongly Agree 51.9% 14 5. An understanding and an ability to apply moral reasoning in applied psychology settings.

Response Percent

Response Count

12

Strongly Disagree 0.0% 0 Disagree 0.0% 0 Neutral 7.4% 2 Agree 40.7% 11 Strongly Agree 51.9% 14 2. Overall, how well do you believe the psychology department has trained you for a career in the field of psychology?

Response Percent

Response Count

Not well at all 3.7% 1 Not well 7.4% 2 Neutral 14.8% 4 Well 66.7% 18 Extremely well 7.4% 2 3. Are you interested in graduate school either now or later in your career?

Response Percent

Response Count

No, not interested 3.7% 1 Yes, interested now 44.4% 12 Yes, interested later 55.6% 15 4. Overall, how well do you believe the psychology department has trained you for attending graduate school in a field related to psychology?

Response Percent

Response Count

Not well at all 3.7% 1 Not well 11.1% 3 Neutral 33.3% 9 Well 44.4% 12 Extremely well 7.4% 2