Assessment in the Biology Department in AY 2007-8

Caroline SolomonFriday December 5

Biology Department Outcomes

• The development of the biology department outcomes took several years (thanks to Dr. Ann Powell who led us in this effort) before finally being polished by Bette Landish

Biology Department Outcomes

1. Demonstrate knowledge of current theories of biology and concepts basic to biology, including: Levels of complexity (molecular/cellular through population/communities/ecosystems); biological principles and processes; connections among concepts across disciplines (physical sciences, mathematics, social sciences).

2. Demonstrate critical thinking skills through: Identifying appropriate questions, issues or problems to be explored; analyzing, interpreting and evaluating evidence; constructing well-supported, clearly articulated arguments and conclusions.

Biology department outcomes (con’t)

3. Demonstrate information literacy and research skills through effective use of: Sources of information in biology including published literature and scientific databases; computer applications to acquire and (continued) analyze experimental data and develop graphic models.

4. Demonstrate experimental research methods including: Scientific methods and instrumentation; safe and appropriate use of laboratory equipment; experimental design; data analysis; familiarity with professional standards in science.

Biology department outcomes (con’t)

5. Demonstrate effective communication with varied audiences and media by: Providing clear structure and transitions; demonstrating substantial understanding via research, credible sources and supporting evidence; demonstrating audience-appropriate purpose, agenda, language and style.

6. Demonstrate preparation for future career and educational goals by: awareness of personal competencies (strengths and weaknesses); ability to work in a team to accomplish a goal; understanding professional and ethical behavior.

Four year plan for assessmentOutcome AY 2007-8 AY 2008-9 AY 2009-10 AY 2010-11

#1: Biology Content

ETS Field Test; Senior Exit Surveys

#2: Critical Thinking

Development of assessment tools

#3 : Information literacy

Begin assessment

#4 Experimental research methods

Begin assessment

#5 Effective communication

Writing & Presentation rubrics

#6 Career Preparation

Internships portfolios

Demonstrate knowledge of current theories of biology and concepts basic to biology, including: Levels of complexity (molecular/cellular through population/communities/ecosystems); biological principles and processes; connections among concepts across disciplines (physical sciences, mathematics, social sciences).

(1) Direct measurement: ETS Biology Field Test(2) Indirect measurements:

(a) Revised student evaluations to reflect outcomes (b) Exit surveys from seniors (2007 & 2008)

Outcome #1: Biology content

Results from ETS Biology Field Test

*areas that need to be worked on the most

Student distribution

*Subscores are reported; (% compared to institutional mean scores that includes seniors from 345 domestic institutions)

Our next step… closing the loop for Outcome #1

Weaknesses: biochemistry, cell biology & botany/plant physiology

How do we change our curriculum or courses to address these weaknesses?

Department decided to:

(1) emphasize botany and plant physiology more in second semester of introductory biology (BIO 108) and in ecology (BIO 241)(2) strongly encourage students to take cell biology or molecular biology; offer the course as early as Spring 2009

Outcome #5: Effective Communication

Demonstrate effective communication with varied audiences and media by: Providing clear structure and transitions; demonstrating substantial understanding via research, credible sources and supporting evidence; demonstrating audience-appropriate purpose, agenda, language and style.

(1) Direct measurement: Rubrics for lab reports, literature reviews & presentations (2) Indirect measurement: Exit surveys from seniors (2007 & 2008)

Assessment tools for this outcome

(1) Writing a. Lab report rubricb. Literature review rubric

(2) Presentationsa. Class presentation rubric

Lab report rubrics

• Two different lab rubrics were used during AY 2007-2008

• Why did we revise the lab report rubric for Spring 2008?– A lab report rubric workshop was held in

December 2007; the scoring for three different student products were not consistent among department members

Lab report rubric used in Fall 2007

Revised lab report rubric

Lab Reports – Fall 2007

BIO 103; n= 48BIO 108; n= 32BIO 231/241; n = 17

We identified these problem areas in December 2007; Intro & discussion involve critical thinking

Lab Reports – Spring 2008

BIO 104/GSR 150; n= 85BIO 110; n= 33BIO 211,221, Capstone; n =30

Note difference in scales from Fall 2007 (5 points) to Spring 2008 (4 points)

A closer look…..Title and Abstract

Need to figure out

why

Need to figure out

why

With the new lab report rubric – we can now see things in more detail

A closer look….Introduction

Decline because SQ is

given in earlier courses?

Decline because SQ is

given in earlier courses?

A closer look….Materials & Methods

Senior Exit survey: Did the program teach you how develop your own experiments? 2007 – 3.7; 2008 – 2.5 Did the program prepare you to use various laboratory instruments? 2007 – 4.2; 2008 – 3.6

Students need more

work in developing their OWN

experiments

Students need more

work in developing their OWN

experiments

A closer look….Results

Senior Exit survey: Did the program teach you how to analyze, interpret and evaluate evidence? 2007 – 4.7; 2008 - 3.1

Did the program teach you how to develop good graphs and charts? 2007 – 4.2; 2008 – 2.4

Students need more practice interpreting & creating their own graphs

Students need more practice interpreting & creating their own graphs

A closer look….Discussion & Conclusion

Senior Exit survey: Did the program teach you how to support your arguments and conclusions? 2007 – 4.4; 2008 – 2.8

Still needs improve-

ment

Still needs improve-

ment

A closer look….Quality of Writing

A closer look….Citation & Bibliography

Improvements were made in this category from fall to spring

Synthesis of results of writing assignments• Decrease in ability from freshman year to upper

level courses:– To write a good abstract– Identification of scientific question

• Areas that need improvement– Development of own experimental procedures

(identification of controls)– Data analysis; graphing– Discussion & Conclusion (critical thinking)

Our next step… closing the loop for Outcome #5

• The discussion of the results from the lab report rubric produced a healthy discussion about comfort levels of different faculty members about Excel (data analysis & graphing) and defining levels of expectation of mastery in Excel were needed

Presentations

• Assessment of presentations during Fall 2007 were compounded by the use of different rubrics by different professors, no recording of presentations

• In January 2008, the department developed a presentation rubric that was used in Spring 2008

Presentations – Fall 2007

Note: For most categories n= 20, but some had n= 26 due to different rubrics; for better understanding of categories see journal club rubric (BIO 231, 241, 401)

Presentations – Spring 2008

n=3

n=8

Improvements that needed to be made in collection of data for presentations

• Identification of weaknesses:– Content– Use of visual media– Bilingual communication (sign clarity)

• Often people did not do all the categories

• No recording of presentations; collection of evidence– A new protocol to record presentations for Fall 2008

Focus for AY 2008-2009

Four year plan for assessmentOutcome AY 2007-8 AY 2008-9 AY 2009-10 AY 2010-11

#1: Biology Content

ETS Field Test; Senior Exit Surveys

#2: Critical Thinking

Development of assessment tools

#3 : Information literacy

Begin assessment

#4 Experimental research methods

Begin assessment

#5 Effective communication

Writing & Presentation rubrics

#6 Career Preparation

Internships portfolios

Outcome # 2: Critical Thinking Demonstrate critical thinking skills through: Identifying appropriate questions, issues or problems to be explored; analyzing, interpreting and evaluating evidence; constructing well-supported, clearly articulated arguments and conclusions.

Suggestions for measurements: (1) Critical thinking components of writing & presentation rubrics (2) Inclusion of data analysis problems and conceptual problems in exams in major courses (following Kitchen et al. 2003) (3) Scoring on analytical problems on the ETS Field Test

Focus for Spring 2009

• Invite Virginia Anderson from Towson to present on how to assess critical thinking in biology (coming January 15th)

• Develop assessment tools for critical thinking (outside of writing rubrics)

Outcome #6: Career Preparation (Ava)

Demonstrate preparation for future career and educational goals by: awareness of personal competencies (strengths and weaknesses); ability to work in a team to accomplish a goal; understanding professional and ethical behavior.

(1) Direct measurement: Alumni surveys (how many are employed in the field after graduation?)(2) Indirect measurements:

(a) Internship portfolio (b) Exit surveys from seniors (2007, 2008 & 2009)

Reflections on assessment

• At the first department meeting of the AY, we had three groups of data to help us guide us in making decisions about the future direction of the department:

(1) Assessment data from the 2 outcomes(2) Preliminary data from internship portfolios(3) An analysis of Biology curriculums around the country at schools of a comparable size to Gallaudet

• Even if we are starting to develop tools for the two student outcomes we want to measure in AY 2008-9, we are continuing to collect data on the two learning outcomes we started last year - it’s an ongoing process!