The STEM Department Evaluation Rubric · The STEM Department Evaluation Rubric ... (Brownell and...

The STEM Department Evaluation Rubric A Tool for Developing a Shared Vision for

Improving Undergraduate STEM Education

Ellen Goldey, PhD William R. Kenan Jr. Professor

Chair of Biology

Wofford College

Plan for the Webinar • Brief Summary of:

– Systems Theory and Undergraduate STEM Reform

– Change Strategies within and beyond STEM

• Overview of STEM Department Evaluation Rubric – please have a printed copy handy

• Interactive polling based on the Rubric

• Concluding thoughts on leadership for STEM Reform

Barriers to Improving Undergraduate STEM Education?

• Most commonly cited barriers:

– Lack of time

– Lack of training

– Lack of incentives

• Other crucial issues gaining attention:

– Systems of influence (Austin 2011)

– Conflicts with professional identity (Brownell and

Tanner, 2012)

teacher researcher

The systems of influence affecting each faculty member’s decision making and professional identity:

Diagram adapted from Austin, 2011

But Change is Possible

Curriculum and Pedagogy

are Disseminated for

Adoption

Development of Reflective

Teachers Who Choose

What to Change

Policies are Enacted to

Promote Change

Stakeholders Collectively

Develop Shared Vision

Modified from Henderson, et al., 2011, Journal of Research in Science Teaching, 48, 952-984

Four Categories of Change Strategies In

Prescriptive Emergent

Four Categories of Change Strategies

Adoption

Teachers Who Choose

What to Change

Promote Change

Less successful despite billions in

NSF Funding

Adoption

Teachers Who Choose

What to Change

Promote Change

Prescriptive Emergent Less successful – hierarchical “us”

vs. “them”

Adoption

Teachers Who Choose

What to Change

Promote Change

Successful but limited scope. May be only

option in contentious

settings

Adoption

Teachers Who Choose

What to Change

Promote Change

Successful, with greatest

likelihood for widespread

reform

A tool for formulating a Shared Vision: The STEM Department Evaluation Rubric

• Quickly build common language for dialogue

Polling Question 1

How confident are you that you could describe the meaning of the word “pedagogy”?

a. Very confident

b. Somewhat confident

c. Not confident at all

Polling Question 2

How confident were you at the start of your academic career that you could describe the meaning of the word “pedagogy”?

a. Very confident

Polling Question 3:

How confident are you that you could describe the meaning of the word “metacognition”?

a. Very confident

Polling Question 4:

How confident were you at the start of your academic career that you could describe the meaning of the word “metacognition”?

a. Very confident

• Focus on research-based factors that improve student outcomes and faculty efficacy

• Identify current strengths and opportunities for improvement

• Identify priorities & set aspirational goals from which to develop an action plan

• Build the grassroots leadership needed to implement the plan

STEM%Department%Evaluation%Rubric*

Baseline Beginning Developing Accomplished Exemplary

Authentic In-

class/lab

Research◊

Laboratory experiments have

known outcomes. Students

perceive that experiments have

"right" answers and will often

attribute unanticipated findings

to "human error."

Only a small fraction of students

gains practice in authentic, open-

ended inquiry, analysis, and

evaluation; usually through

optional, low enrollment

courses. These courses can

easily be avoided entirely.

Authentic inquiry occurs in

several courses; a majority of

students gain some experience

applying primary literature,

predicting outcomes of open-

ended experiments, analyzing

data, interpreting results, and

proposing next steps.

"Accomplished" builds on

"Developing" in that all students

have the opportunity to engage

in authentic inquiry, and the

majority design and carry out

novel, open-ended experiments,

often of their own design.

Students are accustomed to

ended research projects (often

of their own design) and

interpreting results. Articulating

probable explanations and

ensuing research questions is

the norm.

Student

Cognitive Skills

Across the curriculum, students

practice "recall," the lowest-level

cognitive skill (LOCS‡**), and

assignments/exams target this

level. Students' perception is

reinforced that learning is limited

to memorization of facts.

Students typically practice lower-

level cognitive skills (recall,

understand, apply), especially in

beginning courses. Few

instructors consider cognitive

level of assignments or exam

questions.

Students practice higher order

cognitive skills (HOCS; e.g.,

synthesize, evaluate, create) in

some courses, although tests

may still assess LOCS.

Instructors may find creating

HOCS questions difficult.

A good balance of LOCS and

HOCS in assignments in a

majority of courses with

opportunities for students to

practice HOCS assessed during

high-stakes assignments and

exams.

Students regularly practice

HOCS throughout the

curriculum, and instructors are

adept at giving students practice

in preparing for exams and other

graded assignments requiring

Student

Metacognitive

Skills

Students are unreflective of their

own learning strategies and

there is no effort to improve

metacognitive awareness and

empowerment. Attrition risk of

underprepared students is acute

due to unfamiliar early failure.

Rarely are students encouraged

to reflect on their learning

strategies and skills. Study

strategies, when discussed, may

not be specifically geared to

STEM learning or the particular

student's needs.

Students are encouraged in

some courses (e.g., first year

courses) to reflect on their

learning skills and encouraged

to use appropriate learning

strategies‡# that are supported

by research.€

Instructors typically engage

students (esp. first year) in

metacognitive reflection and

practice of research-based,

cognitive strategies. A learning

center may further support

student metacognitive growth.

Instructors regularly integrate

practice of effective

metacognitive strategies within

assignments. Most students

become adept at reflecting upon

and improving their own learning

and coaching younger peers.

Student Core

Competencies

Courses designed around

content delivery (e.g., chapter by

chapter of text), with no

opportunities to practice/build

core competencies.**

Course descriptions include goal

of building students' skills (e.g.,

use of scientific inquiry), but

students rarely practice such

skills. Top students who take

many STEM courses may build

competencies serendipitously.

Attempts to design curriculum

around core competencies

rather than content coverage

yield mixed success and/or face

some resistance. Efforts may be

limited to first year or senior

"capstone" experiences.

Core competencies are targeted

learning outcomes and practiced

in over 50% of courses across

all levels, although efforts may

still be confined within

disciplinary department(s) and

not integrated throughout, and

beyond, STEM.

Fully integrated curriculum

prioritizes competencies

(methods of inquiry, quantitative

reasoning, modeling/simulation,

transdisicplinary thinking,

communication, collaboration,

applying knowledge to civic

problems, etc.) at all levels.

*This rubric was developed for the Partnership for Life Sciences Education (PULSE) for use in workshops led by PULSE Fellows and for use by departments engaged in self-study at their home institution. It is intended as a brief guide

to stimulate discussion, identify department strengths and opportunities for improvement, and introduce just a few of the abundant resources about the topics. Your comments are welcome; contact Ellen Goldey at [email protected]. ◊ Wei and Woodin, 2011. "Undergraduate Research Experiences in Biology: Alternatives to the Apprenticeship Model," CBE-Life Sciences Education, 10, 123 - 131. See also Lopatto, 2010. "Science in Solution: The Impact of

Undergraduate Research in Student Learning," http://web.grinnell.edu/sureiii/Science_in_Solution_Lopatto.pdf, published by Research Corporation for Science Advancement.‡ Crowe, Dirks, and Wenderoth, 2008. "Biology in Bloom: Implementing Bloom’s Taxonomy to Enhance Student Learning in Biology," CBE Life Sciences Education, 7:368-371.

**Anderson and Krothwohl, 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Longmam Publishing.#Hoffmann and McGuire, 2009. "Teaching and Learning Strategies that Work," Science, 325:1203-1204.€ Dweck, 2006. Mindset: The New Psychology of Success, Random House Publishing

Independent

Research

Students are unaware of, and

don't participate in, independent

mentored research.

A few well-prepared students

may seek intra- or extramural

research opportunities (e.g.,

REU) on own initiative, but most

students are unaware of such

opportunities.

Ad hoc advising/encouragement

is given to top students to

consider research opportunities,

but better coordination would

extend such efforts to more

students.

Many students encouraged to

seek research opportunities, but

earlier preparation would

increase acceptance/readiness.

Products (e.g., posters) of

student research are

showcased.

From first year, all students are

prepped for mentored research

opportunities, and over 40%

participate in one or more.

Student research posters and

symposia regularly showcase

student scholars as role models.

Best Pedagogies

Lecturing without student

engagement is practice in

most/all courses and labs. All

knowledge is passively received,

so there is no need to read text

or other sources. Instructor is

"authority."

Traditional lecturing during class

time is the norm, and all

engaging activity occurs during

laboratory sessions. Information

received in class may often be

repeated in lab.

Instructor pedagogies fall into

teacher-centered or learner-

centered categories. Students

may pick among instructors who

"deliver" information and those

that require active learning.

All instructors are attempting to

adopt best pedagogical

practices,§ and lecturing for 50%

or more of class time is rare.

Students actively learn on own

and from each other in most

classes/labs.

Students rarely sit passively

listening to lectures, and they

are engaged in discussion,

guided inquiry, and other

activities in classes/labs.

Typically, knowledge is actively

constructed by students.

Instructor is "coach."

Faculty

Development

Faculty members are unfamiliar

with STEM/Higher Ed

pedagogical research, and there

is no structure/support/incentive

for development of their

knowledge and/or skills.

Some members of the

department are seeking new

knowledge/skills needed for

transforming their program, but

they lack support/time/incentive

for this work.

Faculty learning community

and/or Center for T&L may aid

cadre of practitioners in building

knowledge/skills. Administrative

support is minimally sufficient.

Faculty members discuss

relevant pedagogical literature

and own practices, and a few

may rarely publish own findings.

Incentives available to learn

through Center of T&L,

attendance at conferences, etc.

Pedagogical excellence is

esteemed by the institution and

contributing to the scholarship of

teaching and learning (SoTL) is

highly valued in T&P decisions.

Incentives support such work.

Assessment

Tools do not assess learning

outcomes (e.g., course

evaluations judge instructor

performance rather than student

learning/growth). Assessment

perceived as punitive and

compulsory.

Novel assessment tools may be

used in one or two courses, but

there is minimal administrative

and/or peer interest for these

efforts and findings, and the

focus remains on faculty

performance.

The assessment portfolio may

be narrow and lack the nuanced

insights from indirect methods

(surveys, interviews, etc.) in

favor of commercial, content-

based assessments (e.g., Major

Field Test).

Periodic (e.g., every 5 years)

integration and reflection on

variety of direct and indirect

assessment evidence inspires

episodic reform. Assessment

viewed as essential by some,

necessary evil by others.

Regular (e.g., yearly) reflection

on evidence from diverse

assessment tools guides

continuous efforts to improve

student outcomes. Assessment

perceived as essential and

inspiring.

Dispositions of

Faculty and

Administration

Faculty is change-averse. There

are no safe places for trial and

error. Changes in curriculum

may be dictated to the faculty

and driven by market forces.

There may be an ethos of fear,

suspicion, frustration, and/or

apathy.

Despite awareness of the need

to reform, the faculty and

administration are perceived to

be at odds, and/or there are

strong voices that resist change,

and/or there is poor

communication leading to inertia

and/or distrust.

Pockets of reform may be under

heightened scrutiny, thus

increasing anxiety.

Retrenchment may occur

without encouragement and

opportunities to learn from early

failures. The ethos may reflect

both anxiety and excitement.

A majority of faculty members

are collaborating with

administrators to implement

reform. Financial/market

realities are taken into

consideration, but do not solely

dictate approaches. An ethos of

pride is developing as learning

outcomes improve.

Instructors and administrators

are reflective, open to change,

appropriately skeptical of

change for change's sake, and

risk-tolerant. Research findings

and reflection on assessment

evidence drive continuous

reform. There is a collaborative

ethos of "positive restlessness."

§Bean, 2011. Engaging Ideas: The Professors Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom, Jossey-Bass Publisher; and Smith, Sheppard, Johnson and Johnson, 2005. "Pedagogies of

Engagement: Classroom-Based Practices," Journal of Engineering Education, 94:87-101.

** For sample discussions of core competencies, see the 2009 reports of the AAMC-HHMI Committee Scientific Foundations for Future Physicians and AAAS' Vision and Change in Undergraduate Biology Education: A Call To Action.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Authentic In-

class/lab

Research◊

to "human error."

the norm.

Student

Cognitive Skills

questions.

exams.

HOCS throughout the

Student

Metacognitive

Skills

student's needs.

by research.€

Student Core

Competencies

beyond, STEM.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent Researcher

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

Independent

Research

mentored research.

opportunities.

students.

showcased.

Best Pedagogies

"authority."

repeated in lab.

classes/labs.

Faculty

Development

with STEM/Higher Ed

Some members of the

for this work.

Assessment

compulsory.

performance.

Field Test).

inspiring.

Dispositions of

Faculty and

Administration

apathy.

and/or distrust.

increasing anxiety.

outcomes improve.

What might it mean to transform your program?

• It might mean moving from “Baseline” “Developing” on most factors.

• Or it might mean moving from “Baseline” to “Beginning” on some and “Developing” to “Exemplary” on a few.

• More will improve if there is shared vision, even as individuals work on different factors

Interactive Polling

• Which of the nine factors would you say your department is strongest? (select up to three)

• Which of the nine factors currently receive the least attention in your department? (select up to three)

• Which of the factors would be your top priority for improvement? (select up to three)

Leadership for STEM Reform

• Be resilient, patient, and empathetic.

• Be a role model – take risks, be vulnerable

• Set up assessment plan in advance of change

• Communicate – open up dialogue about successes and disappointments

• Be intentional in developing grassroots leaders – employ scholarship

• Set realistic but ambitious timelines

Recommended References Anderson, W., Banerjee, U., Drennan, C., Elgin, S., Epstein, I., Handelsman, J., Hatful, G.,

Losick, R., O'Dowd, D., Olovera, B., Strobel, S., Walker, G. and Warner I. (2011). Changing the culture of science education at research universities. Science, 331, 152 -153.

Austin, A. E. (2011). Promoting evidence-based change in undergraduate science education. A paper Commissioned by the National Academies National Research Council. http://dev.tidemarkinstitute.org/sites/default/files/documents/Use of Evidence in Changinge Undergraduate Science Education (Austin).pdf, last accessed December 13, 2013.

Brownell, S. E., & Tanner, K. D. (2012). Barriers to faculty pedagogical change: Lack of training, time, incentives, and...tensions with professional identity? CBE - Life Sciences Edcuation, 11, 339 - 346.

Henderson, C. Beach, A., and Finkelstein, N. (2011). Facilitating Change in Undergraduate STEM Instructional Practices: An Analytic Review of the Literature. Journal of Research in College Science Teaching, 48, 952-984.

Kezar, A. & Lester, J. (2009). Supporting faculty grassroots leadership. Research In Higher Education, 50, 715-740.

Pfund, C., Mathieu, R., Austin A., Connolly, M., Manske, B., & Moore, K. (2012). Advancing STEM undergraduate learning: Preparing the nation's future faculty. Change: The Magazine of Higher Learning, 44(6), 64-72.

Savkar, V. and Lokere, J. (2010). Time to Decide: The Ambivalence of the World of Science Toward Education, Nature Education: Cambridge, MA.

Seymour, E., DeWelde, K. and Fry, C. (2011). Determining Progress in Improving Undergraduate STEM Education: The Reformer's Tale. White Paper Commissioned by the National Academy of Engineeringfor the Forum, "Characterizing the Impact and Diffusion of Engineering Education Innovations."

Questions/suggestions [email protected]

Ellen Goldey

The STEM Department Evaluation Rubric · The STEM Department Evaluation Rubric ... (Brownell and...

Documents

Transcript of The STEM Department Evaluation Rubric · The STEM Department Evaluation Rubric ... (Brownell and...