Using  Rubrics  for  GE  and  Major  Assessment  Repor9ng  

Caroline  Breitenberger  Larry  Krissek  

2015  Campus-­‐Wide  Conference  on  Assessment  

Broad  learning  goal:      Par9cipants  will  learn  how  to  develop  and  use  rubrics  for  evalua9ng  and  repor9ng  data  about  course  and  program  learning  outcomes.    

Specific  learning  outcomes:      Par9cipants  will  be  able  to:  

 Define  “rubric”      Design  a  basic  rubric    Explain  how  rubrics  may  be  useful  for  repor9ng      assessment  results  and  for  program  improvement    Describe  a  range  of  different  types  of  rubrics      

Uses  of  Assessment  Results  

Outcomes  Assessment  Results  

Course  improvement  

Program  modifica4on  

Accredita4on  

Inform  stakeholders  

Improve  student  learning  Feedback  to  

students  

Undergraduate  Program  Assessment  in  the  College  of  Arts  and  Sciences  at  Ohio  State  •  General  Educa9on  program  – Assigned  to  ASC  Curriculum  CommiUee  – Handled  by  ASCC’s  Assessment  Panel  – Historically  done  by  course  reports;  some  department-­‐level  reports  now  used  

–  Reports  submiUed  when  requested;  data  should  be  collected  regularly  

•  Major  program  – Assigned  to  ASC  college-­‐level  administrators  –  Reports  submiUed  annually  

Case:  Frustrated  Department  Chair  

Now  the  college  has  asked  me  to  prepare  a  report  about  how  our  students  meet  their  program  learning  outcomes.  How  can  I  get  our  faculty  to  do  this?    

Our  courses  obviously  address  the  program  learning  outcomes!  Why  can’t  we  just  report  student  grades?  

 

hUp://z.about.com/d/learningdisabili9es/1/0/x/0/-­‐/-­‐/frustrated_teacher.jpg  

What  is  a  Rubric?  •  You  may  have  used  a  rubric  for  grading,  like  this  one,  for  

grading  a  physics  lab  notebook:  

•  Or  this  one,  for  grading  a  history  paper:  

hUp://www.uwstout.edu/soe/profdev/upload/Krubric.pdf;  hUp://www.cmu.edu/teaching/designteach/teach/rubrics.html  

What  is  a  Rubric?  •  A  rubric  is  a  descrip4on  of  specific  performance  standards  that  can  be  used  to  provide  consistency  in  evalua4ng  …  almost  anything  

–  Student  papers  –  Presenta9ons  –  Facili9es  –  Adherence  to  safety  guidelines  –  etc.  

Developing  a  Rubric  for  Assessment  Repor4ng  (GE  Educa9on  Abroad  Rubric  as  an  example)  

•  Faculty  establish  expected  learning  outcomes  (ELOs),  and  outline  a  rubric  aligned  to  the  ELOs    –  Other  stakeholders  may  provide  input  (e.g.  Office  of  Interna9onal  

Educa9on,  Service  Learning  Ini9a9ve,  accredi9ng  bodies)      Expected  Learning  Outcomes:  1.  Students  recognize  and  describe  similari4es,  differences,  and  interconnec4ons  between  their  host  country/countries  and  the  U.S.    2.  Students  func4on  effec4vely  within  their  host  country/countries.    3.  Students  ar9culate  how  their  4me  abroad  has  enriched  their  academic  experience.  

    Capstone  (4)   Milestone  (3)   Milestone  (2)   Benchmark  (1)  (ELO1)  Knowledge  of  host  country  and  US:  Culture  and  worldview  frameworks  (ELO2)  Skills  for  effec4ve  func4oning:  a)  Verbal  and  

nonverbal  communica4on    

b)  Problem  solving  

(ELO3)    Enrichment  of  academic  experience:  a)  Knowledge  

b)  Skills  

c)  ATtudes/  Perspec4ves  

     

Developing  a  Rubric  for  Assessment  Repor4ng  (GE  Educa9on  Abroad  Rubric  as  an  example)  

•  Faculty  establish  expected  learning  outcomes  (ELOs),  and  outline  a  rubric  aligned  to  the  ELOs    

•  Define  standards  for  student  achievement  for  each  ELO  

 

hUp://www.assessment.uconn.edu/primer/taxonomies1.html  Describe,  define,  iden4fy,  recognize…  

Compare,  discuss,  interpret,  restate…  

Apply,  demonstrate,  model,  solve…  

Analyze,  categorize,  priori4ze,  test…  

Compile,  design,  integrate,  predict…  

Assess,  cri4cize,  recommend,  validate…  

Examples  of  verbs  that  might  be  used  at  various  levels  of  Bloom’s  taxonomy:  

Developing  a  Rubric  for  Assessment  Repor4ng  (GE  Educa9on  Abroad  Rubric  as  an  example)  

•  Faculty  establish  expected  learning  outcomes  (ELOs),  and  outline  a  rubric  aligned  to  the  ELOs    

•  Define  standards  for  student  achievement  for  each  ELO,  and  desired  level  of  achievement  

   

    Capstone  (4)   Milestone  (3)   Milestone  (2)   Benchmark  (1)  

(ELO1)  Knowledge  of  host  country  and  US:  Culture  and  worldview  frameworks  

Ar4culates  sophis4cated  understanding  of  differences,  similari9es,  and  interconnec9ons  between    cultural  rules  and  prac9ces  of  host  country  and  US.  

Demonstrates  deeper  understanding  of  differences,  similari9es,  and  interconnec9ons  between    cultural  rules  and  prac9ces  of  host  country  and  US.  

Describes  similari9es  and  differences  and  recognizes  interconnec9ons  between  cultural  rules  and    prac9ces  of  host  country  and  US.  

Recognizes  similari9es  and  differences  in  cultural  rules  and  prac9ces  between  host  country  and    US.  

    Capstone  (4)   Milestone  (3)   Milestone  (2)   Benchmark  (1)  

(ELO2)  Skills  for  effec4ve  func4oning:      (a)  Verbal  and  nonverbal  communica4on                                (b)    Problem  solving  

   Ar4culates  a  complex  understanding  of  cultural  differences  in  verbal  and  nonverbal  communica9on    between  host  country  and  US.  Is  able  to  skillfully  nego4ate  a  shared  understanding  based  on  those    differences.              Navigates  host  country  and  overcomes  obstacles  with  confidence  and  ingenuity.  

   Shows  higher  level  understanding  of  cultural  differences  in  verbal  and  nonverbal  communica9on    between  host  country  and  US.  Begins  to  nego4ate  a  shared  understanding  based  on  those  differences.        Navigates  host  country  and  overcomes  obstacles  comfortably.  

   Shows  basic  level  understanding  of  cultural  differences  in  verbal  and  nonverbal  communica9on    between  host  country  and  US.  Shows  awareness  that  misunderstandings  across  cultures  can  occur.        Navigates  host  country  and  overcomes  obstacles  at  basic  level.  

   Shows  minimal  level  understanding  of  cultural  differences  in  verbal  and  nonverbal  communica9on    between  host  country  and  US.                        Struggles  to  navigate  host  country  at  basic  level  and  to  overcome  obstacles.                      

Developing  a  Rubric  for  Assessment  Repor4ng  (GE  Educa9on  Abroad  Rubric  as  an  example)  

•  Faculty  establish  expected  learning  outcomes  (ELOs),  and  outline  a  rubric  aligned  to  the  ELOs    

•  Define  standards  for  student  achievement  for  each  ELO  

•  Decide  on  data  to  be  collected  – Direct  measures:  standardized  tests,  classroom  assignments  (pre-­‐  and  post-­‐tests,  embedded  ques9ons),  evaluated  body  of  work  (porjolio,  capstone  project)    

–  Indirect  measures:  surveys  and  interviews,  syllabus  review,  grade  review  

   Suggested  prompt  for  an  end-­‐of-­‐course  student  reflec9on  paper:      

Please  write  a  thoughjul  four-­‐page  (double-­‐spaced,  typed)  reflec9on  paper  that  considers  the  following  aspects  of  your  Educa9on  Abroad  experience:    1.  What  are  the  most  striking  similari9es,  differences,  and  interconnec9ons  you  have  encountered  between  your  host  country/countries  and  the  U.S.?  

2.  How  have  you  developed  an  ability  to  func9on  effec9vely  within  your  host  country/countries?  (Think  of  the  ways  you  communicated  verbally  and  non-­‐verbally  in  your  host  country  and  how  you  overcame  challenges.)    3.  Overall,  how  has  your  Educa9on  Abroad  experience  enriched  your  academic  experience?  Use  concrete  examples.  

 

•  required  for  assessment  purposes    •  the  instructor  may  (or  may  not)  choose  to  include  this  assignment  as  one  of  the  

assignments  a  student  completes  for  his/her  final  grade.  

GE  Educa4on  Abroad  Category  requires  an  end-­‐of-­‐course  assignment  which  can  take  many  forms  (paper,  video,  blog,  etc.)  

Addi4onal  Notes  Developing  a  Rubric  for  Assessment  Repor9ng  

 

•  Faculty  establish  expected  learning  outcomes  (ELOs),  and  outline  a  rubric  aligned  to  the  ELOs  (other  stakeholders  may  provide  input)  

•  Define  standards  for  student  achievement  for  each  ELO  •  Decide  on  data  (direct  and  indirect  measures)  to  be  collected  

•  UCAT  consultants  can  facilitate  process    •  Some  kind  of  standardiza4on  among  evaluators  may  be  valuable  

Report  examples  

•  Educa9on  Abroad  category  •  One  course  is  targeted  to  freshmen,  who  go  to  England  for  one  week;  other  is  a  capstone  course  for  students  pursuing  a  specific  minor,  who  go  to  Europe  for  3  weeks  

•  Both  courses  used  the  Educa9on  Abroad  assessment  rubric  to  report  assessment  results  

    Capstone  (4)   Milestone  (3)  Milestone  (2)   Benchmark  (1)  (ELO1)  

Students  recognize  and  describe  similari4es,  differences,  and  interconnec4ons  between  their  host  country/countries  and  the  U.S.  

   

6    

19    

   

(ELO2)  

Students  func4on  effec4vely  within  their  host  country/countries.  

   

13    

12    

(ELO3)    

Students  ar4culate  how  their  4me  abroad  has  enriched  their  academic  experience.  

2        

7    

16    

Freshmen  –  one  week  in  London  (25  students)  

    Capstone  (4)   Milestone  (3)  Milestone  (2)   Benchmark  (1)  (ELO1)  

Students  recognize  and  describe  similari4es,  differences,  and  interconnec4ons  between  their  host  country/countries  and  the  U.S.  

 1  

6  11  

19  1  

 1  

(ELO2)  

Students  func4on  effec4vely  within  their  host  country/countries.  

 8  

13  3  

12  3  

(ELO3)    

Students  ar4culate  how  their  4me  abroad  has  enriched  their  academic  experience.  

2      6  

7  8  

16    

Freshmen  –  one  week  in  London  (25  students)  Capstone  course  –  three  weeks  in  Europe  (14  students)  

Instructors’  narra9ves  “The  second  theme  was  that  English  memory  [of  WWII]  …  concentrated  on  the  local  and  quo9dian  experiences  of  the  war,  rather  than  on  high-­‐blown  ideological  or  poli9cal  elements  …”    

Proposed  course  changes  to  provide  “…  greater  congruence  between  pre-­‐departure  course  materials  and  planned  experiences  in  London.”    Freshmen  –  one  week  in  London  (25  students)  Capstone  course  –  three  weeks  in  Europe  (14  students)  

Major  program  assessment  -­‐  WGSS  Women’s    Gender  and  Sexuality  Studies  learning  goals:  

1.  Student  will  be  able  to  think  cri9cally  about  women’s  issues,  gender  and  sexuality  

2.  Student  will  be  able  to  pursue  interdisciplinary  inquiries  about  gender  and  sexuality  and  to  pose  ques9ons  that  cut  across  disciplinary  fields  

3.  Student  will  be  able  to  perform  women’s,  gender  and  sexuality  studies  research  through  developing  an  argument,  organizing  data  and  evidence  for  that  argument,  and  expressing  ideas  in  wri9ng  

•  Sample  of  student  papers  from  capstone  course:  WGSS  4575,  Senior  Seminar  

•  Three  faculty  members  who  serve  on  undergraduate  studies  commiUee  review  these  papers  using  rubric  

(1)    No  

(2)    Basic  

(3)  Intermediate  

(4)  Advanced  

Student  will  be  able  to  think  cri4cally  about  women’s  issues,  gender,  and  sexuality  

Shows  no  comprehension  of  concepts;  liUle  

knowledge  of  gender  and  sexuality

Shows  comprehension  of  concepts  related  to  

gender  and  sexuality  and  some  applica9on  of  ideas  

Shows  comprehension  of  concepts  related  to  gender  and  sexuality,  

applies  and  analyzes  ideas  

Shows  comprehension  of  concepts  related  to  gender  and  sexuality;  applies,  analyzes  and  

evaluates  ideas Student  will  be  able  to  pursue  interdisciplinary  inquiries  about  gender  and  sexuality  and  to  pose  ques4ons  that  cut  across  disciplinary  fields  

Does  not  connect  various  disciplines  or  understand  gender  and  sexuality  as  

interdisciplinary  

Recognizes  various  disciplines  with  a  focus  on  gender  and  sexuality  but  does  not  make  clear  

connec9ons

Connects  various  disciplines  with  an  ar9culated  focus  on  gender  and  sexuality

 

Synthesizes  various  disciplines  through  an  

ar9culated  and  analy9cal  focus  on  gender  and  

sexuality Student  will  be  able  to  perform  women’s,  gender  and  sexuality  studies  research  through  developing  an  argument,  organizing  data,  displaying  evidence  for  that  argument,  and  expressing  ideas  in  wri4ng  

Does  not  provide  clear  arguments  or  suppor9ve  

evidence;  does  not  express  ideas  clearly

Describes  arguments  and  organizes  data  but  lacks  suppor9ve  evidence  and  clear  wriUen  expression

Demonstrates  an  ability  to  develop  arguments  

supported  with  evidence  and  clear  wriUen  

expression

Ar9culates  sophis9cated  arguments,  analyzes  

interconnected  evidence  and  develops  clear  discussion  of  ideas

Examples  of  Repor9ng  Rubrics  •  AAC&U  VALUE  (Valid  Assessment  of  Learning  in  

Undergraduate  Educa9on)  Intercultural  Knowledge  and  Competence  rubric  

•  PULSE  (Partnership  for  Undergraduate  Life  Sciences  Educa9on)  Curriculum  Alignment  rubric  

•  Ohio  State  University  GE  Service  Learning  assessment  rubric  

•  Rutgers  –  21st  Century  Challenges  rubric  •  Maryland  –  Scholarship  in  Prac9ce  rubric  

In  your  group,  pick  one  rubric  per  person,  and  discuss  the  rubrics  with  each  other;  considering:  What  sort  of  program  could  be  assessed  using  this  rubric?  

Case:  Frustrated  Department  Chair  

•  Faculty  ar9culate  learning  outcomes  and  differen9ate  various  levels  of  student  achievement  of  expected  learning  outcomes  

•  Communicate  shared  expecta9ons  for  students  to  demonstrate  that  they  achieve  expected  learning  outcomes  

•  Align  student  learning  ac9vi9es  with  expected  learning  outcomes  •  Establish  common  metrics  which  allow  flexibility  of  course  design  

and  student  work  •  Opportunity  for  faculty,  students,  and  administrators  to  discuss  

progress  toward  and  achievement  of  expected  learning  outcomes  •  Opportunity  for  informed  course  and  program  improvement        

hUp://z.about.com/d/learningdisabili9es/1/0/x/0/-­‐/-­‐/frustrated_teacher.jpg