Dr.  Mary  Herrington-­‐Perry  Office  of  Assessment  &  Accredita:on    April  5th,  2017  Faculty  Center  for  Teaching  Excellence  

Presented  By  

Wri:ng  an  Exemplary  Student  Learning  Summary  Report  

Assump/ons  1.  Your  program  already  has  learning  outcomes,  a  curriculum  

map,  and  an  assessment  plan.  .  .  right?  2.  You  have  assessed  or  will  assess  one  or  more  of  your  

program’s  learning  outcomes  in  2016-­‐17.  3.  You  have  analyzed  or  will  analyze  the  results  of  these  

assessments.  4.  You  have  shared  and  discussed  the  results  with  the  

program’s  faculty  (or  will  do  so  before  the  semester’s  end).  5.  And  thus  you  will  be  ready  to  write  the  Student  Learning  

Summary  Report–  which  is  based  on  items  1-­‐4–  by  your  dean’s  deadline  .  .  .  right?  

Student  Learning  Summary  Report  Part  One:  Summary  of  Assessment  Ac/vi/es  

 

1.  What  learning  outcome(s)  did  you  assess  this  year?    2.  What  assignments  or  ac:vi:es  did  you  use  to  

determine  how  well  your  students  aXained  the  outcome?    

3.  What  expecta:ons  did  you  establish  for  students’  achievement  of  the  outcome?    

4.  What  were  the  actual  results?    5.  Who  was  responsible  for  collec:ng  and  analyzing  the  

results?  How  were  they  shared  with  the  program’s  faculty?    

1.  Learning  Outcomes  Student:  What  exactly  do  you  expect  from  me?    

Instructor:  I  want  you  to  be  able  to  demonstrate  proficiency  in  using  geological  research  tools,  including  Geographic  Informa:on  Systems  and  Global  Posi:oning  Systems.  

Outcomes:  Examples  •  Define  the  terminology  of  tax  accoun:ng.  •  Ar/culate  the  historical  assump:ons,  central  ideas,  and  dominant  

cri:cisms  of  the  key  drama:c  periods  using  illustra:ons  from  per:nent  plays.  

•  Apply  basic  surveying  techniques  for  construc:on  layout  and  control.    

•  Iden/fy  the  mo:va:on  behind  behavioral  issues  and  develop  interven:ons  to  address  inappropriate  behaviors.  

•  Conduct  a  comprehensive  and  systema:c  assessment  of  health  and  illness  parameters  in  complex  situa:ons.  

•  Communicate  ideas  clearly  and  coherently  in  wriXen  work  and  oral  presenta:ons.                      

Which  is  the  stronger  program-­‐level  learning  outcome  .  .  .  and  why?  

A.  Students  will  ar/culate  and  apply  ethical  principles  to  real  world  situa:ons.  

B.  Students  will  understand  ethical  principles  in  real  world  situa:ons.  

Outcomes:  More  Examples  Knowledge/Comprehension:  Defines  advanced  prac:ce  competencies  for  selected  popula:ons.    Analysis:  Analyzes  the  significance  of  advanced  nursing  knowledge  as  it  relates  to  selected  popula:ons.      Applica/on:  Provides  safe,  cost-­‐effec:ve,  and  culturally  adap:ve  advanced  prac:ce  nursing  for  special  popula:ons.      Evalua/on:  Evaluates  outcomes  of  advanced  prac:ce  nursing  interven:ons,  methods,  or  strategies.      Crea/on:  Synthesizes  theore:cal  frameworks  used  in  the  integra:on  of  knowledge  from  related  sciences  and  humani:es,  clinical  knowledge,  and  nursing  sciences  as  the  founda:ons  of  advanced  nursing  prac:ce.    

Exemplary  Learning  Outcomes  .  .  .  1.  Are  specific,  measurable,  student-­‐centered,  program-­‐

level  outcomes  that  span  mul:ple  learning  domains.  2.  Directly  integrate  with  Founda:onal  Studies  learning  

outcomes  or  Graduate  Learning  Goals.  3.  Reflect  the  most  important  results  of  program  

comple:on.  4.  Are  consistent  across  different  modes  of  delivery.  5.  Are  regularly  reviewed  (and  revised,  if  necessary)  by  the  

faculty  and  other  stakeholders.  6.  Are  embedded  in  courses  and  experiences  across  the  

curriculum  so  that  students  have  sufficient  opportunity  to  master  them.  

2.  Assessment  Measures  Learning  Outcome:  Explain  the  central  ideas  and  dominant  cri:cisms  of  the  key  drama:c  periods  using  illustra:ons  from  per:nent  plays.    Poten/al  Measures:  •  Act  as  a  spokesperson  for  one  of  the  drama:c  periods  and  par:cipate  in  a  debate  

about  which  one  is  superior,  based  on  the  key  works  wriXen  in  that  period.  •  Write  and  illustrate  a  graphic  novel  that  explores  one  of  the  drama:c  periods.  •  Take  on  the  role  of  literary  cri:c  and  present  the  strengths  and  weaknesses  of  one  

of  the  periods,  based  on  its  major  works.  •  Write  an  original  play  that  incorporates  the  key  literary  devices,  topics,  and  style  

of  a  selected  drama:c  period.  •  Complete  a  class  evalua:on  that  asks  them  to  rate  how  well  they  learned  course  

concepts.  

Examples  of  Measures  Direct  Measures  •  Case  studies  •  Clinical  experiences  •  Discussion  boards    •  Exams/Licensure  Exams*  •  Exhibi:ons  •  Internships  •  Lab  reports  •  Lesson  plans  •  Performances  •  Porholios  •  Presenta:ons  (solo,  group)  •  Projects  (solo,  group)  •  Research  papers  

Indirect  Measures  •  Alumni  surveys  •  Exit  interviews/surveys  •  Focus  groups  (students,  alums,  

employers)  •  Graduate  school  placement  •  Job  placement  rates  •  Papers  accepted  for  publica:on  •  Research  presented  at  

conferences  

Direct  or  Indirect?  A.  Licensure  examina:on  B.  Focus  group    C.  Membership  in  a  professional  organiza:on    D.  Group  presenta:on  of  culmina:ng  project  E.  Lab  report  F.  Student  teaching  experience  G.  Case  study  H.  Senior  recital  I.  Take-­‐home  exam  J.  Capstone  course  pass  rate  

Exemplary  Assessment  Measures  .  .  .  1.  Are  (mostly)  direct.  2.  Provide  a  complete  picture  of  student  learning.  3.  Are  clear  enough  to  show  that  they  are  appropriate  measures  of  

the  outcomes  being  assessed.  They  ALIGN  with  the  measures.  4.  Include  high-­‐impact  prac:ces  and  ac:vi:es  that  mirror  what  

future  employers  will  expect  them  to  be  able  to  do.  5.  Allow  performance  to  be  gauged  over  :me,  not  just  in  a  single  

course.  6.  Are  assessed  by  rubrics,  checklists,  keys,  etc.  that  provide  specific  

evidence  of  what  students  know  and  can  do  (and  don’t  know/can’t  do).  

7.  Are  accompanied  by  clear,  specific,  and  appropriate  expecta:ons  for  performance.    

3.  Performance  Expecta/ons    

•  At  least  70%  of  students  will  earn  a  score  of  75%  on  the  comprehensive  exam  (which  is  coded  to  outcomes).  

•  100%  of  students  will  earn  a  score  on  the  licensure  exam  that  is  equal  to  or  beXer  than  the  na:onal  average  (95%).  

•  90%  of  respondents  will  rate  their  sa:sfac:on  with  the  program  at  “4”  (sa:sfied)  or  higher  on  a  five-­‐point  scale.  No  respondents  will  rate  their  sa:sfac:on  below  a  “3”  (somewhat  sa:sfied).  

•  The  average  GPA  of  majors  will  increase  by  1%  a  semester  for  three  years,  beginning  with  spring  2017  (2.25).  

•  Students  will  aXend  at  least  three  career  development  workshops  in  their  senior  year.  

Tying  the  Pieces  Together,  #1    •  Outcome:  Analyze,  interpret,  and  cri:cally  assess  empirical  

evidence  in  sociological  research.    •  Measure:  Oral  presenta:on  analyzing  the  misuses  of  

sta:s:cs/data  in  print  media  (SOC  3XXX).  Presenta:ons  are  assessed  through  primary  trait  analysis.    

•  Performance  Expecta/on:  Students  will  earn  a  minimum  ra:ng  of  3  out  of  5  (competent)  in  each  of  four  categories  on  the  rubric  (content,  organiza:on,  presenta:on  skills,  technical  skills).  The  content  sec:on  specifically  looks  at  students’  ability  to  iden:fy  and  analyze  misrepresenta:ons  of  sta:s:cs.  

4.  Results    A.  You’re  assessing  students’  knowledge  of  key  concepts  via  a  

comprehensive  exam  delivered  in  the  senior  capstone  course,  and  the  performance  goal  is  that  90%  of  students  will  earn  a  score  of  at  least  75%.  You’ll  collect:  

 

–  Each  student’s  overall  score  and  his  or  her  score  on  each  of  the  outcome-­‐specific  exam  ques:ons  

 

B.  You’re  assessing  students’  oral  communica:on  competency  via  a  presenta:on  assigned  in  mul:ple  sec:ons  and  assessed  by  a  common  rubric,  and  your  performance  expecta:on  is  that  100%  of  students  will  earn  a  minimum  ra:ng  of  3  out  of  5  (competent)  in  each  of  four  categories  on  the  rubric.  You’ll  collect:    

–  Copies  of  the  completed  rubrics,  whose  scores,  sub-­‐scores,  and  key  comments  you’ll  enter  in  a  spreadsheet    

Using  a  Meta-­‐Rubric  to  Track  Results  

 

Dimension   Organiza/on  (10  points)    

Content  &  Development  (50  points)    

Grammar  &  Mechanics  (30  points)    

 Audience  Awareness  (10  points)    

Ave.  Score  on  Assignment  10,  Fall  2016  (N=48)    

8  (80%)    

38  (76%)    

20  (67%)    

5  (50%)    

Ave.  Score  on  Assignment  10,  Spring  2017  (N=60)    

9  (90%)    

42  (84%)    

23  (77%)    

7  (70%)    

Tying  the  Pieces  Together  #2    •  Goal:  Students  will  earn  a  minimum  ra:ng  of  3  out  of  5  

(competent)  in  each  of  four  categories  on  the  rubric  .  .  .  .    –  Results:  75%  of  students  met  or  exceeded  the  benchmark  (N=38),  so  the  goal  was  not  aXained.  (Detailed  results  are  available  here.)  

 •  Goal:  80%  of  students  will  pass  the  Indiana  Principal  Licensure  

Exam,  which  is  a  requirement  for  Na:onal  Recogni:on  through  our  Accredita:on  Agency.  –   Results:  The  goal  was  met:  48  of  54  (91%)  candidates  passed  the  exam  in  the  period  from  June  1,  2015  to  May  24,  2016.  (Detailed  results  are  available  here.)  

 

A  Note  About  Results    What  the  numbers  mean  is  even  more  

important  than  the  numbers  themselves.      Good  results  answer  the  ques:on,  “What  specifically  did  assessment  help  you  discover  about  student  learning?”  (To  be  con:nued  in  Part  Two.)      

 

Ques/ons  About  Part  1  of  the  SLSR?  

SLSR  Part  Two:    Engagement  &  Improvement  

 

1.  What  discoveries  has  assessment  has  enabled  you  to  make  about  student  learning,  as  well  as  about  the  curriculum,  departmental  processes,  your  assessment  plan,  etc.?    

2.  What  changes  and  improvements  will  you  make  in  response  to  your  discoveries  (or  the  assessment  coordinator’s  feedback  on  your  previous  SLSR)?  

3.  What  will  your  assessment  plan  focus  on  in  the  coming  year?  

Ge_ng  Down  to  the  Details  Organiza/on  (10  points)  

Content  &  Development  (50  points)  

Grammar  &  Mechanics  (30  points)  

Audience  Awareness  (10  points)  

Average  Score  on  Assignment  10,  Fall  2016  (N=48)  

8  (80%)   38  (76%)   20  (67%)   5  (50%)  

Strengths    Thesis  and  topic  sentences  generally  clear      Transi:ons  typically  effec:ve  

Claims/ideas  supported  with  evidence  from  experts  

Fewer  comma  splices  and  agreement  errors  than  at  beginning  of  semester  J  

Recognizes  that  audience  may  have  a  different  posi:on  on  the  issue  

Needed  Improvements  

Structure  immature/predictable  (5-­‐paragraph  essays)  

Analysis  lacks  higher-­‐order  skills  such  as  synthesis  and  reflec:on  

Immature  sentence  structure      Weak  ability  to  paraphrase    

Argument  insufficient  to  change  audience’s  posi:on  

Tying  the  Pieces  Together  #3  •  Goal:  Students  will  earn  a  minimum  ra:ng  of  3  

(competent)  .  .  .  .    •  Results:  .  .  .  thus  the  goal  was  not  aXained.  .  .  .  •  Analysis  and  Plans  for  Improvement:  While  students  did  not  

achieve  the  benchmark,  results  con:nue  to  show  that  they  perform  beXer  in  the  spring  than  in  the  fall.  The  faculty  believes  spring  students  have  an  advantage  because  they  already  have  completed  SOC  XXX,  which  requires  them  to  make  both  individual  and  group  presenta:ons,  and  that  these  students  consequently  are  more  confident  speakers  with  beXer  teamwork  skills.  Beginning  in  2017,  SOC  3XX  will  become  a  spring-­‐only  course.  If  student  performance  improves,  we  will  adjust  workloads  permanently.  

Tying  the  Pieces  Together  #4  •  Goal:  70%  of  students  will  earn  a  score  of  75%  or  beXer  .  .  .  .    •  Results:  .  .  .  Overall,  the  goal  was  met.  •  Analysis  and  Plans  for  Improvement:  When  all  three  

ques:ons  are  collec:vely  considered,  the  75%  benchmark  was  not  met.    Only  20  students  (64.52%)  achieved  the  target  score  of  33.75.    More  problems  were  added  to  the  curriculum  this  year,  and  during  the  next  academic  year,  flipping  the  class  will  be  aXempted  to  encourage  students  to  take  charge  of  their  learning.    Approximately  50%  more  :me  will  be  spent  working  problems  in  class.    

Tying  the  Pieces  Together  #5  •  Goal:  Seniors  will  achieve  an  average  of  3.9/5.0  on  the  preceptor  evalua:on.  .  .  .    •  Results:  Seniors  achieved  a  mean  score  of  4.41/5.0  .  .  .  All  standards  were  met.  

•  Analysis  and  Plans  for  Improvement:  Because  results  show  that  preceptors  rate  students’  knowledge  and  clinical  skills  higher  than  faculty  do,  :me  will  be  given  during  preceptor  development  this  summer  to  ensure  that  all  preceptors  are  evalua:ng  students  in  a  similar  manner.  It  is  interes:ng  that  students  overwhelmingly  scored  higher  on  prac:cal  skills  examina:ons  than  standardized  pa:ent  evalua:ons.  Prac:cal  skills  examina:ons  focus  on  isolated  bits  of  knowledge  and  clinical  skill.  The  standardized  pa:ent  evalua:ons  are  real-­‐:me  pa:ent  encounters  where  students  have  to  incorporate  several  components  of  knowledge  and  skill  into  pa:ent  care  (similar  to  what  clinical  prac:ce  is  really  like).  These  findings  highlight  that  more  :me  should  be  spent  in  the  classroom  and  laboratory  serngs  to  allow  students  to  incorporate  mul:ple  components  of  pa:ent  care  in  a  manner  more  similar  to  clinical  prac:ce  (e.g.,  teaching  standardized  pa:ents,  simula:ons  with  task  trainers  .  .  .  .).  

Exemplary  Engagement  &  Improvement  

1.  Involves  all  program  faculty  in  collec:ng  and  analyzing  results.  2.  Provides  clear,  specific,  and  complete  details  about  data  collec:on,  

analysis,  and  interpreta:on  to  demonstrate  the  validity  and  usefulness  of  the  assessment  process.    

3.  Shows  that  students  generally  are  achieving  the  performance  standards  expected  of  them  and  demonstrate  con/nuous  improvement  on  those  they  have  yet  to  achieve/achieve  less  well.  

4.  Entails  regular  and  specific  reflec:on  from  stakeholders  on  students’  recent  achievement  of  performance  goals  and  the  history  and  impact  of  previous  plans,  ac:ons,  and  results.  

5.  Results  in  the  development  and  implementa:on  of  per:nent  recommenda:ons  for  improvement  in  student  learning  (and  also  outcomes,  assessment  measures,  performance  goals,  data  collec:on  processes,  etc.).  

6.  Is  visible.    

Related  Requirements  of  the  Higher  Learning  Commission  

•  The  ins:tu:on  has  clearly  stated  goals  for  student  learning  and  effec:ve  processes  for  assessment  of  student  learning  and  achievement  of  learning  goals.  

•  The  ins:tu:on  assesses  achievement  of  the  learning  outcomes  that  it  claims  for  its  curricular  and  co-­‐curricular  programs.  

•  The  ins:tu:on  uses  the  informa:on  gained  from  assessment  to  improve  student  learning.  

•  The  ins:tu:on’s  processes  and  methodologies  to  assess  student  learning  reflect  good  prac:ce,  including  the  substan:al  par:cipa:on  of  and  other  instruc:onal  staff  members.  

   

Ques/ons  About  Part  2?  

Mary.Herrington-­‐Perry@indstate.edu  

hXps://www.indstate.edu/assessment  (forms,  direc:ons,  sample  reports)