STREAMLINING PROGRAM LEARNING OBJECTIVE ASSESSMENT … · 4 SeniorThesis M,"A" M,"A" M,"A"...
Transcript of STREAMLINING PROGRAM LEARNING OBJECTIVE ASSESSMENT … · 4 SeniorThesis M,"A" M,"A" M,"A"...
STREAMLINING PROGRAM LEARNING OBJECTIVE ASSESSMENT WITH TARGETED ACTIVITIES AND DESCRIPTIVE RUBRICS Carrie Menke, Ph.D. Physics, LPSOE University of California, Merced
Abstract • The undergraduate physics program has five program learning
objectives (PLOs): (1) physical principles, (2) mathematical expertise, (3) experimental technique, (4) communication and teamwork, and (5) research proficiency. With one PLO assessed each year, we have just completed our first cycle. Our approach strives to maximize the ease and applicability of our assessment practices while maintaining faculty's flexibility in course design and delivery. Objectives are mapped onto the core curriculum and identified coursework is collected as direct evidence. We have found that descriptive rubrics lend themselves to more efficient assessment, higher inter-rater reliability, and can be applied to course and program-level assessment. The presentation will outline our progress, success with a descriptive presentation rubric, and our ongoing work with rubrics applied to more abstract PLOs that utilize students' written work.
2 Streamlining Program Assessment Research Week 2014
Outline • Assessment Cycle • Physics PLOs • Curriculum Matrix & Signature Assignments • Rubrics
• Developing: Types & Sources • Refining: Inter-rater reliability • Applying: Course and Program Assessment
• Successes • Challenges
3 Streamlining Program Assessment Research Week 2014
Assessment Cycle
4
Hybrid of Suskie, CIRTL Network, Wiggins & McTighe
Laura Martin, Introduction to Assessment Presentation, CGS Workshop, January 2013
Streamlining Program Assessment Research Week 2014
Physics PLOs 1. Physical Principles. Students will be able to apply basic physical principles—including
classical mechanics, electricity and magnetism, quantum mechanics, and statistical mechanics—to explain, analyze, and predict a variety of natural phenomena.
2. Mathematical Expertise. Students will be able to translate physical concepts into
mathematical language. Furthermore students will be able to apply advanced mathematical techniques (e.g., calculus, linear algebra, probability, and statistics) in their explanations, analyses, and predictions of physical phenomena.
3. Experimental Techniques. Students will be able to take physical measurements in an
experimental laboratory setting and analyze these results to draw conclusions about the physical system under investigation, including whether their data supports or refutes a given physical model.
4. Communication and Teamwork Skills. Students will be able to clearly explain their
mathematical and physical reasoning, both orally and in writing, and will be able to communicate and work effectively in groups on a common project.
5. Research Proficiency. Students will be able to formulate personal research questions
that expand their knowledge of physics. Students will be able to apply sound scientific research methods to address these questions, either by researching the current literature or developing independent results.
5 Streamlining Program Assessment Research Week 2014
Curriculum Matrix: Courses & PLOs
6
Physics Core Courses
Year Course Title
Program Learning Objec9ves Signature Assignment or Indirect Evidence Support
1 2 3 4 5 Physical Principles
Mathema9cal Exper9se
Experimental Techniques
Communica9on & Teamwork
Research Proficiency
1 Introductory I I I I W, T: I I 1 Introductory II I I I W, T: I I 2 Introductory III I I R W, T: R R
2 Classical Mechanics R R, A W: R, A O: I R, A PLO 2: Final exam: quanBtaBve
PLO 4, 5: Literature review 2 – 3 Thermodynamics R, A R PLO 1: Final exam: conceptual 3 – 4 Electrodynamics R R, A PLO2: Final exam: quanBtaBve 3 – 4 Modern Physics Lab R R, A W, O, T: R R PLO 3: Technical report
3 – 4 Quantum Mechanics R, A R/M O, T: R, A R PLO 1: Final exam: conceptual
PLO 4: Group video 4 Senior Research M M (M, A) (T: M) (M, A) PLO 3, 5: Advisor feedback 4 Senior Thesis M, A M, A M, A W, O: M, A M, A Senior Thesis & PresentaBon All Indirect Evidence A A A A A Campus: CRTE-‐led Focus Group All Indirect Evidence A A A A A Campus: IPA Senior Exit Survey
Streamlining Program Assessment Research Week 2014
I = Introduce, R = Reinforce, M = Master
Curriculum Matrix: Development
7
Physics Core Courses
Year Course Title
Program Learning Objec9ves Signature Assignment or Indirect Evidence Support
1 2 3 4 5 Physical Principles
Mathema9cal Exper9se
Experimental Techniques
Communica9on & Teamwork
Research Proficiency
1 Introductory I I I I W, T: I I 1 Introductory II I I I W, T: I I 2 Introductory III I I R W, T: R R
2 Classical Mechanics R R, A W: R, A O: I R, A PLO 2: Final exam: quanBtaBve
PLO 4, 5: Literature review 2 – 3 Thermodynamics R, A R PLO 1: Final exam: conceptual 3 – 4 Electrodynamics R R, A PLO2: Final exam: quanBtaBve 3 – 4 Modern Physics Lab R R/M, A W, O, T: R R PLO 3: Technical report
3 – 4 Quantum Mechanics R, A R/M O, T: R, A R PLO 1: Final exam: conceptual
PLO 4: Group video 4 Senior Research M M (M, A) (T: M) (M, A) PLO 3, 5: Advisor feedback 4 Senior Thesis M, A M, A M, A W, O: M, A M, A Senior Thesis & PresentaBon All Indirect Evidence A A A A A Campus: CRTE-‐led Focus Group All Indirect Evidence A A A A A Campus: IPA Senior Exit Survey
Streamlining Program Assessment Research Week 2014
I = Introduce, R = Reinforce, M = Master
Curriculum Matrix: Assessments
8
Physics Core Courses
Year Course Title
Program Learning Objec9ves Signature Assignment or Indirect Evidence Support
1 2 3 4 5 Physical Principles
Mathema9cal Exper9se
Experimental Techniques
Communica9on & Teamwork
Research Proficiency
1 Introductory I I I I W, T: I I 1 Introductory II I I I W, T: I I 2 Introductory III I I R W, T: R R
2 Classical Mechanics R R, A W: R, A O: I R, A PLO 2: Final exam: quanBtaBve
PLO 4, 5: Literature review 2 – 3 Thermodynamics R, A R PLO 1: Final exam: conceptual 3 – 4 Electrodynamics R R, A PLO2: Final exam: quanBtaBve 3 – 4 Modern Physics Lab R R/M, A W, O, T: R R PLO 3: Technical report
3 – 4 Quantum Mechanics R, A R/M O, T: R, A R PLO 1: Final exam: conceptual
PLO 4: Group video 4 Senior Research M M (M, A) (T: M) (M, A) PLO 3, 5: Advisor feedback 4 Senior Thesis M, A M, A M, A W, O: M, A M, A Senior Thesis & PresentaBon All Indirect Evidence A A A A A Campus: CRTE-‐led Focus Group All Indirect Evidence A A A A A Campus: IPA Senior Exit Survey
Streamlining Program Assessment Research Week 2014
I = Introduce, R = Reinforce, M = Master, A = Assessment
Curriculum Matrix: Signature Assignments
9
Physics Core Courses
Year Course Title
Program Learning Objec9ves Signature Assignment or Indirect Evidence Support
1 2 3 4 5 Physical Principles
Mathema9cal Exper9se
Experimental Techniques
Communica9on & Teamwork
Research Proficiency
1 Introductory I I I I W, T: I I 1 Introductory II I I I W, T: I I 2 Introductory III I I R W, T: R R
2 Classical Mechanics R R, A W: R, A O: I R, A PLO 2: Final exam: quanBtaBve
PLO 4, 5: Literature review 2 – 3 Thermodynamics R, A R PLO 1: Final exam: conceptual 3 – 4 Electrodynamics R R, A PLO2: Final exam: quanBtaBve 3 – 4 Modern Physics Lab R R/M, A W, O, T: R R PLO 3: Technical report
3 – 4 Quantum Mechanics R, A R/M O, T: R, A R PLO 1: Final exam: conceptual
PLO 4: Group video 4 Senior Research M M (M, A) (T: M) (M, A) PLO 3, 5: Advisor feedback 4 Senior Thesis M, A M, A M, A W, O: M, A M, A Senior Thesis & PresentaBon All Indirect Evidence A A A A A Campus: CRTE-‐led Focus Group All Indirect Evidence A A A A A Campus: IPA Senior Exit Survey
Streamlining Program Assessment Research Week 2014
I = Introduce, R = Reinforce, M = Master, A = Assessment
Curriculum Matrix: Indirect Evidence
10
Physics Core Courses
Year Course Title
Program Learning Objec9ves Signature Assignment or Indirect Evidence Support
1 2 3 4 5 Physical Principles
Mathema9cal Exper9se
Experimental Techniques
Communica9on & Teamwork
Research Proficiency
1 Introductory I I I I W, T: I I 1 Introductory II I I I W, T: I I 2 Introductory III I I R W, T: R R
2 Classical Mechanics R R, A W: R, A O: I R, A PLO 2: Final exam: quanBtaBve
PLO 4, 5: Literature review 2 – 3 Thermodynamics R, A R PLO 1: Final exam: conceptual 3 – 4 Electrodynamics R R, A PLO2: Final exam: quanBtaBve 3 – 4 Modern Physics Lab R R/M, A W, O, T: R R PLO 3: Technical report
3 – 4 Quantum Mechanics R, A R/M O, T: R, A R PLO 1: Final exam: conceptual
PLO 4: Group video 4 Senior Research M M (M, A) (T: M) (M, A) PLO 3, 5: Advisor feedback 4 Senior Thesis M, A M, A M, A W, O: M, A M, A Senior Thesis & PresentaBon All Indirect Evidence A A A A A Campus: CRTE-‐led Focus Group All Indirect Evidence A A A A A Campus: IPA Senior Exit Survey
Streamlining Program Assessment Research Week 2014
I = Introduce, R = Reinforce, M = Master, A = Assessment
Experimental Skills Development
Streamlining Program Assessment Research Week 2014 11
Revised
Year Course Title PLO 3
Expt. Techniques 1 Introductory I I 1 Introductory II I 2 Introductory III R 2 Classical Mechanics
2 – 3 Thermodynamics 3 – 4 Electrodynamics 3 – 4 Modern Physics Lab R/M 3 – 4 Quantum Mechanics 4 Senior Research (M) 4 Senior Thesis M
I = Introduce, R = Reinforce, M = Master
Original
Year Course Title PLO 3
Expt. Techniques 1 Introductory I I 1 Introductory II I 2 Introductory III I 2 Classical Mechanics
2 – 3 Thermodynamics 3 – 4 Electrodynamics 3 – 4 Modern Physics Lab R/M 3 – 4 Quantum Mechanics 4 Senior Research (M) 4 Senior Thesis
Teamwork Development
Streamlining Program Assessment Research Week 2014 12
Revised
Year Course Title PLO 4
Teamwork 1 Introductory I I 1 Introductory II I 2 Introductory III I 2 Classical Mechanics
2 – 3 Thermodynamics 3 – 4 Electrodynamics 3 – 4 Modern Physics Lab I 3 – 4 Quantum Mechanics R 4 Senior Research (M) 4 Senior Thesis
I = Introduce, R = Reinforce, M = Master
Original
Year Course Title PLO 4
Teamwork 1 Introductory I I 1 Introductory II I 2 Introductory III I 2 Classical Mechanics
2 – 3 Thermodynamics 3 – 4 Electrodynamics 3 – 4 Modern Physics Lab I 3 – 4 Quantum Mechanics 4 Senior Research (M) 4 Senior Thesis
Descriptive Rubrics
13 Streamlining Program Assessment Research Week 2014
!"#$%&#'()*+,)+-).,(
!"#$%&$'!()$)*+,+%-*'./0(%&'
/0&1&2%&0(3$4&'(5&$0(6(7&04'( 78%9&(:;(7$9<((
1(%+)(%,' 1,2$+-*)' 3%4)$+-*)' 5)*&"6,(7' !--('8' 9' :' ;'
<(=,*%>,+%-*',*?'!4,**%*='
=(>&99?#&;82&#(1%0@A%@0&(81(82A9@#&#(>8%B(A:2C&2%8:2$9(&9&4&2%1(D1%$%&4&2%(:;("0:E9&4F(E$A<G0:@2#F(4&%B:#1F(0&1@9%1($2#(A:2A9@18:2HI(J%@#&2%(4:C&1(&$189K(E&%>&&2(198#&1F($2#(4$82%$821($(A:2181%&2%9K(14::%B("$A&(%B0:@GB:@%("0&1&2%$%8:2I(
L&2&0$99K(>&99?1%0@A%@0&#(%$9<(>8%B(4:1%(&9&4&2%1("0&1&2%I(J%@#&2%(4:C&1(G&2&0$99K(>&99(E&%>&&2(198#&1F(4$82%$821($(14::%B("$A&(%B0:@GB(4:1%(:;(%B&("0&1&2%$%8:2I(
J:4&(1%0@A%@0$9(&9&4&2%1("0&1&2%F(1"&$<&0(1:4&%84&1($""&$01(@2;$4898$0(>8%B(198#&(A:2%&2%($2#(1&M@&2A&I(7B&(@2&C&2("$A&("&08:#8A$99K(#&%0$A%1(;0:4(%B&("0&1&2%$%8:2I(
N$A<(:;($(A9&$0(1%0@A%@0&(82(%B&(%$9<I(J98#&1(1&&4(#81:0G$28O&#($2#+:0(1"&$<&0(81(@2;$4898$0(>8%B(%B&80(A:2%&2%I(7B&(@2&C&2("$A&(#&%0$A%1(;0:4(%B&("0&1&2%$%8:2($2#(4&11$G&(D8I&I(:C&09K(9:2G(82%0:($2#(0@1B&1(%B0:@GB(A:2A9@18:21HI(
!()$)*+,+%-*'@+#4)'
J"&$<&0(81(A9&$0($2#(A:2;8#&2%I(L8C&1($("0:;&118:2$9(84"0&118:2I((
P9&$0(1"&&ABF($2#(M@8A<9K(:C&0A:4&1(:AA$18:2$9(9$"1&1(82(A:2;8#&2A&(:0(B&18%$%8:2I(
J:4&>B$%(2&0C:@1(:0(B&18%$2%(1%K9&F(E@%(G&%1(%B&(4&11$G&($A0:11I(J:4&(;9$>1(8I&I($C:8#1(&K&(A:2%$A%F(9::<82G($(;9::0+1A0&&2(:0(4@4E982GI(
Q&0K(2&0C:@1F(B&18%$2%(:0(#81R:82%&#(1%K9&F(>B8AB(82%&0;&0&1(>8%B($E898%K(%:(A:44@28A$%&(82;:04$%8:2(%:($@#8&2A&I(
A$)'-B'C,*=/,=)'
!1&1(#&1A08"%8C&F(1A8&2%8;8A(9$2G@$G&(%B$%(81(2:%(:C&0%9K(SR$0G:2KIT(P:2A&"%1($0&(A9&$0($2#("0:;&118:2$99K(&U"9$82&#I(
!1&1(4:1%9K(#&1A08"%8C&F(1A8&2%8;8A(9$2G@$G&F($2#(&U"9$2$%8:21($0&(4:1%9K("0:;&118:2$99K($2#(A9&$09K(&U"9$82&#I(
V$18A(9$2G@$G&(AB:8A&1F($""0:$AB82G("0:;&118:2$9(&U"9$2$%8:21F(E@%(4&11$G&(81(1%899(A9&$0I((W8284$9(;899&01(DS@4THI(
N$A<1(&U"&A%&#(1A8&2%8;8A(C:A$E@9$0KI((W$K(@1&(4$2K(;899&01(DS@4THF(184"981%8A+(R@C&289&(9$2G@$G&F(9&$#82G(%:(@2A9&$0(1%$%&4&2%1I(
D%$/,4'E%?$'' =8#1($0&(A9&$0F(>&99(:0G$28O&#($2#(&2B$2A&(%B&("0&1&2%$%8:2(18G28;8A$2%9KI(
=8#1(&2B$2A&(%B&("0&1&2%$%8:2(E@%(>8%B(1:4&(;9$>1(D8I&I(;:2%(18O&1F(A:2;@182G(9$K:@%1H(
=8#1($0&($#&M@$%&(E@%(2:%(>&99(982<&#(%:(%B&("0:R&A%($2#(A:2%$82(1&C&0$9(;9$>1I(P$2(E&(#81%0$A%82GI(
=8#1($0&(#81:0G$28O&#F("::09K(AB:1&2F(#&%0$A%(;0:4(%B&("0&1&2%$%8:2($2#(4&11$G&I(
1)*+(,4'3)$$,=)'
/@0":1&(:;(0&1&$0AB(81(A9&$09K(1%$%&#($2#(4&%B:#1(R@1%8;8&#(A9&$09K($2#(A:2A81&9K(
/@0":1&(:;(0&1&$0AB(81(1%$%&#($2#(982<&#(%:(4&%B:#1(E@%(1:4&%84&1("::09K(
L&2&0$9(%B&4&(:;(0&1&$0AB(81(82#8A$%&#I(3:(R@1%8;8A$%8:2(:;(4&%B:#1I(
/@0":1&(:;(0&1&$0AB("::09K(&U"9$82&#(:0(2:%($0%8A@9$%&#I(
E??%+%-*,4'1-66)*+$F'
Sources for Rubrics
14 Streamlining Program Assessment Research Week 2014
!"#$%&!''()*&#+*!)%,#$(-%"(."*& !"#$%"#&$'(!"#%)*'"(+$,-&).&$/"(*)/*$0)-1&2))/13"#4
$
/012324253% Oral communication is a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the listeners' attitudes, values, beliefs, or behaviors.
50)-1)*"#.$)#&$&(/"1#)4&6$*"$)..'4($)$7&#"$*"$)(8$9"#:$.)%,-&$"#$/"--&/*'"($"! $9"#:$*;)*$6"&.$("*$%&&*$<&(/;%)#:$=/&--$"(&>$-&0&-$,&#!"#%)(/&3$
% &6784530%4
'290845308%3 2
.03:;<6=>%1
!=?632@64253% Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable and is skillful and makes the content of the presentation cohesive.
Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable within the presentation.
Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is intermittently observable within the presentation.
Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is not observable within the presentation.
$63?A6?0% Language choices are imaginative, memorable, and compelling, and enhance the effectiveness of the presentation. Language in presentation is appropriate to audience.
Language choices are thoughtful and generally support the effectiveness of the presentation. Language in presentation is appropriate to audience.
Language choices are mundane and commonplace and partially support the effectiveness of the presentation. Language in presentation is appropriate to audience.
Language choices are unclear and minimally support the effectiveness of the presentation. Language in presentation is not appropriate to audience.
/092B0=C% Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation compelling, and speaker appears polished and confident.
Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation interesting, and speaker appears comfortable.
Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) make the presentation understandable, and speaker appears tentative.
Delivery techniques (posture, gesture, eye contact, and vocal expressiveness) detract from the understandability of the presentation, and speaker appears uncomfortable.
DA775=423?%'640=269% A variety of types of supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that significantly supports the presentation or establishes the presenter's credibility/authority on the topic.
Supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that generally supports the presentation or establishes the presenter's credibility/authority on the topic.
Supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make appropriate reference to information or analysis that partially supports the presentation or establishes the presenter's credibility/authority on the topic.
Insufficient supporting materials (explanations, examples, illustrations, statistics, analogies, quotations from relevant authorities) make reference to information or analysis that minimally supports the presentation or establishes the presenter's credibility/authority on the topic.
&034=69%'0886?0% Central message is compelling (precisely stated, appropriately repeated, memorable, and strongly supported.)
Central message is clear and consistent with the supporting material.
Central message is basically understandable but is not often repeated and is not memorable.
Central message can be deduced, but is not explicitly stated in the presentation.
VALUE Rubrics. Assessing Outcomes & Improving Achievement: Tips & Tools for Using Rubrics, T. L. Rhodes ed., Association of American Colleges & Universities, 2010.
Rubrics: Research Skills
15 Streamlining Program Assessment Research Week 2014
Level 1
Students research at the level of a closed inquiry* and require a high degree of structure/guidance
Level 2
Students research at the level of a closed inquiry* and require some structure/guidance
Level 3
Students research independently at the level of a closed inquiry*
Level 4
Students research at the level of an open inquiry* within structured guidelines
Level 5
Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline
A. Students embark on inquiry and so determine a need for knowledge/understanding
Respond to questions/tasks arising
explicitly from a closed inquiry.
Respond to questions/tasks
required by and implicit in a closed
inquiry.
Respond to questions/tasks
generated from a closed inquiry.
Generate questions/aims/
hypotheses framed within structured
guidelines.
Generate questions/aims/hypotheses
based on experience, expertise and
literature.
B. Students needed information/data using appropriate methodology
Collect and record required
information/data using a prescribed
methodology from a prescribed
source in which the information/
data is clearly evident.
Collect and record required
information/data using a prescribed
methodology from prescribed
source/s in which the information/
data is not clearly evident.
Collect and record required
information/data from self-selected
sources using one of several
prescribed methodologies.
Collect and record self-determined
information/data from self-selected
sources, choosing an appropriate
methodology based on structured
guidelines.
Collect and record self-determined
information/data from self-selected
sources, choosing or devising an
appropriate methodology with self-
structured guidelines.
C. Students critically evaluate information/data and the process
information/data
Evaluate information/data and
the inquiry process using simple
prescribed criteria.
Evaluate information/data and the
inquiry process using prescribed
criteria.
Evaluate information/data and the
inquiry process using criteria related
to the aims of the inquiry.
Evaluate information/data and the
inquiry process comprehensively
using self-determined criteria
developed within structured
guidelines.
Evaluate information/data and the
inquiry process rigorously using
self-generated criteria based on
experience, expertise and literature.
D. Students organise information collected/generated and manage the research process
Organise information/data and
manage the research process
according to a simple prescribed
structure.
Organise information/data and
manage the research process
according to prescribed structures.
Organise information/data and
manage the research process by
adapting provided structures.
Organise information/data and
manage the research process using
provided guidelines.
Organise information/data and
manage the research process
using self-determined protocols in
accordance with the discipline.
E. Students synthesise and analyse and apply new knowledge
Synthesise and analyse information/
data to reproduce existing
knowledge in prescribed formats.
curiosity.
Synthesise and analyse information/
data to reorganise existing
knowledge in standard formats.
Ask relevant, researchable
questions.
Synthesise and analyse information/
data to construct emergent
knowledge.
Ask rigorous, researchable
questions based on new
understandings.
Synthesise, analyse and apply
knowledge gaps.
Synthesise, analyse and apply
gaps or extend knowledge.
F. Students communicate knowledge and the processes used to generate it, with an awareness of ethical, social and cultural issues
Use mainly lay language and
prescribed genre to demonstrate
required knowledge and
understanding for lecturer/teacher
as the audience.
language and prescribed genre
to demonstrate self-selected
knowledge and understanding
from a stated perspective and for a
language and appropriate genre
to demonstrate knowledge and
a scholarly perspective and for a
Use the language of the discipline
and appropriate genre to address
knowledge and understanding gaps
from several perspectives for a self-
selected audience.
Use the language of the discipline,
choosing appropriate genre to extend
knowledge and understanding, from
diverse perspectives for a range of
audiences.
Research Skill Development Framework
www.adelaide.edu.au/clpd/rsd - [email protected]
A conceptual model to make explicit the incremental and cyclic development of student research skills
LEVEL OF STUDENT AUTONOMY
FAC
ET
OF
INQ
UIR
Y
Concept by John Willison and Kerry O'Regan, design by Peter Murdoch and Nik Cornish, Centre for Learning and Professional Development. Facets derived from ANZIIL(2004) standards, reworked using the Bloom, et al (1956) Taxonomy and implementation trials by Eleanor Peirce and Mario Ricci. © The University of Adelaide, October 2006.
* closed = lecturer speci!ed. open = student initiated. Lecturers and teachers determine scope of inquiry and standard required; student achievement determines the Level their research actually attains. For example, the provision of an open inquiry within structured guidelines (Level 4) in the First Year University context will see some students providing evidence of Level 1 attainment for a speci!c facet, with others demonstrating Level 2, Level 3 or Level 4, depending on their degree of rigour.
Persuasiv
e
Creativ
e
Organised
Critical
Determ
ined Curio
us
J. Willison, K. Regan, The Research Skills Development Framework, www.adelaide.edu.au/rsd/framework, 2006.
Level 1
Students research at the level of a closed inquiry* and require a high degree of structure/guidance
Level 2
Students research at the level of a closed inquiry* and require some structure/guidance
Level 3
Students research independently at the level of a closed inquiry*
Level 4
Students research at the level of an open inquiry* within structured guidelines
Level 5
Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline
A. Students embark on inquiry and so determine a need for knowledge/understanding
Respond to questions/tasks arising
explicitly from a closed inquiry.
Respond to questions/tasks
required by and implicit in a closed
inquiry.
Respond to questions/tasks
generated from a closed inquiry.
Generate questions/aims/
hypotheses framed within structured
guidelines.
Generate questions/aims/hypotheses
based on experience, expertise and
literature.
B. Students needed information/data using appropriate methodology
Collect and record required
information/data using a prescribed
methodology from a prescribed
source in which the information/
data is clearly evident.
Collect and record required
information/data using a prescribed
methodology from prescribed
source/s in which the information/
data is not clearly evident.
Collect and record required
information/data from self-selected
sources using one of several
prescribed methodologies.
Collect and record self-determined
information/data from self-selected
sources, choosing an appropriate
methodology based on structured
guidelines.
Collect and record self-determined
information/data from self-selected
sources, choosing or devising an
appropriate methodology with self-
structured guidelines.
C. Students critically evaluate information/data and the process
information/data
Evaluate information/data and
the inquiry process using simple
prescribed criteria.
Evaluate information/data and the
inquiry process using prescribed
criteria.
Evaluate information/data and the
inquiry process using criteria related
to the aims of the inquiry.
Evaluate information/data and the
inquiry process comprehensively
using self-determined criteria
developed within structured
guidelines.
Evaluate information/data and the
inquiry process rigorously using
self-generated criteria based on
experience, expertise and literature.
D. Students organise information collected/generated and manage the research process
Organise information/data and
manage the research process
according to a simple prescribed
structure.
Organise information/data and
manage the research process
according to prescribed structures.
Organise information/data and
manage the research process by
adapting provided structures.
Organise information/data and
manage the research process using
provided guidelines.
Organise information/data and
manage the research process
using self-determined protocols in
accordance with the discipline.
E. Students synthesise and analyse and apply new knowledge
Synthesise and analyse information/
data to reproduce existing
knowledge in prescribed formats.
curiosity.
Synthesise and analyse information/
data to reorganise existing
knowledge in standard formats.
Ask relevant, researchable
questions.
Synthesise and analyse information/
data to construct emergent
knowledge.
Ask rigorous, researchable
questions based on new
understandings.
Synthesise, analyse and apply
knowledge gaps.
Synthesise, analyse and apply
gaps or extend knowledge.
F. Students communicate knowledge and the processes used to generate it, with an awareness of ethical, social and cultural issues
Use mainly lay language and
prescribed genre to demonstrate
required knowledge and
understanding for lecturer/teacher
as the audience.
language and prescribed genre
to demonstrate self-selected
knowledge and understanding
from a stated perspective and for a
language and appropriate genre
to demonstrate knowledge and
a scholarly perspective and for a
Use the language of the discipline
and appropriate genre to address
knowledge and understanding gaps
from several perspectives for a self-
selected audience.
Use the language of the discipline,
choosing appropriate genre to extend
knowledge and understanding, from
diverse perspectives for a range of
audiences.
Research Skill Development Framework
www.adelaide.edu.au/clpd/rsd - [email protected]
A conceptual model to make explicit the incremental and cyclic development of student research skills
LEVEL OF STUDENT AUTONOMY
FAC
ET
OF
INQ
UIR
Y
Concept by John Willison and Kerry O'Regan, design by Peter Murdoch and Nik Cornish, Centre for Learning and Professional Development. Facets derived from ANZIIL(2004) standards, reworked using the Bloom, et al (1956) Taxonomy and implementation trials by Eleanor Peirce and Mario Ricci. © The University of Adelaide, October 2006.
* closed = lecturer speci!ed. open = student initiated. Lecturers and teachers determine scope of inquiry and standard required; student achievement determines the Level their research actually attains. For example, the provision of an open inquiry within structured guidelines (Level 4) in the First Year University context will see some students providing evidence of Level 1 attainment for a speci!c facet, with others demonstrating Level 2, Level 3 or Level 4, depending on their degree of rigour.
Persuasiv
e
Creativ
e
Organised
Critical
Determ
ined Curio
us
Rubrics: Research Skills
16 Streamlining Program Assessment Research Week 2014
Based on J. Willison, K. Regan, The Research Skills Development Framework, www.adelaide.edu.au/rsd/framework, 2006.
Mentor Questionnaire for PLO#5: Research Proficiency 3
Attributes The student with research skill!
Level I Students research at the level of a closed inquiry* and require a high degree of structure/guidance
Level II Students research at the level of a closed inquiry* and require some structure/guidance
Level III Students research independently at the level of a closed inquiry*
Level IV Students research at the level of an open inquiry* within structured guidelines
Level V Students research at the level of an open inquiry* within self-determined guidelines in accordance with the discipline
“!research questions that expand their knowledge of physics.”
Synthesize and analyze information/data to reproduce existing knowledge in prescribed formats.
Synthesize and analyze information/ data to reorganize existing knowledge in standard formats.
Synthesize and analyze information/ data to construct emergent knowledge.
Synthesize and analyze information/ data to fill recognized knowledge gaps.
Synthesize, analyze, and apply information/ data to fill self-identified gaps or extend knowledge.
1. Characteristics of Mentee:
a. How does the mentee exhibit the characteristics for the level you highlighted above?
b. What would the mentee need to do to reach the next level? And how is that higher level expected, needed, or otherwise appropriate for the mentee’s education and/or career goals?
2. Opportunities within their project/working in your research group: a. How have you gauged the mentee’s expansion of their knowledge?
3. Other examples: a. Describe any other instances (if any) where the mentee has formulated personal research questions.
Inter-rater reliability • PLO: Physical Principles • Distribution Matrix • Correlation coefficient: 0.78
Streamlining Program Assessment Research Week 2014 17
Mitchell E A U
Men
ke E 6 1 0
A 2 15 3 U 0 1 6
Unacceptable Acceptable Excellent • Knowledge of basic physical
principles is missing. • Knowledge of basic physical
principles evident, but o Application is missing. o Significant errors exist in their
application. o Example: student can write down
Maxwell’s equations, but cannot calculate the magnetic field around a wire.
• Knowledge and/or application of two or more physical principles are confused.
• Knowledge of basic physical principles is evident.
• Those principles are applied correctly, o although some errors exist. • Misconception in knowledge or
application of more subtle feature of principle may exist.
• Knowledge of basic physical principles is evident.
• Those principles are applied correctly. o although minimal errors
may be present. • Evidence that more subtle
aspects of physical principles known and correctly applied.
Course and Program
Streamlining Program Assessment Research Week 2014 18
Conclusions & Actions of Assessments • Program Learning Objectives: Important discussion on what constitutes research
proficiency (PLO 5), led to focus on information literacy in early years and encouraging students to engage in undergraduate research early.
• Curriculum Matrix & Evidence: • Allowed us to identify gaps in learning objective development opportunities with Experimental
Techniques (PLO 3) and Teamwork (PLO 4). • Evidence is consistently collected and spread out over all semester-long core courses.
• Pedagogical Validation: Existing writing assignments and presentations effectively support Communication skills (PLO 4).
• Pedagogical Adoptions: • Conceptual questions explicitly integrated into final exams to separate principles (PLO 1) from
mathematics (PLO 2). • Video project adopted in Quantum Mechanics to provide in-class teamwork opportunities (PLO 4). • Stronger emphasis on data analysis in Intro. III provide reinforcement for experimental techniques (PLO
3).
• Applying Rubrics to Course and Program: • Robust program rubrics are applicable to course-level work and allowing for quicker program
assessment when taken in aggregate. • Inter-rater reliability checks are needed to ensure program applicability.
Streamlining Program Assessment Research Week 2014 19
Challenges • Organization • Student writing • Robust rubrics • Participation
• Faculty • Students
Streamlining Program Assessment Research Week 2014 20
References 1. Laura Martin, Introduction to Assessment Presentation, CGS Workshop,
http://crtecertificatespring2013.files.wordpress.com/2012/12/introduction-to-assessment-1-14-2013-final.pdf, January 2013
2. L. A. Suskie, Assessing Student Learning: A Common Sense Guide (Jossey-Bass, San Francisco, CA 2009).
3. Center for the Integration of Research, Teaching, and Learning (CIRTL
Network), Teaching-as-Research (TAR): Developmental Framework, www.cirtl.net/CoreIdeas/teaching_as_research, 2013.
4. G. Wiggins, J. McTighe, Backward Design in Understanding by Design (Assn.
for Supervision & Curriculum Development, Alexandria, 2005). 5. VALUE Rubrics. Assessing Outcomes & Improving Achievement: Tips & Tools
for Using Rubrics, T. L. Rhodes ed., Association of American Colleges & Universities, 2010.
6. J. Willison, K. Regan, The Research Skills Development Framework,
www.adelaide.edu.au/rsd/framework, 2006.
Streamlining Program Assessment Research Week 2014 21