Second day sessions

First sessionSecond session Third Session

How to write LOs at Program and Course Levels

Relationships between Teaching Methods and Assessments Methods and LOs

KPIs in General and Application and Itegration with LOs

74

How to write learning outcomes for How to write learning outcomes for program levels and courses?program levels and courses?

Session 1Session 1

NCAAANCAAA

75

Where do L.O. come from?

Learning OutcomesLearning Outcomes flows out flows out from……from……

……students’ needs assessmentstudents’ needs assessment… … and employers needsand employers needs

(cause and effect chart)(cause and effect chart)76

EffectsEffects Learning Outcomes Learning Outcomes

CauseCause StudentStudent

NeedsNeeds

77

Where do L.O. come from?

Learning OutcomesLearning Outcomes flows out flows out from……from……

……the Mission Statementthe Mission StatementWhat must students do to demonstrate that What must students do to demonstrate that

the Institution/Program Mission is the Institution/Program Mission is accomplished?accomplished?

(cause and effect chart)(cause and effect chart)78

EffectsEffects Program and CourseProgram and Course Learning OutcomesLearning Outcomes CauseCause MissionMission

NeedsNeeds

79

Theory Into Practice5 Questions for Instructional Design5 Questions for Instructional Design

1.1. What do you want the student to be able What do you want the student to be able to do? (Outcome)to do? (Outcome)

2.2. What does the student need to know in What does the student need to know in order to do this well? (Curriculum)order to do this well? (Curriculum)

3.3. What activity will facilitate the learning? What activity will facilitate the learning? (Pedagogy)(Pedagogy)

4.4. How will the student demonstrate the How will the student demonstrate the learning? (Assessment)learning? (Assessment)

5.5. How will I know the student has done How will I know the student has done this well? (Criteria)this well? (Criteria)8

0

Let’s Write a Learning Outcome

• We’re taking a friend desert camping We’re taking a friend desert camping for the first time (not roughing it too for the first time (not roughing it too much).much).

• What do they need to know?What do they need to know?

81

1. What do you want the student to be able to do?This question asks you to This question asks you to

develop the outcome.develop the outcome.

For Example: Student Student identifiesidentifies, , consultsconsults and and evaluatesevaluates reference books appropriate to reference books appropriate to the topic in order to locate the topic in order to locate background information and background information and statistics.statistics.

82

Learning Outcomes Formula

Verb Verb

oror

Action PhraseAction Phrase “In Order In Order To”To”

=Great Great

Learning Learning OutcomesOutcomes

What students need to know?““Student identifies, Student identifies, consults and evaluates consults and evaluates reference books reference books appropriate to the topic”appropriate to the topic”

Why do they Why do they need to know need to know this?this?““locate background locate background information and information and statistics.”statistics.”

In Order to

+

OrOr

Why?

83

Establish DraftEstablish DraftProvide PrepareProvide PrepareTabulate WriteTabulate WriteSchedule UpdateSchedule UpdateAudit ArticulateAudit ArticulateAlign CollectAlign CollectConstruct GenerateConstruct GenerateListList Produce ProduceCompile DocumentCompile DocumentDemonstrate DevelopDemonstrate Develop

Suggested VerbsSuggested Verbs

84

Consider MaximizeConsider MaximizeMaintainMaintain Reflect ReflectContinueContinue Review ReviewEnsure Enlarge Ensure Enlarge Understand ExamineUnderstand ExamineStrengthen ExploreStrengthen ExploreEncourage DeepenEncourage Deepen

Some of these verbs can be used if tied to Some of these verbs can be used if tied to specific actions or quantificationspecific actions or quantification

Verbs Not To UseVerbs Not To Use

85

Example 1

Poor Learning Outcome:Poor Learning Outcome:• Students will name the three types of

rock in order to differentiate among the three.

Good Learning Outcome:• Students will compare and contrast Students will compare and contrast

the characteristics of the three types the characteristics of the three types of rocks in order to differentiate of rocks in order to differentiate among the three. among the three.

86

Example 2

Bad Learning Outcomes:Bad Learning Outcomes:• Discover that Al Fawo University offers a welcoming

and helpful environment which can fulfill their educational, cultural and social needs in order to recognize the university’s role in lifelong learning.

• Use OVID in order to access materials not available Use OVID in order to access materials not available at Al Fawo university Library.at Al Fawo university Library.

Good Learning Outcome:Good Learning Outcome:• Utilize retrieval services in order to obtain materials Utilize retrieval services in order to obtain materials

not owned by Al Fawo University Librarynot owned by Al Fawo University Library.87

Let’s Write a Learning OutcomeLet’s Write a Learning Outcome

• We’ll concentrate on how to build a We’ll concentrate on how to build a tenttent

• Why do we want our friend to be able Why do we want our friend to be able to properly build a tent?to properly build a tent?

88

Let’s Write a Learning OutcomeLet’s Write a Learning Outcome

• Now let’s write the learning Now let’s write the learning outcomeoutcome

• What is our verb (use the right What is our verb (use the right domain)?domain)?

• Why?Why?

89

Let’s Write an Assignment-Specific Learning Outcome

What does the student need to know?What does the student need to know?• Why do they need to know this?Why do they need to know this?

See example and activity See example and activity 90

Student needs Learning outcomes

Student needs example: “Student needs Student needs example: “Student needs to learn how to fish in the ocean to to learn how to fish in the ocean to survive on the island.”survive on the island.”

1.1.What are some learning outcomes that What are some learning outcomes that fit this student’s needs?fit this student’s needs?

2.2.Create a student need for your Create a student need for your specialized course.specialized course.

3. Create five learning outcomes that 3. Create five learning outcomes that are directly based on this need. are directly based on this need. Bubble MapBubble Map9

1

Student Need

92

Relationship between teaching methods Relationship between teaching methods and assessment methods with learning and assessment methods with learning

outcomes.outcomes.

Session 2Session 2

NCAAANCAAA

93

Teaching Methods and LOs• Focuses on learning outcomes in debates on

teaching methods in higher education• “Teaching methods” are not an end in themselves,

they are a means to an end.• They are the vehicle(s) we use to lead our students

towards particular learning outcomes.• We evaluate our teaching methods against the

learning outcomes that we are seeking for our students.

94

Teaching Methods and LOs

• The first step in operationalizing it is to clarify the learning outcomes at which we are aiming.

• The second step involves developing a contingency approach to the choice of teaching methods whereby “fitness for purpose”

95

Teaching Methods and LOs• When selecting any teaching and learning method it

is important to ensure that the method will enable the students to achieve what are intended as learning outcomes.

• There are different kinds of methods available:– effective in building up subject knowledge– contribution to developing generic skills

96

Quality of Teaching

NCAAA Standard 4, paragraph 4.6:

• Teaching must be of high quality with appropriate strategies used for different categories of learning outcomes and student learning styles.

• Differentiated Instruction 97

Teaching Methods MappingTeaching Methods Mapping

Courses Codes

101102103104105106107108

TeachingMethods

LectureSmall GroupsDiscussionProjectActivityDebateResearchLabGuest ExpertDemonstration9

8

Key for Learning Outcomes

Learning outcomes must be…. measurablemeasurable and meaningful to be assessed accurately. meaningful to be assessed accurately.

Who will know? Who will know? Student? Faculty? Student? Faculty?How will I know? How will I know? Evidence EvidenceWhat evidence is needed? DemonstrateWhat evidence is needed? Demonstrate

Key is ASSESSMENTASSESSMENT

99

NCAAA: L.O. + AssessmentNCAAA: L.O. + Assessment

NCAAA Standard 4, paragraph 4.4:NCAAA Standard 4, paragraph 4.4:

Student assessment processes must be appropriate for the intended learning outcomes and effectively and fairly administered with independent verification of standards achieved.

100

Quality Assurance of Assessment Quality Assurance of Assessment of Learning Outcomesof Learning Outcomes….….

May be achieved:• By direct observation – inspection of

assessments before they are set, and of samples of marked work

• Indirectly – by examining the specification of assessment processes

• Via feedback – from students, from employers, from external examiner, from professional bodies.

101

NCAAA Domains of Learning

Strategies

Knowledge Knowledge FactsFacts Concepts, theoriesConcepts, theories ProceduresProcedures

Cognitive SkillsCognitive Skills Apply skills when askedApply skills when asked Creative thinking and problem Creative thinking and problem solvingsolvingInterpersonal Skills and Interpersonal Skills and ResponsibilityResponsibilityResponsibility for own learningResponsibility for own learningGroup participation and leadershipGroup participation and leadershipAct responsibly-personal and Act responsibly-personal and professional situationsprofessional situationsEthical standards of behaviorEthical standards of behaviorCommunication IT and Numerical Communication IT and Numerical SkillsSkills Oral and written communicationOral and written communication Use of ITUse of IT Basic math and statisticsBasic math and statistics

Psychomotor SkillsPsychomotor Skills

Assessment Strategies…Assessment Strategies…

102

Quality Assessment of L.O.sQuality Assessment of L.O.sRequiresRequires

• That a range of assessment methods is used, so as to cover different LOs and different learning styles

• That specifications for assessment, and criteria for grading, are transparent and appropriate to the level of the course within the NQF

• That second-marking or moderation is routinely used

• That there is clarity about the grounds on which students may appeal a mark, and that students are aware of this10

3

Quality Assessment of L.O.sQuality Assessment of L.O.sRequires….Requires….

• That deadlines for the return of marked work are published and adhered to

• That full and constructive feedback on assessments is provided to students, linked to the LOs and to the assessment criteria

• That guidelines for dealing with plagiarism, and with requests for time extensions and mitigating circumstances, are clear and are adhered to

104

Start at the EndStart at the End Teaching strategy and methods depend on the assessment methods utilized. The assessments utilized depends on the learning outcome VERB that guides the assessment process to validate student learning and direct teaching methods.

Teaching Assessment Teaching Assessment LearningLearning Objective Objective Student Student Methods Methods Methods Methods Outcomes Outcomes Content Content Needs Needs

105

AssessmentsAssessments Teaching-Teaching- MethodsMethods

LearningLearningContentContent

ObjectiveObjective

11stst Learning LearningOutcome Outcome 22ndnd LO LO

55thth LOLO

3rd LO

44thth LO LO

106

Introductory Assessment MapIntroductory Assessment Map

107

Intermediate Assessment MapIntermediate Assessment Map

108

Assessment MapAssessment Map

109

NCAAANCAAAKey Performance Indicators,

Application and Integration with Learning Outcomes

Session 3110

……a type of measure of performancea type of measure of performance……a Key Success Indicatora Key Success Indicator (KSI) (KSI)…a measure of results and efficiency

Quantifiable performance measures used to define success and measure progress toward the achievement of goals.

What are KPIs???

111

Success IndicatorsSuccess IndicatorsBubble Map Group ActivityBubble Map Group Activity(think Quality Indicators)(think Quality Indicators)

……for a carfor a car……for a jobfor a job……for dinnerfor dinner……for a business tripfor a business trip……for a housefor a house……for a vacationfor a vacation

112

Good KPIs are Good KPIs are SMARTSMART• Specific – clear and well-defined• Measurable – show distance from

attainment• Agreed – shared by stakeholders• Realistic – fit available time, money,

etc• Time-based – set a specific time-scale

Sometimes ‘achievable’ is used, not ‘agreed’,but ‘achievable’ implies ‘realistic’,

so you might use ‘achievable’ and ‘relevant’113

FIRST……

An An intended learning outcome intended learning outcome must be must be selected and then a KPI may be chosen selected and then a KPI may be chosen which best reflects thewhich best reflects the intended student intended student performance performance ... or intended student intended student achievement achievement … or the student outcome… or the student outcome

114

KPIs are like….KPIs are like….

……like like yeastyeast when it permeates when it permeates bread and makes it rise…bread and makes it rise…

……like like saltsalt seasoning food, it seasoning food, it changes the entire flavor..changes the entire flavor..

……like like lightlight in the darkness.. in the darkness.. They create a context for actionThey create a context for action

115

KPIs are not independent

KPIs do not stand alone or operate in a KPIs do not stand alone or operate in a vacuum; a clear understanding of vacuum; a clear understanding of what is possible is requiredwhat is possible is required “upper “upper and lower limits.”and lower limits.”

What are “limits”???What are “limits”???(ie., speed limits for cars)(ie., speed limits for cars)

KPIs are “Married” to KPIs are “Married” to benchmarksbenchmarks

116

Limits are…

LimitsLimits refer to “BENCHMARKSBENCHMARKS” which are essential to make KPIs useful; both internal and external…

They provide a basis for They provide a basis for ANALYSISANALYSIS……or as we see in the sporting or as we see in the sporting

business…business… “competitive analysis”

117

SebringSebring Racetrack Benchmarks

Example:Example:

DaytonaDaytona500500

SebringSebring

118

Benchmarks prevent Benchmarks prevent race car crashesrace car crashes

Benchmark Cones Control SpeedBenchmark Cones Control Speed Cars are forced to slow down in order Cars are forced to slow down in order to safely make the hairpin curve. to safely make the hairpin curve. How does How does this apply to teaching and learning?this apply to teaching and learning?

=

119

Benchmarking for Benchmarking for quality: Why?quality: Why?

• Benchmarking fixes reference points relative to other measurements

• In the context of quality, benchmarking specifies reference values for KPIs

• Benchmarking is used to ensure that the KPIs are realistic

• Ensure KPIs are sufficiently demanding

120

Benchmarking for Benchmarking for quality: quality: What?What?

• Judgments Judgments about quality are based on comparisons with similar good institutions

• The levels of performancelevels of performance identified are benchmarks that a used for comparisons.

• Aggregate national figures for KPIs can also serve as benchmarks

• Should establish other benchmarks for matters of importance to them

• Remember:Remember: benchmarks should be both benchmarks should be both internal and externalinternal and external

121

KPI & Benchmarks(for Dr. Bothyna’s nursing program)

Learning Outcome: The nursing student will use new Disposable Gloves at The nursing student will use new Disposable Gloves at all times when nursing patients and whenever body fluids are present. all times when nursing patients and whenever body fluids are present.

KPI The student uses new disposable gloves 100% of the time while The student uses new disposable gloves 100% of the time while nursing patients and whenever body fluids are present.nursing patients and whenever body fluids are present.

Target Benchmark Target Benchmark 100% Goal

KPI Finding Benchmark KPI Finding Benchmark 95% Assessment finding

Internal Benchmark Internal Benchmark 98% Medical College

External Benchmark External Benchmark New Target Benchmark New Target Benchmark

98% Cairo University100% New Goal

AnalysisAnalysis: How is this data interpreted?: How is this data interpreted? What is the improvement plan to reach the new goal?What is the improvement plan to reach the new goal?

122

KPIs for learning outcomesKPIs for learning outcomes

Using a Using a Bubble MapBubble Map, write a course , write a course learning outcome for one of your learning outcome for one of your courses in the center circle. courses in the center circle.

Next write teaching methods for Next write teaching methods for learning outcome in the learning outcome in the connected circles.connected circles.

On the extended lines write an On the extended lines write an assessment method with a KPI and assessment method with a KPI and benchmarks.benchmarks.

123

Extra information that may Help

124

Program Learning Program Learning OutcomesOutcomes

What is essential to this degree??Why are students enrolling??What does the public think??

What the STUDENTSTUDENT expects to know and/or do via the Program…

125

ProgramProgram Outcomes Outcomes

What the What the STUDENTSTUDENT will will achieveachieve and and performperform at the at the end of the program.end of the program.

These are general, specialized learning These are general, specialized learning outcomes for all students in all outcomes for all students in all departments of the college program.departments of the college program.

Institutional learning outcomes Institutional learning outcomes are general are general learning outcomes for all students.learning outcomes for all students.

126

CourseCourse Outcomes Outcomes

What the What the STUDENTSTUDENT will will achieveachieve and and performperform at the at the end of a specific course.end of a specific course.

These are highly specialized learning These are highly specialized learning outcomes for all students in a given course.outcomes for all students in a given course.

127

Purposes of Program and Course Learning Outcomes

Measurable Improvement is #1Measurable Improvement is #1Evaluate programs/courses, not Evaluate programs/courses, not

personnelpersonnelAssist with QA and accreditationAssist with QA and accreditation…Initiate “best practices”…Initiate “best practices”…Set learning to KPIs & benchmarksSet learning to KPIs & benchmarks…

128

Harden, R.M. (2001). AMEE Guide No. 21. Curriculum mapping: a tool for transparent and authenticteaching and learning. Medical Teacher, 23 (2), 123-137.

Hobson, E.H. (2005). Changing pedagogy. Presentation at SACS-COC Institute on Quality Enhancement and Accreditation, Orlando, FL, July 24-27, 2005.

Curriculum Alignment?Consistency and Intentionality

Learning OutcomeLearning Outcome

Learning ObjectivesLearning Objectives

129

ExamplesExamples

Student OutcomeStudent Outcome Student will demonstrate ability to use 10 conventions of safe science research in a science lab.

What are the differences???What are the differences???What are the different teaching methods?What are the different teaching methods?

What are the different assessments?What are the different assessments?

Student OutcomeStudent Outcome Student will be able to identify 10 safety rules

for the science lab in a paragraph.

Teaching Objective Teaching Objective teacher will teacher will teachteach & & demonstratedemonstrate 10 science lab safety rules 10 science lab safety rules

130

ConclusionThank you for your time and reflections

Dr Bothyna Zakaria MurshidDr Greg Maffet

131