BENGKEL PENILAIAN OUTCOME BASED EDUCATION

BENGKEL PENILAIAN

OUTCOME BASED EDUCATION

8 Julai 2013 Dewan Delima, DeTAR Putra

Penceramah: Dr Aishah bt Abu Bakar

Pengarah, Bahagian Pengurusan Pembangunan Akademik

J abatan Pengaj ian Tinggi

1. Reflection - Current assessment practice and the need to change

2. Application of Constructive Alignment at subject level 3. CQI and closing the loop 4. Developmental Framework in programme design

ACTION

ADJUSTMENT

Bengkel Peniiaian Outcome Based Education

Te chIng for Quality Learning

How are they related

• Constructive Alignment

• Outcome-based Education

• Malaysian Qualification Framework

3

Towards MQF - basis for quality assurance of

higher education

In line with internationally recognized good

practices (like Washington Accord for

Engineering Education)

Areas of Evaluation in MQF

1. Vision, Mission, Educational Goals and Learning Outcomes; 2. Curriculum Design and Delivery; 3. Assessment of Students; ------,---------, 4. Student Selection and Support Services;

HEARTofQA5. Academic Staff; 6. Educational Resources; 7. Programme Monitoring and Review; 8. Leadership, Governance and Administration; and

P"""~-"'!!"'"~----~--~" 9. Total Continual Quality Improvement.

Bengkel Penilaian Outcome Based Education

6


Teaching Assessment Curriculum

Lecture ExaminationList of topics to be covered

Tutorial Test

Laboratory Assignment

Teacher's and Student's Perspective

Intended LO -+ [T&LActivities I AssessmentJII

LearningOutcomes Assessment Activities

I Institutional Mission I Stakeholders Needs

c:; Program Educational Objectives (PEO) (desired skills of alumni)

•Program Outcomes (PO) (desired knowledge, skills and attitude of graduating seniors)

(considers attribute as specified by Accreditation body)

• --Outcome Indicators (Assessment Instruments & Methods)~~l Performance Targets (Criteria for acceptable performance)

I

Course Outcomes (CO) Course Learning Objectives


Reasons for writing LO • Help to guide students in their learning as it explain what is

expected of them in turn helping them to succeed in their

studies

• Help staff to focus on exactly what they want students to

achieve in terms of both knowledge and skills

• Provide a useful guide to inform potential candidates and

employers about the general knowledge and understanding

that a graduate will possess.

What are LO • Specific intentions of a programme or module written in

specific terms

• In the Malaysian Quality Framework (MQF), LO is a

statement on what students should know, understand and can

do upon completion of a period of study (bearing in mind the

MQF qualification level: Advanced Diploma, Bachelors,

masters, Doctoral)

10


9

ASSESSMENT theOBEway

Aishah Abu Bakar


Continuous assessment -lack of systematic feedback

SOIL MECHANIC S I

SEMESTER 2 I SES12008-2009

.l. Find~.

I No Malric No Laboratory/)O Cant a98mnl 25

I f, IT1 [,lT2 Int aubllll T1 Cw Tllt

I 1 KEA060002 7.0 7 .0 5.0 19.0 1.5 7.5 3.0

2 KEA060011 10.0 8 .0 10.0 28.0 6.0 10.0 . 3. 5

3 KEA.OG OO 12 10.0 8.0 10.0 28.0 2.5 6.5 4.0

, 4 I<EA060014 10.0 8.0 10.0 28.0 8 .5 1 10.0 5.0

~ KE.A0600 15 10.0 8.0 1

10. 0 28.0 5.0 10.0 4.5

6 KEA06001 7 7.0 7 .0 110.0 24.0 9 .0 6.5 i 5.0

7 KEA060030 7.0 7 .0 10.0 24.0 4 .5 6.5 1 4.5

I 8 KEA060034 7.0 7.0 10.0 24.0 6.0 6.5 4.5

9 KEA060040 7 .0 7 .0 6 .0 20.0 7.0 I

8.5 4.5

sublot

13.0

19.5

13.0

23.5

19.5

20.5

15.5

17.0

20.0

Limited feature in the current system to incorporate assessment data on the other two domains - affective and psychomotor

SOIL 1EC HANIC S I

SE ~E STER 1 I SE SI 2 008 · 2009

I No M.trIoNo LaborOI(or,130 COM ..ssmnt 125 EJonU45 Gu.nd r--

NlTl MT2 NlT3 ..b<ot T1 c 'W Tut . "btot $ 'IIbtot Tot~

" K E AOBOO 3 6 6.0 6 .0 6 . 5 1 ••5 6 . 5 5.5 0 . 0 1 2.0 8 .0 38.5 I

2 K EAOSOO42 6 . 0 6.0 6.5 18. 5 5.0 5.5 0.0 10.5 19.0 48.0

:3 KEA070004 6 .0 6 .0 7 .0 19.0 6 .0 7.5 2.5 16.0 14. 5 49.5

I 4 K EA070010 6 . 0 6.0 7.0 19.0 7.5 8.5 5.0 21..0 38.0 78.0

I 1> K EA070 011 I 6.0 6.0 6.5 1 ••5 5 . 5 10.0 4 .0 19.5 45.0 83. 0

6 KEA070D 12 7.0 7.25. 7 . 5 21.8 5 . 5 10.0 2 . 0 17.5 19. 5 58.. 8

I ' K E ."'-070 01:3 1

23.3 6.0 I 1 6.0 73.3? 7.0 7 .25 9.0 6.0 1 L4 . 0 34.0

8 K E A 070016 7 .0 7.25 6.5 20•• 3.0 6 .0 4 .5 13. 5 18. 0 52.3

9 KEA070018 7.0 7.25 6 ,,5 20.8 5 . 5 I 6.0 5.0 16.5 39. 0 76.3

10 K EA070019 7 .0 7 .25 6 . 5 20•• 8 .Q 6.0 3 .5 17.5 36. 5 II 74.8

I 11 K EA0700 22 7.0 7.25 6.5 20.8 7.0. 6.0 2.0 15.0 25.0 60.8

12 K EA0 70024 I 7 .0 17 . 25 6.5 20.. 1 . 0 10.0 2.0 13.0 28.0 61.8

13 K EA070 0 25 I 7 .0 7 .25 9.0 23.3 5 . 5 6 .0 5 . 0 1 6.5 37.0 76.8

I 14 K EA070026 I 7.0 7.25 7.0 2L3 1 . 5 8 . 5 2.0 ,~ 29.5 62.8

Bengkei Penifaian Outcome Based Education

ICOURSE LEARNING OUTCOMES (CO)

PROGRAMME LEARNING OUTCOMES (PO) METHOD OF

ASSESSMENT PO PO PO PO PO PO PO PO PO PO

DELIVERY

1 2 3 4 5 6 7 8 9 10

Define soil as in engineering context and Lectures and Examination short relate problems associated with the I coursework reportdefinition with Ilocal' soil condition " (PBL) CW Interview

Identify and differentiate the different types of soil and their properties and

Lectures and Short test and

classify soil using British and I or Unified I I laboratory'i \'

laboratory work Soil Classification System. work/report

Conduct laboratory tests for Demonstration Direct observation determination of soil index and

, ,,' I and laboratory on laboratory" " compaction, work, work/report

Solve calculation problem using mechanics involving physical properties, -.; -.; Lecture and Examination & compaction, seepage and effective tutorials graded tutorial. stress. Showthe use of soil mechanics concepts

Lecture,in engineering works ,j 'i , i coursevvork. Short report &

(PBL) group interview


Course V.AEA2132 SOIL MECH ANIC I

legends;

LO Learni ng Outccme Cede Af2 INTEREST

LO Dese Le arnir.g Outcome Desc ription C02 ANAL '(TICAL & CRITI CAL

AM Assessm ent P.lethod COL KNOW LEDGE

CO Cog nitive PS~ OeSER ~nON

AF Affect ive PS2 DYNAMISM

PS $Tfehomotor PS 3 ~lA.NlpuLAnON

17

-~~ .... ....................--,..,-~-....,.- ......_................................-- ......

~-- .........- ...... ......... (\IlOWM\/- Refer to list of assessment

methods extracts from 5 Bloxham and P. Boyd" Developing Effective Assessment in Higher Education A Practical Guide"

18


Possible L.T.As Common I LOs Possible Assessment Tools

I Set reading, lecture, report on field .. Describe .. Assignment, essay question exam trip, write essay

-

Tutorial, activities, write essay, ... Explain .. Assignment, essay question exam, oral presentation

Project, assignment ... Integrate ... Project, assignment

Project, case study ... Apply .. Project, case study, experiment

PBl, case study ... SoJve problem .. Case study, project, experiment

Project, poster ... Design, create .. Project, experiment, poster

Reflective diary ... Reflect .. Reflective diary, portfolio, self-assessment

presentation, debate, role play, A range of oral, writing or listening tasks addressing the IlOs, e.g .

... Communicate ... presentation, debate, role play, reporting, assignment, precis,

paraphrasing, answering questions

The point is not how you are go"ng to teach but how and what you want your students to learn.

NOTE! Many o/these TLAs can be assessments tasks as well. Then you have excellent alignment.

Knowledge Ability to observe and remember previously learned Information; knowledge of specific facts, terms, concepts, principtes, ideas, events., p_laces etc; mastery of subject material

Recall or Recognize information

multiple-choice test, recount facts or statistics, recall a process, rules, definitions; quote law or procedure

arrange, define, describe, label, list, memorize, recognize, relate, reproduce, select, state

2

3

Com prehension Ability to understand Information and grasp material, lTanalating knowledge from one form to another; Interpreting, comparing a nd contrasting materia I; predicting consequences and future trends

Application Ability to use information, learned material, methods, concepts, theories, principles, laws and theories in new situations, problem solving using required knowledge or skills

Understand meaning, restate data in one's own wordS, interpret, extrapolate, translate

Use or apply knowledge, put theory into practice, use knowledge in response to real circumstances

explain or interpret meaning from a given scenario or statement, suggest treatment, reaction or solution to given problem, create examples or metaphors

a theory into practical effect, demonstrate, solve a problem, manage an activity

explain , reiterate, reword , critique, classify, summarize, illustrate, translate, review, report, discuss, re-write, estimate, interpret, theorise, paraphrase, reference, example

use, apply, discover, manage, execute, solve, produce, implement, construct, change, prepare, conduct, perforrJl, react, respond, rolE2play

Bengke/ Peni/aian Outcome Based Education

4

5

6

Analysis Ability to breakdown material and recognition of organization structure; identlrlCation of components and relationships between components, recognition of patterns and hidden meetings

Synthesis (Create/build) Ability to combine parts or apply prior skills and knowledge to produce a new whole; integrate ideas Into a solution; generalize from a given facts; propose a plan action; fonnulate new classification methods.

Evaluation Ability to judge and alsess the value of theories and p ....... ntstion, based on their value, logic or adequacy, for a given purpose; compare and make choices based on reasoned argument, verify the value of evld.nce, recognize BubJectivlty

Interpret elements, organizational principles, structure, construction, internal relationships, quality, reliability or individual components

Develop new unique structures, systems, models, approaches, ideas, creative thinking, operations

Assess effectiveness of whole concepts, in relation to values, outputs efficacy, viability; critical thinking, strategic comparison and review, judgment of relating to external

identify constituent parts and functions of a process or concept, or de-construct a methodology or process, making qualitative assessment of elements, relationships, values and effects; measure requirements or needs

develop plans or procedures, design solutions, integrate methods, resources, ideas, parts; create teams or new approaches, write protocols or contingencies

review strategic options or plans in terms of efficacy, return on investment or cost-effectiveness, practicability; assess sustainability; perform a SWOT analysis in relation to alternatives; produce a financial justification for a proposition or venture, calculate the effects of a plan

analyse, break down, catalogue, compare, quantify, measure, test, examine, experiment, relate, graph, diagram, plot, extrapolate, value, divide

develop, plan, build, create, design, organise, revise, formulate, propose, establish , assemble, integrate, rearrange, modify

review, justify, assess, present a case for, defend, report on, investigate, direct, appraise, argue, project-manage

21

~~,~-;--.'~I",",~~~~ 'L:, ....... ~ "~ '" ~-- .-. ",

~r~ ~= ". .......,. -\ :::;. : ~..;-"; ,.,. ,1$. ··i ' ., ; ..I~. '. .., -" ' . ... ...

~~~~----~--.---:----:-1 Imitation copy action of watch teacher or trainer

another; observe and repeat action, and replicate process or activity

2 reproduce activity carry out task from from instruction or

Manipulation written or verbal

memory instruction

3 Precision perform a task or activity reliably, execute skill

with expertise and to independent of high quality without help assistance or

instruction; able to demonstrate an activity to other learners

-~~ - ..~ .~~ "._._ .1.

copy, follow, replicate, repeat, adhere

re-create, build, perform, execute, implement

demonstrate, complete, show, perfect, calibrate, control,

22


4 Articulation adapt and relate and combine construct, solve, integrate expertise associated activities to combine, coordinate, to satisfy a non- develop methods to integrate, adapt, standard objective meet varying, novel develop, formulate,

req u i rements modify, master

5 Naturalization automated, define aim, approach design, specify, unconscious and strategy for use of manage, invent, mastery of activity activities to meet project-manage and related skills strategic need at strategic level

23

open toReceive listen to teacher or ask, listen, focus, Willingness to experience, willing trainer, take interest in attend, take part, partk:ipate in an activity to attend to a stimulus; to hear session or learning discuss, acknowledge, getting and holding the experience, take notes, hear, be open to, attention of students

turn up, make time for retain, follow, learning experience, concentrate, read, do, participate passively feel

2 Respond react and participate actively in react, respond, seek Actively participates; participate actively group discussion, clarification , interpret, demonstrates interest in an object, activity or active participation in clarify, provide other phenomena, seeks or activity, interest in references and pursues this object, activity or phenomena outcomes, enthusiasm examples, contribute ,

for action, question and question, present, cite, probe ideas, suggest become animated or interpretation excited, help team ,

write , perform

24


3 Value attach values and decide worth and argue, challenge, Value or worth attached express personal relevance of ideas, debate, refute, confront, to an object, actlvlty or phenomena; varies from opinions experiences; accept or justify, persuade, simple acceptance to commit to particular criticise,commitment

stance or action

4 Organise or reconcile internal qualify and quantify build, develop, formulate, Conceptualize conflicts; develop personal views, state defend, modify, relate, values value system personal position and prioritise, reconcile, Compare and contrast, reasons, state beliefs contrast, arrange, and resolve conflict to build a consistent value compare system, emphasis on comparing and synthesizing values

5 Internalize or adopt belief system self-reliant; behave act, display, influence, characterise and philosophy consistently with solve, practice, values personal value set Adopt a vlue system for a length of time that contributes to a particular ' lifestyle' (I.e direct 25behavior)

Things to decide in assessment plan

• Getting the level right

• Types of assessment method to assess learning outcome

• How many assessment

• Using formative assignment

• Scheduling the assessment across the curriculum

• How can plagiarism be reduced

• Building in flexibility 26

More example of assessment plan P9 14/52 HEacademy UK


Any more Quality Assurance requirements that we missed?

3.2 Assessment Methods

3.2.1 Benchmarked Standards

• Assessment must be summative and formative.

• A variety of methods and tools must be used appropriately to assess the learning

outcomes and competendes.

• There must be mechanisms to ensure the validity, reliability, consistency,

currency and fairness of the assessment methods.

Formative and Summative Assessment. What are they really?

Are assessment and evaluation the same?

Continuous Assessment = Formative assessment?

Summative Assessment =Final Examination?


Is it another TEST?

CCSSL definition of Formative assessment is

• a process use by teachers and students Not to eva luate or grading

On-going and• During instruction not after class

• That provide feedback Where I am now, where am I going and how to get there

Using data • To adjust ongoing teaching and learning gathered

• To improved students achievement of Lea rn ing is achieved after

intended learning outcome action taken

Reflection

CURRENT PRACTICE

.-r-' I-

Test I FinalCWgp CW ~ Examinterview report

MULTI POINT SUMMATNE + SUMMATIVE (Developing Effective Assessment In Higher Education, Sue Bloxham and Pete Boyd, Open University Press 2007)


Marking scheme / criteria - how well they are align with LO?

Marker's skills?

What about giving feedback?

Marking Practice - Skills _

C03 - Ability to Conduct laboratory tests for determination of soil index and compaction.

manipulation observation

32 e.g. ASSESSING BY DIRECT OBSERVATION - have you got the skill to assess


l\.1anipulation: 5 - CXJtstarding Shows higl.:bility il m~ulating laboratory or demorstration tools! eq.Jipment! measuring ~paratus e:;pedally on settilg up Q(l>L.QllffuJJ:P.;j experimental! demonstration procedLres. Also shows hg, ability in ~lating materials accorcr.g to av3ability, needs and skills

4 - Excellent Shows ability to ~late welilaxlratory or dernorstration tools! eq..Jprnent! rreasLrilg apparatus especially on settilg up ~..w,fu)Ji:lg experimental! dernorstratioll procedLres . Able to ~late well materials accordi1g to avail.:bility, needs 31d skills.

3 - Gocd Able to ~Iate laboratory or dernoost:ration tods! eqjpment ! rreBSLri'lg apparatus espeaaly on settilg up and perforrTi'lg experimental! demonstration procedres. Able to manipulate materials accordilg to avatability, needs and skills .

2 - Satisfactory AverC¥je ma~ulatio n of laboratory or dernorstration tods! eq..ipment ! rneastrng ~paratus especially on setting up m..w....f.QwjJg experimental! demonstration proceru-es . Average rT1BIi:lulation of materials according to availability, needs and skills.

1 - Needs help DiffICult to ~late laboratory or demorstration tools! eq.Jiprnent! rneasumg ~atus especially on setting up and perforrTi'lg experimental! demonstration procedres. Difficult to rnaripulate materials accordilq to availability, needs and skills.

Observation 5 - CXJtstandlng Shows high .:bility n observng payilg more attention n irrportant procerues, .:DIe to spot changes or LnJSUBI occurrence,ibehavior. Record ·observation correctly and does extra notation on what has been observed.

4 - Excellent Shows ability in observing payilg attention on important procedures, able to spot changes or unusual occurrence/behavior . Record observation and make small notation on what has been observed.

3 - Gocd Shows averC¥je ability il observing paying attention on important proceciJres, averC¥je cblity to spot changes' or l.J!"U>UBI occurrence/behavior . Do minimal record on observation .

2 - Satisfactory Shows little .:bility in observng i~rtant procedures . Do mimal record on observation .

1 - Needs help Does not pay attention on important procerues neither make any record on i,/\,O;Jl; !.Y~

NORM -REFERENCE

Aims to discriminate between students and is designed so that student performance is distributed across a range, so that those who do better on the assessment task receive higher grades than those who do less well

10% - Grade A

30% - Grade B

33

CRITERION -REFERENCE

Students being judge based on the quality of their work on set criteria rather than the performance of other members of their particular cohort.

aBE

Desirable nowadays

Give protection to markers to justify their judgment

Implication ??

Refer to Orthogonal Assessment - UK HE Academy Example


CGPA? What does it tells about students achievement - case study

Should affective and psychomotor be report using numbers?

What will the report card look like in future?

How will it be used.

Establishing Developmental Framework Critical Thinking

Observation Interpretation Judgment Planning

Fouv\'clClt~Ov\'

KV\,owLecl g e f

'cleV\,t~ftJ The

pvobLeVlA f

expLove IV\,tey-pvetClhoV\,s § GoV\,V\,ec,hov\,s

fpv~ov~hze

A LtevVl.,Clhves r.

ev\'vLs~ov\'

stvClteg ~c, IV\,V\,ovCl hoV\,

GOv\'fuseclPerformance FClC,t

Patterns -g,~Cl, secl

~ -- JUVlA-pev

pe y-petuCl L PVCl g VlAClt~C, pevfOVVlAevAv\'ClLtJzev

F~v\'clev 1 D~st~vcguLsV1

Pr~ov~t~zeveLevl1 V\,t § ReLl1te

~ssues ~vveLevl1 V\,t l1SSUIIVl:pt~Ov\'S §

l1 v\'cl Iv\'fovVVtl1tLov\' b~l1ses

~V\,fOVVVtl1t~ov\'

Rel1cl AV\,l1 LtJze Just~ftJ pvos § cov\,scOv\'fl~ct~v\'g

l1ssuVVtptLoV\,sOP~v\'~ov\'s

Interventions

Steps in Critical

Thinking

I stvClteg~c,- Re-V~S~O~V

ArHwLl1te v~s~ov\'

Re~V\,te¥'flvet

~V\,fOVVVtl1t~ov\'

Step 1 Step 2 Step 3 Step 4

Steps for Better Thinking Performance Patterns, http://www.wolcottlynch.cOffi


http://www.wolcottlynch.cOffi

Where will the Program Learning Outcomes be infused across the Curriculum?

Course #1

Course #2

...

Course #N

Industrial Training

Capstone Course

Communicate orally and in writing

Introduce

Emphasize

Use

Use

Assess

Analyze, interpret and evaluate results

Introduce

Emphasize

Assess

Assess

Listen to diverse populations

Introduce

Emphasize

Emphasizel Use

Use

Assess

Demonstrate teaching effectiveness

Introduce

Emphasize Use

Assess

Assess g

7

Table 2 - The changing focus ofa programme as it moves through the different levels of study

Level 1 2 3 4

Subject Consolidation Development Authority Mastery

Skills Leaming Key Independence Research

Intellectual capacities

Knowledge Understanding Manipulation

Application Analysis Synthesis

Evaluation Judgment Reasoning

Formulation Conceptualization Reflection

Ref: UK HE Academy 38


Programme Learning Development Plan For Laboratory Work

Level Aims Materials Method Answer

Demonstration 0 Given Given Given Given

Exercise 1 Given Given Given Open

Structured enquiry 2 Given Given pert or whole

Open or part given

Given

Open Enquiry 3 Given Open Open Open

Project 4 Open? Open Open Open

As one moves up the levels one moves further towards the development of active deep learning

39


Demonstration

Usually done to demonstrate theoretical principles. Demonstrator is usually a lecturer or postgraduate student

Exercise

Tightly structures experiments designed to yield well-known results. Students learn to follow precise instructions and in so doing learn specific techniques of observation and manipulation. Careful reading of instruction c,an often reveal the answers required

Struc;tured Enquiry

Lightly structured experiments which require students to select materials and to develop procedures. Students develop problemsolving and interpretative skills as well as manual and observation skills 40



Open-ended Enquiry

Students identify a problem, formulate the problem clearly, hoose and design experimental procedures, interpret results and consider their implications. The constraints on the student may be time and the range of equipment and materials available. Open-ended enquiries used in miniaturized form the skills of the research scientist. They can be useful for a preliminary for project work .. Students develop problemsolving and interpretative skills as well as manual and observation skills

41


Projects

Based on long experiments of a series of experiments or field studies. The project may be selected by a student, offered by a supervisor, or by local industry or the community. The end products may be a dissertation, design plan, model, computer programme or a simulation. They enable students to explore a field deeply, they develop initiative and resourcefulness, they may stimulate a student's intellectual curiosity and they also develop project and time management skills. Guidelines for project are essentials

42


LO - Soft Skill: Communication

Year 1 : can communicate effectively in a format appropriate to the discipline(s) and report practical procedures in a clear and concise manner

Year 2 : can communicate effectively in a manner appropriate to the discipline(s) and report practical procedures in a clear and concise manner in a variety of formats

Year 3 (honours) : can engage effectively in debate in a professional manner and produce detailed and coherent project report

Master's : can engage confidently in academic and professional communication with others, reporting on action, autonomously and competently

43

8engkel Penilaian Outcome Based Education

DEPARTMENT OF CIVIL ENGINEERING, FACULTY OF ENGINEERING, UNIVERSITI MALAYA

EVALUATION OF COURSE OUTCOME ACHIEVEMENT SEMESTER 1 SESSION 2009/2010

Course Code KAEA2132 Course Title Soil Mechanics I Course Type Departmental Core Course

1. Course Learning Outcome Assessment Plan :

I :

COURSE LEARNING OUTCOMES (CO) METHOD OF DELIVERY

i

METHOD OF ASSESSMENT

I

Examination question

addressing the CO

C01 : Ability to define soil as in engineering context and relate problems associated with the definition with local soil condition

Lectures and coursework

I Examination and short report

Q1

C02: Ability to identify and differentiate the different types of soil and their properties and classify soil using British and I or Unified Soil Classification System.

Lectures and laboratory work

Short test and laboratory work/report

none

C03: Ability to conduct laboratory tests for determination of soil index and soil compaction.

Demonstration and laboratory work.

Direct observation on laboratory work/report

none

C04: Ability to solve calculation problem using mechanics involving physical properties, compaction, seepage and effective stress.

Lecture and tutorials

Examination & graded tutorial.

Q2, Q3, Q4

COS: Ability to show the use of soil mechanics concepts in engineering works

Lecture, active session and coursework.

Short report & group interview

Q1

Formative Assessment (55%)

Assessment method 1 CO addressed Weightage Short Test (soil Classification) CO2 10% Coursework report & interview C01, COS 20% Laboratory (MT1 & MT2) Direct observation and lab report)

C02, C03 I

20%

Graded Tutorial (Flownet) C04 5%

Summative Assessment (45%) (Students to answer 3 ques~ions out of 4 questions given) Assessment method CO addressed Weightage Soil definition and related problem C01 15% Calculation and solving problem related to Compaction Permeability & seepage Effective stress

C04 15% 15% 15%


2. Course Learning Outcome Achievement (CO Achievement)

100

90

- 80

--0 ~ 70

I: cu 60E cu 50>.!!! .I: 40 (.)

'" 300 0 20

10

0

---'8-8.8--------------- ---,;;--81.2

C01 C02 C03 C04 C05

Cou rse Outcome

(a) Indirect measurement via student perception through end of semester survey (* refer to attachment 1: end of semester student survey on Co- average)

100

90

80

- 70 ~ ~-c Q) 60 E Q)

> 50 .~ .s::. u 40« 0 0 30

20

10

0 C01 C02 C03 C04 C05

Course Outcome

(b) Direct measurement via course assessment methods (* refer to attachment 2: CO achievement - PI)

Note: PT= 80% students achieved more than 50% of each CO

Figure 1 : Course Learning Outcome achievement measured through indirect and direct measurements


3. Contribution of Course Learning Outcome (CO) to Programme Outcome (PO) :

I

OJ C))

~ ~

~ :J

~ iii :J

o c: o o :3 C))

IOJ Q;) '(/I (* from direct measurement- course assessment) 'c)) 'Ct n, ,0. c: g g':J

COURSE LEARNING OUTCOMES (CO) PROGRAMME LEARNING OUTCOMES (PO)

ASSESSMENT I

P01 P02 P03 P04 POS POG P07 POB POg P010

C01: Define soil as in engineering context and relate problems associated with the definition with local soil condition

72.2 Examination and I

short report

C02: Identify and differentiate the different types of soil and their properties and classify soil using British and / or Unified Soil Classification System.

88.9 88.9 , Short test and I

laboratory work/report

C03: Conduct laboratory tests for determination of soil index and compaction .

94.4 94.4 94.4 Laboratory work/report

I

C04: : Solve calculation problem using mechanics involving physical properties, compaction, seepage and effective stress.

22.2 22.2 Examination,

I laboratory report & graded tutorial.

COS: Show the use of soil mechanics concepts in engineering works

83.3 83.3 83.3 Short report and

I

group interview I

Percentage carried to PO 72.2 68.5 83.3 83.3 I

4.Students Performance According to University Grading System

MlSSM" 11 SEMRAI BlLAJIGU CALOW JllDGIKlJ'l' S UBJU 11120

SESI I 200t/2010 SD1I:S'11:R; 1

Koo S UB.JJ:K I DEA2132

BIL CALOW : 18

MD.. 55.47

STDDEV 9.55

8

• 6

~~.~~~.....--------.-------------------

2

J • .

•1. o

A- B-t B- C-t c c- D

Figure 2 Distribution of students performance to university grading system

Table 1 Students performance to the various bands

Failed Moderate Good

Band (0 - 49%) (50 - 69%) (>70%)

[C- and below] [C to 8] [8+ to A]

Nos. Students 3 12 3

Percentage (%) 17 66 17

(* total number of students IS 18)


5. Instructor's Comment

a. From Figure 1 (a) students believed that they have achieved all fives course learning outcome.

b. However, actual students performance via direct measurement (Figure 1 (b )), shows there are 2 eo's (Le. C01 and C04) not being achieved in accordance to Performance Target (PT : 80% students in class achieved more than 50% in each CO).

c. C04 was found to be the lowest (22.2%). Analysis shows that this is due to large percentage (30-45% from calculation question and 5% from graded tutorial) of C04 being measured through high stake assessment; that is final examination which may be disadvantage in this case.

d. Achievement of C03 being the highest perhaps due to direct observation using rubric being used.

e. C04 does not seems to cause dramatic negative effect on PO. f. Better assessment method such as interview and critical review report in PBL

assessment activity proved to enable in-depth learning. Implementation of PBL and non-traditional assessment method did not impaired achievement of CO.

6. Future Action Plan For CQI :

a. It is important for C04 in Engineering field to achieved its performance target (PT) . The fact that it is well below PT, assessment method for this CO need to be refined. This may be improved either by not confining large percentage of this CO to final examination or more effort need to be given to prepare students to such assessment method.

b. Keeping C04 assessment method, more calculation exercises are to be given as classroom activities in the next semester (change in classroom instruction I delivery method).

c. The overall design of this course is to be refined further for its constructive alignment. Better examination question setting is to be considered.(Le. make 01 - compulsory and answer 2 calculation questions from 02,03 and 04).

d. Assessment rubrics need to be refined from content-based to outcome-based for several assessment methods employed.

e. Assessing laboratory work via direct observation using rubric need to be improved.

7.1nstructor's Detail :

Name : Dr. Aishah binti Abu Bakar Room : L7-10, Block L, Engineering Tower, UM Tel. No. : 03 - 7967 5302 E-mail : [email protected]


Attachment 1 :

DEPARTMENT OF CIVIL ENGINEERING, FACULTY OF ENGINEERING, UNIVERSITI MALAYA

COURSE OUTCOMES SURVEY FORM SEMESTER 2 SESSION 2007/08

Name

Matrix No.

Su~ect/Code : ____~KA~E=A~2~1=3=2~:~S~0~il~M~e=ch=a=n=ic=s~I~______________________

Please rate your level of achievement of Course Outcomes in the scale from 1 to 5 as described below.

[ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ]

Poor Fair Avera~e Good Excellent

The Course outcomes fior t h· IS sub·IJect are:

No. Course Outcomes (CO) Achievement

1 2 3 4 5

1 Ability to define soil as in engineering context and relate problems associated with the definition with local soil condition

2

Ability to identify and differentiate the different types of soil and their properties and classify soil using British and / or Unified Soil Classification System.

3 Ability to conduct laboratory tests for determination of soil index and soil compaction.

4 Ability to solve calculation problem usmg mechanics involving physical properties, compaction, seepage and effective stress.

5 Ability to show the use of soil mechanics concepts in engineering works

If your rating for any of the outcomes is less than 3, please give comments and suggestions how it can be improved.

Thank you for your cooperation.


Attachment 1 - Student Perception on CO achievement Result

C01 Define soil as in engineering context and relate problems associated with local soil condition

O.OX

S1.3X

1S.Sx

O.OX

O.Ox

O.OX 20.0X 40.0x SO.OX SO.OX

C02 Identify and differentiate the different types of soil and their properties and classify soil using British and I or Unified Soil Classification System

I

100.0X

SS.SX

' S.3X

O.OX

O.OX

O.OX 20.0X 40.0X SO.OX SO.OX

C03 Conduct laboratory tests for determination of soil index and compaction

.S.3X

SS.Sx

25.0X

O.OX

I

O.OX 20.0X 40.0x SO.OX SO.OX


C04 ISol..,e calculation problem using mechanics in..,ol..,ing ph~sical properties. compaction. seepage

I land effecti..,e stress

0 O.OX [1] f--

0 O.OX [2] _S.3X

[3] SS.SX r--4 25.0X

[4] r--

11 68.8x 25.0x .-1511 S.3X O.Ox

,-

; ~

O.OX i--6 ••_.",..._ 3.81 O.Ox 20.0x 40.0x SO.OX SO.Ox r--

Answer rate 16

f---o lS

C05 Show the use of soil mechanics concepts in engineering works

0 O.OX [1] r- r--[2) 31.3x0 O.OX [3)

r- r--4 25.0X 43.S:.-: r- r---7 43_8x [4] 25.0 -:: r- r---

[5]5 31.3x O.OX r-I-

f-f-6 .._.",...... 4.06 O.Ox

f-f-

Answer rate 16 O.OX 10.0x 20.0X lS

30.0X 40.0x 50_OX f- r--I-r-

I j I I I 1 j

O u.rall F indino o f C O (S OIL --............. ""NIC S 1 1

4.30 4.19

4.20

4.10 40S

4.00

3.90 3.81 3.81 3.81

3.S0 .'

3.70

3.60


Attachment 2 : CO Achievement

OJ CIl

~ ~ ~ ::J

~ iii' ::J

oc:

:3 CIl

OJ til en CIl Q

rn Q c: ~ g. ::J

SOIL MECHANICS I-SEMESTER 1 I SESI 2009·2010

f _. T T -

~ ~T -r---,' t ~.=r -r· -[ t -+ -f r-r-~ , - -I I- .

10 10 10 10 10 5 15 15 15 15 No Matrie, No Labor atory/20 Cont assmnt 135 EHam/45 Grand

I MTI MT2 ".....,.t Tl ,Olin CW'R Tut I'll"".' QI Q2 Q3 Q4 S'll~t.t Total

1 KEA06003 6.3 0.0 8.3 14.5 5.0 9.0 0.0 23.5 5.0 1.5 0.0 8.5 36.3

2 KEA07004 6.3 5.0 11.3 7.0 7.0 9.0 4.0 32.0 12.0 8.5 7.0 27.5 70.8

3 KEA08000 6.3 5.0 lU 5.0 ' 8.0 4.5 29.5 1.0 6.0 0.0 7.0 47.8

4 KEAOBOOO 6.3 5.0 11.3 5.0 8.0 7.5 5.0 30.5 1.0 10.0 0.0 11.0 52.8

5 KEAOBOOO 6.3 5.0 lU 5.0 8.0 7.5 4.5 30.0 1.0 5.5 3.5 10.0 51.3 -

6 KEA08000 6.3 5.0 lU 6.0 8.0 7.5 4.0 30.5 10.5 1.0 1.5 13.0 54.8

7 KEA08001 6.8 7.5 14.3 9.0 8.0' 7 .5 5_0 34.5 4.5 8.0 14.0 28.5 75.3

:3 KEAO:3001 6.8 7.5 14.3 7.5 8.0 7.0 4.0 31.5 2.0 3.0 8.0 13.0 58.8

9 KEA08001 6.8 7.5 14.3 8.0 6.0 9.0 4.0 32.0 4.5 3.5 1.5 9_5 55.8

10 KEA08002 6.8 7.5 14.3 6.0 5.0 9.0 4.5 29.5 12.0 8.5 7.0 27.5 71.3

11 KEA013002 6.8 7.5 14.3 5.0 8.0 7.0 4.5 29.5 3.0 6.5 7.0 16.5 60.3

12 KEA08003 6.8 7.5 14.3 5.5 5.0 6.0 5.0 21.5 6.0 7.5 2.0 15.5 56.3

13 KEA08003 7.5 4.0 11.5 5.5 6.0 3.5 4.0 25.0 6.0 7.0 5.0 18.0 54.5

14 KEA08003 7.5 4.0 11.5 4.0 6.0 3.5 5.0 23.5 5.5 6.0 9.5 21.0 56.0

15 KEA08003 7.5 4.0 11.5 5.5 6.0 3.5 4.5 24.5 5.0 5.0 4.0 14.0 50.0

16 KES06002 7.5 4.0 11.5 7.5 8.0 4.0 31.5 2.0 2.0 3.0 7 .• 50.0

17 KES08002 7.5 4.0 11.5 0.0 5.0 6.0 5.0 21.0 7.0 6.5 5.0 18.5 51.0

18 KES08002 7.5 4.0 11.5 2.5 5.0 6.0 3.0 21.5 8.5 2.0 1.5 12.0 45.0

74.5 52.5 69.0 78.0 55.4

CO Achie...",ment : %of ~tudents in class that achie... ed more than 50% of a particular CO

PT =CO Achie... ement : 80 % of students: in class that achie... ed more than 50% of a particular CO

I . t + ,. j I

jI !- I

- .. --+- - -4. CW'lnt MTI MTl Tut CW'lnt J~

.t Q2 r -

CW'R MT2 I MT2 CW'Rl iQl , Tl Q314

Indi ...idual Attainment of CO

COl C01(:,q CO2 C02(%J C03 C03(:'-;) C04 C04(%) C05 C06(%)

0.5 54.3 0.4 35.8 0.3 31.3 0.0 4.3 0.7 70.0

0.8 80.0 0.6 60.8 0.6 56.3 0.6 55.7 0.8 80.0 * 0.8 75.0 0.5 54.2 0.6 56.3 0.2 23.0 0.8 75.0

0.8 17.5 0.5 54.2 0.6 56.3 0.3 32.0 0.8 17.5

0.8 17.5 0.5 54.2 0.6 56.3 0.3 29.0 0.8 17.5

0.7 74.3 0.6 57.5 0.6 56.3 0.1 14.3 0.8 17.5

0.8 17.5 0.8 17.5 0.7 71.3 0.6 63.0 0.8 17.5 P

0.8 75.0 0.7 12.5 0.7 71.3 0 .3 34.0 0.8 75.0

0.8 75.0 0.7 74.2 0.7 71.3 0.3 27.0 0.8 75.0

0.7 74.3 0.7 67.5 0.7 71.3 0.6 57.1 0.7 70.0 P

0.8 75.0 0.6 64.2 0.7 71.3 0.4 42.0 0.8 75.0

0.5 48.6 0.7 65.8 0.7 71.3 0.4 41.4 0.6 55.0

0.4 44.3 0.6 60.0 0.6 57.5 0.5 45.7 0.5 47.5

0.4 42.9 0.5 51.7 0.6 57.5 0.6 58.6 0.5 47.5

0.5 47.5 0.6 56.7 0.6 57.5 0.4 37.0 0.5 47.5

0.5 48.6 0.6 63.3 0.6 57.5 0.3 25.7 0.8 75.0

0.5 51.4 0.4 38.3 0.6 57.5 0.5 47.1 0.6 55.0 e 0.6 55.7 0.5 46.7 0.6 57.5 0.2 18.6 0.6 55.0 P

64.1 58.6 60.3 36.4 67.4

72.2 88.9 94.4 22.2 83.3

NA A A NA A - - - - - - - ---- -

8

Orthogonal assessment This is probably the most radical change in the philosophy of assessment practice that has come as a result of the move to an outcomes model of programme design and assessment (such as described in the UK.SPEC).

Traditional assessment practice involves a number of assignments which togeti1er produce an average mark or grade for the module. Consider the assessment strategy as described in the table below. For the purposes of this analysis, let us assume that there are two assessments, each having four criteria of equal weight (25%). Cmn represents the nth criterion of assignment m. We will assume also that the assessors are using a percentage marking scheme and averaging numerically. The pass mark is 40%. The same argument v%uld apply usi'ng grades, in which case an algorithm would be used for averaging.

Assessment 1 Assessment 2 Criteria used for judgments

C11 C21

C12 C22 C13 C23 C14 C24

A student receives marks as shown in the next table:

Assessment 1 Assessment 2 Average mark (%)

Criteria used 12 9 for jUdgments 12 11

15 10 8 7

Mark for assignment (%)

47 37 42

The student's average mark for the module is 42%. They have failed the second component (37%), but the rules will allow compensation for marks of 35% or above in one component. Therefore compensation will be applied to the second assignment, and the student passes the module.

However, with an outcomes model, the process is different. Consider the model below. In this outcomes based model, the learning outcomes for the module have been established in the module specification template (assume there are four). These are used consistently across the assignments. That is, the judgments for each assignment are made against the

3 Grade drift is a term used to describe the tendency for a marker to unconsciously revise their judgment of performance. If the first assignment marked is of a high standard , then those that follow will suffer by comparison. Alternatively , if the first few scripts are of a lower quality and receive marks or grades, and then a high qual1ity script is encountered , those that follow will receive 100ver grades or marks even if they are of the same quality as the first few. Explicit criteria tend to 'tie down' judgments and make them more consistent

Section 2, page 16 Assessment of Leaming Outcomes Engineering Subject Centre


(relevant) learning outcomes, using the cnteria for each learning outcome. This analysis uses the same marks matrix as in the analysis above, but this need not be the case.

The marks are now averaged across the learning outcomes (not the assignments) to give an average mark (or grade) for each learning outcome.

Assessment 1

Assessment 2

Average mark for each outcome

]%) L01 12 9 42 L02 12 11 46 L03 15 10 50 L04 8 7 30 Averaqe mark (%) 42

The implications for the student are very different in this case. The average mark for the module is still 42%. However, the student has 'failed' on learning outcome 4. That is, they have not demonstrated that they have attained this outcome. There is no compensation for this; students must demonstrate each and every outcome, so this student will have failed the module. Any attempt at recovery through a retaken assignment needs only to forus on learning outcome 4.

Note: 1. In effect, the student has been given two attempts to demonstrate L04 and has failed

on both occasions. 2. Even if the average mark is a fail mark for a learning outcome (30% as in L04), if any

attempt, in any assignment, results in a pass mark (10 or greater in this example), then the student has, at some time during the module, demonstrated this learning outcome, and should IpaSS. For example, marks of 10 and 5 would still result in an average mark of 30% for this learning outcome. This would traditionally be seen as a fail mark, however, the student has demonstrated the outcome in the first assignment, so the module is 'passed'.

Implications for recording and decision -making For the assessor

1. For each assessment, judgments are made (and marks are awarded or grades are assigned) against the learning outcomes being assessed (using the relevant criteria for the LOs).

2. Orthogonal assessment removes any decision-making in regard to compensation, as it is not necessary to compensate for individual components of assessment.

3. There can be no compensation for a 'failed' learning outcome. 4 . It is possible to 'pass' each assignment, but to fail the module. For example, using the

grid above, the student has achieved 47% for assignment 1 (pass). If they achieved 15 for L01 in assignment 2, then they would achieve 43% for this assignment (pass) and an average of 45% for the module (pass). However, they still have not achieved a pass for L04 on either occasion. This is what matters, so they will fail the module.

5. Feedback to the student on each assignment will need to focus on their attainment against the Los as well as the assignment itself. The student will need to be advised that, even though they may have achieved a 'pass' mark for the assignment, they have failed to demonstrate a learning outcome, and they must demonstrate this LO in the remaining assignment


Grade Descriptors for final honours undergraduate

OJ Cb

~ [ ~ ::I

~ iii' ::I

o

i c:

~ g: Q.

~

80-100 AA

70-79 A

60-69 B

50-59 C

40-49 D

35-39 F

30-34 F

0-29 F

Students meet all the requirements for an a grade. There is evidence of exceptional scholarship, including critical evaluation and synthesis of issues and information that generates originality and challenges existing approaches. Accurate and detailed use of range of evidence. Comprehensive knowledge and understanding of theories, principles and concepts. Student has met learning outcomes (LOs) of the assessment with evidence of comprehensive and up-to-date knowledge and understanding of concepts and theories and their relationship. the work shows a detailed appreciation of how aspects of the subject are uncertain, contradictory or limited. The work adopts a well-sustained critical approach using a breadth of evidence, reasoning and reflection. Work shows evidence of a mature and independent approach to problem solving, the student can create appropriate hypothesis and select, justify and use imaginative and innovative approaches in their investigations Student has met the LOs of the assessment with evidence of comprehensive and up-to-date knowledge and understanding of concepts and theories and their interrelationship with an awareness of how aspect s of the subject are uncertain, and contradictory or limited. The work adopts a critical approach using breadth of evidence, reasoning and reflection Work shows evidence that the student can confidently and autonomously in the identification and definition of complex problems and select, justify and use approaches aimed at their resolution. Student has met the LOs of the assessment with evidence of detailed knowledge and understanding of key concepts and theories, including an awareness of the provisional nature of knowledge. The work shows evidence of a general critical approach using individual judgment and reflection, although there is some limitation in the ability to conceptualize and / or apply theory Work shows evidence that student can act without guidance in the identification of complex problems and can apply knowledge and skills to their resolution Student has met the LOs of the assessment, with evidence of knowledge and understanding of key concepts and theories including basic recognition of the complexity of the subject. The work is for the most part descriptive rather than based on argument and logical reasoning. Work shows evidence that the student can apply appropriate learning accurately to complex problems and / or practical contexts. Student has not met all the LOs of the assessment, with only basic knowledge of key concepts and theories and weaknesses in understanding. there is little or no recognition in the complexity of the subject. Work is largely descriptive with some unsubstantiated assertion. Analysis is minimal or contradictory. Unable to always apply learning accurately to complex problems and / or practical contexts. For profesional courses any work which contains evidence of, or reference to, unsafe or dangerous practice should be deemed to fail. Student has not met all the LOs, with inadequate knowledge or understanding of key concepts and theories. there is no recognition of complexity of the subject. The work is descriptive and uncritical, with unsubstantiated assertion and a lack of analysis. Insufficient understanding of given tools/methods to apply learning accurately or safely to complex problems and / or practical contexts. Students has failed the majority of the LOs.

~ source: extract from grade descriptors for levels 0-5, University of Cumbriag:

BENGKEL PENILAIAN OUTCOME BASED EDUCATION

Documents

Transcript of BENGKEL PENILAIAN OUTCOME BASED EDUCATION