4assessing student achievement of learning...

4. Assessing student achievement of learning outcomes

The fourth item in eVALUate asks students their level of agreement with this statement:

The learning resources in this unit help me to achieve the learning outcomes. The learning resources could include print, multimedia and online study materials, and equipment available in lectures, laboratories, clinics or studios.

This document aims to assist teachers to ensure that learning resources facilitate students’ achievement of learning outcomes.

1. What are learning outcomes and why are they so important 2

2. Key to successful learning: aligning assessment with learning outcomes 2

3. Reviewing your assessment strategy 2

4. Designing assessments which allow students to demonstrate their achievement of the learning outcomes

3

5. Key assessment terms 3

6. Common assessment tasks 4

7. Assessing group work 11

8. How can rubrics help with assessment? 17

9. For further assistance 23

Using eVALUate to improve student learning

Teaching Development Unit, LSN, Curtin University of Technology 2

1. What are learning outcomes and why are they so important? Unit learning outcomes are what students are expected to know, understand or be able to do in order to be successful in a unit. Unit learning outcomes begin with an action verb and describe something observable and measurable. Learning outcomes are the most important section of your unit outline. They

• clearly communicate the type and depth of learning students are expected to achieve;

• provide a benchmark for prior learning assessment;

• clearly communicate graduates’ skills to prospective employers; and

• indicate to the students how they might organise their learning programme.

2. Key to successful learning: aligning assessment with learning outcomes

One of the keys to successful learning is the aligned curriculum (Biggs, 2003): this means that learning outcomes are clear, learning experiences are designed to assist student achievement of those outcomes, and carefully designed assessment tasks allow students to demonstrate achievement of those outcomes. This concept is illustrated in Figure 1:

Figure 1. The aligned curriculum

3. Reviewing your assessment strategy Begin by thinking about your current assessment practice. The questions below will help guide your reflection.

• How do you currently assess your students? List the methods you use.

• Is each assessment worth doing and can/do you explain to your students why?

• Can you explain how the assessment methods you currently use are matched to your learning goals/objectives?

• What skills and competencies do you want your students to leave your unit/course with?

• Approximately how much does each assessment process cost students and staff in terms of time taken and resources used?

1. The learning outcomes are clear.

2. The learning experiences (face-to-face and virtual) are designed to help students achieve those learning outcomes.

3. The assessment tasks allow the students to demonstrate their achievement of those learning outcomes.



• Do you feel you might be over assessing? How do you know?

• What criteria do you use? Are they yours, or can you involve students themselves in formulating them?

• Do the students know the criteria? Do they really understand them?

• Is the feedback you give your students clearly related to your assessment criteria?

• How well does the feedback students receive on assessed work help them to know how they are doing?

• How much practice and guidance do students get in the chosen assessment methods?

• What assessments do students enjoy and why?

• How do you know that the students find your assessments useful?

• In what ways do the assessments help your student learn?

• Review your assessment strategy regularly. It can be even more productive when done in partnership with your students and colleagues

4. Designing assessments which allow students to demonstrate their achievement of the learning outcomes

Assessing learning can profoundly shape the educational experiences of students. One of the challenges of effective assessment is to ensure that there is a close alignment between the learning goals, the teaching and learning activities aimed at meeting learning goals and the assessment tasks used to assess whether learning goals have been met. Current best practice includes assessment which is aligned to learning goals which focus not only on content knowledge but also on process and 'generic' skills.

5. Key assessment terms

Assessment The collection of information about the nature and extent of learning outcomes/any procedure used to estimate learners learning. The term is derived from the Latin (assidere meaning “to sit beside”).

Measurement Representation of assessment information by a number or grade on a scale of some kind. Answers the question, How much?

Evaluation The making of judgements about the value of a grade and/or the nature and extent of learning outcomes. Answers the question, How well?

Assessment task

An instrument or systematic procedure by which assessment information is collected.

Formative assessment

Ungraded assessment task used before or during learning to support planning and/or diagnosis and/or to provide feedback about learning progress/offers advice and feedback which does not contribute grades towards the final result.

Summative assessment

Graded assessment task used following learning which counts towards the final result.

Validity Degree to which the assessment task measures what it is intended to measure.

Reliability Degree to which the assessment task consistently yields the same result.



Norm - referenced

Uses the performance of a group of learners to rank order learners or 'grading on the curve'. Number of learners who can receive distinctions, credits, passes or fails is set.

Criterion - referenced

Establishes the criteria for performance and any learner meeting the criteria receives the associated grade. Every student can potentially achieve the highest grade.

Authentic assessment

Assessment tasks which test whether a learner is able to demonstrate their learning outcomes in a situation which is as close as possible to a real world context.

6. Common assessment tasks There are many different assessment tasks you can use to assess your students. The following will help you choose the right assessment task. Remember that whatever assessment task you decide to use, it should be clearly aligned with the learning outcomes and teaching and learning activities in your course.

6.1 Short form test Short form tests are also known as objective tests. They include multiple choice, completion (or cloze), true-false and matching types, of which multiple choice is the most commonly used. A multiple choice test item consists of a statement, called the stem, and several alternative statements one of which is the correct answer and the others are distracters. For example, Formative assessment refers to:

• a practice test

• a test used to determine a grade

• a test used to determine prior knowledge

• a test to monitor learning progress

Pros Cons

Measures wide sample of content Difficult to set items which assess more than recall

Easy and quick to score Time consuming to produce

Good for reviewing content Encourages guessing

Provides formative feedback Restricts creative students

Provides fast feedback Favours male students

Items can be reused Nourishes illiteracy

Marker reliability high Poorly constructed questions can give clues to students

Difficult to interpret wrong answers



Hints for writing multiple choice items

• The stem should consist of a single, clear idea. It should make sense independent of the rest of the question.

• Avoid stems stated in negative terms as these are more difficult to understand and may cause confusion.

• Make sure that all the alternatives are grammatically consistent with the stem and similar in form and length to one another.

• Make the distracters plausible by using common misconceptions and typical student errors.

• If you use the alternatives 'none of the above' and 'all of the above' include them as the incorrect answer about 75% of the time.

• The correct answer should appear without pattern and equally often in each of the alternative positions.

6.2 Short answer test Short answer questions require a brief answer consisting of a phrase, sentence or short paragraph. For example, “Write a brief definition of formative assessment”.

Pros Cons

Measures relatively wide sample of content Relatively difficult to set compared to short form

Reasonably easy and quick to score Difficult to establish criteria

Encourages clear and concise expression Scoring may be subjective

Encourages literacy May encourage guessing

Good for reviewing content Little opportunity to display argument and originality

Items can be reused

Hints for writing short answer questions

• Be clear about what you are asking.

• Avoid using phrases straight from the text book.

6.3 Essay Essays require students to select, organise and integrate material on a given topic. They also test writing skill and the ability to develop an argument and use evidence to support it. Essays may vary from a single page (about 300 typed words) to major assignments of ten pages (3000 words). Essays may be written under timed exam conditions or set as research assignments. For example,

Compare summative and formative assessment with reference to assessing higher order learning as defined by Bloom's taxonomy for the cognitive domain (Bloom, 1956).



Pros Cons

Helps students to develop writing skills May not sample a wide range of content

Can reveal errors in understanding or misconceptions

Questions may not be well thought out

Takes less time to set than short form questions such as multiple choice

Hand written responses may be graded on factors other than the content, such as legibility

Requiring students to write can improve understanding of a topic

Time consuming to grade

Helps students to develop information literacy skills

Consistency in assigning marks is difficult to maintain

Subjectivity may affect fair grading

6.4 Performance test Performance tests involve either a hands on activity, such as using microscope correctly or taking a patient history, or the development of products, such as developing a building design or software package.

Pros Cons

Encourages students to take ownership of the learning process

May be time consuming to set, present and assess

Replicates real world conditions or contexts Can be difficult to determine assessment criteria

Students have the opportunity to demonstrate mastery of what they have learned

Can prompt performance anxiety in students

Can assess a range of skills or outcomes including generic skills

May require additional resources

Allows for a variety of tasks Makes comparison between students difficult since the products may be very different

Encourages active learning Subjectivity may affect fair grading

Hints for using performance assessments

• Ensure that the task is clearly related to unit goals.

• Choose tasks which can be completed within the time allowed.

• Check that any special resources needed are available to all students.

• Make sure that students have the knowledge and skills needed to carry out the task.

• Describe the task and its requirements clearly in writing. Allow time in class for discussion and clarification.

• Provide clear criteria for assessment when the task is set. Consider involving students in deciding on the criteria.

• Require students to provide progress reports or to submit material at various intervals before the final submission date.



• Check on student progress regularly and have contingency plans if something goes wrong.

• If possible involve others in the assessment process (e.g. another instructor, peers or the participants themselves).

6.5 Written report The report is a common way of presenting information and recommendations or conclusions related to a specific purpose. Reports are often used as assessment task because well developed report writing skills are important in many professional contexts. Reports are written based on gathering and analysing information using a discipline specific methodology and format. They can be used to assess laboratory, field work or case studies.

Pros Cons

Replicates real world activity May encourage students to fabricate data to make the report 'look good'

Marking for grading using a template is relatively fast

Inconsistent marking between multiple markers can arise

Gives students practice in writing using a standard format

Can prompt performance anxiety in students

Can assess generic skills such as information and computer literacy

Marks allocated may not reflect time and effort needed to complete the task

Allows for a range topics and foci Producing a report as an assessment task may not align with the learning objectives as well as other assessment tasks

May encourage reflection and problem solving Subjectivity may affect fair grading

Hints for using written reports

• Be clear about how marks are allocated to each section of the report.

• Weight marks according to the learning objectives which the task is assessing.

• State clearly what the report format is, include exemplars of good and poor reports.

• Tell students whether and how language (grammar, spelling, punctuation) will be assessed.

• Keep the number of reports required in a semester to a reasonable number and match the percentage of overall marks to the time and effort needed to produce a good quality report.

• Explicitly teach students effective report writing skills before using the report as an assessment task.

6.6 Project Projects are an extended piece of work involving inquiry based activities. Project may be small or large, undertaken by individuals or in groups and have outcomes such as a report, design, art work, working product.



Pros Cons

Allows for greater student involvement in and responsibility for learning

May be time consuming to develop and mark

Assesses in-context student learning Can be difficult to determine assessment criteria

Encourages initiative, independence and problem solving


Can assess a wide range of skills or outcomes including generic skills particularly time and task management


Provides an opportunity to showcase skills and achievements

Makes comparison between students difficult since the projects may be very different

Is comprehensive, multidimensional and flexible

Allows students to explore a topic in depth

Hints for using projects





6.7 Presentation Presentations are usually made orally to a class on a prepared topic and may include the use of presentation aids such as overhead transparencies, handouts or audiovisuals. This assessment may be undertaken individually or as a group. Presentations may take different forms such as role plays, facilitating group activities, debating, presenting a product, question and answer time, and formal speeches.

Pros Cons


Can be very time consuming of limited class meetings

Marking using criteria is relatively fast and reliable Subjectivity may affect fair grading

Allows for immediate feedback to the student Can prompt performance anxiety in students

Allows for a variety of topics

Provides an opportunity to display argument and originality



Hints for using presentation tasks

• Design and use an assessment pro-forma with weightings for each aspect of the presentation.

• Inform students of the requirements and criteria.

• Set minimum and maximum time limits for each presentation.

• If group presentations are used work out beforehand what is expected from each member of the group and how marks will be distributed among group members.

• With group presentations, include some assessment of the working of the group as well as of the presentation.

• Provide students with opportunities to develop and practice oral skills.

6.8 Poster A poster is a visual representation of a topic or the outcomes of learning activity. They can use different media, including online technology, and can be created individually or in groups.

Pros Cons

Provides an opportunity to display creativity and originality

Can focus unduly on presentation rather than content or understanding



Marking using criteria is relatively fast and reliable

Makes comparison between students difficult since the posters may be very different

Allows for a variety of topics Subjectivity may affect fair grading

Encourages active learning

Has potential for peer assessment

Hints for using poster presentations

• Ensure that the task is clearly related to unit goals.

• Check that any special resources or materials needed are available to all students.

• Make sure that students have the knowledge and skills needed to carry out the task.



• If the task is large, require students to provide progress reports or to submit material at various intervals before the final submission date.





6.9 Journal Journals (also called learning logs or learning diaries) are written by students over a period of time, such as a semester, in order to record and reflect on their personal learning experiences and outcomes. They provide an opportunity for students to express their feelings, thoughts and beliefs about the content and process of learning and themselves as learners using an informal writing style and structure.

Pros Cons


May be time consuming to develop and assess

Encourages self-assessment and reflection Can be difficult to determine assessment criteria

Provides valuable insight into student feelings, thoughts and beliefs



Requires time to establish the required high-trust, low risk environment

Encourages regular and extended writing May raise issues of privacy and confidentiality

Students may resist undertaking regular writing and fabricate or sanitise journal entries

Hints for using journals


• Allow class time for journal writing.

• Suggest areas for students to focus on, possibly using guide questions or statements.

• Provide frequent feedback, especially in the early stages.

• Consider keeping a journal yourself and share entries with your students.

• Acknowledge the value of student comments by responding to journal items.

• Take time to establish a high-trust low-risk learning environment.

• Provide examples of journal entries.

6.10 Portfolio A portfolio is "a purposeful collection of student work that exhibits the student’s efforts, progress and achievements in one or more areas. The collection must include student participation in selecting contents, the criteria for judging merit and evidence of student self-reflection" (Paulson, Paulson & Meyer, 1991, p60).

Pros Cons


May be time consuming to develop and assess

Assesses in-context student learning Can be difficult to determine assessment criteria

Encourages self-assessment and reflection Subjectivity may affect fair grading





Provides an opportunity to showcase skills and achievements

Makes comparison between students difficult since the portfolios may be very different


Hints for using portfolios


• Provide examples of completed portfolios.

• Involve students in selecting portfolio items.

• Include compulsory items which show student learning activities, self-reflection and self-evaluation.

• Use portfolios for different functions at different times of the year.

• Ensure that the length of a portfolio is the equivalent of a written assignment you would normally set, such as an essay or report.





7. Assessing group work Group work can help develop specific generic skills sought by employers such as:

• teamwork skills (skills in working within team dynamics; leadership skills);

• analytical and cognitive skills (analysing task requirements; questioning; critically interpreting material; evaluating the work of others);

• collaborative skills (conflict management and resolution; accepting intellectual criticism; flexibility; negotiation and compromise); and

• organisational and time management skills.

Group work may reduce the workload involved in assessing, grading and providing feedback to students. Provided the model of group work adopted is workable and effective, group/groupwork assessment can reduce/streamline assessment by reducing the number of pieces of work to be assessed. However, the tasks of monitoring and assisting students through the process my also add to overall work load.



7.1 Dimensions of assessing group work There are three basic dimensions to assessment of group work:

• Students' demonstrated abilities of working effectively as a team member (process);

• Students demonstrated knowledge and application of the skills of successful task completion (process); and

• The quality of the output or product of the group assessment task (product).

Groups can produce tangible products such as posters, models or artefacts, formal reports and electronic or other forms of media product. They can also submit records of meetings, planning sheets or other monitoring documents as evidence of their progress. The output can also include (or consist of) a performance or seminar presentation.

The assessment programme of any unit must be designed to assess students' achievement of the unit learning outcomes. If the skills of teamwork are stated explicitly in the learning outcomes, these skills must be learnt, demonstrated and assessed. If the outcomes state that student groups produce a certain type and/or quality of product (presentation, poster or report), students must also be assessed on this according to specified criteria. Staff must consider carefully how and why they are assessing students - and be prepared to provide them with a written justification for each assessment task. This is most important for group-based assessment.

If group work is being used solely to aid learning, to motivate students or to reduce staff marking loads, the rationale for it should be made clear to the students and it should not require student learning that detracts from their achievement of other stated learning outcomes.

Students will also need explicit guidelines on what is acceptable and what is not acceptable in terms of their collaborative versus individual work and assessment. They need to be alerted as to how incidents of academic misconduct, such as plagiarism or collusion, might occur in the exercise and how such incidents can be avoided. More information on this is available in the Curtin policy on Plagiarism. This type of information should be included in the written guidelines for the assigned work.

Assessment can be conducted by external assessors (experts in the field), lecturer or tutor, groups of peers, through self-assessment or by some combination of these. The advantage of a combination of assessors is that it allows multiple perspectives of the work to be judged and it reduces the chance of bias that might be inherent in a single form of assessment.

Students also need to know what will happen should one or more group members withdraw or if one or more contributes so little that it jeopardises the likelihood that the group can complete its task. They also need to be aware of how to deal with conflict within groups and what is the appeal process should students not be able to successfully negotiate among themselves about distribution of marks.

7.2 Planning group assessment There are different ways of allocating marks to individual students for the quality of their contribution to the group-working process or the group work product.

• Which assessment model is appropriate for the tasks you set for your students? In the first instance, decisions about the model of assessment need to made with respect to the following:

• What will be assessed - the product of the group work, the process of the group work, or both (and if the latter, what proportion of the total mark will be allocated to each)?



• What assessment criteria will be used and who will determine them - you, your students or both?

• Who will apply the assessment criteria and determine marks - you, your students (peer and/or self assessment), an external assessor or a combination?

• If groups are to be given a total mark to 'share' according to individual contributions, how will the shared mark be determined and distributed?

7.3 Group assessment criteria: Models and examples Students need to have the assessment criteria specified before they start work on the assessment task. They should know what outcomes they will be expected to demonstrate and how the evidence of their work needs to be documented. Where students are to collaboratively decide the criteria, this process should occur at the beginning and all students then receive a written copy of the agreed criteria.

The four marking models described below represent different ways of recognising the contribution of the individual student. In each case, it is assumed that students have written criteria that enable them to award marks and that, where relevant, they are given an appropriate algorithm to determine the distribution of marks.

Model 1: All students get the same mark or grade regardless of individual effort, contribution or ability

If teams in authentic working environments in your particular area, customarily succeed or fail on the basis of team performance alone, and the contribution of individuals is of little importance, you may want to assess students this way. The marks can be determined by the quality of the product or the process or both. All students in the group are awarded the same mark.

Staff must recognise that equal marks regardless of individual input is likely to be contentious. If proceeding this way, staff must make their expectations explicit in the unit outline. An outcome should be that students learn to accept the (stated) rationale for the type of group assessment and work within the (explicit) guidelines or rules. Most students have only experienced educational environments where assessment is competitive and they are unlikely to understand that it can, or should, be anything but competitive. Staff will need to address students' beliefs and concerns about this.

Example 1:

The marks are determined by quality of the product and each student gets the same mark. For example, a group of students prepares a business plan for a company. The business plan is awarded a mark of 16/20. Each student thus gets 16 marks.

Example 2:

Where there is to be a separate group process mark, all students get an equal share. For example, the group preparing the business plans submits documents that provide evidence about how the group went through the process of researching and assembling the business plan. The group is awarded a mark of 7/10 for the process, so each student gets 7 marks.

Model 2: Students get a proportion of their marks according to individual contribution to the process

Where students' individual contributions to the process are to be assessed, there needs to be a way of doing this reliably and fairly. Students themselves are probably in the best position to make such judgements and can do so quite reliably provided they are given the opportunity and understand the



criteria (G. Brown, Bull & Pendlebury, 1997; James, McInnis & Devlin, 2002b; Race, 1999). The process can be either by self-assessment or peer-assessment or both. The teacher or tutor may also have input. Various examples are outlined below - but each has limitations or associated problems. A more comprehensive account of how this model might be implemented forms Appendix 1, at the end of this document.

There are two ways of determining the marks for individual contribution to process:

• Each group gets a mark, determined by the quality of the product, to share or distribute among themselves (see Example 3); or

• Each group is given a pre-determined mark, irrespective of the quality of the product, to distribute among themselves (see Example 4).

In both of these examples, there is a further variation in the determination of each student's mark - either a fixed number of marks is distributed among the students (variation A) or each student gets a proportion of a fixed number of marks (variation B).

Example 3(A): Distribution of a fixed number of marks

A group of four students prepares an environmental impact report. They get 24/30 marks for the report. They therefore get 96 marks (= 24 x 4) to distribute among themselves for individual contribution to the process of researching and preparing the report. Some students get more than 24, and others less. Specified criteria and a distribution algorithm are needed.

The problems with this method are that: • there is effectively no maximum mark for a given student (unless

it is capped at the maximum - 30 in this example); • high marks for one student reduces the maximum possible marks

for the others; • it is unlikely that students from different groups, who make equal

contribution to their respective groups, will get equal reward for effort; and

• the tendency is for groups to allocate the marks equally, regardless of differing contributions.

Example 3(B): Proportional marks method

In the example above (3A), each member of the group would have available to them the full 24 marks i.e. they would each get a mark out of 24 for the quality of their contribution to the process, regardless of the mark allocated to the others in the group. A mark above the maximum is not possible. Specified criteria are needed (but no distribution algorithm).

The problems with this method are that: • it is unlikely that students in different groups, who make equal

contribution to their respective groups, will get equal reward for effort; and




Example 4(A): Distribution of a fixed number of marks

Groups of five students each prepare materials for, and deliver, a seminar. All groups are given 100 marks (20x5) to distribute according to individual contribution to the preparation process, regardless of the quality of the actual seminar. Thus some students will get more than 20 and some will get less. Specified criteria and a distribution algorithm are needed.

The problems with this method are that:

• there is effectively no maximum mark for a given student (unless it is capped at some arbitrary maximum, which in this example might be 25);

• high marks for one student reduces the maximum possible marks for the others; and


Example 4(B): Proportional marks method

In example above (4A), each member of a group would have available 20 marks i.e. they would each get a mark out of 20, regardless of the mark allocated to the others in the group, for their contribution to the process. A mark above the maximum is not possible. Specified criteria are needed (but no distribution algorithm).

The problem with this method is the tendency for groups to allocate the marks equally, regardless of differing contributions.

Each of these methods implies that students have criteria on which they know that they are to be judged, and that they use these criteria when peer- and/or self-assessing. Students will have a better understanding of the criteria if they are involved in deciding what they should be.

Model 3: Students get a proportion of their marks according to individual contribution to the product

This method is likely to be more controversial depending on how students have allocated (or how they have been directed to allocate) the roles or tasks that directly contribute to the product quality. If teams are working effectively, they will make use of the diverse skills and abilities of the members to maximise the quality of their product. To then penalise any one member because their role is not seen as important as another can be seen as discriminatory. One might also argue that this goes against the philosophy of 'teamwork'.

The assignment context must be one for which it is possible to differentiate between the different roles/tasks that students are required to undertake. The allocation of students to particular roles must be fair, and there must be performance criteria for each role. Students are then judged against the criteria for the role that they performed.



While there might be a number of possible ways of allocating marks, only the one described in example 5 is recommended. In this example, the contribution of each student can clearly be seen and verified.

Example 5:

A group of three students prepares an educational website. One student has the role of website designer/manager, another is the instructional designer and the third researches the content for the site. Each student has criteria for the quality of their aspect of the completed website, and is required to explain or justify their contribution.

The website is awarded 18/20 for web design, 15/20 for instructional design and 17/20 for content. Thus the students get 18, 15 and 17 marks respectively.

Model 4: Students get a proportion of their marks according to their individual knowledge about the process or the product.

An assessment variation, perhaps in addition to one of the previous models, might be that students provide further written evidence of their learning. They might be asked to write an account of their understanding of the process and/or their learning in a separate submission or in response to an examination question. They might also be asked questions that test their individual knowledge about the product. This can serve to support (or otherwise) students' claims about their contribution to processes or product. These will result in extra marking.

Evidence of an individual student's knowledge, can also take the form of reflective journals, purposeful portfolios, global or itemised criterion-based reports (usually proformas), documents (such as minutes of meeting) or working reports. Where portfolios of evidence or learning journals are used, students might need to learn how to collect and archive relevant evidence.

Example 6:

A group of six students undertakes a six-week research project on the geomorphology of a particular region. They will produce a final group report. To ascertain the contribution and learning of each member of the group, the students are required to submit a research diary recording their progress, relevant diagrams and printouts and findings at weekly intervals throughout the six weeks. In addition, there is a question in the exam that asks them to briefly describe the geomorphology of their area and to explain how their research method could have been improved.

7.4 Implementing group assessment Whichever model is chosen, careful thought is required. It might be worth trialling a model with a small class before implementing large-scale changes. There are a number of ways and algorithms for apportioning marks for individual effort and input where this is deemed appropriate.

Three other issues that may arise are: • Allocating students to groups strategically; • Providing support for groups to keep them on-track; and • Dealing with situations in which groups are of unequal size



The most significant recent publication to support assessment in Australian higher education, including group assessment, can be found online at http://www.cshe.unimelb.edu.au/assessinglearning/index.html or in hard copy (James et al., 2002b). This publication provides many further suggestions for pre-empting and managing problems related to group-based assessment.

Group work provides many benefits for students and for staff but it is a process that must be planned and implemented carefully. In making the decision to implement group-based assessment, consider the following:

• Why are students to work and be assessed in groups in this unit?

• What are the learning outcomes?

• How will the groups be constituted?

• What are the assessment criteria for meeting the learning outcomes?

• How will the quality of each student's work be judged?

• What other processes will be put in place to ensure that the assessment process is fair, equitable and manageable?

8. How can rubrics help with assessment?

8.1 What is a rubric? A scoring rubric is a table that makes clear to students the criteria against which their work will be assessed. Students can use it in developing, revising and judging the quality of their own work. Assessors can use it to assess students’ performance either analytically or holistically (see p 21). Perhaps the most important function of a scoring rubric, however, is in providing both formative (ongoing) and summative (after marking) feedback to students and feedback to staff on students’ learning and thus the effect of their teaching (Huba & Freed, 2000).

Assessment is a process that should be integral with student learning and not just something tacked on at the end of a period of learning. If assessment is viewed this way, well-constructed scoring rubrics can play a key role implementing an effective learning and assessment programme

In many instances, the use of a scoring rubric enables students to hand in work that is better in quality than they might otherwise have done. Students do not have to guess or infer what the assessor wants; and assessors are forced to articulate and, if necessary, quantify the most valued outcomes of students’ learning (Stevens & Levi, 2005).

Well-constructed rubrics increase inter-rater reliability through improving comparability among a number of assessors, an important factor to consider when using more subjective assessment tasks. A rubric may be designed to be re-used—the task that the student undertakes, or the content of the task, may change but the scoring rubric may be the same. This ultimately can save a lot of time.

Rubrics are consistent with outcomes-focused education and support a learner-centred approach to teaching, and thus should help in changing our thinking about assessment at Curtin in the future. The use of scoring rubrics is also consistent with the principle of making assessment processes open and accountable. The proviso is that the rubric is well-constructed, valid and reliable, and subject to ongoing evaluation.



Quality of performance

(or achievement levels)

Dimensions of performance

The dimensions of performance may also have sub-categories.

For the rubric to be complete, each aspect of quality must be elaborated with a clear description of defining features of work at each level of mastery. Sometimes this is just represented as the extremes of a continuum from ‘unacceptable’ to ‘excellent’, which is less informative since it leaves the student unaware of exactly what constitutes unacceptable, acceptable, good or excellent work.

8.2 How to develop a rubric

Stages Comment

Decide the Dimensions of Performance or essential elements that must be evident in high quality work [rows].

Rule of thumb: If a student can score highly on all Dimensions but not score well overall, you have the wrong ones. Revise. Discuss with colleagues.

Decide the levels of achievement – number and type [columns].

Various levels of mastery are: • sophisticated, competent, partly competent, not yet

competent • exemplary, proficient, marginal, unacceptable • accomplished, average, developing, beginning • or others that you choose (between 3 and 5).

For each Dimension of Performance, distinguish between acceptable and unacceptable (failing) performance

Write the criteria for unacceptable performance clearly and unambiguously.

For each Dimension of Performance, write clear performance descriptors (criteria) at each achievement level.

Try to determine qualitative and quantitative differences that characterise work or performance at the different levels. More on this below …

Include (if possible) the consequences of performing at each level.

For example, the standard of the work would (or would not) be accepted by the profession or a business (as in a charter) or a professional journal (as in publication guidelines), etc.

Criteria describing elements of performance at each level



Add the rating scheme you will use and apply any weightings.

Consider: • Adding marks ( analytical scheme) or not ( holistic

scheme). [There are arguments for both] • Including weighting criteria if required. • Should marks be awarded for work below the

minimum standard? • What are the criteria for ‘failure’?

Evaluate and revise accordingly. Few rubrics will be constructed perfectly the first time. They are developmental tools and need to be critically evaluated after use.

(Huba & Freed, 2000)

8.3 Differentiating performance levels The hardest part about constructing a rubric is formulating and clearly articulating criteria for different levels of performance (or levels of achievement).

Guidelines: Try to determine objective, qualitative or quantitative differences that characterise work or performance at the different levels. Avoid:

• different grades of the same character (good, better, best etc)

• undefined terms e.g. trivial work, good use of, significant work

• value-laden terms e.g. excellent or poor work

These terms may have meaning for the assessor but don’t tell the student the standard expected nor what they can do to improve.

We can educate students simply by using rubrics

• We reveal to students the standards of the discipline and the profession (and help to inculcate them into professional practice).

• We inform students about the different qualities that comprise good (and poor) work.

• We can involve students in setting the standards that they agree to be judged by.

• We can involve students in using their own words to describe the criteria.

• They provide a means of opening the channels of communication between lecturer/tutor and students.

It is possible for a number of people to give feedback to the students on his/her work—peers, other staff etc—emphasising the different points of view about a piece of work.

One of the more sophisticated but effective uses of a scoring rubric is to have the students collaboratively decide the assessment criteria. This makes them much more aware of the criteria and what they mean, as well as giving students greater ‘ownership’ of the process.

8.4 Holistic versus analytic rubrics Holistic rubrics are a form of qualitative assessment while analytic rubrics are a form of quantitative assessment. Holistic rubrics allocate a grade or performance level to a student simply on the basis of performance against explicit standards (without resorting to marks). Analytic rubrics allocate a



specific number of marks for each criterion and/or performance level. Holistic rubrics are becoming more accepted because they “accommodate more than concrete facts” (Dunn, Morgan, O'Reilly, & Parry, 2004) but are often undervalued by those from a scientific measurement background because they may be seen to be too subjective. Provided, however, that the quality criteria for qualitative assessment are applied, they can be as valid an assessment tool as are analytic rubrics (see end of this document for quality criteria).

The following is a simple quantitative approach to assigning marks to each criterion.

Score all items on a 4-point scale:

0 = task or element not done

1 = task or element done, but clearly incorrect

2 = task or element done, but only partially correct

3 = task or element done, and clearly correct

0s, 1s and 3s will be clearly identifiable; anything else is a 2.

One issue is whether or not to award any marks to ‘unsatisfactory performance’. It might be possible to accumulate enough marks on unsatisfactory performance to tip a student over the magical 50%. An alternative is to set a criterion for passing that states there must be no element for which performance is unsatisfactory (regardless of the overall mark). Another way to limit this is to allocate an ‘overall performance’ Dimension on top of all other Dimensions of Performance.

An example of a rubric from Engineering The following is part of a rubric for an ‘Engine Design’ project in an Engineering course. The content is somewhat independent of the content of the task (see Huba & Freed, 2000, p.159-160)

Levels of Achievement

Criteria

Excellent (A)

4 points

Good (B)

3 points

Needs Improvement (C, D)

2 points

Unacceptable (F)

1 point

Formulation of Design Problem

Formulation and scope of problem

Design problem formulation is clear and well thought out. The problem scope is well defined

The problem formulation is clear, but the scope is not well-defined

The problem formulation is unclear in some respects and does not appear to be well thought out

The design problem is not formulated clearly.

Significance The problem chosen represents a current challenge facing the engine industry. The potential market is large and clearly identified.

The problem represents a current challenge in the engine industry, but the potential market is small or is not clearly defined.

The problem does not represent a current challenge in the engine industry, the market is small or is not clearly defined.

The problem does not represent a current challenge in the engine industry. There is no explanation about who would be interested in the product or why. There is no evidence of the background work (eg market analysis) that is needed to design an engine.

Engineering skill utilisation

Analysis Engineering analysis is detailed and challenging and is used at every stage of the design process.

Engineering analysis is detailed and challenging but some steps do not appear to be supported by calculations.

Some analysis is included, but it is not very detailed or challenging. Many steps are not supported by calculations.

Engineering analysis is used infrequently. When used, appears trivial and leads to obvious conclusions.



An example of a generic (re-useable) rubric for Problem-Solving Problem solving involves developing and testing a method or product for overcoming obstacles or constraints to reach a desired outcome.

Levels of Achievement

Criteria 4 points 3 points 2 points 1 point

Accurately identifies constraints of obstacles

Accurately and thoroughly describes the relevant constraints or obstacles.

Addresses obstacles or constraints that are not immediately apparent.

Accurately identifies the most important constraints or obstacles.

Identifies some constraints or obstacles that are accurate along with some that are not accurate.

Omits the most significant constraints or obstacles.

Identifies viable and important alternatives for overcoming the constraints or obstacles

Identifies creative but plausible solutions to the problem under consideration. The solutions address the central difficulties posed by the constraint of obstacle.

Proposes alternative solutions that appear plausible and that address the most important constraints or obstacles.

Presents alternative solutions for dealing with the obstacles or constraints, but the solutions do not all address the important difficulties.

Presents solutions that fail to address critical parts of the problem.

Selects and adequately tries out alternatives

Engages in effective, valid, and exhaustive trials of the selected alternatives. Trials go beyond those required to solve the problem and show a commitment to an in-depth understanding of the problem.

Puts the selected alternatives to trials adequate to determine their utility.

Tries out the alternatives, but the trials are incomplete and important elements are omitted or ignored.

Does not satisfactorily test the selected solutions.

If other alternatives were tried, accurately articulates and supports the reasoning behind the order of their selection, and the extent to which each overcame the obstacles or constraints

Provides a clear, comprehensive summary of the reasoning that led to the selection of secondary solutions. The description includes a review of the decisions that produced the order of selection and how each alternative fared as a solution.

Describes the process that led to the ordering of secondary solutions. The description offers a clear, defensible rationale for the ordering of the alternatives and the final selection.

Describes the process that led to the ordering of secondary solutions. The description does not provide a clear rationale for the ordering of the alternatives, or the student does not address all the alternatives that were tried.

Describes an illogical method for determining the relative value of the alternatives. The student does not present a reasonable review of the strengths and weaknesses of the alternative solutions that were tried and abandoned.

8.5 Evaluating a rubric Rubrics should be reviewed each time they are used, and then revised accordingly. Questions that can be used to do this are:

• Does it measure the learning outcome(s) that you want measured?

• Does it measure ALL the important outcomes?

• Does it measure unimportant/extraneous outcomes?

• Does the rubric cover the important dimensions of student performance?

• Are the performance levels and scales well-defined – does everyone understand them?

• Is there a clear basis for assigning scores at each scale point?

• Do the ‘excellent’ descriptors describe a high enough performance standard?



• Is there sufficient distinction between each dimension

• Can different scorers apply the rubric consistently?

• Is the rubric fair and free from bias?

• Is the rubric useful, feasible, manageable and practical?

8.6 Performance criteria The following are examples of performance criteria drawn from selected scoring rubrics (all cited in Huba and Freed (2000)):

Criterion: Team skills—Group functioning

Excellent The group functions well. Peer review indicates good distribution of effort. All members are challenged and feel their contributions are valued.

Good The group functions fairly well. Some people in the group believe they are working harder (or less hard) than others, but everyone is contributing.

Needs improvement

The group is still functioning, but each individual; is doing his/her own work and ignoring the efforts of others. There are frequent episodes where one person’s design will not fit with another’s due to lack of communication.

Unacceptable The group functions poorly. All work is the product of individual effort.

Criterion: Practicing ethical standards of the chosen discipline

Exemplary Acts congruently with, and advocates for, the ethical standards of chosen discipline.

Proficient Acts congruently with the ethical standards of chosen discipline.

Marginal Acts congruently with the ethical standards of the chosen discipline. Any violations are relatively minor.

Unacceptable Violates ethical standards of chosen profession. Violations are serious.

Criterion: Knowledge of past economic decisions

Exemplary Your bill reflects economic decisions which have worked in the past, is based on research involving comparisons of pervious eras, and makes reference to actual events.

Proficient Your bill reflects economic decisions which have worked in the past, but your examples are limited. More examples would make your bill more easily understood by the reader.

Marginal Your bill makes little reference to precedents from the past. You need to add some to help the reader decide if the bill would work and should be passed into law.

Unacceptable Your bill makes no reference to precedents from the past. The reader is left wondering if you know what past economic and political practice has been.



8.7 Quantitative Assessment Standards

Validity Assessment of a student’s knowledge and skills usually results in an empirical ‘indicant’ i.e. a number (mark) or a grade that represents the knowledge and skills being assessed. Validity refers to the extent to which the indicant measures what it claims to measure. In educational terms, however, it is often not the indicant/measure that is validated but the purpose for which it is being used that is usually submitted to validation processes. Hence, a mark based on a student’s recall of content knowledge cannot be used as a measure of the student’s higher order skills, such as knowledge application.

Reliability Reliability refers to both the accuracy and precision of measurement. Different tests of a student’s particular knowledge or skills, if administered independently of one another, should all give the same result. Two different assessors should arrive at the same conclusion about a student’s learning. The three facets of reliability refer to the basic rationale (can the knowledge and skills be translated into a measurement?), the procedures for data collection (the assessment tool, its administration and marking) and the statistical procedures following (what we subsequently do with the numbers).

Qualitative Assessment Standards Good qualitative assessments are credible, dependable and confirmable. Assessment is credible when the form of assessment is aligned closely with its corresponding form of learning outcome. Assessment is dependable when subjective methods are applied consistently and consensually. Assessment is confirmable when an audit trail (documentary or electronic) is maintained to enable back-tracking to original criteria-based judgements (adapted from Guba & Lincoln’s (1989) Fourth Generation Evaluation standards).

9. For further assistance Teaching development staff are available for help with individuals or teams.

Dr Beverley Oliver +61 8 9266 2292 [email protected]

Dr Shelley Yeo +61 8 9266 3406 [email protected]

Beatrice Tucker +61 8 9266 1092 [email protected]

Resources to help you

A range of resources to help you to develop strategies to improve items which have been identified as needing improvement, are available from the eVALUate website, or alternatively a hard copy can be forwarded to you by Terri Crowe (x2305 or [email protected]). Resources are available for each of the following items of the eVALUate questionnaire.

1. Communicating clear learning outcomes 2. Creating engaging learning experiences 3. Creating effective learning resources 4. Assessing student achievement of learning outcomes 5. Providing feedback for student learning 6. Improving student perceptions of workload 7. Improving student perceptions of teaching quality



List of References Biggs, J. (2003). Teaching for quality learning at university: What the student does (2nd ed.).

Buckingham: Society for Research into Higher Education and Open University Press.

Bloom, B. S., Ed. (1956). Taxonomy of educational objectives: The classification of educational goals. Book 1: Cognitive domain. London, Longman.

Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education. London: Routledge.

Guba, E. G. and Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park: Sage.

Huba, M. E. & Freed, J. E. (2000). Learner-centered assessment on college campuses : shifting the focus from teaching to learning. Boston: Allyn and Bacon.

James, R., McInnis, C. & Devlin, M. (2002). Assessing learning in Australian universities: Ideas, strategies and resources for quality in student assessment. Melbourne: Centre for the Study of Higher Education.

Paulson, L. F., Paulson P. R., & Meyer C. (1991). What makes a portfolio a portfolio? "Educational Leadership," 48(5), 60-63.

Race, P. (1999). The art of assessing. Retrieved from http://www.city.londonmet.ac.uk/deliberations/assessment/artof.html on 06/12/05.

Stevens, D. D and Levi, A. J. (2005). Introduction to rubrics. Virginia, Stylus.

4assessing student achievement of learning...

Documents

Transcript of 4assessing student achievement of learning...