External Assurance of Learning Models ‐

Accountability and Quality Improvement

ABDC 25 July 2014

www.slideshare.net/markalistairfreeman

Achievement Matters: External Peer Review of Accounting Learning Standards

Australian Business Deans Council

CPA Australia

Institute of Chartered Accountants in Australia

Office for Learning and Teaching

Project team : Adelaide (B.Howieson), Deakin (K.Watty), RMIT (B.O’Connell,

P.de Lange), Sydney (M.Freeman), UWA (P. Hancock), UWS (A.Abraham)

Participants from 17 providers

Websites

achievementmatters.com.au

disciplinestandards.pbworks.com

Disclaimer

The views expressed in this presentation do not necessarily reflect the views of

any of these stakeholders

Standard

“a definite level of excellence or attainment.....

so established by authority, custom, or

consensus” (Sadler, 2013)

eg. Learning standard

a.k.a threshold learning outcome (TLO)

Learning outcome

Learner knows, understands and can do…..

independently, on demand and in a range of

contexts

Definitions

Source Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723

Providers

Graduate outcomes

Students

Self regulation

Market forces

National regulation

Prof regulation

article.wn.com

Need better grad outcomes forAccounting degrees. What changes?

With peer

What evidence prioritise to know

if improving?

Market

22% employers would recruit more grads if higher number “appropriate available”

23% domestic bachelor grad self-report seeking FT employment in AGS.

NESB 38%. International bachelor? Conversion masters?

Private providers compete. eg. internships; small engaging classes; some big brands

(GCA, 2014)

NOT interested in “record of completion”

but “demonstrable competence” say CPA Aust (HES 2009)

Accredited by AACSB +/or EQUIS (CPAA guidelines, 2011)

Lobby by CPAA for Accounting on Skills Occupation List (2014)

Professional

AoL standards (AACSB, 2003)

1/3 Aust graduates applying for

residency sub standard English communication (Birrell, 2007)

39% pressure to pass full fee

students work “not good enough” (NTEU, 2012)

Self regulation

Source: Symington(18Oct2009) at commons.wikimedia.org

9

1.1 The course of study meets the

Qualification Standards.

1.2 There are robust internal

processes for design and approval of the

course of study which take account of

external standards and requirements,

e.g. published discipline standards,

professional accreditation, input from

relevant external stakeholders, and

comparable standards at other higher

education providers.

Legislation 2012

Source: http://curiosoando.com/que-es-el-freno-de-servicio

Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/

5.5 The academic standards

intended to be achieved by

students and the outcomes

actually achieved by students

in the course of study are

benchmarked against similar

accredited courses of study.

External AoL

11

Source Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723

1.5.1 The expected learning outcomes for each course

of study are specified, consistent with the level and

field of education of the qualification awarded and

informed by national and/or international

comparators.

1.5.2 The specified learning outcomes for each

course of study encompass discipline-related and

generic outcomes, including…:

HESP propose

Reference pointsAQF (2nd ed)

Discipline learning outcomes statements

Professional requirements

Martin Morris 20/2/12

http://www.flickr.com/photos/martin55/6929721749/

5.3.3 Review and improvement

activities include regular external

referencing against comparable courses of

study, including by referencing:

a) the progress of student cohorts

through courses of study, attrition rates

and completion times and rates, and

b) the grading of students’ achievement

of learning outcomes for selected units

of study within courses of study

HESP propose

article.wn.com

The BCom is 1 of 5 degrees

up for TEQSA scrutiny in 2015.

Challenge

What evidence will you prioritise?

5 students work given to 6 experienced EE in 4 disciplines. (2013)

Findings

• Only 1 jointly highest (of 5) by all 6 EE in discipline

• 9 of 20 ranked both best and worst (of 5)

Call for calibration again

UK model“The idea that a single external examiner could

make a comparative judgement on the national,

and indeed international, standard of a

programme has always been flawed” (2012)

“Assessment is largely dependent upon professional

judgement and confidence in such judgement

requires the establishment of appropriate forums for

the development and sharing of standards within and

between disciplinary and professional communities”

Tenet 6: Price et al (2008)

UK trial forums hospitality management (Rust 2009)

Literature

….just like research

By + 4 Accounting

2,100 participants nationally38 Australian universities21 private/other providers20 professional/peak bodies and

employersABDC leadership

2010

Accounting graduate

Knowledge

Application

JudgementCommunication

& Teamwork

Self management

Bachelor graduates are able to

exercise judgement

under supervision to solve

routine accounting problems

in straightforward contexts

using social, ethical, economic,

regulatory and global

perspectives

Masters

minimal

emerging/advancedcomplex

^

TLO 1

TLO 2

TLO 3

TLO 4

TLO 5

TLO 1

TLO 2

TLO 3

TLO 4

TLO 5

TLO 1

TLO 2

TLO 3

TLO 4

TLO 5

Blended learning

Online distance learning

Problem-based learning

Team-based learning

Provider CProvider B Provider DProvider AApply TLOs

+

Quantitative

Public sector

Professional Small business

Regional

+

+

International

+

International

International

International

International

TLO 4

TLO 3

TLO 2

TLO 1

TLO 5

Assess TLOs

Achievement Matters project aims

1. Obtain external peer-reviewed evidence of accounting learning

outcomes in all types of higher education providers, benchmarked

against the learning standards.

2. A model process for assessing learning outcomes against standards (that

is sustainable, reliable and efficient), satisfying external quality

assurance needs and motivating continuous improvement.

3. Professional development of academics about assessment.

4. Enhanced understandings in the external environment of our model for

assessing achievement of learning standards

UWA

Curtin

(16)

UniSA

Adelaide

(20)

(2)

(3)

(4)

JCU

USQ

QUT

Griffith

(16)UWS UTS

Macquarie

SCU(65)

RMIT (46)

Monash

Deakin

Kaplan

Holmesglen

Assess Enter ComparePre-F2F

F2F

Consensus Agree

Post-F2F

Apply

Confirm

2011-14 Calibrate

22

Remotely assess

23

Remotely

compare de-

identified rating

& feedback

Calibration – Task validity

Individual confidence pre‐workshop

Individual results pre‐workshop

• Min & max (n=33)

• Mean ±1 SD

Group results at workshop

• Small groups (n=6)

• Consensus

91%

NA A

NA A

Oral communication

(12) (21)

Oral communication

Individual confidence pre‐workshop

Individual results pre‐workshop

• Min & max (n=33)

• Mean ±1 SD

Group results at workshop

• Small groups (n=6)

• ConsensusNM M

NM M

85%

....S2, S3, S4, S5Calibration – UG Student 1

(10) (23)

Date Location Learning standards calibrated Level

Jul 2011 Darwin • Written Communication Bachelor

Sep 2011 Melbourne • Written Communication Master

Feb 2012 Adelaide • Written Communication Bachelor

Sep 2012 Sydney • Knowledge

• Written CommunicationMaster

Feb 2013 Adelaide • Knowledge

• Oral CommunicationMaster

Jul 2013 Perth • Application

• Judgement

• Oral Communication

Bachelor

Feb 2014 Adelaide • Application

• Judgement

• Teamwork

Master

Jul 2014 Sydney • Teamwork

• Self management

• Written communication

Bachelor

Calibration workshops

27

Grounded & trusted suggestions to improve assessments

• Reduced use of group assessable task & same mark

• Increased weighting in grade

• Expanded use of past examples as student calibration activity

• Greater use of self-assessment resources eg. writersdiet.com

• More authentic eg. brief CFO/board & email SH friend

• Added triangulation eg. impromptu Q&A

Standard deviation reduced

• Less likely to have an unreliable review

• Avoid perverse consequences

Calibration – Impact

Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/

Live review

2

Live review report

3

Appendix

Achievement Matters model

1. Useful to prove (to external bodies) & to improve achievement

2. Collaborate with industry & professional bodies

3. Fits culture

4. Reduce potential bias– Review only if participated in calibration against explicit standards

– Experiments show standard deviation halved

– Double-blind peer review with third external if any disagreement

– Student work randomly selected so no risk of ranking sample

5. Efficient– Assessment tasks aligned to discipline standards (not units of study)

– Live reviews immediately after calibration follow annual conference

– Electronic reviews: easy to allocate and collect, aggregate & return

– Professional development & shared resources

– Potential for deans council to coordinate for each discipline

This project will help establish

national agreement on learning

standards between accounting

degree providers

This project will help establish

national agreement on learning

standards between accounting

employers and accounting degree

providers

Future - Accounting

Quality Verification Achievement Matters Inter‐university Review

Primary intent QA QA & QE QA & QE

Scope Selected final UoS Selected degree standards Selected final UoS

Disciplines Multiple (7) Accounting Multiple (12)

Level Bachelor Bachelor + Master Bachelor

HEI grouping Group of Eight 17 HEI across sector 11 unis across sector

Reviewers 1 academic 2 anonymous academics +/or

professionals (3rd potentially)

1 academic per partner

Calibrated Not explicitly Yes by workshops Not explicitly

Products viewed Inputs & outputs in

final unit of study

Tasks & outputs in degree

evidencing standard(s)

Tasks (inputs) & outputs in

final unit of study

Data selection Stratified random Fully random Stratified random

Sample size

outputs

5% all assessments

(available) per grade

5 assessments per standard 1 assessment per grade

(HD/D,C,P,F) = 4

Reviews Manual submission

files & aggregation

Online submission & auto

aggregation

Manual submission files &

aggregation

Authority Institutionally top-

down

Disciplinary ground-up Institutionally top-down

Decide with peersThe Bachelor of Business has been selected as one of five degrees to evidence

the Threshold Standards in the upcoming TEQSA review at your institution in

2015. The best evidence of external assessment is:

A. Use AGS % employed and % satisfied with graduate attribute development

B. Access validated standardised test for final year students to do

C. Pay experienced external academic to verify grades on final year UoS sample

D. Select partner department to mutually verify grades on final year UoS sample

E. Argue existing external accreditation system (eg. AACSB, EPAS) suffices

F. Lobby existing university system (eg. Go8, IRU) to focus in 2014

G. Lobby ABDC to coordinate external review system around TLOs with calibration

> 1?

Future

Thank you