Briefing Session on QAC Quality Audit

16 November 2007

HKBU Quality Audit – Visit Dates

19-23 January 2009(4 days)

(Dec 2008 - visit non-local operations, if needed)

QAC Audit Task Force

Membership

Ex-officio: Prof Franklin Luk, VPA Prof Tsoi Ah Chung, VPRIA Chair: Prof Tony Hung, LC Members: Dean/Director of each Faculty/School (including Graduate School, SCE and AVA), Associate VP of UIC Secretary: Ms Rosalind Chan, AR

QAC Audit Task Force – Terms of Reference1. To oversee and coordinate activities in HKBU, at the institutional a

s well as faculty/school/academy/department levels, in preparation for the QAC audit of January 2009;

2. To brief all relevant units on the details of the QAC audit, and discuss the strategies and actions needed to prepare for the audit;

3. To set an internal timeline for HKBU’s institutional self-review and submission of documents, and ensure that it is followed by all parties concerned;

4. To organise workshops and seminars for all units concerned – as and when needed – on Outcome-Based Teaching and Learning and other aspects of Teaching/Learning and Quality Assurance deemed relevant to the focus of the QAC audit;

Terms of Reference (con’t)

5. To provide assistance to individual units in their preparations for the audit, and monitor and review their progress and submissions;

6. To conduct mock interviews in preparation for the actual audit;

7. To assist in drafting HKBU’s institutional submission to QAC;

8. To serve as liaison with the QAC on arrangements and procedures for the audit;

9. To liaise with other tertiary institutions in Hong Kong on matters of common concern regarding the QAC audit.

Previous UGC Audits of HKBU

1996: TLQPR 1 (Teaching & Learning Quality Process Review)

Report: http://www.hkbu.edu.hk/~ar/staff/qa/tlqpr.htm

2002: TLQPR 2

Report: http://www.hkbu.edu.hk/2nd_tlqpr/

TLQPR 2002 Report on HKBU: Some ‘areas of concern’3.1.2 There is variation across campus in the

extent to which appropriate QA&I processes are in place, and similar variation in the energy with which EQW is being pursued at the unit level.

Further, the EQW may in some cases be viewed as an administrative process (with an attendant culture of compliance) rather than a real and practical basis for the improvement of teaching and learning.

3.1.3 The lack of clarity and transparency associated with systemic efforts to improve QA&I is exemplified by the inability of the Panel to validate that Appendices I – III of the Quality Handbook (“Quality as Transformation”, “Course Aims, Objectives and Learning Outcomes” and “Guidelines on Assessment”) have been taken on board in a meaningful way by rank and file teaching staff as a basis for improving teaching and learning.

3.1.5 (on student assessment) … the great majority of attention has focussed on grading and associated aspects… There is a tendency for it to deflect attention away from other crucial issues.

At the programme level especially, there is the need to use assessment as a focus for directing desired student learning outcomes, and evaluating the extent to which these have been achieved.

3.1.6 … the Panel could not identify clear evidence that the goal of Whole Person Education pervades departmental thinking (apart from the departments with a specific mandate in this respect) to the extent that could be expected, or indeed the Panel was led to expect. Similarly, the Panel could not identify clear evidence for any translation or progression from the broad aim, to objectives, to explicit learning outcomes, or how the University assures itself that this aim of providing Whole Person Education is in fact being achieved.

3.2.3 The University should develop procedures to ensure that student assessment is properly aligned with intended student learning outcomes. Further, there should be an increased emphasis on ensuring that data from student assessment is appropriately used in QA&I processes.

4.3Some units visited had defined the learning

outcomes that they intended students to achieve in a fairly clear manner, and had aligned assessment tasks with these. More typically though there was less coherence between intended learning outcomes and assessment tasks. This is particularly the case with respect to generic competences such as critical thinking, leadership skills, etc; here there are few clear signs that assessment has been specifically designed to promote and monitor the achievement of these kinds of outcomes.

QAC Audit of HKBU

Dates: 19-23 January 2009 Audit Manual: http://

www.ugc.edu.hk/eng/doc/qac/publication/auditmanual.pdf

Scope of QAC Audit

The QAC’s audit activities cover all first degree programmes and above, however funded, offered by UGC-funded institutions (including their continuing education arms and community colleges).

This includes self-financing programmes, joint programmes leading to a Hong Kong award, and the teaching and learning aspects of research degree programmes.

3.1 QAC’s approach to audit

The main objective of QAC quality audits is to assure the quality of student learning in UGC-funded institutions.

The audit examines whether the institution has procedures in place appropriate for its stated purposes, whether it pursues activities and applies resources to achieve those purposes, and whether there is verifiable evidence to show that the purposes are being achieved.

A QAC audit is not a review against a predefined set of standards. It does, however, oblige institutions to articulate and justify the standards they set for themselves, and demonstrate how the standards are achieved.

The quality of student learning is the centrepiece of audit. The QAC audits research and managerial activities only in so far as they affect the quality of teaching and learning.

Some characteristics of the audit model(a) The quality of student learning is the centrep

iece.(b) There is an emphasis on quality enhancement.(c) An audit is viewed as a collaboration between

the QAC and the institution….(e) The audit process involves institutional self-ev

aluation, followed by peer review which makes evidence-based findings.

(f) No attempts to make comparisons among institutions….

Outcome-based approach

The audit process is particularly concerned with the ways institutions articulate and measure the student learning outcomes they expect or aspire to.

The UGC encourages institutions to adopt an outcome-based approach to student learning, and the QAC audit process reinforces this objective.

3.2 Audit methodology

Approach-Deployment-Results-Improvement (ADRI methodology):

1. (Approach) What is the institution’s purpose?

2. (Deployment) How does the institution achieve its purpose?

3. (Results) What evidence does the institution have that its purpose is being achieved?

4. (Improvement) What processes are in place for improvement?

1. What objectives is the institution trying to achieve and on what basis does it set the objectives? The audit checks that the institution articulates overall lear

ning objectives which are consistent with its mission and with the role statement agreed with the UGC. The audit examines how these overall objectives are set and justified, and the extent to which they take cognizance of relevant external reference points…

The audit further examines how the overall objectives are communicated throughout the institution, and the extent to which they are reflected in the desired learning outcomes of individual programmes. The audit also examines how objectives are set for each activity which contributes to student learning

2. How does the institution go about achieving its objectives?

The means of achieving the desired objectives should be expressed in the institution’s plans, policies and procedures, and its activities and deployment of resources should be geared towards achieving its objectives.

The audit also checks that policies and procedures are implemented widely and consistently throughout the institution.

3. How does the institution measure achievement of its objectives and what evidence is there that the objectives are being achieved?

The audit examines the adequacy of the measures (e.g. benchmarks, students’ achievements, performance indicators) used by the institution to demonstrate that it achieves its objectives in each of the focus areas.

Some objectives may be directly measurable (e.g. graduation rates) while others can only be inferred from proxy evidence (e.g. certain graduate attributes being inferred from employer satisfaction).

The audit examines the institution’s performance against its own measures, and checks that the institution systematically collects and analyses evidence of its performance.

4. What processes are in place for improvement? The audit examines whether the institution

analyses its performance in order to effect improvement.

The audit checks that there are robust feedback loops from evidence of performance back to planning and implementation.

3.3 Standards and external reference points … there is public expectation that the standards

are in some sense appropriate and comparable with standards elsewhere.

… in setting its academic standards an institution should take account of external reference points such as public and employer expectations, the requirements of professions, and the standards of other institutions in Hong Kong and elsewhere.

…a QAC audit examines the adequacy of the measures used by an institution to demonstrate that it achieves its objectives.

An important element of this examination is the extent to which the measures allow the institution to place its achievement in a broader context, through the use of benchmarks and other comparative data.

3.4 The scope of audit: Focus Areas Since student learning is the focal point o

f the audit system, audits examine all aspects of an institution’s activities which contribute to student learning quality.

These activities range from planning and policy development, through programme design, approval and review, to teaching, assessment and student support.

11 COMMON FOCUS AREAS

1. Articulation of appropriate objectives Rationale: the institution is unlikely to

achieve high quality student learning unless its objectives are clearly expressed and well understood by staff.

2. Management, planning and accountability

Rationale: achievement of high quality learning requires good planning, clear management structures, and accountability for achieving objectives.

3. Programme development and approval processes

Rationale: the integrity of programme development and approval processes lie at the heart of an institution’s self-accrediting status, and hence at the heart of public confidence in programme quality.

4. Programme monitoring and review Rationale: monitoring and review are esse

ntial to maintaining the quality of programmes; they are also a crucial characteristic of self-accreditation.

5. Curriculum design Rationale: the quality of the curriculum is a

critical contributor to student learning, so it is important to examine how the curriculum is designed (e.g. use of external inputs).

6. Programme delivery, including resources, teaching mode, and the student learning environment

Rationale: student learning depends on how programmes are delivered, on the allocation of sufficient resources, and on the adequacy of the learning environment (e.g. library, IT, student support services).

7. Experiential and other out-of-classroom learning (e.g. leadership development, overseas exchange, work-integrated learning, service learning)

Rationale: these non-traditional activities are increasingly used to further student learning outside the more narrowly understood programme delivery.

8. Assessment Rationale: assessment is a major determin

ant of student learning behaviour and source of information for students and institutions about the achievement of learning outcomes, so it is important to examine whether it systematically reinforces positive learning outcomes.

9. Teaching quality and staff development

Rationale: good teaching is a significant contributor to student learning, so it is important to examine how the institution develops, supports and rewards good teaching by staff.

10. Student participation Rationale: student participation at various

levels in the planning and management processes increases a sense of ownership of the learning experience and improves the quality of learning.

11. Activities specific to research degrees Rationale: there are some teaching and learning

activities (e.g. supervision, examination) which are unique or of particular importance in research degree programmes…. However, audits do not focus on research quality per se, which continues within the remit of the Research Grants Council.

3.5 Sample programme review As part of a QAC audit, the panel examines in dep

th a sample of programmes to seek evidence that the institution’s quality assurance processes are in fact being implemented in practice and that stated outcomes are being achieved

In a sense, each selected programme is treated as a test case of the institution’s performance in the focus areas

The number of programmes selected (normally no more than three) is determined by the audit panel. (cf. Appendix (iii) for more details)

3.6 Institutional self-review

Audits are preceded by a period of institutional self-review culminating in an Institutional Submission to the audit panel.

A major output of the self-review is the Institutional Submission which serves as a principal source of information in the audit…

The draft is submitted for feedback about 6 months before the audit visit, and the final Institutional Submission is submitted about 3 months before the audit visit.

5.4 Self Review

As part of its preparation for audit, the institution is required to review all of its operations which contribute to the quality of student learning.

This self-review is an opportunity for the institution to analyse and reflect on its objectives for student learning, on its procedures, standards and benchmarks, and to check that it has appropriate outcome measures in place.

Some questions for self-review

What student learning outcomes are we trying to achieve? What kind of graduates are we trying to produce, and why? How do our desired learning outcomes relate to our mission and to the role agreed with the UGC? What standards do we have in place, and how are they derived? Are our desired learning outcomes reflected in all programmes?

How are we trying to achieve our desired student learning outcomes? How do our plans align with our desired outcomes? Are our policies and procedures appropriate and widely understood? Are our policies and procedures followed consistently throughout the institution? How do we know?

How do we know we are achieving our desired student learning outcomes? What measures (quantitative or qualitative) do we have in place? Are they appropriate? Do the measures relate to benchmarks or other external reference points? How do we collect evidence of performance? What does the evidence tell us?

How do we respond to what the evidence tells us? What feedback mechanisms do we have in place? How do we know they work? How do we raise aspirations and standards?

7. THE AUDIT VISIT 7.1 Overview The purpose of the Audit Visit is to allow the panel

to test the material presented by the institution through first-hand investigation and personal interactions.

The visit allows the panel to clarify and interpret the material it has been given, to examine evidence, and through meetings with staff, students and other stakeholders, to verify that policies and procedures are carried out in practice.

QAC’s activities during visit:

interviews staff, students and other stakeholders peruses documents requested by the panel progressively reflects on and discusses the

written and verbal material so far presented progressively refines findings and draft

recommendations gives a brief oral presentation of its findings

during an exit meeting with the institution’s leaders The audit visit lasts for 3 to 5 days

Appendices:

(i) HKBU’s Mission Statement

The University is committed to academic excellence in teaching, research and service, and to the development of whole person in all these endeavours built upon the heritage of Christian higher education.

(ii) HKBU’s Role Statement offers a range of programmes leading to the award of first degrees in Arts, Business, Chinese Medicine, Communication Studies, Education, Science and Social Sciences; 

pursues the delivery of teaching at an internationally competitive level in all the taught programmes that it offers; 

offers a number of taught postgraduate programmes and research postgraduate programmes in selected subject areas; 

follows a holistic approach to higher education and emphasizes a broad-based creativity-inspiring undergraduate education, which inculcates in all who participate a sense of human values; 

aims at being internationally competitive in its areas of research strength, and in particular in support of teaching; 

maintains strong links with the community;  pursues actively deep collaboration in its areas of

strength with other higher education institutions in Hong Kong or the region or more widely so as to enhance the Hong Kong higher education system; 

encourages academic staff to be engaged in public service, consultancy and collaborative work with the private sector in areas where they have special expertise, as part of the institution's general collaboration with government, business and industry; and 

manages in the most effective and efficient way the public and private resources bestowed upon the institution, employing collaboration whenever it is of value.

(iii) Selection of sample programmes for QAC Audit 2009

(Section 6.1) The panel selects programmes for sample review, using criteria including the following:

o relevance: the extent to which a programme may shed light on any major audit issues the panel has identified

o distinctiveness: the extent to which a programme breaks new ground or employs innovative modes of delivery

o diversity: selection of programmes in diverse areas (e.g. engineering and humanities) or of diverse types (e.g. self-financing and publicly funded) to provide distinctive samples to examine

o significance: the size of a programme in terms of staff, students or resources, or its strategic importance to the institution s future

o recent review: to examine whether a programme which has been recently reviewed (by the institution or by an external body) has adopted any changes recommended

o coverage: the extent to which the QAC wishes to enhance its knowledge of a wide variety of programme types

(iv) Outcome-Based Teaching & Learning: Useful Links Introduction to OBTL:

http://lc.hkbu.edu.hk/te/doc/preworkshop_reference.doc Workshops on OBTL:

http://lc.hkbu.edu.hk/te/doc/preworkshop01.ppt

http://lc.hkbu.edu.hk/te/doc/preworkshop02.ppt

http://lc.hkbu.edu.hk/te/doc/preworkshop03.ppt UGC-commissioned study reports on OBTL:

http://www.hku.hk/caut/new1/documents/OBA_1st_report.pdf

http://www.hku.hk/caut/new1/documents/OBA_2nd_report.pdf

City U’s OBTL site:

http://tfq.cityu.edu.hk/obtl/

(v) Centres for Teaching & Learning in UGC-Funded Institutions in HK CityU: http://www.cityu.edu.hk/cityu/dpt-admin/edo.htm CUHK: (i) CLEAR: http://www.cuhk.edu.hk/clear/

(ii) QTL: http://www.cuhk.edu.hk/v6/en/teaching/quality.html HKBU: (i) CHTL: http://net3.hkbu.edu.hk/~holistic/

(ii) LC: http://lc.hkbu.edu.hk/te/ HKIEd: http://www.ied.edu.hk/lttc/ HKU: http://www.hku.hk/caut/new1/index.htm HKUST: http://celt.ust.hk/index.html LU: http://www.ln.edu.hk/tlc/index.html PolyU: http://edc.polyu.edu.hk/

To recap

The Quality Audit is an audit of an institution’s Fitness for Purpose in teaching and

learning, where the Quality of Student Learning is the

centrepiece (focus), with an emphasis on Quality Enhancement

The Audit Visit

Allows the panel: To test the material presented by the institution

through first hand investigation and personal interactions

To clarify and interpret the material it has been given

To examine evidence, and through meetings with staff, students and other stakeholders, to verify that policies and procedures are carried out in practice

The Audit Visit – Typical Interviewees

Senior management of the institution and selected members of the Council

Senior staff of committees or departments involved in teaching and learning policies and its implementation

Programme Directors/Coordinators Managers of student support services, and others who

contribute to the student learning experience Teaching staff (various levels and different degrees of

experience) Students and graduates External stakeholders (e.g. employers)

Quality Audit – Key Dates

Jul 2008 Forward draft institutional submission and contents list of Supplementary Material to QAC

Oct 2008 Forward Institutional Submission and Supplementary Material to QAC

Dec 2008 QAC Panel to conduct visit to non-local operations, if required

19-23 Jan 2009 Audit Visit

Mar 2009 Receive draft audit report and submit comments to the draft audit report

Apr 2009 Receive finalized Audit Report and submit Institutional Response to the Audit Report

Jun 2009 QAC to publish the Audit Report

Action PlansBy Dec 2007

– Updated Programme Documents Syllabuses Teaching Methodology & Activities Assessment Methods Quality Assurance Processes

By Apr 2008 – Completion of Faculty/School self-review and finalization* of:

Learning Outcomes at Course level Learning Outcomes at Programme level Re-alignment of Teaching & Learning Activities to

Intended Learning Outcomes Re-alignment of Assessment Methods to Intended

Learning Outcomes Quality Assurance Processes

* please go through consultation and review at the faculty/school level prior to the finalization process

Internal Timeline

May 2008 – Faculty/School Submission to VPA via AQSS

Jun 2008 – Preparation of draft Institutional Submission to QAC

Jul 2008 – Forward draft Institutional Submission to QAC

Oct 2008 – Forward Institutional Submission and

Supplementary Material to QAC

Thank you!