Australian Quality Audit

8/14/2019 Australian Quality Audit

1/87

AUSTRALIAN

UNIVERSITIESQUALITY AGENCY

Audit Manual

Version 3.0

June 2006


2/87

AUQA Audit Manual, Version 3.0

ISBN 1 877090 53 0

Copyright Australian Universities Quality Agency 2006

Level 10, 123 Lonsdale StreetMelbourne, VIC, 3000Australia

Ph 61 3 9664 1000Fax 61 3 9639 7377

[email protected]

www.auqa.edu.au

The Australian Universities Quality Agency receives financial supportfrom the Commonwealth, State and Territory Governments of Australia.


3/87

TABLE OF CONTENTS

INTRODUCTION.......................................................................................................................................1

PART 1: CONTEXT...................................................................................................................................3

1. Introduction to AUQA ........................................................................................................................3

1.1 Context for AUQA ..................................................................................................................3

1.2 AUQAs Objectives.................................................................................................................3

1.3 AUQA Mission, Values and Vision ........................................................................................4

1.3.1 Mission ...........................................................................................................................4

1.3.2 Values.............................................................................................................................4

1.3.3 Vision .............................................................................................................................4

1.4 AUQA Structure......................................................................................................................5

1.4.1 Members.........................................................................................................................5

1.4.2 Directors .........................................................................................................................5

1.4.3 Staff ................................................................................................................................6

1.4.4 Register of Auditors .......................................................................................................6

1.5 Relations with Other Organisations.........................................................................................6

1.6 Quality Improvement...............................................................................................................7

2. Introduction to Quality Audit ............................................................................................................9

2.1 Quality Audit and AUQAs Investigations..............................................................................9

2.2 Internally and Externally Derived Objectives .......................................................................10

2.3 National Protocols for Higher Education Approval Processes..............................................112.4 Self-Review ...........................................................................................................................11

2.4.1 Value of Self-Review ...................................................................................................11

2.4.2 Nature of Self-Review..................................................................................................11

2.5 Other Internal and External Reviews.....................................................................................12

2.6 Approach to Audit .................................................................................................................13

PART 2: PROCEDURES .........................................................................................................................15

3. Stages in an AUQA Audit .................................................................................................................15

4. Before the Audit Visit........................................................................................................................19

4.1 The Audit Process..................................................................................................................194.2 Initiating the Process .............................................................................................................19

4.3 Panel Selection ......................................................................................................................19

4.4 Self-Review ...........................................................................................................................20

4.5 The Audit Panels Investigations (see also sections 3, 5 and 6)............................................21

4.5.1 The Performance Portfolio ...........................................................................................21

4.5.2 Portfolio Meeting..........................................................................................................21

4.5.3 Preparatory Visit...........................................................................................................22

4.5.4 Following the Preparatory Visit ...................................................................................22

5. The Audit Visit...................................................................................................................................25

5.1 Parameters of the Audit Visit ................................................................................................25


4/87

5.2 Typical Visit Format..............................................................................................................25

5.2.1 Panel Preparations ........................................................................................................25

5.2.2 Auditee Preparations.....................................................................................................26

5.2.3 Interview Sessions ........................................................................................................26

5.2.4 Open Session ................................................................................................................27

5.2.5 In-situ Interviews..........................................................................................................275.2.6 Flexibility .....................................................................................................................27

5.2.7 Panel-only Meetings.....................................................................................................28

5.2.8 Exit Meeting .................................................................................................................28

6. The Audit Report...............................................................................................................................29

6.1 General Parameters................................................................................................................29

6.2 Audit Report Format and Writing Process ............................................................................29

6.2.1 Format...........................................................................................................................29

6.2.2 Commendations, Recommendations and Affirmations ................................................29

6.2.3 Writing the Audit Report..............................................................................................30

6.3 Approval of the Audit Report ................................................................................................30

6.3.1 The Definitive Draft .....................................................................................................30

6.3.2 Responsibilities of AUQA Board with Respect to Audit Reports................................30

6.4 The Final Report....................................................................................................................31

6.4.1 The Public Release .......................................................................................................31

6.4.2 Review of Audit Reports ..............................................................................................32

7. After the Audit Visit ..........................................................................................................................33

7.1 Feedback ................................................................................................................................33

7.2 Sharing Good Practices Identified Through Audit ................................................................33

7.3 Audit Report Implementation................................................................................................337.3.1 Approach to Monitoring...............................................................................................33

7.3.2 Progress Report.............................................................................................................33

7.3.3 Subsequent Audit..........................................................................................................35

PART 3: INFORMATION FOR AUDITEES.........................................................................................37

8. Notes for Institutions .........................................................................................................................37

8.1 Institutional Audit Scope.......................................................................................................37

8.2 Institutional Audit Panel Composition ..................................................................................37

8.3 Performance Portfolio ............................................................................................................38

8.3.1 Scope ............................................................................................................................388.3.2 Overall Structure and Format .......................................................................................38

8.3.3 Core Document.............................................................................................................39

8.3.4 Appendices ...................................................................................................................39

8.3.5 Supporting Material......................................................................................................40

8.3.6 Additional Material.......................................................................................................40

8.4 Prior to the Audit Visit ..........................................................................................................41

8.5 Institutional Audit Visit Program ..........................................................................................41

8.5.1 Outline ..........................................................................................................................41

8.5.2 Scope ............................................................................................................................42

8.5.3 Indicative Program for an Institution Audit Visit.........................................................428.6 The Audit Report ...................................................................................................................44


5/87

9. Notes for Accrediting Agencies.........................................................................................................45

9.1 Agency Audit Scope..............................................................................................................45

9.2 MCEETYA National Protocols and Relationship to AUQA Audits.....................................45

9.2.1 Implementation of the National Protocols Through Legislation ..................................45

9.2.2 Protocol 1 .....................................................................................................................46

9.2.3 Protocol 2 .....................................................................................................................469.2.4 Protocol 3 .....................................................................................................................46

9.2.5 Protocol 4 .....................................................................................................................47

9.2.6 Protocol 5 .....................................................................................................................47

9.3 Agency Panel Composition ...................................................................................................48

9.4 Performance Portfolio............................................................................................................48

9.4.1 Scope ............................................................................................................................48

9.4.2 Overall Structure and Format .......................................................................................48

9.4.3 Core Document.............................................................................................................49

9.4.4 Appendices ...................................................................................................................49

9.4.5 Supporting Material......................................................................................................499.4.6 Additional Material ......................................................................................................50

9.5 Prior to the Audit Visit ..........................................................................................................50

9.6 Agency Audit Visit Program.................................................................................................50

9.6.1 Scope ............................................................................................................................51

9.6.2 Indicative Program for an Agency Audit Visit.............................................................51

9.7 The Audit Report...................................................................................................................52

10. Audit Fees...........................................................................................................................................53

PART 4: INFORMATION FOR AUDITORS........................................................................................55

11. Audit Panels Composition, Characteristics and Actions...........................................................55

11.1 Extended Peer Review...........................................................................................................55

11.2 Auditor Characteristics ..........................................................................................................55

11.3 AUQA Honorary Auditors Training.................................................................................56

11.4 Roles and Responsibilities of Panel Members.......................................................................56

11.5 Additional Roles and Responsibilities of the Panel Chair.....................................................57

11.6 Additional Roles and Responsibilities of the AUQA Staff Member.....................................58

11.7 Summary Timeline ................................................................................................................59

11.8 Observers on Audit Panels ....................................................................................................60

11.8.1

Purpose .........................................................................................................................6011.8.2 Policy............................................................................................................................60

11.8.3 Guiding Principles........................................................................................................61

11.8.4 Procedures ....................................................................................................................61

11.8.5 References ....................................................................................................................62

11.8.6 Approvals .....................................................................................................................62

11.9 Information Gathering Techniques........................................................................................62

11.9.1 The Need for Evidence.................................................................................................62

11.9.2 Sampling.......................................................................................................................62

11.9.3 Tracking or Trailing .....................................................................................................63

11.9.4 Triangulation ................................................................................................................63

11.9.5 Styles of Questioning at Audit Visits ...........................................................................64


6/87

12. Conflicts of Interest ...........................................................................................................................65

12.1 Purpose ..................................................................................................................................65

12.2 Policy .....................................................................................................................................65

12.3 Guidelines..............................................................................................................................66

12.4 Approvals...............................................................................................................................66

13. Privacy of Information and Freedom of Information ....................................................................67

13.1 Privacy of Information Policy ...............................................................................................67

13.1.1 Background...................................................................................................................67

13.1.2 Policy............................................................................................................................67

13.2 Freedom of Information Policy..............................................................................................69

13.2.1 Purpose .........................................................................................................................69

13.2.2 Background...................................................................................................................69

13.2.3 Policy............................................................................................................................69

14. Auditing Overseas Activities.............................................................................................................71

14.1 Purpose ..................................................................................................................................7114.2 Policy .....................................................................................................................................71

14.3 Guidelines..............................................................................................................................71

14.3.1 Audit Principles ............................................................................................................71

14.3.2 Seven Factors to Consider ............................................................................................72

14.3.3 Implementation.............................................................................................................73

14.4 Approvals...............................................................................................................................74

15. Administrative Arrangements for Honorary Auditors ..................................................................75

15.1 Purpose ..................................................................................................................................75

15.2 Policy .....................................................................................................................................75

15.3 Guidelines..............................................................................................................................75

15.3.1 Travel Arrangements: Australian-based Auditors ........................................................75

15.3.2 Travel Arrangements: International Auditors...............................................................75

15.3.3 Other Expenses .............................................................................................................76

15.3.4 Alternative Travel Arrangements .................................................................................76

15.3.5 Travel and Medical Insurance and Travel Safety .........................................................76

15.3.6 Accommodation and Expenses.....................................................................................76

15.3.7 Honoraria......................................................................................................................76

15.4 References..............................................................................................................................77

15.5 Approvals...............................................................................................................................77

15.6 Car Rates Policy 004 .............................................................................................................77

15.6.1 Purpose .........................................................................................................................77

15.6.2 Policy............................................................................................................................77

15.6.3 Guidelines.....................................................................................................................77

15.6.4 Approvals......................................................................................................................77

PART 5: ANCILLARY MATERIALS....................................................................................................79

ABBREVIATIONS....................................................................................................................................79

BIBLIOGRAPHY......................................................................................................................................80


7/87

AUQA Audit Manual, Version 3.0 Introduction

Australian Universities Quality Agency 2006 Page 1

INTRODUCTION

The Australian Universities Quality Agency (AUQA) has four Constitutional Objectives. The first is toarrange and manage a system of periodic audits of the quality assurance arrangements of the activities ofAustralian universities, other self-accrediting institutions and State and Territory higher educationaccreditation bodies. AUQAs second Objective is to monitor, review, analyse and provide public reportson quality assurance arrangements in self-accrediting institutions, and on processes and procedures of Stateand Territory accreditation authorities, and on the impact of those processes on the quality of programs (seesection 1.2 for all four Objectives). This Manual sets out AUQAs operating procedures with respect to thesetwo Objectives and is for use by institutions and accrediting agencies, and by AUQA auditors and staff. TheManual has been structured to assist readers in identifying those sections most relevant to them. For brevity,where the same observation applies to audits of both higher education (HE) institutions and accreditingagencies, the term auditee is used.

AUQA audits universities and other self-accrediting institutions (SAIs). AUQA also audits the highereducation approvals functions of State and Territory departments or other authorities and of the Australian

Government. These functions encompass the implementation of all five of the National Protocols for HigherEducation Approval Processes of the Ministerial Council on Education, Employment, Training and YouthAffairs (MCEETYA 2000). In some jurisdictions, these responsibilities are located within a single unit whilein others more than one unit is involved. AUQA uses the term State and Territory accreditation agencies(STAAs) to refer to these bodies collectively. For convenience, AUQA uses the terms accrediting agencyor agency to refer all units responsible for these functions within a single jurisdiction.

AUQA carries out other audits, for example of non self-accrediting institutions (NSAIs). As guidelines forthese audits are still in a developmental phase, this version of the Audit Manual does not specifically addressthe processes for NSAI audits. However, NSAI audits conducted by AUQA will be based on the principlesof audit described in this Manual. The advice for auditors contained in the Manual is generally applicable toaudits of all types of higher education institutions and agencies.

AUQA works with each auditee to tailor the process as appropriate.

AUQA does not impose an externally prescribed set of standards upon auditees, but rather uses, as itsprimary starting point for audit, each organisations own objectives. This approach recognises the auditeesautonomy in setting its objectives and in implementing processes to achieve them. The core task of AUQAAudit Panels is to consider the auditees performance against these objectives. In keeping with AUQAsorientation towards quality improvement, AUQA emphasises the importance of a critical self-reviewundertaken by the auditee as a basis for the audit investigation. Part of AUQAs vision is that by workingcooperatively with its auditees, AUQA will add value to the activities of these organisations.

AUQA has developed a web-based glossary of quality-related terms which readers of this Manual may find auseful resource (see: www.auqa.edu.au/qualityenhancement/index.shtml).

USE OF THIS MANUAL

The AUQA Board has agreed that institutions and agencies to be audited in 2007 should follow this versionof the Manual. Auditees for 2006 should continue to follow Version 2.1 but may follow this revised versionif they choose.

In light of the completion of AUQAs first audit cycle in 2007, it is intended that the Audit Manual will bereviewed early in 2007 and a further version produced for the use of auditees in the second cycle. The nextversion of the Audit Manual will address NSAI audits in detail and will also take account of changes to the

National Protocols for Higher Education Approvals Processes. Users of the Manual are welcome to providefeedback on the Manuals utility to AUQA at any stage.


8/87


9/87

AUQA Audit Manual, Version 3.0 Part 1: Introduction to AUQA


PART 1: CONTEXT

1. INTRODUCTION TO AUQA

1.1 Context for AUQA

The Australian Universities Quality Agency (AUQA) is a core part of a total national quality assurance (QA)framework for Australia.

AUQA was established in 2000 by the group of Ministers of Education of Australia and each of the sixStates and two Territories, acting jointly through the Ministerial Council on Education, Employment,Training and Youth Affairs (MCEETYA).

1.2 AUQAs Objectives

The Objectives of the company as stated in its Constitution are:

1. to arrange and manage a system of periodic audits of quality assurance arrangementsrelating to the activities of Australian Universities, other Self Accrediting Institutions(SAIs) and State and Territory higher education accreditation bodies;

2. to monitor, review, analyse and provide public reports on quality assurance arrangementsin Self Accrediting Institutions, and on processes and procedures of State and Territoryaccreditation authorities, and on the impact of those processes on the quality ofprograms;

3. to report on the criteria for the accreditation of new universities and non-university

higher education courses as a result of information obtained during the audit ofinstitutions and State and Territory accreditation processes; and

4. to report on the relative standards of the Australian higher education system and itsquality assurance processes, including their international standing, as a result ofinformation obtained during the audit process.

AUQAs core task, set out in Objectives 1 and 2, is to carry out quality audits of Australias universities,other self-accrediting institutions and accrediting agencies. In each case, the audit is of the wholeorganisation, and it therefore addresses the effectiveness of the organisations quality systems for all itsactivities. The audits are conducted as described in this Manual.

AUQA carries out other activities from time to time in fulfilment of Objective 3, drawing on information

from a number of audits. The reports specified by Objective 4 are also occasional productions, based oninformation accumulated from a large number of audits, from other public data and from internationalsources.

AUQAs activities as outlined above were originally set out by The Hon Dr D. A. Kemp in December 1999when Commonwealth Minister for Education, Training and Youth Affairs (Kemp 1999). They weresubsequently embedded in the Constitution of AUQA after it was established by MCEETYA in March 2000.


10/87

Part 1: Introduction to AUQA AUQA Audit Manual, Version 3.0

Page 4 Australian Universities Quality Agency 2006

1.3 AUQA Mission, Values and Vision

AUQA has developed the following statements to guide and direct its actions.

1.3.1 Mission

By means of quality audits of universities and accrediting agencies, and otherwise, AUQA will providepublic assurance of the quality of Australias universities and other institutions of HE, and will assist inimproving the academic quality of these institutions.

1.3.2 Values

AUQA will be:

Thorough: AUQA carries out all its audits as thoroughly as possible. Supportive: recognising institutional autonomy in setting objectives and implementing processes to

achieve them, AUQA acts to facilitate and support this.

Flexible: AUQA operates flexibly, in order to acknowledge and reinforce institutional diversity. Cooperative: recognising that the achievement of quality in any organisation depends on a

commitment to quality within the organisation itself, AUQA operates as unobtrusively as is consistentwith effectiveness and rigour.

Collaborative: as a quality assurance agency, AUQA works collaboratively with the accreditingagencies (in addition to its audit role with respect to these agencies).

Transparent: AUQAs audit procedures, and its own quality assurance system, are open to publicscrutiny.

Economical: AUQA operates cost-effectively and keeps as low as possible the demands it places oninstitutions and agencies. Open: AUQA reports publicly and clearly on its findings in relation to institutions, agencies and the

sector.

1.3.3 Vision

AUQAs judgements will be widely recognised as objective, fair, accurate, perceptive, rigorous anduseful: AUQA has established detailed and effective procedures for audit, that include auditorappointment and training, extensive and thorough investigation, and consistent implementation.

AUQA will work in partnership with institutions and accrediting agencies to add value to their

activities: AUQA audit is based on self-review, acknowledges the characteristics of the institution oragency being audited, and accepts comment from the auditee on the best way of expressing the auditfindings.

AUQAs advice will be sought on matters related to quality assurance in higher education: AUQAwill carry out consulting activities, including workshops, publications, and advising, and will publishand maintain a database of good practice.

AUQA will be recognised among its international peers as a leading quality assurance agency:AUQA will build international links to learn from and provide leadership to other agencies, and willwork with other agencies to the benefit of Australian institutions.


11/87



AUQA has committed itself to the Principles of Good Practice for Higher Education Quality AssuranceAgencies, developed by the International Network of Quality Assurance Agencies in Higher Education(INQAAHE), see www.inqaahe.org. In 2006, AUQA was reviewed against its Objectives and Vision, andthese Principles.

1.4 AUQA Structure

AUQA is established under Corporations Law as a not-for-profit company limited by guarantee.

1.4.1 Members

AUQAs Members are the nine Ministers responsible for higher education in the Commonwealth, the Statesand the Territories. Section 5 of AUQAs Constitution describes the requirements concerning meetings ofMembers of the company. In accordance with Corporations Law, an Annual General Meeting of Members isheld.

1.4.2 Directors

AUQA has a Board of twelve Directors, composed as follows:

four Directors elected by the Chief Executive Officers of self-accrediting institutions (SAIs) one Director elected by the Chief Executive Officers of non self-accrediting institutions (NSAIs) three Directors nominated by the Commonwealth Minister responsible for HE three Directors nominated jointly by the State and Territory Ministers responsible for HE (at least one

of whom is a person with extensive knowledge and experience of HE accreditation processes), and

an Executive Director.

The first eleven positions mentioned are appointed by the Members for terms of three years each, while thelast-named is appointed by the Board for such term as the Board determines. The Executive Director is alsoCompany Secretary.

Details of current Directors are available from: www.auqa.edu.au/aboutauqa/abouttheboard/index.shtml

In summary, the functions of the Board are to:

appoint the Executive Director of AUQA plan the strategic direction for AUQA, having in mind the national and international context, and

within the Objects defined by the Constitution

determine the policies of AUQA, within the parameters set by the Constitution monitor the implementation of the policies by the Executive Director and other staff confirm that the operating procedures of AUQA are carried out take responsibility for the performance of the organisation, with respect to meeting the Objects of

the company

appoint auditors to the Register approve the release of Audit Reports (see section 6.3.2) approve the budget of AUQA accept responsibility for the financial performance and reports of AUQA submit to the Members an Annual Report of AUQA, including the audited accounts, and advise the Members on the Constitution of AUQA.


12/87

Part 1: Introduction to AUQA AUQA Audit Manual, Version 3.0


The Board Chair and AUQA staff appoint Audit Panels from the Register, with the Chair acting on behalf ofthe Board. In establishing an Audit Panel, the Board delegates to that Panel the responsibility and authorityfor carrying out an audit according to Board policies and procedures.

1.4.3 Staff

AUQA has a small permanent staff, appointed by the Executive Director. Audit Directors have a backgroundin quality and quality assurance in higher education within Australia and internationally. One of the primaryroles of these staff is to serve on Audit Panels and guide and support Panels through all stages of the auditprocess. Otherwise, they have responsibility for various portfolios of activity related to the operation of theAgency and quality enhancement within the Australian higher education sector. A number of other full-timestaff support the work of AUQA and Audit Panels in various capacities.

Staff profiles and contact details are available from AUQAs website (see www.auqa.edu.au).

1.4.4 Register of Auditors

AUQA has established a pool of people willing to serve on AUQA Audit Panels as honorary auditors (theRegister of Auditors). Honorary auditors can be categorised as belonging to three broad groups: staff fromAustralian universities or other SAIs; individuals from Australian business or industry with high-levelexperience and understanding of planning and quality assurance; and people based outside Australia (usuallyfrom within a university or quality assurance agency). Characteristics of auditors are defined in section 11.2.

From time to time, the AUQA Board requests institutions and agencies to nominate people to be consideredfor appointment and suitably qualified individuals are also welcome to express their interest. Potentialauditors are requested to submit a curriculum vitae and the names of two referees. Referees reports areconsidered and the Board then selects auditors to be appointed. The term of appointment is two years.Auditors are trained by AUQA as a pre-requisite of their appointment to the Register. For Australian-based

auditors this involves attendance at a two-day training workshop (see section 11.3).

Photographs and biographies of current members of the Register are available from the AUQA website atwww.auqa.edu.au/qualityaudit/auditors/body.htm.

1.5 Relations with Other Organisations

In carrying out its work, AUQA has developed and maintains links with an extensive network of nationaland international agencies and institutions.

Within Australia these include the Australian universities, other HE institutions and accrediting agencies; and

other bodies such as MCEETYA, the Department of Education Science and Training (DEST), State andTerritory government departments, the Australian Vice-Chancellors Committee, the National QualityCouncil, the Australian Qualifications Framework Advisory Board, professional associations, and staff andstudent associations. AUQA is frequently consulted for advice and its staff are often invited to speak atmeetings and conferences.

Major points of contact outside Australia include HE quality agencies, networks of institutions andorganisations active in HE policy such as the Organisation for Economic Cooperation and DevelopmentProgramme on Institutional Management in Higher Education (OECD/IMHE) and United NationsEducational, Scientific and Cultural Organisation (UNESCO). AUQA is a full member of INQAAHE and isan active participant in the Networks activities. In addition, AUQA is serving currently as the secretariat forthe Asia-Pacific Quality Network (APQN), a group of quality assurance agencies in the Asia-Pacific region

(see www.apqn.org). In keeping with its Objects, AUQA undertakes international consulting projects as timeallows.


13/87



1.6 Quality Improvement

One of the central roles of AUQA is to encourage and assist the growth of an organisational culture in highereducation that values quality and is committed to continuous improvement. This is evidenced in AUQAsapproach to audit which emphasises self-review and ongoing interaction with auditees after the publicationof an Audit Report.

Other mechanisms used by AUQA for assisting in the improvement of quality in the higher education systeminclude:

compiling and publishing information on concepts and practices in QA in higher education identifying, recording, codifying and disseminating good practice in higher education through the

AUQA Good Practice Database (www.auqa.edu.au/gp)

providing advice and consultation conducting and commissioning research into quality and standards and their achievement identifying and investigating issues of current or likely national concern in relation to quality and

standards

facilitating inter-institutional discussion through electronic discussion lists and otherwise; leading the organisation of the annual Australian Universities Quality Forum running seminars and workshops publishing papers and articles, and making submissions to government and other agencies.

These activities are not restricted to supporting Australian higher education institutions and accreditingagencies, but are made available as a consulting service within Australia and internationally.


14/87


15/87

AUQA Audit Manual, Version 3.0 Part 1: Introduction to Quality Audit


2. INTRODUCTION TO QUALITY AUDIT

2.1 Quality Audit and AUQAs Investigations

Quality audit is defined as a systematic and independent examination to determine whether activities and

related results comply with planned arrangements and whether these arrangements are implementedeffectively and are suitable to achieve objectives (Standards Australia & Standards New Zealand JointTechnical Committee QR/7 1994) This definition has been adopted by AUQA as the foundation of itsapproach to audit.

General thinking about quality has progressed from controlling quality in the 1950s, to assuring quality inthe 1980s, to managing for continuous quality improvement today. This contemporary approach viewsquality management as operating an integrated system that enables an organisation to continually reflect on,and improve, its performance. Cognisant of this, AUQAs audit method considers four dimensions for anygiven issue: Approach, Deployment, Results and Improvement (ADRI).

The Approach includes the trail from an organisations mission, vision and values through to more specific

goals and objectives and the planned arrangements for how these will be achieved. The latter may culminatein written policies and procedures. Broad audit questions include:

What is this organisation about? What is it trying to achieve? Does it understand its context and capabilities? How does it know its objectives are appropriate? Are they set against appropriate benchmarks? How will the objectives be achieved? Is the approach aligned and communicated throughout the organisation and more widely?

In understanding an auditees approach, AUQA Audit Panels are likely to discuss with the auditee such

things as the ways in which particular objectives have been decided, the factors that were taken into accountin their development and the stakeholders that were consulted.

The Deployment dimension considers whether, and how effectively, the approach is being put into effect.Broad audit questions include:

Is the approach being implemented in the best possible manner? According to whom? How exactly does the organisation know this to be the case? If the approach is not being deployed, why not, and how is this managed? Are staff appropriately trained, and resources appropriately deployed, to fulfil the approach?

The Results dimension looks at an organisations results as a means of determining how well thedeployment is achieving the planned approach. Broad audit questions include:

What are the results for the stated objectives? Does the organisation understand why and how it achieved those particular results?

The Improvement dimension focuses on whether the organisation is actively and continuously engagedwith understanding its performance in each of the A-D-R dimensions, and is using this understanding tobring about improvements. Broad audit questions include:

Does the organisation know how it can improve? How does it know this? How is it acting upon this knowledge?

It is insufficient to audit only one or a combination of two or three of these dimensions because of theirfundamentally integrated nature. For example, it is insufficient to audit only deployment because this is


16/87

Part 1: Introduction to Quality Audit AUQA Audit Manual, Version 3.0


meaningless without considering what the deployment was designed to achieve and what it actually isachieving. Similarly, it is insufficient to audit only results because the stakeholders, and the organisationitself, need assurance that the results (and their consequential effects) are achieved by design rather thanby accident. Moreover, even if results look impressive, perhaps they could be significantly better if thedeployment was more effective. In conclusion, AUQA Audit Panels are concerned about both processesand outcomes.

2.2 Internally and Externally Derived Objectives

As each auditee will have systems that are relevant to its own objectives and character, the actualprocedures used and the way they are implemented will vary from auditee to auditee. Amid this variety,AUQAs anchor point for drawing conclusions on quality is always the objectives of the auditee, togetherwith any externally set objectives. For institutions, such external objectives include those set out in theAct or Regulation under which the institution is recognised, other relevant legislation, and the NationalProtocols for Higher Education Approval Processes (MCEETYA 2000). The Australian QualificationsFramework is also an important element of the context in which Australian institutions operate.Accrediting agency objectives include the MCEETYA National Protocols and various other legislated

requirements.

A number of guidelines, codes of practice or codes of conduct relating to specific aspects of HE havebeen promulgated by various bodies. Examples include the Australian Vice-Chancellors CommitteesProvision of Education to International Students: Code of Practice and Guidelines for Australian

Universities and MCEETYAs Good Practice Principles for Credit Transfer and Articulation fromVET to Higher Education. To the extent that such guidelines are adopted by institutions, they becomepart of institutional policies (and implicitly, therefore, part of an institutions objectives). AUQA maytherefore enquire of an institution whether such guidelines have been adopted or adapted and investigatethe extent to which the institutions objectives in this regard are being met.

Given that AUQA accepts an auditees objectives as the starting point of the audit, it is explicitly theresponsibility of the auditee to devise a systematic process for evaluating its objectives with respect tocriteria which may include relevance, desirability, feasibility, distinctiveness, measurability, etc. In orderto check its own policies, procedures and practices, to learn whether it is achieving its objectives, and todetermine how to improve its performance, an institution or agency must have in place appropriatequantitative and qualitative measures and indicators. Measurements give information about individualitems, but also about processes. For this reason AUQAs audit is in part process-based, with outcomesproviding information on the effectiveness of the processes. Attention is also paid to inputs, as these arealso integral to the process effectiveness.

Processes are more than documentation, but institutional practice and knowledge must be sufficientlyformalised so that it does not reside solely in the minds of individuals. This approach also incorporates an

appreciation of risk assessment and risk management. Furthermore, the audit is concerned with theauditees self-monitoring to identify the need for and consequences of doing things differently.

It is sometimes claimed that all external review, whether audit, accreditation or assessment, is the enemyof creativity, since it must err on the side of caution in terms of accepted methods and outcomes. AUQAavoids this danger by encouraging institutions and agencies to devise their own characteristic qualitymanagement systems and any necessary revisions to them, rather than detailing checklists to be followed.


17/87



2.3 National Protocols for Higher Education Approval Processes

In March 2000, MCEETYA approved a set of five National Protocols for Higher Education Approval

Processes. The purpose of setting out these Protocols was to ensure consistent criteria and standardsacross Australia in such matters as the recognition of new universities, the operation of overseas highereducation institutions in Australia, and the accreditation of higher education courses to be offered by non-self accrediting providers (MCEETYA 2000 Preface). The Protocols are not set out in full here, but areavailable from the MCEETYA website: www.mceetya.edu.au. They are currently being revised.

In auditing an Australian university, AUQA has regard to whether the universitys objectives areconsistent with the established criteria for a university (Protocol 1). In auditing an accrediting agency,AUQA has regard to whether the agencys objectives take account of all the Protocols, particularly thecriteria for accrediting higher education courses in non self-accrediting institutions (Protocol 3). AUQAchecks that the agency is using the Protocols when relevant and applying them appropriately. For furtherdetail, refer to section 9.2.

2.4 Self-Review

2.4.1 Value of Self-Review

AUQA bases each audit on a critical self-review (also referred to as self-evaluation, self-audit) by theinstitution or agency subject to the audit. Such a self-review not only enables the auditee to supply theinformation required by AUQA, but also has the potential to lead to improvements even without AUQAsinvolvement.

AUQA attempts to be as flexible as possible in scheduling Audit Visits to accommodate the preferencesof individual auditees, within the constraints of AUQAs resources. To minimise duplication andrepetition, AUQA is also willing to discuss with auditees the format of self-review reports.

Effective quality audit is a composite process that is owned and implemented by both the auditee and theexternal quality body, with the latter performing a verification, reporting and enhancement role. AUQAsemphasis on meaningful self-review has several merits, including:

recognition of auditees autonomy and responsibility recognition of the diversity of auditees enabling different approaches to self-monitoring initiation and/or maintenance of a process of critical self-development, and production of information, some of which may not normally be evident.

The evaluation and checking of the self-review by the AUQA Audit Panel will:

respect the objectives and values of the auditee contribute towards the auditees process of self-learning for an institution, elucidate how it judges its teaching and research in relation to national and

international standards, and

for an agency, elucidate how it defines and measures its effectiveness.

2.4.2 Nature of Self-Review

There is no single model for self-review. Indeed, the first decision might be whether an organisation

interprets self-review as a process or as an event. Self-review might be an ongoing process that is builtinto all activities in a sense of continuous improvement; it might be an ongoing process that leads to a


18/87

Part 1: Introduction to Quality Audit AUQA Audit Manual, Version 3.0


report to the governing body for confirmation or redirection; it might be a long event that leads to majorchange; it might be a brief event that provides a snapshot for comparison with an earlier snapshot; and so on.

In the current context, a second dimension for auditees is whether the review (event or process) isdesigned principally for the organisations internal needs, or primarily in preparation for an AUQA audit.AUQA strongly urges institutions and agencies to design and implement quality systems that are bestsuited to their own needs. The extent to which the organisation builds in explicit awareness of theexistence of occasional AUQA audit should be determined by the approach that is most effective for theorganisation itself. As self-review is an essential feature of an organisations quality system, this tooshould be designed primarily for the organisation rather than for AUQA.

Regardless of these considerations, some useful questions for an organisation to ask about processeswithin its quality system include:

what are the objectives of the process? how do these objectives relate to the organisations objectives? what are the main steps in the process? who has responsibility for the process? Are others also responsible for stages in the process?

how is the process implemented? what documentation is required for, or associated with, the process? how is the effectiveness of the process monitored? What indicators are used? what is the current state of achievement of the objectives as revealed by the indicators and

monitoring?

what analysis of the strengths and weaknesses of these quality assurance arrangements isperformed? What does this indicate about the process effectiveness in achieving intendedoutcomes?

what plans are in place (or proposals made) for development or improvement?

AUQA provides advice to assist in quality improvement in the HE sector, and is willing, when possible,

to advise on approaches to self-review.

2.5 Other Internal and External Reviews

Many institutions have developed rigorous internal audit processes that are themselves attuned toreflection, evaluation and development, and matched to the time and resources available. AUQA takesinto consideration the robustness of such locally devised systems.

Also, by compulsion or choice, institutions are subject to increasing numbers of external quality-relatedchecks and reviews. There is a variety of models of QA, such as ISO 9000 and quality awards. There arelegal and professional requirements, such as professional accreditation and government audits and reports.Some institutions find value in obtaining certification or accreditation from international or foreignagencies, associations or institutions. An institution needs to have these checks integrated into its ownquality system, and to that extent, AUQAs audit of the institutions QA processes will embrace theseother reviews as well.

Auditees are free to include in their Performance Portfolio contributions from other groups as evidence ofthe achievement of their objectives. Such groups may include professional associations, employers,students, and community groups. To the extent possible, AUQA takes into account any other internal orexternal reviews undertaken by auditees and their results, and integrates them into its own conclusions,rather than simply duplicating. To assist with this, auditees are requested to provide, in their PerformancePortfolio a schedule of the external and internal quality reviews undertaken in the five year period prior tothe audit (see sections 8.3 and 9.4 for institutions and agencies, respectively).


19/87



2.6 Approach to Audit

AUQA emphasises the obligation of all parties to exhibit professional conduct and integrity at all timesthroughout the audit process. AUQA expects that auditees will behave appropriately in interactions withAUQA and in their approach to the audit process. AUQAs expectations of auditors are described insection 11.


20/87


21/87

AUQA Audit Manual, Version 3.0 Part 2: Stages in an AUQA Audit


PART 2: PROCEDURES

3. STAGES IN AN AUQA AUDIT

The following timeline shows the typical sequence and time scale for the stages of an AUQA audit,starting from the time of selection. Variations may well occur, depending on the context of the audit.

Fuller details on the various stages are given in subsequent sections of this Manual (see especiallysections 17).

Note: AD (abbreviation of Audit Director) denotes the AUQA staff member on the Audit Panel. This may be the

Executive Director or an Audit Director.

Timeline Activity Agent

At least 9months prior to

Auditees are selected for audit AUQA Board

Portfoliosubmission

Auditees are notified of their selection for audit and are invited toprovide a short written statement of any particular distinguishingfeatures or contextual issues that should be taken into account byAUQA in its selection of possible Audit Panel members (seesection 4.2)

In the case of the audit of an institution, the state/territory ministeris notified of the audit.

AUQA/Auditee

Preliminary negotiation of preferred Audit Visit timing begins AUQA/Auditee

At least 3months prior to

Portfoliosubmission

Auditee asked to review names of potential Panel members forpossible conflicts of interest (see section 4.3)

Auditee

Audit Panel appointed and dates finalised. Details confirmed toauditee

AUQA

Performance Portfolio submitted to AUQA Auditee

Performance Portfolio distributed to Panel members AUQA

Week 1

Hotel and travel bookings made for Panel members to attendPortfolio Meeting

AUQA

Panel members provide initial brief written comments on the

Portfolio

PanelWeek 3

Comments circulated to all Panel members AD

Portfolio Meeting of Panel to discuss approach to be adopted forthe audit (including visits to different campuses, onshore partnersand offshore sites) and identify any further information required.This meeting occupies a full day.

PanelWeek 4/5

Panel informed of administrative/travel arrangements for the AuditVisit

AD/AUQA

Week 5 Notes from Portfolio Meeting prepared and preliminary Audit Visitprogram drafted. Documentation sent to other Panel members forrapid comment and response

AD/Panel


22/87

Part 2: Stages in an AUQA Audit AUQA Audit Manual, Version 3.0


Week 6 Draft Audit Visit program, requests for further information required,details of offshore sites to be visited (if any) & parameters forPreparatory Visit sent to auditee

AD

For Agency Audit Only: AUQA surveys a selection of theinstitutions within the agencys jurisdiction. Some Panel membersmay visit some of these institutions

AUQASome Panel members

Finalise program for Preparatory Visit AD/Auditee

Week 6/7 Preparatory Visit : Panel Chair and AD visit auditee to discussdraft Audit Visit program, requests for further information, logisticsfor Audit Visit & any other related matters

AD/Panel Chair/

Auditee

Week 8 Auditee invoiced AUQA

Auditee sends AUQA any further information requested Auditee

Final Audit Visit program sent to auditee AD

Week 9

Final Audit Visit program and further information requested sent tothe Panel

AD

Week 10 Auditee sends provisional list of names and positions ofinterviewees for each Audit Visit session to AUQA for comment

Auditee

Week 10 /11 For Institution Audit Only: Visits to selected partner operations(if any)

Some Panel members

Week 10/11 For Agency Audit Only: Results of surveys are collated. Follow-up for further information (if necessary)

AD

Further brief written comments and full interviewee list for AuditVisit sent to Panel members

AD

Panel members send AD comments on further documentationreceived

Panel

Week 11/12

AD and Panel Chair review process AD/Panel Chair

Week 12/13 For Institution Audit Only: Report of any visit(s) to onshore oroffshore partners circulated to all Panel members

AD

Week 14/15 Audit Visit (varies in length from 2 to 5 days; see section 10) Auditee/Panel

Week 16 Panel may comment on style and format of Portfolio and logisticsof Audit Visit for information of the auditee

AD

Week 16 All interviewees at the Audit Visit invited to comment on the audit AUQA

Week 16/17 Audit Report drafted and circulated to other Panel members AD

Week 18/19 Panel members return comments on draft report to AD Panel

Week 20/21 Report revised and circulated to other Panel members AD

Panel members return comments on draft report to AD PanelWeek 22/23

Audit Report revised and sent to:i) auditee (for comment on fact and emphasis; see section6.3)ii) AUQA Board (for information)iii) other Panel members (for comment if desired)

AD

Week 25 Auditee responds to definitive draft of Audit Report Auditee

Audit Report finalised AD/Panel Chair

Week 27

Report sent to AUQA Board for approval of release AD

Report approved if no objections AUQA BoardWeek 28/29

Report printed AUQA

Week 29 Report sent to auditee and Panel (embargoed for up to two weeks,

as agreed with auditee)

AD


23/87

AUQA Audit Manual, Version 3.0 Part 2: Stages in an AUQA Audit


Week 31 Audit Report publicly released

Placed on AUQA website and distributed to interestedpersons/bodies

AUQA

Week 32+ Feedback on audit process sought by AUQA from auditee contactperson and Panel members

Auditee/Panel

AUQA Board Chair telephones vice-chancellor or head of agencyto invite any brief oral comments to supplement written feedback

AUQA Chair

Good practices noted for inclusion in the AUQA Good PracticeDatabase

AUQA staffWeek 32+

Short report on the audit prepared for presentation to the nextAUQA Board meeting

AD

Approximately18 months after

AuditReport release

AUQA writes to auditee requesting a Progress Report to AUQAshowing actions taken in response to the Audit Report.

Auditee provides the Progress Report, which is considered byAUQA.

Auditee publishes Progress Report on its own website

AUQA provides a link from its website to the Progress Report onthe auditees website

AUQA

AuditeeAUQA

Approximately5 years after the

Audit Visit

AUQA reviews auditee responses to Audit Report in the context ofthe next audit

AD


24/87


25/87

AUQA Audit Manual, Version 3.0 Part 2: Before the Audit Visit


4. BEFORE THE AUDIT VISIT

4.1 The Audit Process

A quality audit is not synonymous or coterminous with the visit of the Audit Panel to the auditee. The

process starts within the institution or agency itself, and should ideally be integrated into the totality ofthe auditees QA system (see section 2.4). For AUQA, too, there is much more to an audit than the AuditVisit(s).

AUQA is required to make judgements about the appropriateness and effectiveness of quality assuranceplans and processes, and their relation to nationally and internationally accepted good practices. These

judgements are based in the first place on the Performance Portfolio, then tested and elaborated throughoral (and usually further written) evidence. The oral evidence is gathered during the Panels visit(s) to theauditee.

In order to evaluate the effectiveness of the QA procedures, it is necessary for the auditors to investigatethe results of applying these procedures and they therefore check the extent to which an institution is

achieving its mission and objectives, or an agency is meeting its legislative requirements. Auditees areexpected to have the necessary checking mechanisms in place and be able to demonstrate to AUQA thatthe procedures are being used and are working (or that there are plans for addressing any shortfall).

4.2 Initiating the Process

Once the AUQA Board has approved the audit schedule, the Executive Director of AUQA selects one ofAUQAs academic staff members to assume primary responsibility for each audit. The process of thataudit is then initiated by AUQA approaching the Chief Executive Officer (CEO) of the institution, or theDepartmental Head responsible for the agency, to discuss a mutually acceptable provisional schedule forthe audit. AUQA attempts to arrange with each institution and agency an audit schedule that fits with the

auditees own plans. The audit schedule embraces the need for the auditee to carry out whatever self-review it deems appropriate, prior to the external audit by AUQA. Typical time scales and responsibilitiesare set out in section 3. In brief, however, AUQA requires at least three months notice of the submissionof the auditees self-review report and the Audit Visit takes place approximately three months after theplanned submission date.

AUQA also invites the CEO or Departmental Head to provide a short, confidential written statement (ofapproximately 500 words) describing any particular distinguishing features of the institution/agency orcontextual issues that he/she feels should be taken into account by AUQA in its selection of possibleAudit Panel members.

Finally at this stage, the auditee is asked to nominate a person who will be AUQAs main point of contact

for the operational purposes of the audit.

4.3 Panel Selection

From AUQAs Register of Auditors (see section 11), AUQA staff and the Chair of the AUQA Boardselect potential members for an Audit Panel, bearing in mind the scope of the audit, any specialcharacteristics of the auditee (including those identified by the auditee itself, see section 4.2 above), andthe need to have a Panel that is coherent and balanced in background and experience. Among otherconsiderations, the shortlist includes members from within the Australian higher education sector;members with a background in business or industry; and members from outside Australia. Specialconsideration is also given to including prospective auditors with the characteristics required to be Panel

Chair in the shortlist.


26/87

Part 2: Before the Audit Visit AUQA Audit Manual, Version 3.0


Once a Panel shortlist has been constructed, the auditee is invited, in confidence, to indicate whether it isaware of any reason why any of the potential auditors should not be involved in the audit. Valid reasonscould involve known or probable conflicts of interest (see section 12) or inappropriateness for thecharacter of the auditee. AUQA makes the final decision on Panel composition, taking into account anyvalid concerns of the auditee. It also asks prospective auditors to declare formally any matters that couldpose a conflict of interest in their serving as an auditor for the particular institution or agency (see section12.2). If the prospective auditor responds in the affirmative, AUQA may remove her/him fromconsideration, or, having considered the reasons, decide that in fact no conflict exists.

The Audit Panel is then formally appointed. The typical composition of institution and agency Panels isdetailed in sections 8.2 and 9.3, respectively.

One Panel member is appointed by AUQA as Panel Chair, a role which carries with it a number ofadditional responsibilities (see section 1.1). The AUQA staff member on the Panel also has a number ofroles in addition to those of other Panel members (for details, see section 11.6). The AUQA staffmembers first task is to establish, with the auditee and the Panel, the timetable for the Panels meetingsand visits and to ensure that appropriate arrangements are made for these. The auditee is also alerted to

the existence of the photographs and short biographies of each Audit Panel member available from theAUQA website.

As part of its quality improvement and information dissemination roles, AUQA is willing to permitobservers on Audit Panels. At most one observer is permitted on any Panel, and the concurrence of theauditee and the Panel Chair is required (for policy statement on observers on Audit Panels, see section11.8).

Unless otherwise arranged, all communication between the auditee and the Audit Panel is via the AUQAstaff member on the Panel.

4.4 Self-Review

As suggested in section 1, AUQA bases its audit on the auditees own self-review and its outcomes. Thestarting point for self-review, as later for the Audit Panels investigation, will be the objectives the auditeehas formulated. The Audit Panel, in evaluating and checking the self-review, will consider theeffectiveness of the auditees quality system and the extent to which its objectives are being met. Ingeneral terms, the self-review should address questions implied or suggested by AUQAs terms ofreference, such as, What are our internal quality procedures? Are they appropriate? Are theyeffective? How do we know?.

In addition to carrying out a self-review and writing a Performance Portfolio, some institutions andagencies prepare for AUQA audits in other ways such as a trial audit, conducted by one or more people

internal to the auditee and/or contracted by the auditee for the purpose. As an aspect of self-review, thiscan be helpful, and is akin to the widespread use of internally initiated but externally conducted reviews.However, AUQA does not expect auditees to carry out a preparatory trial audit. If it is to be undertakenat all, AUQA suggests that this should be as part of the auditees own quality system, not as training forAUQA audit.

Students have a vital role in the institutional audit process. AUQAs view is that because quality isessentially the responsibility of the institution, the most important involvements for students areparticipation in the self-review process and ongoing suggestions for improvement. In addition, AUQAinterviews students extensively during the Audit Visits.

The self-review results in the submission to AUQA of a written portfolio (the Performance Portfolio) that

briefly outlines the auditees quality system and an appreciation of its effectiveness, with evidence.


27/87



Sections 8.3 and 9.4 set out the anticipated form and content of a Performance Portfolio for institutionsand agencies respectively.

4.5 The Audit Panels Investigations (see also sections 3, 5 and 6)

4.5.1 The Performance Portfolio

Once the Performance Portfolio is submitted to AUQA, the staff member on the Panel distributes it toother members of the Audit Panel. The auditors examine the Portfolio to determine the effectiveness ofthe auditees quality system, and hence how much detailed checking is required. Even in the case of anapparently strong system, the Panel will wish to probe selected areas to obtain evidence that internalcontrol procedures are being applied as described and to check that they are effective. On receipt of thePortfolio, Panel members are requested to prepare brief written comments which are circulated to theentire Panel prior to the Portfolio Meeting.

AUQA expects that, within an institution, there should be a broad understanding of and commitment to

the Performance Portfolio. In analogous fashion, an agencys Performance Portfolio should showevidence of interaction with the agencys client institutions, and AUQA consults these stakeholders in thecourse of its audit.

The Performance Portfolio may be read at two levels. At one level, the auditor is reading for informationon the auditees quality system and forming preliminary views on that system. At another level, theauditor is forming an opinion on the quality of the self-review and the depth of the analysis itself, andattempting to answer questions such as:

How thorough is this Performance Portfolio? How perceptive is it? Does it show evidence of a genuine, useful self-review? Has the data presented been validated, and, if not, how may the Panel do this? Does the Portfolio clearly identify strengths and weaknesses? Does the Portfolio propose appropriate actions on identified weaknesses?

It cannot be too strongly emphasised that the reporting of an area for improvement in a PerformancePortfolio is not itself a weakness, especially if it is coupled with evidence that the auditee has consideredways of removing or overcoming the issue. While there may always be some things that only an externalinvestigation can reveal, the more rigorous the self-review and the more honest and frank the Portfolio,the less there is for the Audit Panel to detect de novo, and the more the visit can concentrate onverification and validation.

4.5.2 Portfolio Meeting

Between two and four weeks after receipt of the Performance Portfolio, the Audit Panel assembles for aone-day meeting at AUQAs offices (the Portfolio Meeting) to:

review AUQAs audit procedures discuss the Performance Portfolio in detail plan the outline of the audit investigation identify any further information or clarification required from the auditee or other sources decide which information is needed in advance of, and which at, the Audit Visit decide which visits to overseas or local activities (for institutional audits) or to institutions (for

agency audits) will be necessary, and

decide whom to interview at the Audit Visit(s).


28/87

Part 2: Before the Audit Visit AUQA Audit Manual, Version 3.0


The Panel uses this meeting to identify provisionally specific topics for investigation and approaches toadopt. Sampling for an institution audit may be of particular academic departments or schools, programsor processes. In the case of an agency audit, the Panel identifies which of the agencys institutions andprograms to investigate in depth.

A program for the Audit Visit to the auditee is then mapped out by the AUQA staff member inconsultation with other Panel members. The staff member also produces a document detailing the issuesidentified by the Panel, grouped by topic, and the requests for further information identified at thePortfolio Meeting. The list of further information required and the provisional Audit Visit program aresent to the auditee for discussion at the Preparatory Visit (see section 4.5.3). Further notes for auditors onthe Portfolio Meeting are included in section 11.4.

4.5.3 Preparatory Visit

Two or three weeks after the Portfolio Meeting, the Panel Chair and the AUQA staff member visit theauditee (the Preparatory Visit). AUQA has four primary purposes for this meeting. They areto:

discuss the provisional Audit Visit program check the appropriateness of selections and

combinations of interviewees, clarify the role and responsibilities of various bodies, ensure that theprogram for the Audit Visit addresses the Panels needs while also being feasible for the auditee,and, for institutional audits, discuss plans for visits to the auditees overseas or localoperations (if the Panel has agreed on such visits)

discuss the further information required by the Panel this might typically include questions ofclarification (to which there are usually relatively short answers) and requests for furtherdocuments. The auditee may prepare provisional answers to the questions and assemble possibledocuments in advance of the Preparatory Visit, and these can be reviewed at this visit to seewhether they will meet the Panels requirements. The Panel and auditee will agree on the mostappropriate means of the auditee providing answers to the questions. This may be orally or inwriting at the Preparatory Visit, or in writing soon after. Some documentation may be made

available on site at the time of the Audit Visit check whether there are any sensitive issues of which the Panel should be aware, and review the logistics for the Audit Visit (including viewing the proposed meeting room, etc.).

The above activities can usually be achieved by Panel members meeting with the auditees nominatedcontact person, although the auditee may well wish to involve others. The Preparatory Visit typicallybegins with a brief meeting between the Panel members and the CEO (or Departmental Head responsiblefor the agency). The AUQA staff member and auditee contact person work closely to produce a suitableagenda.

The auditee may wish to use the occasion of the Preparatory Visit for its own purposes (such as allowinga larger group of staff to meet the Panel members). This is quite acceptable to AUQA, provided the

purposes outlined above are also achieved, and on the understanding that the additional activities are notincompatible with the audit process (that is, they must not constitute a pre-audit).

4.5.4 Following the Preparatory Visit

Immediately after the Preparatory Visit, the AUQA staff member finalises the exact groupings of staff,students and external stakeholders to be included in the Audit Visit program and the auditee thenconcludes the arrangements for the interviews. A definitive Audit Visit program should be produced noless than two weeks before the date of the Audit Visit.


29/87



During the period between the Portfolio Meeting and the Audit Visit, the activities of the Panel varydepending on whether the audit is of an institution or agency (for more detail see sections 8.4 and 9.5respectively).

In the week before the Audit Visit, Panel members produce and send to the AUQA staff member theircomments on the additional documents received since the Portfolio Meeting and may participate indeveloping further questions for the Audit Visit interviews.


30/87


31/87

AUQA Audit Manual, Version 3.0 Part 2: The Audit Visit


5. THE AUDIT VISIT

5.1 Parameters of the Audit Visit

The Panels second plenary meeting is the Audit Visit itself. The main purpose of the Audit Visit is toallow the Panel to test the statements and descriptions in the Performance Portfolio and acquire furtherinsight into the auditees operation through first-hand investigation and personal interaction. Those whomeet the Panel are invited to explain the auditees strengths and discuss the difficulties being faced. Intheir role as colleagues and peers, Audit Panel members seek to establish a genuine dialogue with thosethey meet.

The visit allows the Panel to obtain further evidence and to interpret and judge the evidence it has beengiven. This includes, for an institution, such features as the thoroughness of program planning, thesupport of students, the support of staff, the attitude of students, the intellectual atmosphere and thecommitment to quality. From a thorough reading and analysis of a considerable amount of documentarymaterial, the interviews during the Panel visit, and the Panels own reflections on them, are the

culmination of the process through which the Panel reaches its conclusions.

The visit program is devised to permit the Panel to carry out such investigations and test such hypothesesas it feels necessary. Interviewees can reasonably be expected to be asked about anything within the scopeof the audit. The program is sufficiently flexible to give time for the provision of further information orfor the Panel to arrange further interviews with specific people, as the need for these emerges during thevisit.

In accordance with AUQA policy, Audit Panels are not permitted to accept gifts from auditees.Hospitality provided to the Panel during visits should be moderate.

5.2 Typical Visit Format

The length of an AUQA Audit Visit varies, depending on the size and complexity of the auditee, fromtwo to five days. (For detail on the typical length of visits to institutions and agencies, see sections 8.5and 9.6 respectively.) The following sections consider in detail various aspects of a typical visit.

5.2.1 Panel Preparations

The first event of the Audit Visit is often a visit by the Panel to the institution on the day beforeinterviews formally commence. A tour of the institutions campus (or campuses) may be included but themain purpose is for the Panel to inform itself of the further documentation that has been made available

on site.

The second event of the Audit Visit is a private meeting of the Panel. This meeting, usually two or threehours in length, has several purposes. These are to:

discuss the further information received since the Portfolio Meeting, including for an agency audit,the responses to the surveys (see section 3)

note the availability of further materials for inspection or reference at the site of the visit and anyinformation obtained as a result of examining this material prior to the private meeting

share new ideas following further reflection on the Performance Portfolio plan, in detail, the approach to be adopted for the interviews (particularly for the first day), and review the logistics for the Audit Visit.

By the end of this meeting, there may still be significant disagreements between Panel members onsubstantive issues. Such differences must be resolved by the end of the visit, and plans should be made


32/87

Part 2: The Audit Visit AUQA Audit Manual, Version 3.0


for questioning and other forms of investigation to achieve this. The Panel Chair and AUQA staffmember on the Panel have particular responsibility for ensuring that resolutions are achieved in Panel-only sessions during the visit.

5.2.2 Auditee Preparations

To ensure that the institutional community is informed about AUQA and its role, AUQA providesinformation to the auditee about the audit process for wide distribution. Biographies and photographs ofAudit Panel members are also available on AUQAs website. In addition, most auditees brief their staffabout the audit scope and general audit process. AUQA encourages this practice, as it is most helpful ifstaff are aware of the purposes of the audit, how it relates to the auditees own quality managementsystem and what may and may not reasonably be expected from the audit process. For instance, AUQAhas no mandate for investigating individual cases or hearing appeals.

In addition to this general briefing, an auditee may wish to hold a meeting of those individuals who willbe interviewed by the Audit Panel. This can be useful to familiarise interviewees with the audit process,so they can feel at ease and contribute more fully. Such briefings may also occur immediately before each

group of interviewees meets the Panel. Care should be taken, however, to ensure that such preparationdoes not slip over from briefing interviewees to training them in what to say and/or how to behave.Any such coaching is usually very evident to the Panel and may reduce its confidence in intervieweesresponses. In any case, and as noted above, the Panel may ask interviewees about anything within thescope of the audit, not merely the topics that may seem obvious for the particular group.

AUQA finds that auditees sometimes ask each group of interviewees about their experience immediatelyafter their interview with the Audit Panel. This debriefing may be appropriate if it allows intervieweesto report on their experience of the Panel (whether it acted in a professional manner, listened to them, wascourteous, etc.). Indeed, AUQA seeks feedback on such matters each day of the Audit Visit from theauditees contact person. However, AUQA strongly discourages debriefing meetings in whichinterviewees might feel obliged to participate and divulge their specific responses to Panel questions.AUQA provides an assurance that information provided orally to the Audit Panel will not be attributed.The auditee is expected to respect this aspect of the audit process, which could be undermined by adetailed obligatory debriefing. It is even less appropriate to expect interviewees from groups that aresomewhat independent of an institutions management (eg student or staff associations or communitygroups) to participate in such debriefing. In addition to any of these considerations, auditees should beaware that interviewees responses in any debrief may not accurately capture what was actually said in theinterview or the context in which it was said.

5.2.3 Interview Sessions

During the Audit Visit, the Panel mostly operates as a single group, but may split into subgroups for visits

to different areas or campuses or for meetings with large groups in lunchtime sessions. While the overallprinciple is that the Panel should stay together for interview sessions, the Panel may also split intosubgroups when there would otherwise be a diverse group of interviewees. For institutional audits, briefvisits by individual Panel members to parts of the campus may occur in parallel with the ongoinginterview sessions (see section 5.2.5). Arrangements for such activities are discussed with the auditee atthe Preparatory Visit.

AUQA recognises that interviewees often hold a number of different responsibilities, but in the short timeavailable to the Panel it is inefficient and unhelpful to see the same person many times. The Panel wouldnot generally expect to meet anyone more than twice, and preferably only once.

When the group being met is selected from a wider population (for instance, a selection of undergraduatestudents, or a selection of community representatives), AUQA may ask the auditee to be responsible for

Australian Quality Audit

Documents

Transcript of Australian Quality Audit