György BAZSA President, HAC

ENQADebriefing/brainstorming meeting for the

training of experts for agency reviews

Barcelona, 17 March 2009

A possible option: grading scheme for judging

compliance with criteria

Basic quantitative requirements - of a fine wine:

ethanol 9 – 13 %sugar 0 – 100 g/lpH 3 – 4

- of a nice hostess:bust size 80 – 99 cmhip size 60 – 65 cmwaistline 75 – 90 cm

- of an excellent chemist:publications minimum 50citations minimum 200impact factor minimum 40

But still there is a need for the evaluation of its/her/his quality by experts!

Now, what about a good quality assurance agency?

i. Membership

REGULATIONS OF ENQA(as from 26 September 2008)

TITLE III. INVOLVEMENT IN ENQA

a) Full MembershipFull Membership of ENQA is open to quality assurance

agencies that meet the relevant (‘eight’) criteria de-scribed in Annex I Membership Provisions document.

b) Candidate MembershipCandidate Membership is available to quality assurance

agencies that meet the criteria described in Annex I Membership Provisions document. (i.e. ‘substantial compliance with a majority of the criteria”)

ANNEX 1 – Membership Provisions

CHAPTER I. CRITERIA FOR FULL MEMBERSHIP

a) Full Membership of ENQA is open to quality assu- rance agencies in the field of higher education from EHEA member states that have been operating and conducting actual evaluation activities for at least two years.

Before being accepted as a Full Member, an applicant agency must satisfy the Board that it meets the eight criteria, listed below. The applicant agency will thereby also meet the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). - (8 or 15 criteria?)

b) An applicant agency may apply for Candidate Membership rather than Full Membership in the first instance. The Board will grant such member-ship if it believes

- that the applicant demonstrates, through its application, substantial compliance with a majority of the criteria and further believes

- that the applicant will be able to demonstrate, through an acceptable external review submitted within no more than two years Candidate Membership, full compliance with the criteria for Full Membership.

QAAdocuments QAAexpert report

Candidate2y m. FullFull5y5y member member

ENQA

ENQA

non non

non non

ii. Evaluation of the compliances by the expert panel

Dictionary and stages of compliance

Compliant: in accordance with a specified body or rules (usually used in combination):

Fully (compliant): entirely or wholly

Substantially (compliant): of ample or considerable amount, quantity, size, etc.

Partially or partly (compliant): in part, to some extent or degree, not wholly; being such in part only, not total or general, incomplete

Non-compliant: failure or refusal to comply, as with a law, regulation, or term of a contract.

ENQA National reviews (quide):

summarised conclusion stating whether the agency is

- fully or substantially compliant;

- partially compliant; or

- non-compliant.

ESU

The study visit team believes that objectivity is best served by using a three level marking system.

- Compliance, where QA implements the ESG in an effective manner;

- Partial compliance, where QA practices are based on the correct interpretation of the ESG, positive aspects have been identified but the manner of implementation is not effective enough.

- No compliance, where ARACIS practices are based on misinterpretations of the ESG …;

EUA

- fully compliant

- substantially compliant

- partly compliant

- (non-compliant)

Summary (in the case of ARACIS): substantially compliant

March 2009!: Guidelines for external reviews of ENQA member and applicant agencies- fully compliant: the Agency adheres to the letter

(literal interpretation) and spirit (intent) of thecriterion/ESG standard;

- substantially compliant: if the criterion/ESGstandard is not fulfilled to the letter, the Agency might still be considered substantially compliant if the principle/spirit of the stipulated criterion/ESG standard is followed in practice;

- partially compliant: the Agency does not fully or substantially comply with the criterion/ESG standard;

- non-compliant: the Agency does not comply with the criterion/ESG standard.

The way in which the membership criteria/ESG are to be interpreted by the review panel and the Board

A possible option: grading scheme for judging compliance with different criteria

EvaluationEvaluation GradingGrading

FullyFully compliant compliant + + ++ + +

SubstantiallySubstantially compliant compliant + + / -+ + / -

PartiallyPartially compliant compliant + / - - + / - -

NonNon-compliant-compliant / / - - -- - -

The grading is proposed by the review panel

„The Review panel will need to exercise its judgement, in the light of all the evidence before it, and the intentions of the ENQA membership criteria / ESG, as to whether substantial compliance is being achieved, taking account of the circumstances in which the Agency is required to operate (see section 3.6.2). There is little point in adopting a ‘hard line’ position in respect of compliance with the ESG if, by doing so, trustworthy and credible agencies are prevented from gaining Full membership of ENQA through no fault of their own.”

„There may, as a result, be occasions where a strict reading of the ENQA membership criteria / ESG would indicate substantial rather than full compliance with one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. This may occur when national legislation requires a different approach, or where an alternative way of achieving an objective is used, or because of other justifiable reasons.”

„Where an agency is found to be either partially compliant (+ - -) or non-compliant (- - -) with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country(ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account.”

A possible option for 15 criteria: grading scale for granting ENQA membership

Decided by the BoardProposed by the review panel

PointsPoints LimitLimit Overall Overall grading?grading?

Membership Membership grantedgranted

45 – 4145 – 411 partly c.1 partly c.

0 non-c.0 non-c.Fully ?Fully ? Full memberFull member

40-39-3840-39-38 Board decidesBoard decides

37-36-3537-36-352 partly c.2 partly c.

1 non-c.1 non-c.Substan-Substan-

tially ?tially ?Candidate Candidate membermember

34-33-3234-33-32 Board decidesBoard decides

31 31 →→ ------ Non Non Non-memberNon-member

PointFully 3+

Subst 2+

Partly 1

Non 0

Sum

Full member

41

12 2 1 0 41

11 4 0 0 41

41

13 1 0 1 !!! 41 13 0 2 !!! 0 41

Full or candidate member

403938

11 1 3 0 38

10 3 2 0 38

9 5 1 0 38

8 7 0 0 38

Candidate member

373635

11 1 1 2 36

10 2 2 1 36

9 4 1 1 36

8 5 2 0 36

7 8 !!! 0 0 37

  PointFully 3+

Subst 2+

Partly 1

Non 0

Sum

 Candidate or non-member

 

343332

10  0 2 3 !!! 32

9 1 3 2 32

8 3 2 2 32

8 2 4 1 32

7 5 1 2 32

7 3 5 0 32

6 6 2 0 32

Non-member31

8 3 1 ! 3 ! 31

8 2 3 ! 2 ! 31

8 1 5 ! 1 ! 31

iii. Further proposals

- renumbering of ENQA criteria

- public list of data

- publicity of ENQA evaluation

- QA vs. QP

1 1 Executive summary 2+ - Glossary3 2 Introduction (giving contextual information: reason(s) for the commis-

sioning of the review; the place of the agency in the quality assurance structure of its jurisdiction; the main functions; the engagement of the agency with the ESG);

4 3 Findings (each membership criterion should be discussed separately)4.1 3.1 ENQA criterion 11 1: Activities / ESG 3.1, 3.34.2 3.2 ENQA criterion 11 2: External qual. assurance proc./ESG 2 4.3 3.3 ENQA criterion 22 3: Official status / ESG 3.24.4 3.4 ENQA criterion 33 4: Resources / ESG 3.4 4.5 3.5 ENQA criterion 44 5: Mission statement / ESG 3.5 4.6 3.6 ENQA criterion 55 6: Independence / ESG 3.6

4.7 3.7ENQA criterion 66 7: External quality assurance criteria and processes used by the members / ESG 3.7

4.8 3.8 ENQA Criterion 77 8: Accountability procedures / ESG 3.8 5+ 4 Any sections relating to additional Terms of Reference of the review6 5 Conclusion and recommendations (?)7 6 Annexes (Key pieces of evidence – i.e. extracts from legislation,

policies and procedures etc. – may be added as appendices)

Further proposals (out of scope of the title):

ENQA should prepare a list of compulsory data and information about QAA to be posted permanently on member agency’s homepage (partly in English).

Application, self-evaluation report, date of site visit and composition of visiting panel should be posted on the QAA’s homepage like on ENQA homepage.

QA vs. QP: quality assurance is crucial, but qua-lity of the „product” (learning outcomes etc.) is the main aim. Should we move to QP direction?

Thank you for your kind attention!

ENQA Board decisions: (see Scrutiny form)

3. Full members’ mandatory cyclical review in each 5 year:

- full membership re-confirmed for another 5 years or

- candidate membership granted in order to conform with full membership criteria in 2 years or

- neither full, nor candidate membership is granted?