Transparency of European higher education through public quality ...
Transcript of Transparency of European higher education through public quality ...
1
occasional papers
21
TransparencY oF eUropean HiGHer eDUcaTion THroUGH pUBlic QUaliTY
assUrance reporTs(eQarep)
Final report of the project
Tiia BacH, ĐUrĐica DraGoJeViĆ , peTer FinDlaY, sTepHanie HerinG, liia laUri, orla lYncH, ZeYnep olcen, MaiKi UDaM
21
TransparencY oF eUropean HiGHer eDUcaTion THroUGH pUBlic QUaliTY
assUrance reporTs(eQarep)
Final report of the project
Tiia BacH, ĐUrĐica DraGoJeViĆ , peTer FinDlaY, sTepHanie HerinG, liia laUri, orla lYncH, ZeYnep olcen, MaiKi UDaM
ISBN978-952-5539-74-5ISBN978-952-5539-75-2ISSN1458-1051
ThepresentreportcanbedownloadedfromtheENQAwebsiteatwww.enqa.eu/index.php/publications
©EuropeanAssociationforQualityAssuranceinHigherEducationAISBL2014,BrusselsENQAAISBL,AvenuedeTervuren36-38bte4,1040Brussels,BelgiumQuotationallowedonlywithsourcereference.
Coverdesignandpagelayout:double-id.comEditedbyMariaKeloandLindseyKerber
Brussels,Belgium,2014
ThisprojecthasbeenfundedwithsupportfromtheEuropeanCommissionintheframeworkoftheLifelongLearningProgramme.
Thispublicationreflectstheviewsonlyoftheauthor,andtheCommissioncannotbeheldresponsibleforanyusewhichmaybemadeoftheinformationcontainedtherein.
5
TaBle oF conTenTsForeword……………………………………………………………………………………......................….……………..4
Chapter1: IntroductiontotheEQArepproject....................................................................................................5
Chapter2: Introductiontoqualityassurancereports.........................................................................................7
Chapter3: Understandingthestakeholders’perspectiveontheuseandusefulnessofexternalqualityassurancereports.................................................................10
Chapter4: Analysisofqualityassuranceagencies’currentpracticesinreportingtheoutcomesofthequalityassuranceprocedures.................................................................... 26
Chapter5: Europeanguidelinesforsummaryreportsofexternalqualityassuranceproceduresandgenericrecommendationsforcomprehensivereports................................42
Chapter6: Featuresofagoodexternalqualityassurancereport................................................................ 47
Chapter7: Conclusionsandrecommendations................................................................................................ 53
Acknowledgements.............................................................................................................................56
References.............................................................................................................................................. 57
ANNEXES...............................................................................................................................................58
6
ForeWorDTheevolutiontowardsamoreknowledge-basedsocietyandstrongcompetitioninthelabourmarkethasresultedinanincreasinginterestinhighereducationinEurope.Giventhelargenumberofhighereducationinstitutions(HEIs)andstudyprogrammesonthemarket,choosingaprogrammeandaninstitutionhasbecomemoredifficult.Asaresult,students,otherstakeholdersandthepublicatlargearelookingformoredetailed,reliableandcomparableinformationonthequalityofindividualstudyprogrammes,facultiesandHEIs.
AsstatedintheEuropeanCouncilconclusionsof12May2009,regardingastrategicframeworkforEuropeancooperationineducationandtraining(ET2020),oneofthefourstrategicobjectivesfortheframeworkisto“improvethequalityandefficiencyofeducationandtraining”.Qualityassurance(QA),whichplaysanimportantroleinfosteringthetransparencyofEuropeanhighereducation,canbeconsideredanimportantmeanstoachievethisobjective.Inresponsetothegrowingneedforaccessibleandreliableinformationoninstitutionsandstudyprogrammes,ENQAhascarriedouttheEU-funded“TransparencyofEuropeanhighereducationthroughpublicqualityassurancereports"(EQArep)”project.TheoverallaimoftheprojectistounderstandtheneedsoftherelevantstakeholdersandtoimprovetheQAreportsproducedbyQAagenciesasaresult.TheprojecthasbeenimplementedincollaborationwiththeSwissCenterofAccreditationandQualityAssuranceinHigherEducation(OAQ),Quality&QualificationsIreland(QQI),the(Croatian)AgencyforScienceandHigherEducation(ASHE)andtheEstonianHigherEducationQualityAgency(EKKA).
ThisreportpresentstheresultsoftheactivitiescarriedoutinthecontextoftheEQArepproject.Indoingso,thereportsetsoutthecurrentpracticesofQAagenciesinpublishingtheoutcomesoftheirevaluations(QAreports)andanoverviewoftheuseandusefulnessoftheseQAreportsfromastakeholderperspective.Bothanalysesarebasedonthefindingsofsurveysandworkshopscarriedoutaspartoftheproject.
TheprojecthasresultedinasetofEuropeanGuidelinesforsummaryQAreportsaddressedtoQAagenciesandasetofgenericrecommendationsforthedraftingofcomprehensivereports.TheseGuidelines,intandemwiththeEuropeanStandardsandGuidelines(ESG),aimtoprovideaframeworkfortheQAagenciestoworkwithin.TheGuidelinesspecificallyrecommendthatallsummaryreportsproducedbyQAagenciesprovidesimilarandcomparabletypesofinformation,tohelpthebeneficiariesbetterunderstand,compareandinformchoice,whilecontributingtothetransparencyofEuropeanhighereducation.
Onbehalfoftheprojectconsortium,Iwouldliketothankthecontributorstothisproject.Thesearetherespondentstothesurveys,participantsoftheworkshops,membersofthesteeringgroupandmembersoftheadvisoryboard.
IwouldalsoliketotakethisopportunitytoencourageallENQAmemberagenciestoconsidertheoutcomesofthisprojectfortheirownwork,thuscontributingtotheshapingofthefutureofQAreports.
Padraig WalshPresident of ENQA
7
Chapter 1:
inTroDUcTion To THe eQarep proJecTZeynep Olcen, European Association for Quality Assurance in Higher Education (ENQA)
responding to a need for better information
Asaresultofincreasingcompetitioninthelabourmarketandtheshifttowardsasocietywhereknowledgeisthekeytosuccess,highereducationinstitutions(HEIs)arethecenterofattentionforstudentsandotherstakeholdersmorethanever.TheincreasedinterestanddemandforHEIsraisesconcernsregardingthequalityandtheaccessibilityofinformationaboutinstitutionsandstudyprogrammes.Stakeholdersrequiretransparent,reliableandcomparableinformationaboutinstitutionsandprogrammesinordertomakewell-informeddecisions.
Amongothertoolsprovidingtransparentinformation,suchastheBolognatransparencytoolsorrankings,qualityassurance(QA)playsamajorroleincontributingtothetransparencyofEuropeanhighereducation.Withinthisperspective,QAreportsareconsideredanimportantsourceofreliableandcomparableinformation.AsmentionedintheStandards and Guidelines for Quality Assurance in the European Higher Education Area(ESG),externalQAreportsshouldbemadepublic.However,thecontent,structureandpublishingchannelsofthesereportsvaryconsiderably,andtherearenocommonguidelinesforQAagenciestofollow.ThissituationlimitsthepotentialofreportsincontributingtothecomparativedimensionandtransparencyofhighereducationattheEuropeanlevel.
InordertofostertheroleofQAreportsasasourceofreliableandcomparableinformationandtocontributetotheoveralltransparencyofEuropeanhighereducation,theEuropeanAssociationforQualityAssuranceinHigherEducation(ENQA)decidedtocarryoutaprojectentitled“TransparencyofEuropeanhighereducationthroughpublicqualityassurancereports(EQArep)”.Theprojectwassettoinvestigatethecurrentstateofthecontent,structureandpublicationofQAreportswhiletryingtounderstandtheexpectationsanddemandsofstakeholderswithregardtothesereports.Onthebasisoftheresults,theprojectdevelopedasetofrecommendationsforusebyQAagencies.
InadditiontoENQAastheleadpartner,theprojectconsortiumcomprisedfourQAagencies:theSwissCenterofAccreditationandQualityAssuranceinhighereducation(OAQ),Quality&QualificationsIreland(QQI),CroatianAgencyforScienceandHigherEducation(ASHE)andtheEstonianHigherEducationQualityAgency(EKKA).TheprojectwassuccessfulinobtainingfundingfromtheEuropeanCommissionLifelongLearningProgramme.
methodology
Theproject’sprimaryobjectivetodevelopcommon“EuropeanGuidelinesforexternalQAreports”requiredtwoimportantactions:1)analysingandunderstandinghowthe
8
stakeholdersperceivetheuseandusefulnessofexternalQAreportsproducedbyQAagencies,and2)mappingthecurrentpracticesofQAagenciesinpublishingexternalQAreportsintermsofcontent,structureandpurpose.
Withinthisperspective,thefirstpartoftheprojectmethodologyconsistedofconsultingvariousstakeholders-namelystudents,employers,governmentsandHEIs-throughanonlinequestionnairedevelopedbytheprojectworkinggroup.Theaimofthequestionnairewastwofold:1)tomapthecurrentuseofinformationregardingqualityinHEIsandstudyprogrammes,and2)toidentifytheexpectationsofthestakeholdersregardingthereports.Theanalysisoftheonlinesurveywaspresentedanddiscussedduringaninteractiveworkshopwithstakeholdersfromdifferentbackgrounds.
ThesecondactivityfocusedonmappingthecurrentreportingmethodsandpracticesofQAagencies.Followingthesamestructureasforpartone,anonlinesurveywasdevelopedbytheprojectworkinggroupandsenttoENQAmembersandaffiliates.Thesurveyquestionsfocusedonthepurpose,structure,contentandpublicationchannelsoftheQAreports.Inaddition,theprojectworkinggroupperformedanin-depthanalysisofaselectedsampleoftwentyQAreportsbasedonacommongridinordertoconsolidateandframethefindingsofthequestionnaire.ThefindingsoftheonlinesurveyandtheanalysisofthesampleofreportswerepresentedanddiscussedinaninteractiveworkshopwithQAagencies.
Theresultsofbothanalysessupportedtheprojectworkinggroupinidentifyingareaswhereamismatchexistedbetweenthecurrentpracticesandtheexpectationsofthebeneficiarygroups.BasedontheneedsandrequirementsexpressedbythestakeholderswithregardtoexternalQAreports,theprojectworkinggroupdevelopedasetofrecommendationsforQAreportsandspecificallyforsummaryQAreportsinsofarastheircontent,structureandaccessibilityareconcerned.
Theprojectconsortiumconsideredthatanimportantdistinctionhadtobemadebetweencomprehensive(“full”)reportsandsummaryreports.Infact,thegroupconsidersthatthemainpurposeofinstitutionalandprogrammecomprehensiveQAreportsistofacilitatearevieworaccreditationdecisionandtoserveasatriggerforenhancementatthereviewedinstitution.TheirprimaryusersarethereforetheHEIsorprogrammesunderreview.Thesummaryreports,ontheotherhand,aremainlyproducedtoprovideconciseandeasilyreadableinformationtothegeneralpublic.Theprojectconsortium,basedonthefindingsoftheprojectactivities,cametotheconclusionthatattemptingtouniformalisecomprehensivereportsmightresultinthelossofrelevantimportantinformationtotheinstitutionorareducedusefulnessofthereportstotheirmainusers.Therefore,whilethisreportprovidessomerecommendationsinsofarascomprehensivereportsareconcernedandenlistsgoodfeaturesofcomprehensivereports,theguidelinesthathavebeendevelopedconcentrateonsummaryreportsspecifically.Whilenotsuggestingfullstandardisationofsummaryreports,theprojectconsortiumconsidersthatinordertobettermeettheneedsofstakeholders,ahigherdegreeofcomparabilitybetweensummaryreportswouldbebeneficial.
ENQAinvitesitsmembersandaffiliates(whererelevant)tomakegooduseofthedevelopedrecommendationsinordertoenhancetheinformationvalueandaccessibilityoftheirreportsandthuscontributetotheincreasedtransparencyofEuropeanhighereducation.
9
Chapter 2:
inTroDUcTion To QUaliTY assUrance reporTsOrla Lynch, Quality and Qualifications Ireland (QQI)
Context and purpose of Qa reports
Theproductionandpublicationofreportsarecommonunifyingfeaturesofallexternalqualityassurance(QA)proceedingsintheEuropeanHigherEducationArea(EHEA).Standard2.5oftheStandards and Guidelines for Quality Assurance in the European Higher Education Area,20091(ESG)requiresthat“reportsshouldbepublishedandshouldbewritteninastylewhichisclearandreadilyaccessibletoitsintendedreadership.Anydecisions,commendationsorrecommendationscontainedinreportsshouldbeeasyforareadertofind.”TheproductionofthesereportsallowsforarecordofaQAreviewatprogrammeorinstitutionlevel.Thefrequentlyassertedcoreobjectiveforreportswhicharetobemadepublicistransparency.IntheEHEA,theoriginsofQAcanbetracedtotwomainsources:attemptstosolveproblemscentredonquestionsofqualityandconsiderationsrelatingtotheimprovementofthesystemsthatunderpinhighereducation.Consequently,QAhastraditionallyservedtwomainpurposesinhighereducation:enhancementandaccountability.ThequestionremainsastowhethertransparencyisapurposeofQAonparwithaccountabilityandenhancementorwhetheritisakeyprincipleorstandardforQA.Theexistenceofstandard2.5onQAreportsintheESGmayindicatethelatter.
IntheEHEA,whereexternalQAregimesareincompliancewiththeESG,varietyandsteadychangearekeyfeatures2.ThemostcommonexternalQAproceduresareaccreditationandevaluationofprogrammes,followedatasignificantdistancebyevaluationandaccreditationofinstitutions,thoughthegapisclosing.Ninetypercentofagenciesapplymorethanoneapproachand75percentofagencieshavechangedorarechangingtheirapproach.ThevariabilityofnationalagendasmeansthattheemergenceofafullyunifiedexternalQAsysteminEuropeisunlikely.
Overthecourseofrecentyears,inmoreeconomicallystraitenedcircumstances,thestakesaroundexternalQAhavebeenraised,andexternalQAintheEHEAhasbecomemorevisible.Duetocompetitionanddiversification,qualityhasbecomeacoresuccessfactorforinstitutionalsuccess.Guaranteeingacertainlevelofqualityorenhancingthequalityofaprogrammeoraninstitutionhasbecomeanintegralpartofregularmanagementandexternalmarketingofhighereducationinstitutions(HEIs).PossiblyderivingfromtherequirementforcompliancewiththeESG,thereisastrongeremphasis1 Thune,C.(2009).Standards and guidelines for quality assurance in the European Higher Education Area, 3rd edition. Helsinki:
EuropeanAssociationforQualityAssuranceinHigherEducation.2 Rauhvargers,A.,Deane,C.,&Pauwels,W.(2009).Bolognaprocessstocktakingreport2009.Report from working groups
appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvain-la-Neuve.Brussels:LifelongLearningProgramme,EuropeanCommission.
Westerheijden,D.F.,etal.(2010).The Bologna Process Independent Assessment-The first decade on working on the European Higher Education Area-Volume 1 Detailed assessment report. CHEPS/INCHER-Kassel/ECOTEC.Availableat:www.ond.
vlaanderen.be/hogeronderwijs/bologna/2010_conference/documents/IndependentAssessment_executive_summary_overview_conclusions.pdf;
Crozier,F.,Grifoll,J.,Harris,N.,Kekalainen,H.,&Malan,T.(2011).Evaluation of the reports on agency reviews (2005-2009). ENQA Occasional Papers 16.Helsinki:EuropeanAssociationforQualityAssuranceinHigherEducation.Availableat: www.enqa.eu/indirme/papers-and-reports/occasional-papers/ENQA_Occasional%20paper_16.pdf
10
onQAand,inparticular,onreportinginrelationtoQAasasourceofinformationtoinforminternalmanagementdecision-makingandtoinformanexternalpublicaboutthequalityofaninstitution.ThereisgrowinginterestindetailedandreliableinformationonthequalityofindividualstudyprogrammesandHEIs,sometimesforcomparisonpurposesratherthandescriptionofsingleprogrammesorinstitutions.
QAisahighlydynamicanddiversefieldwithdivergentrequestsfromstakeholders.HowcanQAandreportingonQAmeetthischallenge?InthefirsttenyearsoftheESG,thefocushasbeenonmethodology.Isittimenowforashiftinfocusfrommethodologytowardspurpose?AndarethepurposesofexternalQAsufficientlyclear?AstudybyDavidWoodhouse(2010)indicatesthataconfusingandheterogeneousarrayofstatedpurposeshaveemergedforexternalQAagencies3.ItisvalidtoassertthatasinglepurposeforexternalQAwouldnotbeappropriate.ArecentevaluationofexternalreviewsinIreland,theReviewofReviews4,inkeepingwiththeWoodhousereport,foundthatthepurposesofexternalQAreviewswerenotclear.Furthermore,theReviewofReviewspositedthatbeforemattersrelatingtoreviewmethodscanbeproperlyorevenadequatelyaddressed,theremustbeclaritywithrespecttopurpose.
WhilsttheexistingpurposesofexternalQAarenotclearorunified,theydoappeartoconvergearoundanumberofkeythemes,whichare:
• ToassistHEIsinassuring/enhancingtheirquality,indevelopingtheirinternalstructuresandproceduresandinachievingvariousaims
• Toinvestigatethe‘quality’ofprogrammesorinstitutions• Tocheckcompliancewithcertain(legal)requirements• Toassesseffectiveness/successofcertainpolicies/reforms• Toprovideindependentinformationforcomparingprogrammes/HEIs• Toprovideindependentinformationfordecision-making(funding,enrolment,
collaborativework)• Toprovideindependentinformationforcertainconstituencies• ToprovideindependentinformationaboutqualityofHEIs/programmes/
HEI-systems
Theintendedaudiencesforreportsarenothomogeneous.Reportscanbevariously,andsometimessimultaneously,addressedtoHEImanagement,teachers,students,employers,cooperationpartners,politicaldecision-makers,ormediaandsocietyatlarge.
InthesecontextsitisworthgivingdeliberateconsiderationtoreportingonexternalQA.Ifreportsaretobefitforpurpose,thenexternalQAreportsarerequiredtomeetvaryingpurposesacrosstheEHEA,alignedtodifferentnationalpriorities.Furthermore,evenwithinasinglestateoragencytherecanbearangeofpossiblycompetingpurposestowhichexternalQAisaligned.TheconsequenceofthisisthatvariouskindsofinformationarerequiredfromreportsacrosstheEHEA,andsometimestherearediverserequirements,orpurposes,forasinglereport.TheESGstandard2.5forreportingrequiresthat“reportsshouldbepublishedandshouldbewritteninastyle,whichis
3 Woodhouse,D.(November2010).Is there a globally common understanding to Quality Assurance?ESU Board Meeting 59 Seminar: Quality Assurance. PresentationconductedfromJerusalem,Israel.
4 QualityandQualificationsIreland(QQI)(2014).Review of Reviews: Report of the Independent Review Team. Availableat: www.qqi.ie/Downloads/Reviews/Review%20of%20Reviews/12639-QQI%20Review%20of%20Reviews-WEB.pdf.
11
clearandreadilyaccessibletoitsintendedreadership.Anydecisions,commendationsorrecommendationscontainedinreportsshouldbeeasyforareadertofind.”BasedonthecurrentdraftfortherevisedESG5,thenewversionwilllikelygofurther,statingthat“fullreportsbytheexpertsshouldbepublished.”Therevisedversionmayalsoinclude(intheGuidelines)arecommendationtoallowforthereporttoinclude“featuresofgoodpractice,demonstratedbytheinstitution”.Thecurrentversionoftherevisedguidelinesalsoindicatesthat“thepreparationofasummaryreportmaybeuseful.”
Asoutlinedabove,thediversityofpurposesforexternalQAthathaveemergedmeansthattherearevariedandheterogeneousneedsandaudiencesforQAreports.Allagenciesproducereportsasanoutcomeoftheirreviews.WhatarethekeyfeaturesofareportthatprovidesareliableaccountoftherevieweventwhileassuringthatthevariouspurposesofexternalQAhavebeenaddressed?Furthermore,howcanareportadequatelyinformadiverseaudience?Isonereportsufficient?Orshouldtherebedifferentreportsfordifferentaudiences?Thisprojectemergedasadirectresponsetoanecessitytodeliberateonreportingingeneral,thesequestionsinparticularandtheconsiderationofalloftheseinabroadercontextoftransparency.
5 E4Group,incooperationwithEI,BusinessEurope,andEQAR(2014).Revision of the ESG.Availableat:http://revisionesg.wordpress.com.
12
Chapter 3:
UnDersTanDinG THe sTaKeHolDers’ perspecTiVe on THe Use anD UseFUlness oF eXTernal QUaliTY assUrance reporTsMaiki Udam, Liia Lauri, and Tiia Bach, Estonian Higher Education Quality Agency (EKKA)
introduCtion and methodology
Thesurveyontheuseandusefulnessofexternalqualityassurance(QA)reportsfordifferentstakeholderswasdirectedatallmainstakeholdergroupsinhighereducationQA:students,potentialfutureemployers,governments,andhighereducationinstitutions(HEIs)themselves.Thepurposeofthesurveywastoidentifyandcomparetheexactinterestsofthevariousstakeholdersregardinginformationaboutthequalityofinstitutionsandprogrammes.
Thequestionnaire(Annex1)wassentbytheEQAreppartnerstostakeholderswithintheirrespectivecountriesinFebruary2013andtoadditionalstakeholdersbytheENQASecretariat.ThefirstpartofthequestionnairedealtwiththecurrentuseoftheinformationaboutqualityinHEIsandstudyprogrammes.Thesecondpartofthequestionnairefocusedontheexpectationsofstakeholders:whatinformationconcerningthequalityofaHEItheyneed,aswellaswhereandinwhatformattheinformationshouldbepresented.Thelastthreequestionsconcernedinformationabouttherespondent.AworkshopforstakeholdersorganisedinMay2013exploredthefindingsofthesurveyandcollectedfurtherinsightsontheexpectationsofdifferentstakeholdersandthepossibleformat/templateofanassessmentreport.
Stakeholdersweregroupedasfollows:
• representativesofHEIs• students• publicauthorities/governmentoffices• employers• funders/investors• other
13
Intotal,therewere127respondentsfrom15countries(Table1).Table 1. Number of respondents by country
CouNTry resPoNdeNTs
Ireland 37
Estonia 35
Switzerland 21
Croatia 15
Romania 3
Italy 3
France 3
Slovenia 2
Bulgaria 2
UnitedKingdom 1
Netherlands 1
Hungary 1
Germany 1
Bosnia-Herzegovina 1
Austria 1
ThemostactiverespondentswererepresentativesofHEIs,witharesponserateof70percent.Theresponserateofstudentsandpublicauthoritieswas28percentand21percent,respectively.Onlythreeemployerscompletedthequestionnaire,contributingtoa10percentresponserate(seeTables2and3). Table 2. Number of respondents, by stakeholder groups
GrouP resPoNdeNTs
Highereducationinstitution
78
Student 24
Publicauthorities 10
Other 12
Employer 3
Funder/Investor 0
Table 3. response rate, by stakeholder groups
GrouP Nº asked Nº resPoNded resPoNse raTe
HEI 112 78 70%
Student 86 24 28%
Publicauthority 48 10 21%
Employer 30 3 10%
14
results of the surVey
Below,theresultsofthesurveyarepresented.Allresponsesareincludedinthegeneralstatistics(e.g.Figure1);inthesegmentedstatistics(e.g.Figure2),onlytheresponsesfromstudents,publicauthoritiesandHEIsarepresented,astheresponseratefromemployerswastoolow.
reasons for searChing for information about the Quality of a hei and its study programmes
ThefirstquestionexploredthereasonswhystakeholderssearchforinformationaboutthequalityofHEIsandstudyprogrammes.Theanswersindicatethatthemainpurposeisdecidingonpossiblefurtherstudies(31%ofallresponses),butalsofindingpartnersamongotherHEIsandevaluatingthequalityofgraduatesforrecruitmentpurposesreceivedrelativelyhighscores–23percentand16percent,respectively(Figure1).Expectedly,mostofthestudentslookedforinformationonfurtherstudies,andthegreatestamountofHEIrepresentativeswereinterestedinfindingcollaborationpartners(Figure2).
However,24percentofrespondentsselected“otherpurposes”.Thesepurposeshavebeensummarisedandgroupedintothreecategories:
• TolearnaboutinternalQAsystemsinotherHEIs• Forcomparison/benchmarking(ofsimilarprogrammes,orQAprocedures)• Todoresearch
31%
16% 23%
6%
24%
For what purposes have you searched for information about the quality of a HEI and study programme?
To decide on possible further studies
To evaluate the quality of graduates for recruitment purposes
To find partners among HEIs
To decide on investments/funding/sponsorship to a HEI or its unit
Other
Figure 1. reasons for searching for information about the quality of higher education.
Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?
15
34
2
21
22
4
5
40
2
1
6
3
2
29
6
5
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HEIs
Public authorities
Students
For what purposes have you searched for information about the quality of a HEI and study programme?
To decide on possible further studies
To evaluate the quality of graduates for recruitment purposes
To find partners among HEIs
To decide on investments/funding/sponsorship to a HEI or its unit
Other
Figure 2. reasons for searching for information, by stakeholder groups.
Current sourCes of information
Therespondentswereaskedaboutthemainsourcesofinformationtheyusedtocollecttheinformationwhichtheysought.Byandlarge,themainsourcefordifferentstakeholderswerethewebsitesofHEIs(seeFigure3).Thesecondmostpopularsourceforinformationcamefromfriends,colleagues,parents,etc.Assessmentreportsappearedtobethethirdmostpopularsourceofinformationconcerningthequalityofaninstitutionorprogramme.Onlyveryfewrespondentsmarkedsocialmediaasasourceforthiskindofinformation,anditwasslightlymorepopularamongstudentscomparedtoothergroups(Figures4,5,6).
Intheadditionalcomments,NARICandalumniwerementionedasdistinctivesourcesofinformation.
56
31
31
49
5
83
51
59
69
51
41
39
11
26
16
20
63
1
0 20 40 60 80 100 120 140
Information from/opinions of friends, colleagues, parents etc.
Various rankings/league tables
Government reports/publications
Assessment reports provided by QA agencies
Social media
Websites of HEIs
Sources to get information about the quality of HEI and study programmes
O,en Once in a While Never
Figure 3. sources for acquiring information about the quality of HeIs and study programs.
Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?
WhichsourcesdoyouusuallyuseforgettinginformationaboutthequalityofHEIsandstudyprogrammes?
16
62
2
34
22
21
40
5
25
26
40
52
46
37
1
1
50
14
10
18
8
3
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Websites of HEIs
Social media
Assessment reports provided by QA agencies
Government reports/publications
Various rankings/league tables
Information from/opinions of friends, colleagues, parents etc.
Other (please name below)
HEIs: Which sources do you usually use for getting information about the quality of HEIs and study programmes?
Often Once in a while Never
Figure 4. sources for acquiring information: HeIs.
Figure 5. sources for acquiring information: Public authorities.
HEIs:WhichsourcesdoyouusuallyuseforgettinginformationaboutthequalityofHEIsandstudyprogrammes
PublicAuthorities:WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIsandstudyprogrammes?
17
Figure 6. sources for acquiring information: students.
reasons why not to use assessment reports
SeveralrespondentshadneverusedQAreportsasaninformationsource.Thesurveysoughtinsightfromrespondentswhoindicatedthattheyneverusedassessmentreportsastowhythesesourceswerenotutilised.Thirty-sixpercentofrespondentsindicatedtheyeitherdidnotknowwheretofindthereports,ortheydidnotknowaboutthem(Figure7),suggestingthatstakeholdersarenotsufficientlyawareofthereportsandthekindofinformationtheymayprovide.Incaseof20percentoftheresponses,respondentsfoundtheneededinformationelsewhere,andincaseof15percentofresponses,itwasclaimedthatthereportsdidnotcontainnecessaryinformation.One-fifth(21%)ofresponsesrevealedthatthereportsareeithertoolongortoocomplicatedtounderstand.Interestingly,thedifferentstakeholdergroupshadquitesimilarreasonsfornotusingtheQAreportsasasourceofinformation(Figure8).
Additionalcommentsindicatedthelackoftimetoconsultreportsasareason,butalsothefactthatthereportsarenotalwayspubliclyavailable.
Students:WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIsandstudyprogrammes?
18
20%
14%
3%
9% 15%
22%
14% 3%
If you do not use reports by QA agencies as a source of information, please explain why.
I found the needed information elsewhere
The reports are too long
There are no reports in English/language I understand
The reports are in a too complicated language
The reports do not contain the information I need
Did not know where to find them
Did not know about them
Other
Figure 7. reasons why not to use reports published by Qa agencies.
9
1
6
16
2
6
9
2
5
7
1
3
2
0
1
9
2
5
15
0
5
2
0
0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HEI
Public authorities
Students
Did not know about them Did not know where to find them
The reports do not contain the information I need The reports are in a too complicated language
There are no reports in English/language I understand The reports are too long
I found the needed information elsewhere Other (please name below)
Figure 8. reasons why not to use reports published by Qa agencies, sorted by stakeholder group.
helpfulness of assessment reports
IndividualswhouseassessmentreportspublishedbyQAagenciesasonesourceofinformationwereaskedwhethertheyfindthereportsandtheinformationprovidedinthemhelpful.
One-thirdofrespondents(32%)fullyagreedthatthereportsareahelpfulsourceofinformation,whileanadditional51percent“slightlyagree”withthestatement.Sixpercentofresponsesshowthatrespondentsdidnotfindthereportshelpfulatall(seeFigure9).Thestakeholdergroupspossess,onceagain,quitesimilarviews(Figure10).
Thisquestionraisedverymanycomments,55intotal.Themainideasexpressedinthecommentsaresummarisedbelow:
Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy.
Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy.
19
• Themainadvantageofthesereportsisthefactthatalltheinformationisgatheredinoneplace,andthesourceistrustworthy.Unfortunately,thereportsaresometimestootechnical.
• Somereportsarefartoostandardisedanddonotcontainsufficientinformationaboutwhattheevaluationteamactuallyfound.
• Thefinaldecisionoftheagencyonlyshowsifthestandardsarefulfilled–withoutanyrankingordegreeofperformanceoftheevaluatedprogrammes.
• Agencyreportsdifferinstyleandcontentconsiderably-amorestandardisedinternationalapproachwouldbeveryhelpful.
• Thelanguageusedistoocomplicatedandwouldneedtobechangedtomakethereportsmoreuser-friendly.
Ingeneral,thereportsarecomprehensiveandconsideredtocoverallrelevantareas.Atthesametime,easiercomparability(e.g.throughinternationalstandardsforreports)anduser-friendlinessisneeded(includinglanguage,lengthetc.).
32%
51%
6%
11%
If you use reports by QA agencies, do you find them helpful in providing information about HEIs/programmes?
Fully agree
Slightly agree
Fully disagree
Slightly disagree
Figure 9. Helpfulness of reports by Qa agencies.
Figure 10. Helpfulness of reports by Qa agencies, sorted by stakeholder group.
Ifyouusereportsby qualityassuranceagencies, doyoufindthemhelpfulinprovidinginformationaboutHEIs/programmes?
Ifyouusereportsbyqualityassuranceagencies,doyoufindthemhelpfulinprovidinginformationaboutHEIs/programmes?
20
plaCes to find reports published by Qa agenCies
Morethanhalf(56%)ofresponsesindicatedthatrespondentsfindthereportsonwebsitesofQAagencies.However,alargeproportionoftherespondents(36%)alsofindthereportsonwebsitesofHEIs(Figure11).Amongthestudents,nearlyhalfofthemusethewebsitesofHEIstofindthereports(Figure12).
36%
56%
8%
If you use reports by QA agencies, where did you find the assessment reports?
Websites of HEIs
Websites of QA agencies
Other
Figure 11. Places to find the reports by Qa agencies.
44
4
14
67
11
16
12
1
1
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HEI
Public authori0es
Students
If you use reports by QA agencies, where did you find/access the assessment reports?
Websites of HEIs Websites of QA agencies Other (please name)
Figure 12. Places to find the reports by Qa agencies, sorted by stakeholder group.
information expeCted by different stakeholders
Foronequestion,therespondentswereaskedtoindicatewhatkindofinformationtheynormallyneedtomakedecisionsregardingfurtherlearning,partnerships,comparisonswithotherinstitutions,etc.Themajorityofrespondents(96outof127)named‘contentofstudyprogrammes’asthesinglemostimportantpieceofinformation,followedby‘accreditationstatusofinstitutions/studyprogrammes’(80respondents)and‘strategicplanning,managementandgovernance’(78respondents)(Table4).Theoverall
Ifyouusereportsbyqualityassuranceagencies,wheredoyoufindtheassessmentreports?
Ifyouusereportsbyqualityassuranceagencies,wheredoyoufind/accesstheassessmentreports?
21
prioritiescorrespondwiththepreferencesofrespondentsfromHEIs,withtheexceptionofstudentsandpublicauthoritieswhoexpressedslightlydifferentneeds.Althoughstudentsalsoindicate‘contentofstudyprogrammes’asthemostimportantitem,theirsecondconcernis‘employabilityofgraduates’(9thimportantforHEIsandthemostimportantforpublicauthorities)followedby‘studentsupportsystem’and‘qualificationsofteachingstaff’.Publicauthoritiesvaluedequally‘contentofstudyprogrammes’,‘accreditationstatus’,‘qualificationsofteachingstaff’,‘studentsupportsystem’and‘financialresources’,placingthemallasthesecondmostimportantfollowing‘employabilityofgraduates’.Unimportantforallstakeholderswas‘institution’spositioninleaguetables’and‘historyandtraditions’(Table5).
Table 4. Information needed to make decisions regarding further learning, partnerships, comparisons with other institutions, etc.
INFormaTIoN resPoNdeNTs
Contentofstudyprogrammes 96
Accreditationstatusofinstitutions/studyprogrammes
80
Strategicplanning,management,governance 78
Qualificationsofteachingstaff 74
Internalqualityassurancesystem 73
Studentsupportsystem 67
Employability/employmentofgraduates 66
Reputationofteachingstaff 59
Numberofresearchgrants,publications,citations 57
Applicationandadmissionstatistics 57
Conditionofinfrastructure 54
Institution’sabilitytoresponddiversestudents’needs
46
Financialresources 45
Historyandtraditions 37
Institution’spositioninleaguetables 29
Other 4
22
Table 5. Information needed, by stakeholder group
HeI Public authorities
students
Contentofstudyprogrammes 69(1) 9(2) 27(1)
Accreditationstatusofinstitutions/studyprogrammes
62(2) 9(2) 13(9)
Strategicplanning,management,governance 61(3) 8(7) 14(8)
Internalqualityassurancesystem 56(4) 7(8) 12(11)
Qualificationsofteachingstaff 53(5) 9(2) 17(4)
Studentsupportsystem 47(6) 9(2) 18(3)
Numberofresearchgrants,publications,citations 46(7) 7(8) 9(13)
Reputationofteachingstaff 44(8) 4(13) 16(6)
Employability/employmentofgraduates 40(9) 10(1) 21(2)
Applicationandadmissionstatistics 40(10) 6(10) 15(7)
Conditionofinfrastructure 35(11) 6(10) 17(4)
Institution’sabilitytorespondtodiversestudentneeds
32(12) 6(10) 12(11)
Historyandtraditions 30(13) 1(15) 8(14)
Financialresources 28(14) 9(2) 13(9)
Institution’spositioninleaguetables 22(15) 2(14) 6(15)
Other(pleasenamebelow) 6(16) 1(15) 1(16)
preferable format of information
One-third(32%)ofrespondentsindicatetheywouldliketoreceiveinformationaboutthequalityofaHEIanditsstudyprogrammesina short, concentrated summary describingthemainstrengthsandareasforimprovement.Twenty-sevenpercentexpecttoseecomparativedatawithotherinstitutions,and25percentprefernumericaldatadesignatingthemostimportantaspectsoftheinstitutionanditsprogrammes.Only14percentareinterestedincomprehensivereportsprovidingextensiveinformationaboutstrengthsandareasofimprovementinmanagementandcoreprocessesofaHEI(seeFigure13).Nosignificantdifferencescouldbedetectedamongthedifferentstakeholdergroups(Figure14).Someadditionalcommentsreferredtotheuseofmultimediaasafacilitatedentrywaytoinformation.
Inthebreakoutgroupsduringthestakeholders’workshopinTallinnon6-7May,itwasdiscussedthatthenationalagencieswithinEuropeprioritisetheirownuniquenationalagendastherebymakingcomparisonbetweendifferentcountriesimpossible.Consequently,itseemsthatcomparisonscanreallyonlybemadebetweenHEIswithinonecountry.Thestudentsagreed,furthermore,thattheprovisionofcomparabledatawasnotataskofQAagencies:inQAprocedures,aninstitution’soraprogramme’sperformanceiscomparedagainstsetstandards,notagainstotherinstitutions.
23
Fortheemployers,theinstitutionalreportsareofnorelevanceandarethereforenotused,asemployersdonothaveanyspecificneedforacquiringinformationaboutinstitutions’internalQAmechanisms.Themostrelevantinformationfortheprofessionalworldis‘performance’,whichistranslatedintoquantitativeindicatorsthroughvariousrankings.
Allstakeholdergroupsagreedthatanassessmentreportshouldalsoincludeasummaryreportshowingtheoutcomeoftheassessment,themainstrengthsandweaknessesofaninstitution/programmeandtherecommendationsforfollow-upactivities.Thisisessential,asthecomprehensivereportsareconsideredtoolongandnotintendedfortheneedsofawiderreadership.Atthesametime,theinstitutionsandprogrammessubjecttoQAproceduresneedcomprehensivereports,andtheirneedsshouldnotbesacrificedforthesakeofotherpotentialreaders.
Employersrecommendedthefollowingcontentandformatbeconsideredforinclusioninthesummaryreportsofstudyprogrammes:
• Contextofthequalityassessment(voluntary/obligatory;accreditation/evaluation;periodofvalidity;qualitylabels?evaluatedbynational/internationalpanelagainstnational/internationalstandards;single/jointprocedure;accreditationstatusofofferingHEI;etc.);
• Syntheticprogrammedescription(specialfeatures,innovativecharacter,relevance,specificities);
• StatementsonachievementoftheintendedlearningoutcomesmatchingwithgivenQFlevel;
• Profileofstrengthsandweaknesses;• Linktocomprehensivereport;• Linktothewebsitewherethestudyprogrammecanbefound
Thegroupsagreedthatthelengthofasummaryreportshouldbekeptlimited,atapproximatelytwopages.Informationconcerningthecontextshouldbeprovidedinschematicform(ratherthanindiscursiveform),whiletheprogrammedescriptionshouldbediscursiveandnormallynolongerthanfivelines.Careshouldbetakenwiththeuseofquantitativedataorfiguresthatcouldbeeasilymisusedandmisinterpreted.Strengthsandweaknessesshouldbeprovidedinatablefocusingonthemainoutcomesoftheassessment.Inallgroupsitwasagreedthatatemplateorstandardreportingstructuremightbehelpful.
Thediscussionsinthestakeholders’workshopunderlinedtheimportanceofmakinginformationeasilyaccessible,readable,andevencomparable.Atthesametime,itwasunderlinedthatasdifferentQAprocesseshavedifferentsubjects(institutionsvs.programmes)anddifferentpurposes(accreditationvs.audit),asinglereporttemplatewouldnotbepossible,nordesirable.SomemaincharacteristicsofagoodcomprehensivereportcouldbeidentifiedattheEuropeanlevel,whilethedetailsshouldbediscussedanddeterminedinthenationalcontextinconsultationwiththestakeholders.Inaddition,thestakeholdergroupsagreedthattheinstitutionsaretheprimaryusersoftheQAreports,andthereportsshouldthusaddresstheirneedsfirst,beforeconsideringtherequirementsofotheraudiences.Areportsummarywasconsideredasapossiblewaytomeettheneedsofallgroups.
24
25%
32%
27%
14% 2%
In what format would you like to get this information?
Table(s) of numerical data indicating the most important aspects of a HEI/programme
A short, concentrated summary describing the main strengths and areas for improvement of a HEI/programme
Comparative data with other institutions
A comprehensive report providing extensive information about strengths and areas for improvement in management and core processes (study process, research and development), explaining also the possible reasons for a given situation
Other
Figure 13. Preferable format of information.
50
7
16
70
9
16
50
10
18
28
3
10
3
1
1
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HEIs
Public authori<es
Students
In what format would you like to get this information?
Table(s) of numerical data indicating the most important aspects of a HEI/programme
A short, concentrated summary describing the main strengths and areas for improvement of a HEI/programme
Comparative data with other institutions
A comprehensive report providing extensive information about strengths and areas for improvement in management and core processes (study process, research and development), explaining also the possible reasons for a given situation Other (please name below)
Figure 14. Preferable format of information, sorted by stakeholder group.
Inwhatformatwouldyouliketogetthisinformation?
Inwhatformatwouldyouliketogetthisinformation?
25
preferable sourCes of information
Similartotheanswerscorrespondingtothequestionwhichinquiresastocurrentlyusedsources,themostpreferablesourceforacquiringinformationconcerningthequalityofaHEIistheinstitution’sownwebpage(47%ofresponses)followedcloselybywebpagesofQAagencies(40%).Socialmediaisnotapreferredsourceforinformationonthequalityofinstitutions,withonly5percentofallrespondentsgivingitpriority(Figure15).
Whencomparingdifferentstakeholdergroups,itisevidentthatwhileabouthalfofallgroupspreferinstitutions’webpages,studentsdifferfromHEIsandpublicauthoritiesintheirexpectationsregardingothersources:onlyabout20percentofstudentsseekinformationfromthewebpagesofQAagencies.Atthesametime,theyname‘socialmedia’andeven‘printedreportsinlibraries’moreoftenthanotherstakeholders(Figure16).Somecommentssuggestthatitisnotnecessarytopublishthereportonmorethanonewebsite(e.g.thatofaQAagency),butthateventuallylinkstothepublishedreportshouldbemadeonotherrelevantwebpages(e.g.thatofaninstitution/programme).
8% 5%
40%
47%
Where would you like to find this information?
Printed reports in libraries/QA agencies/HEIs
Social media (please name the most preferred source, e.g. Facebook, Twitter, etc.)
Webpages of QA agencies
Webpages of HEIs
Figure 15. Preferable sources of information.
Wherewouldyouliketofindthisinformation?
26
81
9
27
70
11
17
5
0
9
8
0
10
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HEIs
Public authorities
Students
Where would you like to find this information?
Webpages of HEIs
Webpages of QA agencies
Social media (please name the most preferred source, e.g. Facebook, Twitter, etc.)
Printed reports in libraries/QA agencies/HEIs
Figure 16. Preferable sources of information, sorted by stakeholder group.
ConClusions
Basedontheresultscollectedinthesurveyandatthestakeholders’workshopinMay2013,itispossibletodrawthefollowingconclusionsregardingtheinformationneedsanduseofQAreportsbythestakeholders:
• Thereisnosignificantdifferencebetweentheexpectationsanduseofinformationbetweenthedifferentstakeholders(i.e.HEIs,publicauthoritiesandstudents),exceptexpectationsregardingthetypeofinformationwhichissearched.
• VerylowresponseratefromemployersindicatestheydonotseethemselvesasatargetgroupforQAofhighereducationandareprobablynotfrequentusersofQAreports.
• QAreportsarethethirdsourceofinformationafterwebsitesofHEIsandfriends/colleagueswhenseekinginformationaboutthequalityofaninstitutionoritsprogrammes.
• AwarenessabouttheexistenceofQAreports,andlocationsofwheretheycanbefound,isnotwidespread,therebysignificantlyhinderingtheiruseandinformationalvalueforawidergroupofstakeholders.
• Reportsare,ingeneral,ahelpfultooltogetinformation,buttheycanbemorecomparableanduser-friendly,especiallyintermsoflengthandthelanguageused.
• Reportsshouldcontaineasilycomparabledataintheformatofshort,concentratedsummariesandtableswithquantitativedata.
• ReportsshouldbeaccessiblebothonthewebpagesoftheinstitutionsandtheQAagencies,preferablywithlinkstoeachother.
• SomemaincharacteristicsofagoodcomprehensivereportcouldbeidentifiedattheEuropeanlevel,whilethedetailsshouldbediscussedanddeterminedinthenationalcontextinconsultationwiththestakeholders.
Wherewouldyouliketofindthisinformation?
0
27
• InstitutionsaretheprimaryusersofQAreports,andreportsshouldthusaddresstheirneedsfirst,beforeconsideringtherequirementsofotheraudiences.Areportsummarywasconsideredasapossiblewaytomeettheneedsofallgroups.
28
Chapter 4:
analYsis oF QUaliTY assUrance aGencies’ cUrrenT pracTices in reporTinG THe oUTcoMes oF THe QUaliTY assUrance proceDUres Stephanie Hering, Swiss Center for Accreditation and Quality Assurance in Higher Education (OAQ)
baCkground, Context and methodology
Asoneoftheprojectactivities,theprojectteamsurveyedEuropeanqualityassurance(QA)agenciesontheircurrentpracticesinthepublication,purpose,structureandcontentofQAreports.ThesurveyoftheQAagenciesconsistedoftwoparts:
1. AquestionnaireforallENQAagencieswassentduringsummer2013,including41fullmembersand45affiliates.Atotalof50responseswerereceived(Annex2).
2. In-depthanalysisofasampleofapproximately20reports,conductedduringautumn/winter2013-2014byENQAandtheprojectpartneragenciesfollowingasharedanalysisgridinordertoconsolidateandframethefindingsofthequestionnaire.
Furthermore,theresultsofthesurveyandreportanalysiswerepresentedandfurtherdiscussedataworkshopforagencyrepresentativesinJanuary2014,inZurich,Switzerland.
Somequestionsinthequestionnaireturnedouttobeirrelevantornotsignificant.Inthefollowingsection,onlyrelevantandsignificantresultsareshowninordertokeepthereportcomprehensibleandreadable,highlightingthemostimportantfindings.Thesurveydistinguishedbetweeninstitutionalandprogrammeassessmentreports,andthefocuslayonpublished,publiclyavailablereports.
Awareofthedifferentuseandtargetgroupsforeachtypeofreport,thecurrentanalysishasfurthermoredistinguishedbetweencomprehensiveandsummary reports,whicharedefinedasfollows:
Comprehensive report:
AnextensivereviewreportwhichdocumentsthefullanalyticaloutcomesofagivenexternalQAassessmentprocedure,beitatinstitutionalorprogrammelevel,beitwrittenbyacademicexperts,agencyemployeesoranexternaltechnicalsecretary;anin-depthanalysisuponwhichthemainfindingsarebasedismadeexplicitandisakeycharacteristicofthistypeofreport.Oftenthisistheprimaryreport.
29
Q 4. How many institutional reviews / assessments does your agency conduct on average per year? (n = 46)
summary report:
AnysummarisingformofreportingtheoutcomesofanexternalQAassessmentprocedure,beitasummary,adescription,atableofcomparativedata,afinalproceduralreport,webtextorotherpossibletypesandformsofdescriptiveorschematicreporting.Allkindsofderivativeformsofaprimarycomprehensivereportareincludedhere(exceptexclusivelyyes/noassessmentresults).
results of the surVey
Theagencysurveycontained31questionscoveringthefollowingitems:
• Typeofassessmentsconductedandreportspublished• Publicationandintended/desiredreadership• Editingandpublicationpractices• Structure• Content• Usabilityandutility• Perspectives
Themainresultsofeachofthesearepresentedbelow.
type of assessments ConduCted and reports published
Theagencieswereaskedhowmanyinstitutionalreviewsorassessmentstheycarryout,onaverage,inayear.Asthefigurebelowshows(Figure1),thevariationbetweenagenciesissignificant.
Almostathird(30%)oftheagenciesconductbetween6and15institutionalreviewsorassessmentseachyear.Thesecondlargestgroupofrespondents,28percent,completenoreviewsinayear.Thiscanbeexplainedbythefactthatsomeoftheagenciesarespecialisedinprogrammeassessmentsand/orarenewagenciesjustabouttostarttheirwork.Thirteenpercentoftheagenciesconductmorethan31reviewsorassessmentsperyear.
Figure 1. amount of institutional reviews/assessments per year.
30
Q 5. What kind of report is issued for the institutional review / assessment? (multiple answers, n = 34)
Q 7. How many programme reviews / assessments does your agency conduct on average per year (span of 5-7 years, n = 46)?
Publishedcomprehensivereportsarethemostcommonformtopresenttheresultsofinstitutionalreviews(Figure2).Thisisthecasefor77percentoftherespondents.Forty-onepercentalsoreportontheresultsoftheinstitutionalreviewsinpublishedsummaryreports.Onlythreepercentofagenciesissuenon-publishedsummaryreports,whileasmanyassixagenciesreportedthattheydonotpublishthecomprehensivereport.
Figure 2. Types of reports issued for institutional reviews/assessments.
Programmereviewsaremorecommonthaninstitutionalreviewsamongtherespondents(Figure3).Morethan60percentoftheagenciesconductover30programmereviewsperyear,and35percentimplementbetweenoneand30programmereviewsannually.Onlyfourpercentdonotexecuteanyprogrammereviews.
Figure 3. amount of programme reviews/assessments per year.
31
Figure 4. Types of reports issued for programme reviews/assessments.
Three-quarters(76%)ofthoseconductingprogrammeassessmentspublishedthecomprehensivereports,asissimilartothe77percentforinstitutionalreviews(Figure4andFigure2,respectively).Ofall,31percentdisclosetheresultsofprogrammereviewsintheformofpublishedsummaryreports.
publiCation and readership
ItisevidentthatdifferentQAreportsmeetdifferentgoalsandareintendedfordifferentpurposes(Figure5).Overall,institutional comprehensive reportsareexpectedtofacilitaterevieworaccreditationdecisionsandtosupplyfeedbacktothehighereducationinstitution(HEI)tosupportitsinternalqualityenhancement.Programme comprehensive reportshavetheprimaryobjectivetofacilitatearevieworaccreditationdecision,whiletheymayalsobeusedforenhancementpurposesandtoassuretransparency.
Institutional summary reportsaswellasprogramme summary reports aremainlypublishedtosupplyinformationtothegeneralpublicandtoassuretransparency.Theinformationcontainedinthesummaryreportsisconciseandeasiertoread,butnecessarilylimitedindetailanddepth.
Figure 5. Purpose of reports.
Q 8. What kind of report is issued for the programme review / assessment? (multiple options, n = 45)
Q 10. What is the main purpose of the reports? (multiple options, n = 48)
32
Differentstakeholdersusedifferentreports(Figure6),andindeedthedifferentreportsaredesignedtomeettheneedsoftheidentifiedtargetgroups.AccordingtotheperceptionsoftheQAagencies,institutionsandofficialauthoritiesorgovernmentalbodiesusethereportsmostfrequently.Overall,itappearsthatcomprehensivereportsaremorefrequentlyusedthansummaryreports,andtheformerarefundamentallyimportantforHEIsandofficialauthorities.AccordingtothesurveyedQAagencies,programmecomprehensivereportsare,aftertheHEIsandofficialauthorities,themostimportantforstudents.
Figure 6. users of published reports.
Theagenciesagreethatthereportsshouldideallybeusedmore,byallstakeholders.Inparticular,theagencieswouldliketohavemorestudentsandprofessionalorganisationsoremployersamongtheirreadership(Figure7).Additionally,themediaisconsideredarelativelyimportantfuturetargetuserofQAreports.Agencieswouldalsoliketoseethecomprehensivereportsbeusedmorethantheyarenowandseemtogivelessimportancetothesummaryreports.Ontheotherhand,thestakeholders(inparticularthoserepresentingtheworldofwork)expressedastrongpleaforclearandcomparablesummaryreportsinthesurveyofstakeholders(seeChapter3).
Q 11. In your opinion, which stakeholder groups currently tend to use your published reports? (multiple options, n = 48)
33
Figure 7. Potential readership, sorted by stakeholder group.
DiscussingthequestionofthecurrentandpotentialfuturereadershipofpublishedQAreportsduringtheworkshopinZurichinJanuary2014,somerespondentspointedoutthatitmaynotbenecessarytohavedifferentreportstomeettheneedsofdifferentstakeholders,butthatthedifferentpartsofthesamereportmaybeofinteresttodifferentusergroups.Inordertoputthisintopractice,thereportswillneedtobewell-structuredandtheirpurposeandcontentclearlyexplainedtoallpotentialusers.Ontheotherhand,someotheragenciesareworriedthattryingtomeettheneedsofdifferentreaderswouldleadtothemodificationofthefullreports,arguingitisimportanttomaintainthemfortheirprimaryusergroups,i.e.theinstitutionitselfandtherelevantpublicauthorities.Manyagenciesexpressedawarenessofthedifficultyinmeetingthedifferentinformationneedsofstakeholdersandcommentedthattheyarecurrentlyconsideringdifferentcommunicationandpublicationstrategiestomeetthoseexpectationsbetter.Anotherimportantchallengeherederivesfromthefactthateventostakeholdersthemselves,itisnotalwaysnoryetclearwhatexactlytheywouldandcouldexpectfromaQAreport(seesectionentitled“Usabilityandutilityofreports”onpage37).
editing and publiCation of reports
Currently,agenciespublishtheQAreportstheyproducemainlyinthenationallanguage(s)(Figure8).Atthesametime,Englishhasbecomeanimportantworkinglanguageforagencieswhichutiliseinternationalexpertsandisgaininggroundasthepublicationlanguageofcomprehensivereportsinparticular.Languagesotherthanthenationalone(s)andEnglishseemnottoberelevantinthiscontext.Consideringthatforthemajorityofagenciestheinstitutionsthemselvesand/orofficialauthorities/governmentalbodiesareconsideredasthemostimportantreadershipofQAreports,itcomesasnosurprisethatthenationallanguageisthefirstandforemostlanguageinwhichthesereportsaredrafted.
Q 12. Who might be your potential or future readership? Which stakeholders would you like to use your reports more? (multiple options, n = 47)
34
Figure 8. Languages in which reports are made available.
Mostoftenthepanelofexperts,thepeerleaderandthesecretaryofthepanelofexperts(sometimesaQAagencyemployee)arethemainauthorsofthereports(Figure9).Forcomprehensivereports,theroleofthepanelisstronger,whilesummaryreportsmaybeeditedbythesecretaryofthepanelofexpertsand/oranagencyemployee.
Figure 9. Writers of reports.
Q 14. In which language(s) are your reports available? (n = 48)
Q 15. Who writes the reports? (n = 48)
35
PreparingQAreportscanbehighlytime-consuming.Thesurveyaskedtheagenciestoprovideanestimateontheaveragetimethatittakestoproduceareport(ofthetypeusedbytheagency),consideringthetotalinputoftimebetweenallthoseinvolved(panel,QAagencystaff,etc.).Unsurprisingly,thepreparationofthecomprehensivereportsrequiresthehighesttimeinvestment(Figure10).Mostagenciesestimatetheyspendatleast16hourstowriteaninstitutionalcomprehensivereport.Thecompletionofacomprehensiveprogrammereportisnearlyastime-consuming.Agenciesindicatedthatsummaryreportsaremostlywritteninlessthan5hours.Thismayneverthelessconstituteasignificantadditionaltimeexpenditure,inparticularforagenciesthatmaynotpublishsummaryreports.
Figure 10. Time spent to complete a report.
ThereportsaremostoftenpublishedonthewebsitesofQAagencies(Figure11).ThewebpagesofotherpublicauthoritiesandofHEIsareoflessinterest,andagenciesdonotoftenhaveinformationonwhetherinstitutionspublishthereportsontheirownwebsitesornot.Thehardcopiesofreportsareconsideredirrelevantbytheagencies,orarenotapriorityconsideringalsotheadditionalcostsassociatedwithprintingreports.Atthesametime,thestakeholdersconsiderthewebsitesoftheHEIsasthemainsourceofinformation,andthereforeitseemsthatinordertoreachawidergroupoftheintendedreadership,atleastlinkstotheagencywebsiteshouldbeprovidedontheinstitutions’websites(seeChapter3).
Q 16. How much time on average does it take to complete a report (summing up the time invested by all authors* )? (n = 48)
0-5
36
Figure 11. Location of published reports.
struCture of reports
Nearlyallagencieshavetemplatesavailableforwritingreports(Figure12),whichmeansthatapre-setstructureisfollowedthroughoutthedraftingprocessforallreportsofthesametypology.Templatesexistforallkindsofdifferentreportsbutareconsideredthemostimportantandrelevantfortheprogrammecomprehensivereports.
Figure 12. Templates for reports.
Thelengthofthereports(Figure13)variesaccordingtothetypeofreport,buttheyoftenfollowguidelinessetbytheagency.Themajorityofcomprehensivereportsfallintothe‘31pagesormore’category(55%ofinstitutionalreviewsand33%ofprogrammereviews).Noinstitutionalcomprehensivereportsareshorterthan6pages.
Q 17. Where are the reports published? (n = 47)
Q 19. Does your agency have a template for the writing of reports? (n = 47)
37
Forprogrammeassessments,thereportstendtobeslightlyshorter,withonly25percentfallingintotherangeof31ormorepagesinlength.Theshorterlengthofprogrammereportsisunderscoredbythefactthatthemajorityfallbetweensixand15pages.Summaryreportsareshort,astheirnameindicates,andtheyaremostoftenundersixpages.
Figure 13. Length of reports.
Reportsuseavarietyofformatstopresentinformation(Figure14).Discursive,detailedinformationismainlyusedforcomprehensivereports,andprovisionofkeydataisparticularlyimportantforprogrammecomprehensivereports.Summaryreportsusenearlynotablesofnumericaldataandschematiccomparativedata.
Figure 14. Formats of reports.
Q 20. What is the average length of the reports (please provide indicative number of pages)? (n = 48)
Q 21. In which format is the report information presented?
(Please choose as many options as applicable, n = 46)
38
Content of reports
Reportsprovideawiderangeofinformationandaddressdifferentrequirementsandinformationneeds.Principally,theagencieswanttoprovideinformationonthefinaloutcomesofanassessment,proposesuggestionsforqualityenhancementandpublishgeneralfindingsintheirreports(Figure15).Also,thedescriptionofformalcriteria,qualitystandardsandQAsystemsisofgreatimportance.Thesameappliestotheinformationaboutsitevisitsandthelearningenvironment.SomeagenciespointedoutthatthecontentofthereportsisconstantlyadjustedbasedonfeedbackandrecommendationsfromthereviewteamsandfromtheHEIs.Accordingtothefeedbackreceivedthroughtheagencyworkshop,insomecountries,theagenciesarenotallowedtopublishinformationordatathatcouldbeusedtocreaterankingsatthenationallevel.
Figure 15. Information provided in reports.
Q 23. What information does your agency systematically provide in the reports? (n = 47)
39
usability and utility of reports
Anoverwhelmingmajorityoftheagencies(46outof47)statedthatthecomprehensivereportsneedtobefirstandforemostusefulfortheHEIs(Figure16).HighimportanceisalsogiventotheroleofQAreportsinprovidingclearandtransparentinformationonthequalityofinstitutionsandprogrammes.Easyaccesstothereportsisalsocrucial.Agenciesdonotgive,overall,highimportancetotheusefulnessofthereportsforfundingdecisionsorforthemedia.
Figure 16. Comprehensive reports.
Themainpurposeofthesummaryreportsistoprovideclearandtransparentinformationinaneasilyreadableandaccessiblemanner(Figure17).Thesummaryreportsmayalsobeusefulforthepublicauthoritiesorthemedia.
Q 25. How would you best describe your comprehensive reports? (n = 47)
40
Figure 17. summary reports.
perspeCtiVe of agenCies on their reports
main strengths and weaknesses of the published reports
WhenaskedaboutthemainstrengthsandweaknessesrelatedtotheirQAreports,theagenciesidentifytransparencyandrecommendationsforqualityimprovementasthemainstrengthsofthepublishedreports.Ontheotherhand,theformatandconsistencyofthereports,aswellasthecomplicatedlanguageoftenused,areconsideredasimportantareasofimprovement.
Strengths:
• Transparency,opentothepublic(7)• Recommendationsforqualityimprovement(6)• Easilyreadable(5)• Qualityandlevelofinformation(4)• UsefulinformationforHEIs(4)• Format(3)• Easyaccess(3)
Otherpositivecharacteristicsmentionedinclude:clearstructure;coverageofawideareaofsubjects;availabilityinboththenationallanguageandinEnglish;fullinformationonthemembersofexpertpanelsandtheirexpertise;andthevisualpresentationofinformation.
Q 26. How would you best describe your summary reports? (n = 23)
41
Weaknesses:
• Formatandconsistencyofthereportschanges,whichdependsoftenontheexperts(4)
• Useofcomplicatedlanguage(3)• AvailabilityofreportsonlyinEnglish(2)
Othershortcomingsidentifiedbytheagenciesinclude:complexityofinformationprovided;notenoughinformationonevaluationprocedures;delaysinfinishingthereportsduetoexpertsbeinglateinsubmittingtheirparts;delaysbetweendecisionandpublicationofthereport;reportsareonlyavailableinthenationallanguage;reportsarenotusefulforstudentsandemployers;summariesdonotincludethekeydataoftheinstitution;andHEIsdonotpublishthereportsontheirwebpage.
elements that would inCrease the quality and usefulness of reports in the ehea
Therespondentswerealsoaskedwhichelements,accordingtotheirexpertise,couldincreasethequalityandusefulnessofreportsproducedbyQAagenciesintheEuropeanHigherEducationArea(EHEA).Sixrespondentsagreethatcomparabilityofreportswouldbemostwelcome.Inaddition,fiveagenciesmentionedtheuseofclearlanguageandtheavailabilityofaEuropeantemplate,orEuropeanGuidelinesforreportsasimportantimprovementstothereportsattheEuropeanlevel.
Inaddition,thefollowingpointsweremade:
• Clearterminology(4)• Availableforthepublic(4)• Goodstructureandeasilysearchable(4)• Englishreports(3)• Reportsshouldbemoreclearlytargeted(3)• OutliningbestpracticeintheHEIs;Opinionofforeignexpert/s(3)• Availabilityofrecommendations(2)
possible Challenges and risks in trying to inCrease tr ansparenCy and Compar ability of reports
Throughouttheproject,theprojectteamhasbeenawareofthepotentialriskoflimitingtheusefulnessofreportsbytryingtoadaptthemtowiderordifferentreaderships.Theagencieswherethusaskedtoexpresstheirviewsonthepotentialrisksorchallengesrelatedtotheimprovedaccesstomorecomparablereports.
Somequotesfromthesurveycollectingthevoiceswere:
• ComparisonofreportswillturnQAintoanotherformofranking.Reportsshouldneverbecomparable,onlytheprocedureand/orthedecisionmakingshouldbecomparable,i.e.consistent.
• Reportscontainanexcessiveamountofdatawhichmakestheirusedifficultandwhichmayleadtospeculationandunfaircompetition.
• Universitiescouldperceivetheprocessasanexerciseof“revealingitall".Itcanshowparticularweaknessesthatcouldmakesomestudentsthinktochooseanotheroption.
42
• GoodcomprehensivereportsinboththenationallanguageandEnglishrequirealotofworkandexpertiseandwell-establishedagencies.Publishingreportsrevealsnotonlythequalityoftheinstitutionorprogrammebutalsooftheagency.
• TransparencyisnotalwaysintheHEI’sbestinterest,asthereportsalsoshowtheirweaknesses.Thus,greatertransparencywouldleadtomoreprotestsagainstpublicationorlessopennessintheprocedures.
• Legalframeworksmightlimitthewayreportsarepublished.• Tomakereportsmorecomparative,allagencieswouldneedtochangetheircurrent
practices.• QAreportsdealwithverycomplexissues.Oversimplifyingthecontentisnot
appropriate.ItisimportanttodistinguishbetweenHEIsthemselvesandotherpotentialusersofthereports.
• Agenciesaredealingwithalargenumberofprocesseseveryyearandusealargenumberofexperts.Itwillbedifficulttoensurefullconsistencyofthereports.
• The(national)systemsareverydiverseandarealsoatdifferentdevelopmentstages,asaretheneedsandaimsofdifferentQAsystems.Thismakescomparisondifficultandrisky.
ConClusions
Basedonthefindingsofthesurveyandthein-depthanalysisofasampleofQAreports,conclusionscouldbedrawnasfollows:
key findings• Publishedcomprehensivereportsarethemostcommonformtopresenttheresults
ofinstitutionalreviews.• Thevariationinthenumberofpublicationsofinstitutionalreviewsperyearamong
agenciesishigh.• Reportsmeetdifferentgoalsandareusedbydifferentstakeholders:
– Comprehensivereportsfacilitaterevieworaccreditationdecisionsandsupplyfeedbacktotheinstitutionsforqualityenhancement.
– Summaryreportssupplyinformationtoageneralpublicandassuretransparency.
• QAreportsaremainlywrittenbyapanelofexpertsandrequiresignificanttimeinput.
• Webpages,primarilythoseoftheQAagencies,arethemostcommonmediumofpublication.
• Themajorityofagencieshavetemplatesavailableforthewritingofreports.• Comprehensivereportsareoftenlongerthan30pages.Programmereportsare,on
average,slightlyshorterthaninstitutionalreports.• Reportsprovideawiderangeofinformationandaddressdifferentrequirements.• QAreportsvarygreatlybetweencountriesandagenciesbutalsobetween
programmeandinstitutionalreports.Inaddition,comprehensiveandsummaryreports-andeveneditingpractice,structure,content,length,readability,accessibility,etc.-withinasingleagencydifferconsiderably.
• QAreportsare–atthemoment–notcomparable.
43
views on quality reports
EssentialqualitiesofagoodandusefulQAreport:
• Clearstructure• Introductiontotheframeworkandstandards/guidelines• Informationonprocedure/reviewandreviewpanel/author(s)• DetailedinformationaboutHEI/programme• Adequatecomplexity-reduction(withoutoversimplification)• Carefuluseofterminology/language• Pleasantlayout• Easilyaccessible
risks and Challenges
Possiblerisksandchallengesintryingtoincreasetransparencyandcomparabilityofreports:
• Transparencyitself:misuseormanipulationofinformation• Comparability/benchmarking• Losesightoftheactualpurposeofthereports(usefulnessfortheinstitution)• Overratingthepotentialvalueandinterestforwiderpublic/layreader• Oversimplification• Standardisation:comparingappleswithpears• Languageissues
44
Chapter 5:
eUropean GUiDelines For sUMMarY reporTs anD Generic recoMMenDaTions For coMpreHensiVe reporTs oF eXTernal QUaliTY assUrance proceDUresĐurđica Dragojević, Agency for Science and Higher Education (ASHE)
european guidelines for summary Quality assuranCe reports
Theresultsofthesurveyontheuseandusefulnessofqualityassurance(QA)reportsconductedbythestakeholdersshowthatthereisaclearneedformorecomparableandaccessibleinformationtobeprovidedbytheQAagencies.WhileoneoftheinitialaimsoftheprojectwastodevelopaEuropeantemplateforcomprehensivereports,thiswasdeemedbytheprojectconsortiumasunfeasibleduetothedifferencesinQAproceduresandsystemsofhighereducationandthespecificneedsofstakeholdersandauthoritiesrelatingtotheparticularpoliticalframeworkofexternalQAinhighereducation.Approximatelyone-thirdoftherespondentstakeholdersexpressedtheneedforsummaryreportstobepublishedontheagencies’websitesinordertoserveasanintroductiontothedatacollectedbytheQAagenciesinthecomprehensivereportsandtoprovidecomparableinformationtothewiderpublic.Thisisinlinewithstandard2.5ofthecurrent(and2.6ofthedraftrevised)ESGonthepublicationofQAreports.
Wehopethattheseguidelines,togetherwiththeESGstandardonthepublicationofQAreports,willencouragealargenumberofQAagenciestopublishsummaryreports.Whilefullstandardisationisavoidedbysuggestingtheuseofguidelinesratherthantemplates,theseguidelinesaimtoensurethatallsummaryreportsprovidesimilarandcomparableinformation,thusincreasingtheirusefulnessfortheintendedusergroupsandthetransparencyoftheagencies’work.
1. Typesofsummaryreports
Dependingonthetypesofevaluationsperformed,anagencymaywishtohaveacommontemplateforsummaryreportsforallofitsprocedures,or,inlinewiththecomprehensivereports,aseparatetemplateforinstitutionalandprogrammeevaluationsummaryreports.
Thetargetaudiencesofsummaryreportsarevarious–fromstudentsandprospectivestudentsandtheirparentstoemployers,otherhighereducationinstitutions(HEIs),governmentbodiesandmediarepresentatives.Asummaryreportshouldprovideclearandsuccinctinformationontheevaluation,whichshouldbeunderstandabletomembers
45
ofthegeneralpublicandsufficientasafirstsourceofinformation,withouttargetinganygroupspecifically.Targetingspecificgroupswouldrequiredevelopmentofanumberofdifferenttemplates,whichmightreducetheircomparabilityandsignificantlyincreasethetimenecessaryforproducingthem,withoutsufficientadditionalbenefits.However,agenciesarewell-advisedtosurveypotentialtargetgroupsontheuseofsummaryreportsandaddadditionalinformationifrequestedbyspecifictargetgroups.
2. Placement
Thesummaryreportsshouldbeaccessibletoaswideanaudienceaspossible.Forthisreason,asinthecaseofcomprehensivereports,itisrecommendedtheybeeasilyfoundbysearchenginesthroughtheuseofkeywords,suchasthenameoftheagencyandtheinstitution(s)and/orprogramme(s)andtermsconnectedwiththetypeofevaluation(e.g.accreditation,audit,etc.).
Thesummaryreportsshouldbeeasilyaccessibleontheagencies’websites,andtheagenciesmaywishtodisseminatethemfurther,forexample,byaskingtheinstitutionstoplacethemontheirownwebsitesorbyplacingthemonotherwebsitesthatprovideinformationonhighereducation(suchasthewebsitesofnationalENIC-NARICs,Studyin...websitesprovidinginformationtoforeignstudents,etc.).
3. Drafting
Toensurehomogeneityandconsistentuseofterminology,itisbestthatreportsarewrittenbyagencystaff.Asummaryreportshouldonlysummariseandpresentinaclearandsuccinctformsomeoftheinformationprovidedinthecomprehensivereport,withoutaddinganyadditionalinformation.Thesummaryreportshouldbesenttothepanelfortheircheckandapprovalsothatwhatishighlightedinthesummaryreportreflectstheoverallimpressionofthepanel.
4. Developingtemplates
Thecorporateimageoftheagencyshouldbeincludedinthesummaryreport,soitiseasilyrecognisableanddistinctive.Thetemplateshouldenablethesummaryreporttobepublishedasawebpage(i.e.thepartoftheinstitutionalwebsitewheretheconnectedcomprehensivereportcanbefound)withaprint-readyversion(e.g.pdf)whichcanbeeasilyprintedorevendistributedinaprintform(e.g.asaleaflet).Whenprinted,asummaryreportshouldnotbelongerthantwopages,inordertoprovideinformationinaconciseandstructuredway.
Giventhatthelayoutandthedesignarecrucialforthereadabilityanduser-friendlinessofthesummaryreports,theagencyshouldbecarefultoincludeheadings/textboxesandotherlayoutsegmentswhichprovidemaximumclarityandeaseofcommunication.Whendevelopingtemplates,theagencymaybewell-advisedtoworkwithcommunicationanddesignexperts.
5. Language
GiventhatoneofthefunctionsofthesummaryreportistoenablecomparabilityandprovideinformationacrosstheEuropeanHigherEducationArea(EHEA),forexample,toinstitutionslookingforpartnersinothercountriesorforeignstudents,summaryreportsshouldbeproducedinEnglish.However,dependingonthelanguage(s)usedby
46
theagency’sstakeholders,theagencymayalsowishtopublishsummaryreportsinthenationallanguage(s)oftheagencyortheinstitution.
Becausethesummaryreportsareaimedatthegeneralpublic,meaningarangeofstakeholdergroups,thelanguageshouldavoidspecialistjargonandprovideexplanationswhenitcannotbeavoided.ThisisespeciallytrueforQAandlegalterminologywhichcanoftengounnoticedbyexperts,whiledecreasingthetransparencyandtheunderstandabilityofthereportbythewideraudience.Inputfromlinguisticexpertsandstakeholderscanbehelpfulinthisaspect.
6. Content–minimumusefulinformationtobeprovidedinthesummaryreports
Summaryreportsshouldcontainbasicinformationontheagencyandtheinstitution/programmeunderevaluation.Thefactthatthesummarieswillbemadeavailableonlinemakesiteasytoinsertlinkstorelatedwebsitesandthecomprehensivereportonwhichitisbased.
Giventhatthesummaryreportisadocumentproducedbytheagency,butalsogiventhetypeofinformationthestakeholdersseekinthistypeofreport,thenamesofthepanelmemberswhohaveperformedtheevaluationneednottobementioned.Instead,thecompositionofthepanelshouldbenotedinordertogiveanideaofthegroupofstakeholdersrepresentedinthepanel(e.g.,academic,student,employer,etc.).
Asindicatedbystakeholdersinthesurvey,layreaderswouldalsoliketobeabletofindinformationwhichisoftenobviousforexpertreaders–individualsfromagenciesandinstitutions–andisthussometimesomittedfromcomprehensivereports.Suchinformationincludesthetypeofevaluation,whetheritisobligatoryorvoluntary,anditsfocus(ifany–e.g.,management,internationalisation,etc.).Hyperlinksshouldagainbeinsertedinthesummaryreporttomakethecomprehensivereporteasytofindforanyoneinterestedinit,possiblyaddingabriefdescriptionofthetypeofinformationthatcanbefoundthere.Othertypesofcontextualinformation–thedescriptionoftheQAsystemwithinthecountry,thecriteriausedintheevaluations,etc.shouldbeaccessibleontheagency’swebsiteandthusnotincludedinthesummaryreport.
TakingintoaccountthefactthatthegeneralpublicoftenlooksatQAreportssimplytoestablishiftheinstitutionand/orprogrammeisaccredited,thisinformationshouldbementionedinthesummaryreports.Inadditiontothis,becauseexternalQAdecisionsoftenhavean‘expirydate’,thedatewhenthereportwaspublishedorthefinaldecisionwasissuedshouldbenotedtogetherwiththevalidityperiod.Thiswillhelpthereadertoeasilyestablishwhentheevaluationwasconductedandwhetheritmaybenecessarytolookfurtherforreportsonmorerecentevaluations.
Althoughreportsshouldprimarilyprovideasummaryofthepanel’sfindingsratherthaninformationaboutevaluatedinstitutionsandprogrammes,somebasicinformationshouldbeprovidedtothereader.Thistypeofinformationwillvaryamongthedifferenttypesofreports,butshouldincludeinformationonthestatusoftheinstitution(e.g.,ifitisprivateorpublic,ifitisauniversityorsomeothertypeofHEI).Forprogrammereports,theBolognaframework(QF-EHEA),theNationalQualificationsFramework(NQF)and,ifapplicable,theEuropeanQualificationsFramework(EQF)leveloftheprogrammeshouldbeindicated,withotherinformationrelevantfortheevaluation(e.g.,ifitisanacademicoraprofessionalprogramme,ajointprogramme,etc.).
47
Content of summary reports: CheCklist
Information on the evaluation
• Thenameoftheagencywhichperformedtheevaluation.• Linktothewebsiteoftheagency.• Typeandfocusofevaluation(e.g.institutionalaudit,auditwithafocuson
internationalisation)andifitwasobligatoryorvoluntary(with,possibly,abriefmentionofthewayitfitsinwiththenationallegalframework/otheractivitiesoftheagency–e.g.,obligatoryaccreditationofstudyprogrammesaccordingtothenationalcriteria).
• Thedatewhenthereport/decisionwasissued,thevalidityperiodandthedateuntilwhichtherecommendationsshouldbeimplemented.
• Linktothecomprehensivereport(possiblywithanoteonitscontent,e.g.“recommendationsforimprovementanddetailedanalysiscanbefoundinthecomprehensivereport”).
• Abriefexplanationonthecompositionofthepanel.
Information on the institution/programme
• Nameoftheinstitution/programme.• Linktotheinstitutionalwebsiteand/orstudyprogrammewebsite.• Typeofinstitution(e.g.privateorpublic,professionaloruniversityforbinary
systems)orprogramme(BolognaFramework,NQFand,ifpossible,EQFlevel,whetheritisprofessionaloracademic,ifitisajointprogramme,etc.).
general reCommendations for ComprehensiVe Quality assuranCe reports
ThefindingsofthesurveyontheuseandusefulnessofQAreportsaddressedtothestakeholdersshowedthatQAreportscanbeattimestechnicalwithacomplicatedlanguageandtoodifferentinstyleandcontent.Thestakeholdersmentionedthatamorestandardisedapproachwouldbehelpful,butthepurpose(supportingtherevieworanaccreditationprocess)andtheverydiverseandcomplexnatureofcomprehensiveQAreportsdoesnotallowthedevelopmentofadetailedsetofcontents,ascouldbedoneinthecaseforthesummaryreports.Comprehensivereportsprovideimportantqualitativeinformationoninstitutionsorprogrammes.Givinganindicationofthecontentforcomprehensivereportswouldlimittheirusabilityandwouldthusresultinthelossofsomevaluableinformationortheusefulnessoftheinformationtothemainusers.Nevertheless,thereisspaceforsomegenericrecommendationsonthestructure,layout,andpublishingofthesereportstothosewhoproducethem.
Itisimportantthatcomprehensivereportsprovideclearandunderstandableinformationinordertomakethemmoreusableandaccessiblefordifferentgroupsofstakeholders,includingthosewithintheinstitutionssubjecttothereview.Inthisperspective,whenproducingcomprehensivereports,oneshouldconsiderthegenericrecommendationsbelow:
48
1. structure
• Thecomprehensivereportshouldbewell-structuredandeasytoread.• Alistofcontentsshouldbeprovidedatthebeginningofthereportinorderto
facilitatethefindingofspecificinformation.• Clearheadings,sub-headingsandnumberedparagraphsshouldbeusedinorderto
clearlyindicatethedifferentsectionsofthereport.
2. Content
• Thereportshouldclearlydescribetheprocess,recommendations,commendationsandconditionsclearlyidentifiableandthecontextofthereview.
• Thereportshouldusekeyterminologyconsistently.• Itisimportantthatthereportcontainsaglossaryexplainingtechnicaltermsfor
thosewhoarenotfamiliarwithQA.• Thereportshouldidentifyandpresenttheperson(s)whowrotethereportaswell
asthecompositionofthereviewteam.
3. Publication and dissemination
• Thereportshouldbemadeeasilyaccessible:itshouldbepublishedonlineandinaneasilyprintableformat.
• Thedecisions,institutionalresponsesandfollowupdocumentsshouldbepublishedtogetherwiththereport.
• ThereviewedHEI/programmeshouldbeencouragedtopublishthereportontheinstitutionalwebsite,oratleast,toprovidealinktotheagencywebsitewherethereportispublished.
Inadditiontothesefindings,thereflectionsonQAreportsbyPeterFindlayinthenextchapterprovidesomeinterestingpointstoconsiderinrelationtoQAreports.
49
Chapter 6:
FeaTUres oF a GooD eXTernal QUaliTY assUrance reporT – soMe THoUGHTsPeter Findlay, Quality Assurance Agency for Higher Education (QAA)
This collection of reflections and questions about the quality assurance (QA) reports produced by agencies is based on a presentation, part serious, part light-hearted and occasionally provocative, made at the EQArep workshop held in January 2014. While these reflections draw on some twenty years of working with QAA and its predecessor, the ideas here do not necessarily represent either the policy or practice of QAA as an agency.
qa questions for reports
Someoftheperennialandwell-triedquestionsforQAalsoapply,ofcourse,tothequalityofouragencyreports:
• Forwhatisitintended?(aims,readership)• Whyarewedoingitthatway?(methodandformatadopted)• Isthatthebestway?(evaluationofthemethod)• Howdoweknowitworks?(achievingtheaims,impactofreports)• Howcoulditbeimproved?(reviewing,evaluating,planning)
a report is a message
Areportis,intermsofstandardcommunicationtheory,justanotherkindofmessage.Itseffectivecommunicationwilldependonthreethings:thesender(theexpertteam,theagency),themedium(theformofthereport,themeansofitspublication)andthereceiver(theuniversityandotherstakeholderssuchasstudentsandemployers).
Messagesarealsogovernedbytherulethatthepurpose,thecontentandthereadershiporaudiencearethreeinter-relatedfactorswhichwilldetermineeachother.Anyreporthastotakeintoaccountitsmostimportantaims,itslikelyreadership,andthebestforminwhichtoaddressthese.
Communicationtheoryusestheterms‘encoding’and‘decoding’withregardtomessages.Forourconsiderations,‘encoding’means:
• Knowingthepurposeofthereport• Knowingthecontentofthereport• Knowinghowitwillbewrittenandthestagesofproduction• Knowingwhowill/shouldreadit• Knowinghowandwhentheywillreadit
50
and‘decoding’implies:
• thatthereportisreceivedandreadinamanagedprocess• thatthereaderunderstandstheprocessinvolvedandtheconventionsofthereport• thatthereaderknowswhatisexpectedfromthereportintermsofactionstobe
takenorinformationtobeused.
who is it for?
Anagencythereforeneedstobeveryclearindecidingtheaimsofitsreportsandtheirintendedreadershipandusers.Unfortunately,thatisoftennotasimplematter.Itwouldbereasonablystraightforwardifreportsweresimplywrittenfortheexclusiveattentionofhighereducationinstitutions(HEIs),butthereareotherinterestedparties–governments,students,employers,etc.whomayalsobeamongthetargetreadership.Areviewreportmayneedtoaddressalloftheseinterests,butcanasinglereportdothatsuccessfullywithsuchdifferentexpectationsandneedsinvolved?Amajorquestionforourreportsishowtomeettheneedsofthesedifferentstakeholders.
Answerstothisknottyproblemthathavebeentried:
• Separatereportsforseparatestakeholders• Asummarysectionofthereportwithclearoutcomes,separatedfromthemore
detailedmainbodyofthereport• Differentpartsofthereportfordifferentreaders• Ashortreportonthewebsiteandalongerreportfortheinstitution• A‘checklist’formatforthekeycriteria,withacommentsboxforthedetails,where
required.
the report is the result
Weneedtokeepinmindthatanagency’sreportistheendresult,oroutcomeofawholeprocess;thecharacterandqualityofthereportwillthereforedependonthequalityofmanyothercontributingaspects:
• Agencymanagement,policyandmethod• Agencyofficerandreviewteam• Documentationandinformationprovided,accesstoinstitutionalinformation• Visitandconfirmation• Draftingthereport• Editingthereport• Finalisingandagreeingonthereport
Itcaneasilybeforgottenthatcompletingthereportistheprimaryaimofallthesedifferentpartsofthereviewprocess.Itisboththeendproductandthefocusofthewholeprocess.Meetingotherreviewersormanagingtheteamisworthwhileandrewarding;thesitevisitislikelytobeexcitingandinteresting;thepreparationofthereport,ontheotherhand,isisolatedanddesk-based,andwilloftenbeademandingandeventedioustask.Itcanalltooeasilybeforgottenaboutduringtheearlierstagesoftheprocess.
51
Ateverystage,therefore,weneedtothinkaheadtothereport:keepingnotes,makingearlydrafts,checkingevaluationagainstthefacts,workingtowardsthejudgements.Agoodreportresultsfromaplannedprocessleadinguptoitsfinalproduction.
who owns what?
Theremust-fromtheverystartoftheprocess-beclarityaboutwho‘owns’thereport–thatis,whomakesthefinaldecisionaboutwhatisincludedinitornot,andhowjudgementsaremade.Sometimesownershipisshared,andthenthevariousresponsibilitiesmustbecarefullydefined.Doesownershipliewith:
• Theexpertpanel?• Thememberofthepanelwhowritesthereport?• Theagencyofficerwhoeditsthereport?• Theagencycommitteethatapprovesthereport?• Theagencyasalegalentity?• Theinstitution?• Mostimportantly,whatpersonorbodyhastherighttomodifythereportand/orits
judgements?
Candifferentparticipantsintheprocessowndifferentpartsofthereport?Forinstance,itmightbearguedthatthejudgementsinareportareownedbytheexpertsaspeerreviewers,buttheformandcontentofthereportareownedbytheagencythathasultimateresponsibilityforthequalityofpublication.
keep it simple , ******!
Itcanbearguedthatcomplicatedandopaquereportswhicharedifficulttoreadmayserveapurpose,whichwouldbetolimittheunderstandingandreceptionofthereporttoalimitedreadershipwhounderstandtheconventionsandcodesinwhichitiswritten.However,iftheaimistoreachaswideandvariousareadershipaspossible,thenthesimpler,thebetter.
Bysimplicityisnotmeantreducingthesignificanceorcomplexityofthecontent,butensuringthatitisconveyedviaaveryclearstructureandinlanguagewhichcanbeeasilyfollowedandunderstood.
Somedesirableandundesirablefeaturesofsuchareportmightbe:
To aim for:
• Length–asshortaspossible• Agoodsign-postedstructurewithclearheadings• Style–simple,precise,clear,focused• Well-structuredparagraphs‘beginning,middle,end’• Sentences,whicharesimpleandclear(neverlongerthantwolinesoftext!)• Tone–neutral,measured,objective• Content–minimumdescription,maximumanalysisandevaluation(itisdifficultto
seetheusefulnessoflargeamountsofdescriptionofinstitutionalprocedures–theinstitutionitselfknowsaboutthemalready,andotherreportreadersprobablydon’treallywanttoknowverymuchaboutthem,exceptwhethertheyareeffective)
52
• Argumentsclearlybasedonevidence• Regularremindersoftherelevancetothereviewquestions/agenda/framework• Regularsummariesofeachsection• Clearexplanationofbasisforjudgementsandrecommendations
To avoid:
• Toomuchreferencetotheprocess(theexpertteam,thereviewvisit,etc.)• Overlycomplexsentences• Jargonandcliché(especiallythejargonofQAexperts)• Emotiveorstronglyjudgmentalwords• Obscurewords• Ambiguity• Toomuchunnecessarydetail• Speculationaboutfuturedevelopmentsintheinstitution• Subjectivecommentsdrawnonexperts’personalexperience
the politiCs of l anguage
Nationallanguagesofsmallnationswon’thavethesamebreadthofimpactorallowinternationalcomparability(butwillstayintheculture),butthenationallanguagereflectsmoreexactlytheownershipandlocationofthereport.
Translationintothelinguafranca(English)bringsrisksofmistranslation,confusion,andmisunderstanding;therearemanyvarietiesofEnglish(a‘Euroglish’isindevelopmentinQAreports!).Wedon’trealisehowvariedourunderstandingofkeytermsinEnglishcanbe(‘accreditation,’‘expert,’‘validation,’‘assessment’and‘standards’aresomekeytermsthathavedifferentmeaningsindifferentcontexts),andyoucan’talwaysbesurewhatEnglishmeanstothereader.
sChedules
Atimelineforthedevelopmentandsubmissionofthereportisessential.Itshouldincludeaclearindicationofwhateachcontributorisexpectedtoprovideandtheiterationsneededforediting.
Deadlinesaretheplagueandsometimestheagonyofreport-writers.Alwaysmakeyourdeadlinesveryrealistic,andthenaddsomemoredayson,tomakeforgreaterflexibilityinmeetingtheproblemsthatwillcertainlyoccur.
tr ansparenCy… i Can see Clearly now
Transparencyisgenerallyregardedasavirtueandagoalofagoodreport,buttransparencycanharmaswellashelp–whatexactlyistheaimofcompleteclarity?Isthemainaimofthereporttoregulate,controlandcompare,orisittoimprove,enhanceanddevelop?Ifweseektoimprove,andtohelpaninstitutiontosucceed,thenisitalwaysagoodideatopublishthefullfacts,whichmightdamageitsreputation(topublishindetailforthepublictoreadallthatiswrongwithaninstitution?)?Isthealternativeacceptable-foranagency’spublicationpolicytobeadjusted,takingintoaccountthemessagedeliveredbythereport?Isthere,then,atensionbetweentransparencyandenhancement?
53
tight- rope walking – the tension between peer review and regul atory body
Mostagenciesuseareportingmethodwhichinvolvesanexpertteam.Eithertheywritetheearlydraftsofthereportthemselves,ortheirviewsandopinionsaregatheredandincorporatedintoareportbytheagencyofficer.Thisapproachfollowstheprincipleofpeerreview.Initspurestform,apeerreviewapproachusingtheinformedopinionoftheexpertswouldmeanthatthoseexpertsownthereviewandwritethereport.Insuchamethod,individualexpertsmightevenbeallowedtosubmitaminority,dissentingstatement.However,agencieswillusuallywantahigherlevelofcontroloverthereport.Theywillrequireexpertstoreachaconsensusonthereportjudgementsandmayimposeacommonframeworkforallreports.
Considerthepotentialspectrumofpossibilitiesfortheformatofthereport.Atoneextreme,purepeerreviewisclosetothecharacterofapersonalnarrativeaboutimpressionsgainedbyexperts;attheotherendofthespectrumissomethinglikeahighlycontrolledchecklist,pre-definedbytheagency,intowhichtheexpertsplacetheirviewsanddecisions.Aquestionduringallreportprocessesiswheretopositiontheformofthereportbetweenthesetwoextremes.Thesignsofagencycontrolwillbefoundintheamountofshapingstructuregiventothereport:specifiedstandardsandcriteria;requiredheadings;checklistsforcompletion;wordcounts;tables;pre-definedformsofjudgementstatements.
Thereisatensionhere,becausethehigherthelevelofcontrolthatisexertedbytheagency,thelesstheexpertsmayfeeltheirviewsandtheexpressionofthemaretakenseriously.Itisalsoquestionablewhetherasimple‘tick-box’conformityapproachcanaccuratelyreflectthecomplexityofinstitutionalsystemsandtheirdifferentapproaches.Soitisamatterofbalancingfreedomofexpression(ahighvalueinouracademiccontext)asagainstthelevelofbureaucraticcontrol,andthelevelsofconsistencyandcomparabilitybetweenreportsthatarerequiredbytheagency.
Itisalsoworthnotingthatthehigherthelevelofagencycontrol,themorewillbeneededathoroughtrainingandbriefingofexperts,sothattheyunderstandfullywhatisexpectedintheirworkonthereport.
show me the evidenCe ….
Wefollowtheprincipleof‘evidence-basedassessment’,butwhatexactlydowemeanby‘evidence’,andhowdoweuseit?
• Doweonlyuseevidenceprovidedbytheinstitution?• Doestheteamhavetherighttorequestotherevidence?• Canwerequestfromtheinstitutionsevidencethatdemonstratesorconfirmsthe
occurrenceofbadpractice?• Howdoweensurethatevidenceisrecordedandretained?(Photocopyingduring
thevisit?Accesstointernalwebpages?)• Whotakescareoftheevidenceandensuressafekeeping?Canindividualexpertsbe
reliedontodothis?• Istheevidencepublishedwiththereport(whichprobablymakesittoolong)orkept
inreserve?
54
• Maybetheevidence-basecanbeembeddedinearly,unpublishedversionsofthereportascomments,footnotesorparentheses.
Itisworthnotingtheinstitutionsusuallyonlyrequiresightoftheevidencewhenacriticismismade,notwhentheassessmentispositive:inthatcasedoweonlyneedtoretainevidencerelatingtocriticalpointsorproblematicmatters?
who reads it, who needs it?
ItseemsatleastarguablethatveryfewpeopleactuallyreadthroughafullinstitutionalQAreportfromendtoend.Thatis,ofcourse,withtheexceptionoftheever-resilientagencyofficerresponsibleforthereport,whoprobablyreadsitthroughfromendtoendmanytimes!
Whatevidencedoesanagencyhaveoftheeventualreadingandtheactualimpactofitsreports?Weshouldreflectcarefullyonthelikelyreadershipforreports.
Ifwereallyaredealingwithasmallnumberofreaders(morethanten,lessthan50?Orevenlessthanthat?),thensurelyweneedalsotothinkabouttheeconomyofourreportproduction:theexpenditureofeffortinproducingitrelativetotheactualpatternofreceptionanduse.Rememberalsothatresearchshowsthatgenerally,inreadingreportslikeours,theattentionspanofthereaderreduceswitheverypagethatisturned.
Maybemanymorepeoplewillreadthesummaryofthereportorashortversionpublishedonthewebsite.Isthereacaseforarelativelyshortpublishedreportandthenseparatelyanunpublished,moredetailedreportfortheinstitution?Usuallyevidenceofimpactcomesintheformoffollow-upreportsandactionplans.Maybe,then,allweneedforourreadershiparesummaries,judgements,recommendationsandactionplans?
Couldweimprovetheaccessibilityofreportsbyusingmorevisualimagesinthelayout(e.g.graphics,tables,highlighting,andevenpictures)?
Finally,weneedtorememberthatourreaderstoomayneedadviceandeventraininginreadingandusingreportsinthemostproductiveway.Guidancedocumentsabouttheprocedureandthemainfeaturesandaimsofreportsareveryworthwhile.
l ast word (it’ s always good to arrive at it!)
Theperfectreportismorelikelytobeanaspirationthanarealisation:thebestyoucando,inthetimeyouhaveavailable,hastobegoodenough!
55
Chapter 7:
conclUsionsZeynep Olcen, European Association for Quality Assurance in Higher Education (ENQA)
Withthegrowingpublicinterestinhighereducationinstitutions(HEIs)inEurope,accessingqualityandtransparentinformationontheinstitutionsortheprogrammesofferedbytheseinstitutionshasbecomeachallenge.AsoneofthemainBolognatransparencytools,qualityassurance(QA)reportsplayanimportantroleinfosteringthetransparencyofEuropeanhighereducationandprovidingreliableinformationtothestakeholders,butthereiscurrentlyagapbetweenthecurrentpracticesofQAagenciesinpublishingtheirexternalQAreportsandtheexpectationsofstakeholdergroupsinrelationtothesereports.
QAreportsarethethirdsourceofinformationforstakeholdersfollowingtheHEIs’websitesandfriendsorcolleagues.ThereisaclearneedtoraiseawarenessregardingtheexistenceofQAreportsamongthestakeholdersandtoensurethattheyareeasilyaccessible.WhiledifferenttargetgroupssuchasHEIs,studentsandpublicauthoritieshavesimilarexpectationsregardingQAreports,andutilisetheinformationwithinthem,employersdonotnecessarilyconsiderthesereportsasasourceofinformationontheprogrammeortheinstitution.TheoverallimpressionfromthestakeholdersisthatQAreportsareahelpfulsourceofinformation,butthetechnicality(jargonused)andthelengthofthesereportslimittheirpotential.Oneofthemainrequirementsofthestakeholdersisthatthereportsshouldcontaineasilyunderstandableandcomparabledata.However,summaryQAreportsdonotexistinordertoprovidequantitativedataonHEIs(whichcanbeeasilyfoundontheinstitutions’websites)butratherinformationontheevaluationprocessandtheoutcomes.Inaddition,stakeholdersunderlinedtheimportanceofeasyaccesstoQAreportsandrequirethattheybepublishednotonlyontheQAagencies’websitesbutalsoontheHEIs’websites.Withinthisperspective,HEIsshouldbehighlyencouragedtopublishtheirevaluationreportsontheirwebsites.
WhilecomprehensiveQAreportsareverydiverseandaddressprimarilyHEIsthemselves,summaryQAreportsmaybetterservetheneedsofallthedifferenttargetgroups.Thus,thisprojectaimedatdevelopingcommonEuropeanGuidelinesforthesummaryreportsandsomegenericrecommendationsforthestructureandthepublicationofcomprehensivereportsontheEuropeanlevel.Thedetailsandcontentofthecomprehensivereportsaretobedecideduponatthenationallevel.
TheEQArepprojectalsolookedintothecurrentpracticesofQAagenciesinpublishingtheirQAreportsandtherelatedstrengthsandweaknesses.Thesurveyedagenciesidentifiedtransparency(opennesstopublic)andtherecommendationsforqualityimprovementasthemainstrengthsofQAreports.Atthesametime,interestingly,someagenciesalsomentionedeasyreadability,clearformatandeasyaccessasstrengths.ThisisincontradictionwithwhatthestakeholdershaveidentifiedaslimitationswhenusingQAreports.Indeed,thereisadiscrepancybetweentheagencies’andthestakeholders’visionsregardingthereadabilityandtheaccessibilityofQAreports.ThismightbeduetothefactthatQAagenciesaremorefamiliarwiththestructureandterminologyofQAreportsandhavebetterknowledgeonwheretofindthem.Someoftheidentified
56
weaknessesincludetheinconsistencyconcerningtheformatofQAreports(whichverymuchdependsontheexpertsdraftingthem)andtheuseofcomplicatedterminology.TheavailabilityofreportsonlyinEnglishisalsoseenasaweaknessbysomesurveyedQAagencies.However,fromthestakeholders’pointofview,publicationofQAreportsinEnglishisessentialintermsofunderstandabilityandcomparabilityattheEuropeanlevel.Fromthisperspective,thecommonGuidelinesforsummaryreportssuggestthatthereportsshouldbepublishedinEnglishandanyotherlanguageswhichmightberelevant.
WhenquestionedabouttheelementsthatcouldcontributetotheimprovementofthequalityofQAreports,themajorityoftheagenciesemphasisedtheneedforclearterminology,accessibility(availabilitytopublic),goodstructure,cleartargetgroups,availabilityofreportsinEnglishaswellasotherlanguages,inclusionofinstitutions’bestpractices,andrecommendations/commendationsgivenbytheexperts.ThegeneralfindingoftheQAagencies’surveyindicatesthatQAreports(comprehensiveandsummary)varysignificantlyamongcountriesandagenciesbutalsobetweenprogrammeandinstitutionaltypes.Furthermore,theediting,structureandcontent,length,readabilityandaccessibilityofreportsalsodifferatthecountryandagencylevel.
Theconclusionsofthisproject,therefore,focusontheimprovementoftheQAreportsbybringingcloserthecurrentpracticesofQAagenciesinpublishingthesereportsandtheoverallexpectationsofthestakeholders.ThecommonEuropeanGuidelinesforsummaryreportsandthegenericrecommendationsforcomprehensivereportsaretheimportanttoolstoachievethis.
european guidelines for summary reports
Summaryreportstargetawideraudiencethancomprehensivereports,includingstudents,parents,employers,otherHEIsorgovernmentbodies,andthus,theyareexpectedtoprovideclear,understandableandconciseinformationabouttheevaluatedinstitutionorprogramme.Whenproducingsummaryreports,QAagenciesshouldensurethefollowingexpectationsaremet:
• Clearterminology,avoidingprofessionaljargon.• Easyaccesstothewideraudiencebypublishingreportsnotonlyontheagency’s
websitebutalsotheHEI’swebsite.Thetargetgroupsshouldbeabletofindthereportseasilythroughsearchenginesbytypingkeywords.
• Draftingofthereportbyanagency’sstaff,asitisimportanttoensurehomogeneityandconsistency.
• Developingauser-friendlyandeasy-to-communicatetemplatewhichincludestheagency’scorporateimageforvisibilityaswellasclearheadingsandotherlayoutsegments.
• ProducingthereportsinEnglishinordertoreachawiderpublic,especiallyforeignstudentsinothercountries,andforcomparabilitypurposesattheEuropeanlevel,butalsoinanyotherlanguage(s)thattheagencyorinstitutionuses.
• Providingusefulinformationon:1)theevaluationprocess;2)theinstitutionortheprogrammeunderevaluation;and3)themainconclusionsoftheevaluation.
57
reCommendations for ComprehensiVe reports
SincecomprehensivereportstargetmainlytheHEIsandcontainawiderangeofimportantinformationvaryingfromoneagencytotheother,therecommendationsforthesetypesofreportsislimitedtotheirstructure,draftingandthepublicationratherthanthecontent.Thecomprehensivereportsshouldpayattentiontothefollowing:
• Goodstructurewithclearheadings,sub-headingsandalistofthecontentstofacilitatesearchesforspecificinformation.
• Cleardescriptionoftheprocess,statingtherecommendations,commendationsandthecontextofthereview.
• Consistentuseofterminologyandaglossaryexplainingthetechnicalterms.• Easyaccesstothereportonlinethroughpublicationofthereportontheagency
andthehighereducationwebsites.• Publicationoftheinstitutionalresponsesandthefollow-updocumentstogether
withthereport.
ThroughtheanalysisoftheneedsofstakeholdersregardingQAreports,andtheQAagencies’currentpracticesonthepublishingofQAprocedures,theEQArepprojectshedlightontheexistinggapbetweenwhatisexpectedandwhatisreality.BasedonthefindingsandtheconclusionsderivedfromthefindingsoftheEQArepproject,ENQAhighlyrecommendsthattheQAagenciestakeintoconsiderationtheEuropeanGuidelinespresentedinthisreportwhenproducingandpublishingsummaryreports–orwhenconsideringthedevelopmentofsuchreports.Furthermore,theQAagenciesarealsoencouragedtocommunicatethegenericrecommendationsforcomprehensivereportstothoseinchargeofdraftingandtomirrortheircurrentpracticetothoserecommendations.
ThedetailedworkcarriedoutintheEQArepprojectbroughtintofocusdifferentviews,perspectivesandexpectationswithregardtoQAreports.AsQAreportsandtheoverallcontextofhighereducationisconstantlyevolving,QAagenciesarerecommendedtorequestfeedbackfromandperiodicallysurveytheirstakeholdersinordertofurtherimprovetheirreports.
TheoutcomesofthisprojectwillhaveanimportantimpactontheimprovementoftheroleofQAreportsasasourceofreliableandcomparableinformationfordifferenttypesofstakeholdergroupsiftherecommendationsmadearetakenintoaccount.QAagencieshaveanimportantroletoplayinthisprocess.
58
acKnoWleDGeMenTsENQAandtheprojectconsortiummembersaregratefultoalltheEuropeanQAagenciesthattookthetimetorespondtotheagencysurveyandallstakeholders(individualsandorganisations)thatprovidedtheirinvaluableinputthroughtheirresponsestothestakeholders’survey.Inaddition,wesendourwarmestthankstoalltheparticipantsofthetwoworkshops,whocontributedactivelybytheircomments,questionsandideas.
TheEQArepprojectbenefittedfromthesupportandadviceofanAdvisoryBoardcomposedofrepresentativesfromtheEuropeanUniversityAssociation(EUA),EuropeanStudents’Union(ESU),EuropeanAssociationofInstitutionsinHigherEducation(EURASHE)andBusinessEurope.Wearegratefulforthecontributionsandadvicereceivedthroughouttheprojectfromvariousexperts.WewouldliketothankGuyHaugforhisinputontheprojectastheexternalevaluator.TheprojectconsortiumisfurtherindebtedtoPeterWilliamsandAchimHopbachfortheirideasandinput.
59
reFerencesCrozier,F.,Grifoll,J.,Harris,N.,Kekalainen,H.,&Malan,T.,(2011),Evaluationofthereportsonagencyreviews(2005-2009),ENQAOccasionalPapers16.Helsinki:EuropeanAssociationforQualityAssuranceinHigherEducation.Availableat:www.enqa.eu/indirme/papers-and-reports/occasional-papers/ENQA_Occasional%20paper_16.pdf.
E4Group,incooperationwithEI,BusinessEurope,andEQAR,(2014),RevisionoftheESG.Availableat:http://revisionesg.wordpress.com.
QualityandQualificationsIreland(QQI),(2014),ReviewofReviews:ReportoftheIndependentReviewTeam.Availableat:www.qqi.ie/Downloads/Reviews/Review%20of%20Reviews/12639-QQI%20Review%20of%20Reviews-WEB.pdf.
Rauhvargers,A.,Deane,C.,&Pauwels,W.,(2009),Bolognaprocessstocktakingreport2009,Report from working groups appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvainla-Neuve. Brussels:LifelongLearningProgramme,EuropeanCommission.
Thune,C.(2009),Standards and guidelines for quality assurance in the European Higher Education Area, 3rd edition. Helsinki:EuropeanAssociationforQualityAssuranceintheEuropeanHigherEducation.
Westerheijden,D.F.,etal.(2010),The Bologna Process Independent Assessment-The first decade on working on the European Higher Education Area-Volume 1 Detailed assessment report, CHEPS/INCHER-Kassel/ECOTEC.Availableat:www.ond.vlaanderen.be/hogeronderwijs/bologna/2010_conference/documents/IndependentAssessment_executive_summary_overview_conclusions.pdf.
Woodhouse,D.(November2010),IsthereagloballycommonunderstandingtoQualityAssurance?ESU Board Meeting 59 Seminar: Quality Assurance. PresentationconductedfromJerusalem,Israel.
60
anneX i
Questionnaire: the use and usefulness of external Quality assuranCe reports for different stakeholders
TheEuropeanAssociationforQualityAssuranceinHigherEducation(ENQA)iscarryingoutaEuropeanCommissionLLPco-fundedprojectTransparencyofEuropeanhighereducationthroughpublicqualityassurancereports(EQArep)togetherwithfourEuropeanQAagencies:OAQ(Switzerland),QQI(Ireland),ASHE(Croatia)andEKKA(Estonia).
TheaimoftheprojectistodevelopEuropeanstandardsforqualityassurancereportsinordertoensurethattheinformationcontainedinthequalityassurancereportsmeettheexpectationsofthestakeholders.Theoutcomesoftheprojectshallresultinarecommendationtoqualityassuranceagenciesonthecontentandformofinformativeandapproachablequalityassurancereports.
The surveyisdirectedatallmainstakeholdergroupsinhighereducationqualityassurance:students,potentialfutureemployers,governmentsandthehighereducationinstitutionsthemselves.Thepurposeofthesurveyistoidentifyandcomparetheexactinterests of the various stakeholdersasregardsinformationaboutthequalityofinstitutionsandofprogrammes.
Wewouldbemostgratefulifyoucoulddedicatesometimetorespondingtothissurvey.Theapproximatetimeneededis10 minutes.Theresponseswillbeanonymous.
Pleaserelyonyour personal experienceanduseofinformationsourcesonHEI.Pleasenotethatnotallpartsofaquestionwillberelevanttoalldifferentrespondergroups.
Yourcommentswillbeofhighvalueforus.Pleaseprovideasmanyofthemasyouwishinthe comment boxes provided,toexplainyourexperiencebetter.
Pleaseleaveyouremailaddressifyouwishtobeinformedoftheworkshopwhichwillbeorganiseasafollow-upofthesurvey,andwillprovideanopportunitytodiscusstheinformationneedsofdifferentusergroupsfurther.
The deadline for replying is 28th of February.
Warmthanksinadvanceforyourtimeandcontribution
The eQarep consortium
61
i - use of the information about Quality in higher eduCation institutions and study programmes
1. Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?(Multiple answer)
a. Todecideonpossiblefurtherstudies
b. Toevaluatethequalityofgraduatesforrecruitmentpurposes
c. TofindpartnersamongHEIs
d. Todecideoninvestments/funding/sponsorshiptoaHEIoritsunit
e. Other(pleasename)
2. WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIandstudyprogrammes?
a. WebsitesofHEIsoften – once in a while – never
b. Socialmedia(e.g.Facebook)often – once in a while – never
c. Assessmentreportsprovidedbyqualityassuranceagenciesoften – once in a while – never
d. Governmentreports/publicationsoften – once in a while – never
e. Variousrankings/leaguetablesoften – once in a while – never
f. Informationfrom/opinionsoffriends,colleagues,parentsetc.often – once in a while – never
g. Other(pleasename)
3. Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy(pleasechooseallthatapply):
a. Didnotknowaboutthem
b. Didnotknowwheretofindthem
c. ThereportsdonotcontaintheinformationIneed
d. Thereportsareinatoocomplicatedlanguage
e. TherearenoreportsinEnglish/languageIunderstand
f. Thereportsaretoolong
g. Ifoundtheneededinformationelsewhere
h. Other(pleasename)
62
If you do not use reports by quality assurance agencies as a source of information, please continue to Question 8.
4. Ifspecificinformation,pleaseindicatewhatkindofinformationyouwerelookingfor(pleasechooseallthatapply)
a. Generalfindings(e.g.generalrecommendations,overallassessment)
b. Strategicplanning,management,governance
c. Internalqualityassurancesystem
d. Historyandtraditions
e. Applicationandadmissionstatistics
f. Numberofresearchgrants,publications,citations
g. Employability/employmentofgraduates
h. Contentofstudyprogrammes
i. Reputationofteachingstaff
j. Qualificationsofteachingstaff
k. Financialresources
l. Conditionofinfrastructure
m. Studentsupportsystem
n. Institution’sabilitytoresponddiversestudents’needs
o. Accreditationstatusofinstitutions/studyprogrammes
p. Institution’spositioninleaguetables
q. Other(pleasename)
5. Theassessmentreportsprovidetheinformationyouarelookingfor
Fullyagree Slightlyagree Slightlydisagree FullydisagreePlease comment what exactly is good, what is missing, how it can be improved
6. YoufindtheassessmentreportsbyqualityassuranceagencieshelpfulinprovidinginformationaboutHEIs/programmes
Fullyagree Slightlyagree Slightlydisagree FullydisagreePlease comment what exactly is good, what is missing, how it can be improved
7. Wheredidyoufind/accesstheassessmentreports?
a. WebsitesofHEIs
b. Websitesofqualityassuranceagencies
c. Other(pleasename)
63
ii - expeCtations of the information proVided by the Quality assuranCe reports
8. Whatinformationdoyouneedtomakedecisionsforfurtherlearning,partnerships,comparisonswithotherinstitutionsetc.?(Upto5choices)
a. Strategicplanning,management,governance
b. Internalqualityassurancesystem
c. Historyandtraditions
d. Applicationandadmissionstatistics
e. Numberofresearchgrants,publications,citations
f. Employability/employmentofgraduates
g. Contentofstudyprogrammes
h. Reputationofteachingstaff
i. Qualificationsofteachingstaff
j. Financialresources
k. Conditionofinfrastructure
l. Studentsupportsystem
m. Institution’sabilitytoresponddiversestudents’needs
n. Accreditationstatusofinstitutions/studyprogrammes
o. Institution’spositioninleaguetables
p. Other(pleasename)
9. Inwhatformatwouldyouliketogetthisinformation?(chooseone)
a. Table(s)ofnumericaldataindicatingthemostimportantaspectsofaHEI/programme
b. AshortconcentratedsummarydescribingthemainstrengthsandareasforimprovementofaHEI/programme
c. Comparativedatawithotherinstitutions
d. Acomprehensivereportprovidingextensiveinformationaboutstrengthsandareasforimprovementinmanagementandcoreprocesses(studyprocess,researchanddevelopment),explainingalsothepossiblereasonsforagivensituation
e. Other(pleasename)
64
10.Wherewouldyouliketofindthisinformation?(pleasechooseallthatapply)
a. WebpagesofHEIs
b. Webpagesofqualityassuranceagencies
c. Socialmedia(pleasenamethemostpreferredsource,e.g.Facebook,Twitter…)
d. Printedreportsinthelibraries/qualityassuranceagencies/HEIs
e. Other(pleasename)
iii - information about the respondent
11. Whatisyourcountryofresidence?(ChoiceofthecountriesofEHEA)
12.Whichofthefollowinggroupsdoyourepresent?
a. Highereducationinstitution
b. Publicauthorities/Governmentoffice
c. Funder/Investor
d. Employer
e. Student
f. Other(pleasename)
13.TheresultsofthissurveywillbepresentedandfurtherfocusgroupinterviewsconductedinaworkshopinTallinn,Estonia,onmay 6-7, 2013.Ifyouareinterestedinparticipatinginthisevent,pleasewriteyournameande-mailaddressbelow.Theprojectwillcoverthetravelcostsoftheselectedparticipants.
• Name(optional)• E-mailaddress(optional)
thank you for your Contribution !
65
anneX ii
Questionnaire: the publiCation of Qa results: purpose, struCture and Content
survey on the use and usefulness of quality assur anCe reports
ENQAiscarryingouttheprojectTransparency of European higher education through public quality assurance reports (EQArep),togetherwithfourmemberagencies:ASHE(Croatia),EKKA(Estonia),OAQ(Switzerland)andQQI(Ireland).TheaimofthisprojectistoenquireoncurrentneedsandpracticesaboutthepublicationofexternalQAreportsandevaluatethepossibilityofdevelopingEuropeanstandardsforthesereportsinordertoensurethattheinformationprovidedbythemmeetstheneedsofstakeholders.
ThissurveyhasbeendevelopedforQualityAssuranceAgencies(ENQAmembersandaffiliates).ItsoutcomeswillbecomplementedwiththeanalysisofasampleofpublishedqualityassurancereportsandfurtherdiscussedinaworkshopinSwitzerlandduringwinter2013-14.
Wewouldbegratefulifyouwoulddedicatesometimetorespondthissurvey.Theapproximatetimeneededis20minutes.
Pleasenotethatyouranswerscanbesavedandreviewedanytimebeforeyourfeedbackissubmitted.Youranswerswillexclusivelybeusedfortheprojectpurposesandtreatedwithdiscretion.
Thesurveywillbeopenuntil[XX.XX.XX]
Thankyouinadvanceforyourtimeandpreciouscontribution.
The eQarep consortium
6661
Respondent Quality Assurance Agency (QAA): …please insert weblink to the agency
Name and role of the person of reference for this survey: …
Please note that for the purpose of the present survey:
- a distinction between institutional and programme external QA reports is made; - the focus is on published, publicly available reports;
There is a diversity of practices in writing and publishing reports across the EHEA, accordingly, in this survey, we distinguish between
- “comprehensive report” and “summary report(s)”:
comprehensive report: An extensive review report which documents the full analytical outcomes of a given external QA assessment procedure, be it at institutional or programme level, be it written by academic experts, agency employees or an external technical secretary; an in-depth analysis upon which the main findings are based is made explicit and represents a key characteristic of this type of report. Often this is the primary report.
summary report(s): Any summarising form of reporting the outcomes of an external QA assessment procedure, be it a summary, a description, a table of comparative data, a final procedural report, web text or other possible types and forms of descriptive or schematic reporting. All kinds of derivative forms of a primary comprehensive report are included here.
Section 1 - Type of assessments conducted and reports published
1. How many institutional reviews/ assessments does your agency conduct on average per year (covering a time span of 5-7 years)?
0
1-5
6-15
16-30
31+
2. What kind of report is issued for the institutional review/ assessment? (Please choose as many options as applicable)
Non published comprehensive report Published comprehensive report
Non published summary report(s) Published summary report(s)
3. Additional comment/ further explanation:
respondent Quality assuranCe agenCy (Qaa)weblink to the agency Name and role of the person of reference for this survey: …
67
62
4. How many programme reviews/ assessments does the agency conduct on average per year (covering a time span of 5-7 years)?
0
1-5
6-15
16-30
31+
5. What kind of report is issued for the programme review/ assessment? (Please choose as many options as applicable)
Non published comprehensive report Published comprehensive report
Non published summary report(s) Published summary report(s)
6. Additional comment/ further explanation:
Section 2 – Publication and Readership
7. What is the main purpose of the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Facilitate a QA/ accreditation decision
Supply feedback to a HEI for quality enhancement
Supply information to the general public
Assure transparency Agency accountability Other (specify):
63
8. In your opinion, which stakeholder groups currently tend to use your published reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies
Media Other (specify):
9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies
Media Other (specify):
10. Additional comment/ further explanation
Section 3 – Editing and publication of reports
11. In which language(s) are your reports available?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
National language(s) English Other (specify):
68
63
8. In your opinion, which stakeholder groups currently tend to use your published reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies
Media Other (specify):
9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies
Media Other (specify):
10. Additional comment/ further explanation
Section 3 – Editing and publication of reports
11. In which language(s) are your reports available?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
National language(s) English Other (specify):
63
8. In your opinion, which stakeholder groups currently tend to use your published reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies
Media Other (specify):
9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
HEIs Students (and their parents) Official authorities/ Governmental bodies
Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies
Media Other (specify):
10. Additional comment/ further explanation
Section 3 – Editing and publication of reports
11. In which language(s) are your reports available?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
National language(s) English Other (specify):
64
12. Who writes the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Panel of experts Peer leader of the panel of experts
Secretary of the panel of experts (QA employee)
Secretary of the panel of experts (external subcontractor)
Other (specify):
13. How much time on average does it take to complete a report?
approximate number of working hours
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
-5 6-15 16-30 31+
14. Where are the reports published?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Webpages of the agency Webpages of HEIs Webpages of other public authorities
Social media (specify): Hard copy Other (specify):
15. Additonal comment/ further explanation
Section 4 – Structure of reports
16. Does your agency have a template available for the writing of reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
yes no Other (specify):
64
12. Who writes the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Panel of experts Peer leader of the panel of experts
Secretary of the panel of experts (QA employee)
Secretary of the panel of experts (external subcontractor)
Other (specify):
13. How much time on average does it take to complete a report?
approximate number of working hours
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
-5 6-15 16-30 31+
14. Where are the reports published?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Webpages of the agency Webpages of HEIs Webpages of other public authorities
Social media (specify): Hard copy Other (specify):
15. Additonal comment/ further explanation
Section 4 – Structure of reports
16. Does your agency have a template available for the writing of reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
yes no Other (specify):
69
64
12. Who writes the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Panel of experts Peer leader of the panel of experts
Secretary of the panel of experts (QA employee)
Secretary of the panel of experts (external subcontractor)
Other (specify):
13. How much time on average does it take to complete a report?
approximate number of working hours
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
-5 6-15 16-30 31+
14. Where are the reports published?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Webpages of the agency Webpages of HEIs Webpages of other public authorities
Social media (specify): Hard copy Other (specify):
15. Additonal comment/ further explanation
Section 4 – Structure of reports
16. Does your agency have a template available for the writing of reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
yes no Other (specify):
65
17. What is the average length of the reports?
number of pages
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
1 2-5 6-15 16-30 31+
18. In which format is the report information presented? (Please choose as many options as applicable)
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Discursive detailed information Discursive consolidated information
Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):
19. Additional comment/ further explanation
Section 5 – Content of reports
20. What information does your agency systematically provide in the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Assessment procedure/ national legal framework
Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes
Accreditation status General information and key data of institution
Profile and content of programme
Fulfillment of formal criteria or quality standards
General findings
65
17. What is the average length of the reports?
number of pages
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
1 2-5 6-15 16-30 31+
18. In which format is the report information presented? (Please choose as many options as applicable)
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Discursive detailed information Discursive consolidated information
Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):
19. Additional comment/ further explanation
Section 5 – Content of reports
20. What information does your agency systematically provide in the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Assessment procedure/ national legal framework
Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes
Accreditation status General information and key data of institution
Profile and content of programme
Fulfillment of formal criteria or quality standards
General findings
70
65
17. What is the average length of the reports?
number of pages
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
1 2-5 6-15 16-30 31+
18. In which format is the report information presented? (Please choose as many options as applicable)
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Discursive detailed information Discursive consolidated information
Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):
19. Additional comment/ further explanation
Section 5 – Content of reports
20. What information does your agency systematically provide in the reports?
Institutional comprehensive report
Institutional summary report(s)
Programme comprehensive report
Programme summary report(s)
Assessment procedure/ national legal framework
Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes
Accreditation status General information and key data of institution
Profile and content of programme
Fulfillment of formal criteria or quality standards
General findings
66
(commendations, recommendations) Profile of strengths and weaknesses
Management, strategic planning Financial resources Quality assurance system Learning environment Employability of graduates Suggestions for quality enhancement
Other benchmarking information, please specify:
Other (specify): Other (specify):
21. Additional comment/ further explanation
Section 6 – Usability of reports
22. How would you best define your comprehensive reports?
strongly agree
agree neutral disagree strongly disagree
don’t know
easily accessible
easily readable
provides clear and transparent information on the quality of an institution/ programme
useful for HEIs
useful for public authorities
useful for funding decisions
useful for students
useful for employers
71
66
(commendations, recommendations) Profile of strengths and weaknesses
Management, strategic planning Financial resources Quality assurance system Learning environment Employability of graduates Suggestions for quality enhancement
Other benchmarking information, please specify:
Other (specify): Other (specify):
21. Additional comment/ further explanation
Section 6 – Usability of reports
22. How would you best define your comprehensive reports?
strongly agree
agree neutral disagree strongly disagree
don’t know
easily accessible
easily readable
provides clear and transparent information on the quality of an institution/ programme
useful for HEIs
useful for public authorities
useful for funding decisions
useful for students
useful for employers
67
useful for media
23. How would you best define your summary reports?
strongly agree
agree neutral disagree strongly disagree
don’t know
easily accessible
easily readable
provides clear and transparent information on the quality of an institution/ programme
useful for HEIs
useful for public authorities
useful for funding decisions
useful for students
useful for employers
useful for media
24. How do you get external feedback on the quality and usefulness of your reports?
25. Additional comment/ further explanation
Section 7 – Perspectives
26. In general, what do you think are the main strengths and weaknesses of the published reports of your agency?
72
67
useful for media
23. How would you best define your summary reports?
strongly agree
agree neutral disagree strongly disagree
don’t know
easily accessible
easily readable
provides clear and transparent information on the quality of an institution/ programme
useful for HEIs
useful for public authorities
useful for funding decisions
useful for students
useful for employers
useful for media
24. How do you get external feedback on the quality and usefulness of your reports?
25. Additional comment/ further explanation
Section 7 – Perspectives
26. In general, what do you think are the main strengths and weaknesses of the published reports of your agency?
73
68
27. Please name some elements that, from your professional perspective , would increase quality and usefulness of reports.
28. What do you think are possible challenges and risks in trying to increase transparency and comparability of reports?
Thank you for your contribution!
74
anneX iii
list of respondents to the surVey on the publiCation of Qa results: purpose, struCture and Content
• AccreditationAssociationforMedicalEducation,Turkey• AccreditationCommission(ACCR),CzechRepublic• AccreditationOrganisation(NVAO),TheNetherlandsandFlanders• Akkreditierungs-,Certifizierungs-undQualitätssicherungs-Institut(ACQUIN),Germany• AgenciaparalaCalidaddelSistemaUniversitariodeCastillayLeón(ACSUCYL),Spain• Agenced'évaluationdelarechercheetdel'enseignementsupérieur(AERES),France• Agencepourl'EvaluationdelaQualitédel'Enseignementsupérieur(AEQES),Belgium• AgênciadeAvaliaçãoeAcreditaçãodoEnsinoSuperior(A3ES),Portugal• AgencianacionaldeEvaluacióndelaCalidadyAcreditación(ANECA),Spain• AgencyforDevelopmentofHigherEducationandQualityAssurance(HEA),Bosnia
&Herzegovina• AgencyforQualityAssuranceintheGalicianUniversitySystem(ACSUG),Spain• AgencyforScienceandHigherEducation(ASHE),Croatia• AgenziaperlaCertificazionediQualitàel’AccreditamentoEUR-ACEdeiCorsidiStudioinIngegneria(QUACING),
Italy• AgencyforHigherEducationQualityAssuranceandCareerDevelopment(AKKORK),Russia• AgenturfürQualitätssicherungdurchAkkreditierungvonStudiengängen(AQASe.V.),Germany• AgenţiaRomânădeAsigurareaCalităţiiînÎnvăţământulSuperior(ARACIS),Romania• AgènciaperalaQualitatdelSistemaUniversitarideCatalunya(AQUCatalunya),Spain• AssociazioneItalianaperlaFormazioneManageriale-ItalianAssociation forManagementDevelopment(ASFOR),Italy• AkkreditierungsagenturfürStudiengängederIngenieurwissenschaften,derInformatik,
derNaturwissenschaftenundderMathematike.V.(ASIINe.V.),Germany• AustrianQualityAssuranceandAccreditationAgency(AQ),Austria• CenterforQualityAssessmentinHigherEducation(SKVC),Lithuania• CentralEvaluationandAccreditationAgency(ZEvA),Germany• Commissionforaccreditationandqualityassurance(CAQA),Serbia• TheCommissionforTechnologyandInnovation(CTI),France• EstonianHigherEducationQualityAgency(EKKA),Estonia• EuropeanAccreditationAgencyfortheLifeSciences(EAALS)• EuropeanAssociationforPublicAdministrationAccreditation(EAPAA)• EuropeanCouncilforBusinessEducation(ECBE)• EuropeanCouncilonChiropracticEducation(ECCE)• EvaluationsagenturBaden-Württemberg(evalag),Germany• FoundationforInternationalBusinessAdministrationAccreditation(FIBAA),Germany• FinnishHigherEducationEvaluationCouncil(FINHEEC),Finland• HungarianAccreditationCommittee(HAC),Hungary• InstitutionalEvaluationProgramme(IEP)• KosovoAccreditationAgency(KAA),Kosovo• NationalAccreditationAgencyoftheRussianFederation(NAA),Russia• NationalCentreforPublicAccreditation(NCPA),Russia• NationalCenterforProfessionalEducationQualityAssurance(ANQA),Armenia• NationalEvaluationandAccreditationAgencytotheCouncilofMinisters
oftheRepublicofBulgaria(NEAA),Bulgaria• NorwegianAgencyforQualityAssuranceinEducation(NOKUT),Norway• OrganfürAkkreditierungundQualitätssicherung(OAQ),Switzerland• PolishAccreditationCommittee(PKA),Poland• PublicAgencyforAccreditationofHigherEducation(PAAHE),Albania• TheQualityAssuranceAgencyforHigherEducation(QAA),UK• QualityandQualificationsIreland(QQI),Ireland• TheCouncilforHigherEducation(CHE),Israel• TheDanishAccreditationInstitution(AKKR),Denmark
76
the present publiCation is based on the findings of the “Transparency of european higher education through public quality assurance reports (eQarep)” project. This report investigates the current state of content, structure and publication of quality assurance reports while trying to understand the expectations and demands of stakeholders with regard to these reports. This report sheds light on the existing gap between what is expected by the stakeholders and the reality of the situation. in addition, the report provides Qa agencies with a set of european Guidelines for summary reports and some generic recommendations on the structure and publication of the comprehensive reports in order to support these agencies in producing reports with understandable and comparable information.
oCCasional papers 21
isBn 978-952-5539-74-5 (paperbound)isBn 978-952-5539-75-2 (pDF)issn 1458-1051