1 Outcomes of the Nuremberg Peer Learning Activity on Indicators and Benchmarks in Guidance, 27.-...

1

Outcomes of the NurembergPeer Learning Activity on Indicators

and Benchmarks in Guidance, 27.- -28. October 2005

presented at the ELGPN-WP 4 – Peer Learning MeetingHelsinki, 19. – 20. October 2009

Karen Schober

2Schober, Indicators and Benchmarks in Guidance – Results from Nuremberg Meeting 2005Presentation ELGPN –PLM Helsinki, Oct. 19-20, 2009

Background/ purpose of the Nuremberg PLA (1)

• Special activity of the EU-Expert Group LLG in the context of the establishment of a “Standing Group on Indicators and Benchmarks” in the Education and Training 2010 – Program

• Purpose of the ExpGr. was to draw attention to the guidance issue and provide guidance related indicators + benchmarks

• In addition to existing previous work CEDEFOP commissioned a “Study on Indicators and Benchmarks in Guidance” by Peter den Boer and Jarp Scheerens (on ELGPN website, background material WP 4),

• Survey was carried out among experts in EU-member countries on national policies and practices related to guidance indicators and benchmarks



• Because of the limited responses additional national surveys were commissioned in order to improve the basis (publication ??)

• The PLA in Nuremberg was designed as a small group meeting to discuss the outcomes of the studies and selected national “interesting practice” examples

• The aim was to give recommendations to the Expert Group on further work to be done in that field and to contribute to the Standing Group on Indicators and Benchmark Group



• From the results of the studies 6 countries were chosen who had some experience in implementing indicators and benchmarks in (at least one) area of career guidance to present their recent work: DE, DK, FI, FR, IR, UK

• Jennifer Wannan (CEDEFOP) facilitated the meeting and Ronald Sultana as invited expert drafted the results and conclusion (on the ELGPN website, background material of WP 4)


Issues of Discussion

• Framework and questionnaire of the Study on “Indicators and Benchmarks”

• Pros and cons of implementing indicators and benchmarks in guidance and resulting challenges

• “Interesting practice” from the 6 countries

• Conclusions and recommendations for further work of the Expert Group, including possible changes in the “Common Reference Tools”


Framework of the study/questionnaire

• Descriptive approach, no comprehensive theoretical model of a quality development circle like the one we discuss today

• Attempt to generate indicators from a matrix of

• type of provider / type of service and target group (according to educational/employment situation)

• operational level (individual – organisational – national) and type of indicator (input – process – output – context)

• and a subsequent attempt of prioritisation of the proposed indicators


Types of providers Types of servicesTarget groups

youth in formal education/

training

adults in continuing education/ training

unemployed employed

education/ training institutes

Information

Individual advice and counsellinglearning & career management competency building

enterprises

See above

s.a.

s.a.

employment agencies

specialised organisations/centres

community / welfare organisations


INPUT PROCESS OUTPUT CONTEXT

INDIVIDUAL LEVEL

1.1 Number andcharacteristics of

participants

1.2 Qualification of staff

3.1. Client satisfaction

3.2. Personal development

3.3. Policy related output

4.1. Personal characteristics

ORGANISATIONAL LEVEL

1.3. Availability and typology of service providers

2.1. Content of services 2.2. Quality management & delivery

1

4.2. Entrepreneurial climate

4.3. Infrastructure

NATIONAL LEVEL

1.4. Financial means

1.5 Policy

2.3. Organisation of co-ordination and control

2.4. Coherence

1

4.4. Social, educational and labour-market situations


Interesting Practice in six countries (1)

• Denmark presented a preliminary version of what was presented in the Berlin PLM 2009: an education sector based top down QA-System for the career centres on the basis of quantitative indicators.

• Finland presented the results of the 2002 evaluation of guidance in comprehensive and upper secondary schools on the basis of internet based questionnaires to all relevant actors including students, parents, teachers, principals in an “multi-angled” approach.

• Germany presented the PES-monitoring and controlling system for the guidance services of the PES: a top down model of quantitative indicators which are controlled against a standard set by target agreement between the local offices and the regional and central office, integrated into a balanced score card.


Interesting Practice in six countries (2)

• France presented “les mission locales” a government programme to improve the educational, vocational and labour market integration of young people by intensive case management and the measures taken for quality assurance.

•Ireland presented the monitoring and controlling system of the Adult Education Guidance Initiative (AEGI) – a government programme to assist the lifelong learning strategy: a top down model of a large number of qualitative and quantitative indicators with a quarterly reporting system, established for the time of the programme.

• UK presented an overview over the nationwide existing monitoring and QA-Systems in IAG services in education and employment: the majority uses external regulated standards (“Matrix”/”Charter Mark”).


Indicators and Benchmarks (1) Pros Cons

• need to evaluate effectiveness• potential of enhancing quality• establish performance targets• potential for benchmarking against those standards• improve comparability• user feedback as a monitoring and feedback mechanism• useful in discrete areas as part of a holistic approach• integration of quantitative and qualitative data • “triangulation” approach

• danger bureaucratic overload• danger of overcrowded data• danger of overemphasising measurable quant. Indicators• methodological difficulties of measuring phenomenological realities• professional role of GP can lead to resistance and by that to limited validity of data• danger of misuse by managers justifying reduction of resources • in general: danger of “perverse effects”


Indicators and Benchmarks - Challenges

• generating an appropriate and consolidated national data base across the various sectors and providers

• agreement on definitional problems (i.e. Output, outcome, throughput etc.)

• solving methodological problems of validity and reliability

• data protection issues (privacy, ethical issues)

• avoiding overcrowded databases and too much indicators

• formative and developmental approach rather than one-dimensional

• holistic, integrated approach of quantitative and qualitative indicators

with the client/user needs at the centre

• focus on new guidance paradigm (lifelong, life wide, career management skills, seamlessness, enhanced sector collaboration)


Key Recommendations and Decisions of the Expert Group (1)

• work on indicators be clearly situated within a broader evaluation culture of lifelong guidance and

• be clearly linked to qualitative analysis;

• indicators be designed from a formative perspective and

• linked to quality assurance;

• indicators take account of the new paradigm of guidance;

• as a first step EU guidelines to be drawn up for the improvement of the evaluation of guidance and data collection at national level covering all key aspects of guidance;



•In the medium to long-term perspective, EU level indicators should be established on

• access to guidance services

• take-up of guidance provision,

• user satisfaction and

• learning and career management competences

• Data on the short-term, medium-term & long-term

• outputs and outcomes from career guidance and

• evidence of the economic and social impact of career guidance.



•small sub-group be set up, charged with

• identifying which key aspects of guidance should be measured statistically and which require other means;

• providing examples of good practice and new methodologies;

•identifying actions which the various stakeholders could take at EU and national level;

•developing proposals for EU level indicators on access, satisfaction & competence development;

•exploring the feasibility of setting an EU benchmark for guidance.



• supported the conclusions of the Nuremberg meeting, especially concerning the need for use to be made of the data collected to improve services.

• agreed with all the recommendations, with the exception of the idea of an EU benchmark, which was felt to be unrealistic.

• decided to discus with DG EAC to establish a sub-group within the Standing Group on Indicators and Benchmarks with representatives from the Expert Group (Steffen Jensen and a provider/practitioner representative from Learndirect)

• Need to collect more evidence base for the impact of career guidance on clients’ outcomes and achievements



• Continue work on quality assurance and organise another small mutual learning meeting on QA-Systems, inviting countries who have made progress in implementing QA-Systems in guidance (Copenhagen, Jan. 26, 2006)

• Draft findings by Ronald Sultana (on the ELGPN website, background material of WP 4) including a suggestion for amendments / changes in the European Reference Tools

• After the end of the Expert Groups mandate further work on QA, indicators and benchmarks was postponed to the newly established ELGPN


Thank you

for your attention !

Kontakt: [email protected]

1 Outcomes of the Nuremberg Peer Learning Activity on Indicators and Benchmarks in Guidance, 27.-...

Documents

Transcript of 1 Outcomes of the Nuremberg Peer Learning Activity on Indicators and Benchmarks in Guidance, 27.-...