How Career Development Services Are Assessed! The Canadian research group on measuring the impact of...

How Career Development Services Are Assessed!

How Career Development Services Are Assessed!

The Canadian research group on measuring the impact of career development services

(Robert Baudoin, Lynne Bezanson, Bill Borgen, Liette Goyer, Bryan Hiebert, Vivian Lalande, Kris

Magnusson, Guylaine Michaud, Céline Renald, and Michel Turcotte)

***Names listed in alphabetical order***

Canadian group - impact of services -05

2

Why this interest?Why this interest?

• International symposia in 1999 and 2001• The OCED study on career development, 2002

-2004• The International Centre for Career

Development and Public Policy, 2003• Canada-wide symposium on career

development, continuous education and human resource development, 2003


3

What do we mean by career development?

What do we mean by career development?

• The aim of high-calibre career development services is to help individuals to:• become informed about the various training and

work options; • choose the options that correspond best to their

interests, values, aptitudes and beliefs as they make occupational choices and identify their aspirations in an uncertain and changing environment.


4

What career development services contribute

What career development services contribute

• High-calibre services that can:• contribute to an individual’s well-being and

overall development.• stimulate motivation for studies and work• help maintain good mental health for work and

studies• create a sound foundation for building effective

skill development and training systems.


5

Benefits of career developmentfor society

Benefits of career developmentfor society

• Spin-off benefits may include:• Fewer learners operating in “trial and error” mode

• Higher success rates in programs, courses, and obtaining certificates and degrees

• Less time on social assistance

• Lower healthcare costs

• Fewer social drop-outs

• Happier employees, less sick leave, higher retention rates, greater involvement in training and learning


6

Why is assessment important?Why is assessment important?

• To obtain funding

• To have a clearer idea of actual results as opposed to expected results

• To request the investment needed to achieve the expected results.


7

Lack of solid dataLack of solid data

• At this time, we unfortunately don’t have much information on how career development services are assessed.

• Before suggesting ways of solving the problem, it is crucial to know more about the current state of assessment practice and its impact.


8

Responses to the need for solid data

Responses to the need for solid data

• Fall 2004: online survey:• How do service providers assess the impact of

their services?• Our goal: To better understand the state of

current practice in Canada with respect to documenting the results of career development activities and services.


9

The purpose of the survey: 4 main questions

The purpose of the survey: 4 main questions

1. How important is it to assess the impact of career development services?

2. How can the impact of these services be determined?

3. What are the results from such services?

4. How can these results be measured?


10

Data collectionData collection

• Questionnaire for agencies (online)

• Questionnaire for practitioners (online)

• Questionnaires in both official languages

• Telephone interviews with employers and policymakers


11

What we mean by “results”What we mean by “results”

• A reaction or specific product that results from an activity

• A result can be one or other of the following : • Changes in the client’s skills: demonstrated by

changes in the client’s knowledge, abilities or characteristics

• Changes in the client’s situation: for example, job situation, educational situation, etc.

• General changes for clients and the community:• Changes in terms of financial situation, social integration,

impact on families, community economic development, etc.


12

What we mean by “activities”What we mean by “activities”

Any activity carried out with the intention of helping clients to make changes to their individual situations – in other words, any action that produces a result for a client or a group of clients.


13

Survey sampleSurvey sample

• 173 agencies; 214 practitioners; 9 policymakers and 7 employers

• The types of service offered include all customary services – individual interviews, group sessions, assessment, LMI and employment development

• Target age-group: mostly young people and adult job-seekers, followed by young people at the post-secondary level or finishing high school

• Type of agency: mostly NGOs (50.3%), followed by non-educational provincial organizations (23.3%)

• Agency size: 41% with over 10 employees; 20% with 6-10 employees; and 39% with fewer than 6 employees


14

What importance is put on measuring the results or effectiveness of the services provided?

What importance is put on measuring the results or effectiveness of the services provided?

97.6% of the agencies responded that it was “quite important” (24.4%) or “very important” (73.2%)

84% of the agencies report on the impact or effectiveness of their services.


15

There is a significant relationship between the type of service provider and the importance placed on

measuring impact and determining its actual value

There is a significant relationship between the type of service provider and the importance placed on

measuring impact and determining its actual value

• Agencies: Primary and secondary schools are less inclined to respond “very important” and post-secondary institutions are less inclined to respond “quite important.”

• Practitioners: Practitioners in educational institutions are less inclined to respond “very important,” and the NGOs are more inclined to respond “very important.”


16

Agencies and practitioners: What are the 3 most important results you report on?

Agencies and practitioners: What are the 3 most important results you report on?

1. Changes in the client’s job or educational situation;**

2. Development of skills; financial independence, self-confidence; **

and, to a lesser extent:3. Number of clients served4. Client satisfaction5. Programs completed6. Service delivery7. Cost/benefit ratio


17

Agencies and practitioners: Types of evidence for results

Agencies and practitioners: Types of evidence for results

• Assessment/follow-up reports: Reports from client or a partner; investigations, interviews

• Frequencies: number of clients served/month; number of clients finding jobs; number of action plans; number of clients completing programs

To a lesser extent:

• Observation of changes in the client• Cost/benefit analysis• No difference between agency and practitioner responses• 33% of the practitioners did not respond


18

Comments receivedComments received

• Assessment is difficult• Certain results can’t be measured

• Too much importance put on employment statistics – to the detriment of other types of results

• There is need for research on career development services and on specific populations


19

What results do you obtain that are not measured or reported?

What results do you obtain that are not measured or reported?

• Developing the client’s ability to act • Skill development (e.g., self-confidence, independence,

changed attitudes towards their future and their perception of the labour market)

• Financial independence• Creation of support networks• More job opportunities and other kinds of opportunities

for clients

To a lesser extent: benefits for the community; client satisfaction; increase in demand for political lobbying (agency); program implementation and development (agency); repeat clients (practitioner); professional development (practitioner)


20

What proves that you have achieved these results?

What proves that you have achieved these results?

• Anecdotal evidence• Verbal reports from clients• Verbal reports from employers

• Observations• Observed changes in the client• Observed changes in the client’s attitudes

• Recurring theme in the data: belief that several of the results observed or reported are either not quantifiable or are very difficult to measure (and, as a result, are not taken seriously)


21

What difficulties do you experience in proving or measuring results?

What difficulties do you experience in proving or measuring results?

• Complexity• Difficulty in determining the results of the services provided• Belief that certain results cannot be measured

• Assessment considered to be of little importance• Lack of allotted resources (money and time)

• Little recognition of the importance of collecting solid data• Lack of training in evaluation methods

• Difficulty in obtaining feedback from clients• Loss of contact with clients; clients don’t want to give

feedback


22

Difficulties (cont’d)Difficulties (cont’d)

• Lack of uniformity or agreement between the agencies and the funders on the type of results to be collected

• Little linkage between service delivery and assessment (agencies)

• Absence of assessment protocols or formal processes for completing effectiveness evaluations

• Lack of experience in, and limited access to, models of best practices (practitioners).

• Rapidity of changes (practitioners)


23

What things would you like to report on, if given the opportunity?

What things would you like to report on, if given the opportunity?

• The social impact of services on individuals, families, the community, and employers

• Development of the ability to act, and client growth and satisfaction• Changes in behaviour and attitudes

• Quality of services• Components of the process; time needed to achieve the

goals• Longitudinal impact• Link between local and national results

The other side of the coinThe other side of the coin

What do policymakers and employers think?


25

What do policymakers want?What do policymakers want?

• Client results• e.g., client satisfaction, intelligent decisions, and client’s ability to identify

career opportunities.

• Indicators of client results • e.g., number of jobs obtained, number of successful transitions

• Services that make a difference in client’s lives; the change would not have occurred without the service provided

• Value added of the services in terms of return on investment or economic improvement

• Longitudinal studies to make comparisons between those who receive services and those who don’t


26

Other information that policymakers are interested in

Other information that policymakers are interested in

• How services are delivered

• Number of clients served; number of places offered in programs, client assessment

• Level of service provided and degree of correspondence between level of service and the client’s expressed need

• Monitoring and assessing contracts


27

What do employers want?What do employers want?

• Employees who are well-trained, competent, committed, motivated and satisfied

• Increased productivity, less employee turnover, skill-based job descriptions, improved internal employee mobility

• Feedback from employees and service providers on what worked and what didn’t


28

What policymakers and employers agree on

What policymakers and employers agree on

• Need to collect solid data linking results and services

• Need to develop better methods for assessing results.


29

Next stepsNext steps

• Develop detailed impact assessment models

• Develop clear, valid and reliable tools

• Create an analytical framework for comparing data (meta-analysis)

• Create a mechanism for disseminating and promoting service impact assessment information.

Thank you!Thank you!

The Canadian research group on measuring the impact of career

development services

How Career Development Services Are Assessed! The Canadian research group on measuring the impact of...

Documents

Transcript of How Career Development Services Are Assessed! The Canadian research group on measuring the impact of...