General National Survey of CES Membership · ever General National Survey of CES Membership to...

Canadian Evaluation Society National Membership Survey 2016

ONP39TOL-24 Page 2

General National Survey of CES Membership

Prepared for the

Canadian Evaluation Society November 21st, 2016

TABLE OF CONTENTS

STRATEGIC OBSERVATIONS .............................................................................................. 1

1.0 INTRODUCTION AND METHODOLOGY ..................................................................... 2

2.0 PROFILE OF RESPONDENTS ..................................................................................... 3

3.0 SURVEY RESULTS ...................................................................................................... 4

3.1 EVALUATION PRACTICE AND MEMBERSHIP WITH CES .................................... 4

3.1.1 Main Reasons for Joining CES ............................................................................. 4

3.1.2 Levels of Member Engagement ............................................................................ 5

3.1.3 Interest in Becoming More Involved with CES ...................................................... 6

3.1.4 Recent Involvement with CES Activities- Higher Levels of Participation .............. 7

3.1.5 Recent Involvement with CES Activities- Lower Levels of Participation ............... 8

3.1.6 Evaluation-Related Work as a Proportion of Total Workload ................................ 9

3.1.7 Role of Evaluation in Work Performed by CES Members ................................... 10

3.1.8 Length of Time Working in Evaluation ................................................................ 11

3.1.9 Incidence of Credentialed Evaluators ................................................................. 12

3.1.10 Anticipated Time Frame for Applying for Credentialed Evaluator Designation ... 13

3.2 PROFESSIONAL DEVELOPMENT ........................................................................ 14

3.2.1 Average Time Spent on Evaluation-Related Professional Development ............ 14

3.2.2 Appeal of CES Training Formats ........................................................................ 15

3.2.3 Members’ Interest in Pursuing Selected CES Courses Over the Next Two Years

...................................................................................................................... 16

3.2.4 Appeal of Specific Topic Areas for Training - Highest Levels of Appeal ........... 17

3.2.5 Appeal of Specific Topic Areas for Training -Lower Levels of Appeal ............. 18

3.2.6 Average Annual Expenditure on Evaluation-Related Professional Development 19

3.2.7 Sources of Funding for Evaluation-Related Professional Development ............... 20

3.2.8 Attendance at CES National Conferences .......................................................... 21

3.2.9 Likelihood of Attending 2017 CES Annual Conference in Vancouver ................ 22

3.3 APPEAL AND USAGE OF CES MEMBER BENEFITS .......................................... 23

3.3.1 Appeal and Usage of CES Member Benefits - Knowledge and Research

Resources ..................................................................................................... 23

3.3.2 Appeal and Usage of CES Member Benefits - Professional Development ......... 24

3.3.3 Appeal and Usage of CES Member Benefits - Perks and Group Member

Discounts ...................................................................................................... 25

3.3.4 Suggested Improvements to CES National Membership Perks, Products or

Services ........................................................................................................ 26

3.4 ATTITUDES TOWARDS BEING A MEMBER OF THE CES .................................. 27

3.4.1 Attitudes Towards Being a Member of the CES – Overall Participation ............. 27

3.4.2 Attitudes Towards Being a Member of CES - Perceived Value and Benefits ..... 28

3.4.3 Attitudes towards CES Membership– Networking and Connections to CES ...... 29

3.4.4 Attitudes Towards CES Membership- Making a Difference ................................ 30

3.5 COMMUNICATIONS .............................................................................................. 31

3.5.1 Sources of Information about CES Activities ...................................................... 31

3.5.2 “E-Blast” - CES National Weekly Newsletter Readership ................................... 32

3.5.3 Suggested Improvements to “E-Blast” Newsletter .............................................. 33

3.5.4 Perceived Importance of Emails on CES Employment-Related Opportunities ... 34

3.5.5 Visitation to CES National Website ..................................................................... 35

3.5.6 Main Uses of CES National Website .................................................................. 36

3.5.7 Suggested Improvements for CES National Website ......................................... 37

3.5.8 Perceived Adequacy of Current Communications from CES .............................. 38

3.6 PERFORMANCE APPRAISAL OF CES IN SELECTED AREAS ........................... 39

3.6.1 Performance Appraisal of CES in Selected Areas – Highest Scores ................. 39

3.6.2 Performance Appraisal of CES in Selected Areas – Lower Scores .................... 40

3.6.3 Incidence of Member Contacts with CES Representatives ................................ 41

3.6.4 Satisfaction with Interactions with CES Representatives ...................................... 42

3.6.5 Suggested Improvements for Better Responding to CES Member Queries ....... 43

3.6.6 Overall Member Satisfaction with CES ............................................................... 44

3.6.7 Opportunities for CES to Better Meet Members’ Professional Needs ................ 45

3.6.8 Member Satisfaction with Local CES Chapters .................................................. 46

3.6.9 Member Satisfaction with Local CES Chapters – By Region .............................. 47

3.6.10 Opportunities for Local Chapters to Better Meet Members’ Needs .................... 48

3.6.11 Intentions Regarding CES Membership Renewals ............................................. 49

3.7 RECOMMENDATIONS ........................................................................................... 50

3.7.1 Suggestions for Promoting Membership with CES ............................................... 50

3.7.2 Recommendations for Better Engaging Under-Represented Groups ................. 51

3.7.3 Final Recommendations and Comments ............................................................ 52

APPENDICES

Ü Questionnaire Ü Detailed Tabular Results Ü Verbatims


ONP39TOL-24 Page 1

STRATEGIC OBSERVATIONS CES members provided some very valuable insights regarding evaluation practice in Canada and membership with their professional association. The 2016 General National Survey of CES Membership was conducted among a sample of 405 CES members, for a 30% response rate. The Marketing Research and Intelligence Association policy limits statements about margins of sampling error for most online surveys. The margin of error for a comparable probability-based random sample is +/- 4.1%, 19 times out of 20. Totals may not add up to 100 due to rounding.

A look at evaluation practice in Canada showed a fairly even split between respondents who spend three-quarters to all of their work on evaluation and those who spend less than three-quarters of their work on evaluation. This of course reflects respondents’ perceptions of what is considered “evaluation-related” and may require greater clarification for research participants in the future as evaluation practice expands into “demand capacity” and “evaluative thinking” and as monitoring and assessment become more integrated into the role of evaluators and the competencies required by those working in the field.

The proportion of respondents who have worked in evaluation for ten years or less slightly outnumbered those who have been working in the field for over ten years. Close to one-third of respondents participating in the survey are Credentialed Evaluators (CEs) and almost three-quarters of these CEs have been working in the field for ten years or more.1

The main reasons respondents joined their professional association was for professional development opportunities and because participating in CES gave them a sense of belonging to a community of practice. Respondents were twice as likely to say they were engaged as compared to unengaged with CES though less than one-quarter of survey participants were interested in becoming more involved. Virtually all respondents read CES-related newsletters, emails and articles while around one-half of those surveyed attended CES Chapter events. Participation in a number of CES national and Chapter activities was higher among respondents who have been members for over three years.

The most appealing member benefits for roughly three-quarters of survey participants were the professional development offerings such as the Chapter learning events, Annual CES conference and CES webinars. A similar proportion of respondents found CES’ knowledge- and research-related offerings valuable, notably having immediate access to the Journal of Canadian Program Evaluation and the grey literature database. Professional development was seen as a critical service provided by CES and was rated as a valuable member benefit for roughly three-quarters of survey participants. It was impressive to note that this was an area where CES also received highest accolades. Survey participants were also pleased with how CES was developing a community of practice for those involved in evaluation and felt CES was effectively advocating for increased awareness and understanding of the use of evaluation. Areas where performance could be improved were in expanding and diversifying membership and encouraging Chapters to collaborate with each other and with the national organization.

The most popular source of information about CES activities was the weekly “E-Blast” newsletter and more than four-in-five respondents reported reading the newsletter at least once a month. Just slightly fewer survey participants felt the employment-related emails sent out by CES were important, particularly those earning less than $75K annually. A strong majority of those surveyed visited the national CES website at least once a month, most often to access evaluation-related information or

1 When interpreting the results, it should be noted that Credentialed Evaluators represent approximately 20 percent of CES members and that the proportion of CEs participating in the study is higher (31%).


ONP39TOL-24 Page 2

resources, the calendar of events, webinars or CES Annual Conference information. Overall, nearly all respondents felt they were receiving the right amount of information from CES, though one-tenth felt it was not enough and five percent thought it was excessive.

Three-quarters of those surveyed expressed overall satisfaction with the CES and virtually all respondents intend to renew their membership next year. Moving forward, it will be important to engage relatively newer members and those expressing lower levels of satisfaction with the organization by better promoting member benefits, identifying sources of discontent and encouraging these cohorts to avail themselves of the diverse range of products and services CES has to offer to its membership.

The good response rate demonstrates an engaged membership with a keen interest in the future of the organization. The insights gleaned from this membership survey will provide the CES Board of Directors and standing Committees, as well as the CES Secretariat, Executive and Chapters with accurate and actionable results for planning and management of CES moving forward.

1.0 INTRODUCTION AND METHODOLOGY Probe Research was commissioned by the Canadian Evaluation Society (CES) to conduct its first ever General National Survey of CES Membership to assess CES members’ and evaluation stakeholders’ satisfaction with the variety of education and training-related activities, networking events and other CES program and service offerings available at the national and regional levels. The research identified specific needs of CES members and other evaluation stakeholders and assessed members’ expectations, concerns and priorities for the organization.

The insights gleaned from this research will provide direction to the CES Board of Directors, Standing Committees and to the CES Secretariat, Executive Director and Chapters and will be used to inform the implementation of the “CES Strategic Plan 2015-2018” which outlines the organization’s priorities moving forward. The study included a background literature review of key industry trends driving the need for specific evaluation-related products and services required by those working in the profession. Effective communications and outreach efforts for communicating with and attracting members and evaluation stakeholder groups to be actively involved in the organization, participate in CES events and benefit from professional development and networking opportunities were also examined.

This survey was conducted online between October 6th and October 27th, 2016 with responses from 405 members.

The Canadian Evaluation Society sent out an email advising members that they would be contacted by Probe Research Inc. to participate in the 2016 National CES Membership Survey and encouraging their participation. Pretests (n=51) were conducted and subsequently Probe Research sent emails to CES members inviting them to share their views on important issues regarding the CES via a unique link to the survey. Library members (n=38) and those members who requested not to be contacted for surveys on their membership form (n=301) were not sent survey invitations. Upon consideration, CES decided to extend the invites to this second cohort to offer them an opportunity to participate in the membership survey. A total of 1,376 survey invitations were sent out (including pretests) and there were 30 emails that bounced back. Two reminder emails were sent out to all potential survey respondents. The online survey was conducted using the Fluid Surveys platform. All results were tabulated, open-ended questions were coded and data analysis was conducted using SPSS.

The findings presented here cover topline results and only statistically significant variations in attitudes and behaviours among survey sub-populations considered pertinent for the research are included. CES will be undertaking its own more detailed analysis of Committee and Chapter perspectives.


ONP39TOL-24 Page 3

2.0 PROFILE OF RESPONDENTS The following table outlines the demographic characteristics of those who responded to the survey:

TABLE 1

PROFILE OF RESPONDENTS

% of Total Respondents (n= 405)*

Gender (n=345) Men 30 Women 70

CES Chapter (n=405) British Columbia & Yukon 13 Alberta & NWT 12 Saskatchewan 6 Manitoba 4 Ontario 18 National Capital Region & Nunavut 23 Quebec 8 Atlantic Canada 12 International Members 3

Age (n=336) 35 years or younger 29 36-45 years 36 Over 45 years 35

Income (n=313) < $75K 20 $75K-$124K 35 $125K or more 45

Education (n=388) Less than a Masters 10 Masters 71 PhD 19

Years of Membership with CES (n=405) Less than 1 year 14 1 year to less than 3 years 22 3 years to less than 5 years 12 5 years to 9 years 25 10 years or more 26

CES Membership Categories (n=402) Individual member 79 Senior member (60+ years) 10 Student member (full-time registered student) 8 International 3

Language (n=405) English 93 French 7

*DK/NS Removed


ONP39TOL-24 Page 4

3.0 SURVEY RESULTS 3.1 Evaluation Practice and Membership with CES This section of the report examines CES members’ current involvement with evaluation and explores their motivations for joining their professional society.

3.1.1 Main Reasons for Joining CES

Survey participants were presented with a list of nine response options (including “other- specify” and “don’t know”) and were asked to rank their top three reasons for being a member of CES. An examination of the top three responses reveals that three-quarters of research participants (75%) joined for professional development reasons. The second most frequently mentioned top three responses was that membership gives me a sense of belonging to a community of practice (60%). These were followed by the opportunity to network (49%) and that membership with CES gives me credibility (47%) which were each mentioned by roughly one-half of respondents.

Figure 1

• Respondents who have been CES members for ten years or longer were more likely to indicate that they are members because it gives them a sense of belonging to a community of practice compared to relatively newer members who have been with the organization for less than three years (76% versus 47% respectively).

MainReasonsforJoiningCES-Top3Mentions-

Q. A1 “What are the main reasons you are a member of CES?” (n=405)

0% 20% 40% 60% 80%

For professional development opportunities

Gives me a sense of belonging to a community ofpractice

Opportunity to network

Membership with CES gives me credibility

Membership fees are paid for by my employer

To receive a discount on fees(conferences/workshops)

I felt obligated to join

Other

Don't know

75%

60%

49%

47%

18%

17%

10%

6%

1%

Base: Total Respondents


ONP39TOL-24 Page 5

3.1.2 Levels of Member Engagement

Survey participants self-assessed their levels of engagement with CES and were given examples that corresponded with each response option. Two-in-five respondents considered themselves to be somewhat engaged (38%) with the organization. Survey participants were twice as likely to say they were engaged (including “somewhat” and “very engaged”) as compared to unengaged (including “not very” and “not at all engaged”) with the organization (59% versus 25% respectively).

Figure 2

• Above average levels of disengagement were evident among respondents aged 36 to 45 years of age (38% “not engaged” including 9% who are “not at all engaged”).

• More than twice as many research participants who expressed high levels of satisfaction with CES were engaged with the organization as compare to those expressing lower levels of satisfaction with CES (81% versus 35% respectively).

• While three-quarters of Credentialed Evaluators (CEs) surveyed were engaged with CES, only one-half of non-CEs participating in the research were similarly engaged (78% versus 51% respectively).

LevelsofMemberEngagementQ. C1. “Individuals have different levels of engagement with CES. Would you say you are…Very engaged – (e.g.

hold a Board position or sit on a sub-committee); Somewhat engaged– (e.g. attend training sessions/keep up-to-date about CES activities on a regular basis); Neither engaged nor unengaged; Not very engaged (e.g. occasionally visit the website and read updates); Not at all engaged – (e.g. pay dues and have little interaction).” (n=405)


0%

20%

40%

60%

Very engaged(Board orcommitteemember)

Somewhatengaged(regularly

attendsessions/keep

up-to-date)

Not veryengaged

(occasionallyvisit

website/readupdates)

Not at allengaged (pay

dues)

Neitherengaged norunengaged

Don't know

21%

38%

19%

6%13%

3%


ONP39TOL-24 Page 6

3.1.3 Interest in Becoming More Involved with CES

We asked CES members participating in the survey if they were interested in becoming more involved with CES. One-half said they were not interested (52%), just under one-quarter said they were interested (22%) and one-quarter said they did not know (25%).

Figure 3

• Students (41%) and respondents aged 18 to 35 years (35%) were among those expressing an interest in being more involved with CES.

• Three times as many respondents who have been with the organization for less than three years were interested in increasing their involvement with CES compared to those who have been members for ten years or longer (34% versus 11% respectively). This should not be misconstrued as demonstrating higher levels of disinterest with CES among longer-term members. Indeed, the study shows that many individuals who have been members with CES for ten years or longer are already very involved with the organization. Instead, the results demonstrate that a significant proportion of these long-term members participating in the research are not interested in further expanding their time commitment beyond their current levels of involvement.

Levels of Interest in Greater Involvement with CES Among Current Members

Yes, 22%

No, 52%

Don't know, 25%


Q.C6. “Are you interested in being more involved with CES? If so, then please provide your email below so that the Society can follow-up with you when opportunities arise.” (n=405)


ONP39TOL-24 Page 7

3.1.4 Recent Involvement with CES Activities- Higher Levels of Participation

Respondents were asked to indicate which CES-related activities they had been involved with over the past 12 months. Multiple mentions were accepted.

Highest levels of involvement were reported for reading CES-related newsletters/emails/articles (88%). Virtually all respondents participated in this activity and this was fairly consistent across different sub-populations. Around one-half of respondents said they attended Chapter events (47%). One-third of those surveyed attended Chapter Annual General Meetings (35%). Around one-fifth were Chapter Board Members or Committee members (22%) while a similar proportion presented at Chapter events or national events (20%).

Figure 4

• Levels of participation in all these activities except reading CES-related newsletters, emails, articles and materials were higher for respondents who have been CES members for ten years or more compared to those who have been with the organization for less than three years.

• Senior members participating in the research were more were more likely than average to attend the national CES Annual General Meeting (33%).

RecentInvolvementwithCES-RelatedActivities– HigherLevelsofParticipation-Q.C2 “People get involved in different ways at different point in time with CES. Which of the following CES-

related activities have you been involved with over the past 12 months? (CHECK ALL THAT APPLY) (n=405)

0% 20% 40% 60% 80% 100%

Reading CES-relatednewsletters/emails/articles/materials

Attending CES Chapter events

Attending my CES Chapter Annual GeneralMeeting

Volunteering as a CES Chapter Board member/AdHoc Committee member

Presenting at CES Chapter events or nationalconferences

Attending the national CES Annual GeneralMeeting

Volunteering to plan a CES annual conference

88%

47%

35%

22%

20%

15%

12%

Base: Total Respondents *Multiple mentions accepted


ONP39TOL-24 Page 8

3.1.5 Recent Involvement with CES Activities- Lower Levels of Participation

Less than one-in-ten members surveyed reported that they were involved with the CES activities listed in the graph below including writing/editing manuscripts (9%), providing feedback into strategic planning (9%), sitting on the National Board/Credentialing Committee (8%), providing mentoring (3%) and other mentions (2%). Two percent of research participants reported that they have not participated in any CES activities while the same proportion said they did not know.

Figure 5

• Senior members participating in the research were more involved in volunteering on the National Board or for National Committee positions (28%).

• Survey participants with a Ph.D. and those who identified themselves as Researchers in evaluation reported higher than average levels of involvement in writing/editing manuscripts for publication compared to members overall (26% each versus 9% respectively).

RecentInvolvementwithCES-RelatedActivities– LowerLevelsofParticipation

Q.C2 “People get involved in different ways at different point in time with CES. Which of the following CES-related activities have you been involved with over the past 12 months? (CHECK ALL THAT APPLY) (n=405)

0% 5% 10%

Writing/Editing manuscripts forpublication

Providing feedback into CESStrategic Planning

Volunteering on CES NationalBoard/Nat'l Committee

Providing mentoring through theCES Mentoring Initiative

Other (specify)

I have not participated in anyCES-related activities recently

Don't know

9%

9%

8%

3%

2%

2%

2%



ONP39TOL-24 Page 9

3.1.6 Evaluation-Related Work as a Proportion of Total Workload

There was a fairly even split between respondents who spend three-quarters to all their work on evaluation (48%) and those who spend less than three-quarters of their work on evaluation (49%). This of course reflects respondents’ perceptions of what is considered “evaluation-related” and may require greater clarification for research participants in the future as evaluation practice expands into “demand capacity” and “evaluative thinking” and as monitoring and assessment become more integrated into the role of evaluators and the competencies required by those working in the profession.

Figure 6

Evaluation-Related Work as a Proportion of Total Workload

Less than one-quarter, 13%

One-quarter to less than one-half, 14%

One-half to less than three-quarters, 22%

Three-quarters to almost all, 24%

All, 24%

Don't know, 3%

Base: Total Respondents-

Q.D5. “Roughly what proportion of your work is evaluation-related?” (n=405)


ONP39TOL-24 Page 10

3.1.7 Role of Evaluation in Work Performed by CES Members

One-half of respondents considered themselves to be producers of evaluation results for their own organization (50%) while a similar proportion were producers of evaluation results for other organizations (49%). One-quarter of survey participants said they were researchers on evaluation (23%) while one-fifth said they were users of evaluation results (21%). Only a small proportion of respondents noted other responses or declined to comment (2% each).

Figure 7

• A significant proportion of research participants working for the government and for not-for-profit organizations were producers of evaluation results for their own organization (85% and 73% respectively).

• Credentialed Evaluators participating in the survey were more likely than average to report being producers of evaluation results for other organizations (62%).

• Three-in-five research participants who have been working in evaluation for 5 years or less (61%) reported that they are producers of evaluation results for their own organization. Only one-third of this cohort (32%) indicated that they are producers of evaluation results for outside organizations. Conversely, two-thirds of research participants who have been working in evaluation for more than 15 years (65%) reported that they are producers of evaluation results for outside organizations. This may suggest that there is a career progression for those working in evaluation from working mostly internally to expanding to work for outside organizations as one becomes a more seasoned evaluation practitioner.

Role of Evaluation in Work Performed by CES Members

D6. “Would you consider yourself a (CHECK ALL THAT APPLY)?” (n=405)


0%

20%

40%

60%

80%

100%

Producer ofevaluation

results for ownorganization

Producer ofevaluationresults for

outsideorganizations

Researcher onevaluation

User ofevaluation

results

Other

50% 49%

23% 21%

2%

38%

62%

19%11%

1%

56%

42%

26% 26%

2%

Total (n=405) CE Non-CE


ONP39TOL-24 Page 11

3.1.8 Length of Time Working in Evaluation

We asked members to report on how long they have been working in evaluation. The proportion of research participants working 10 years or less in evaluation (52%) slightly outnumbered those who have been working in the field for 11 years or more (45%).

Figure 8

• Among Credentialed Evaluators surveyed, one-quarter have been working in evaluation for 10 years or less (26%) while almost three-quarters have been working in the field for 10 years or more (72%). In fact, one-third of Credentialed Evaluators participating in the research have been working in evaluation for over 20 years (33%).

• Close to one-third of self-employed research participants reported that they have been working in evaluation for more than 20 years (31%).

Length of Time Working in Evaluation

Less than 2 years, 10%

2 to 5 years, 20%

6 to 10 years, 22%11 to 15 years, 17%

16 to 20 years, 10%

More than 20 years, 18%

Don't know, 4%


Q.D7. “How long have you worked in evaluation?” (n=405)


ONP39TOL-24 Page 12

3.1.9 Incidence of Credentialed Evaluators

One-third of respondents are not Credentialed Evaluators (33%), close to one-third are CEs (31%) while the remaining one-third are either thinking of becoming one (25%), in the process of applying (8%) or their application is pending (1%).2

Figure 9

• Sub-populations reporting the highest incidence of CEs included those working in the field of education (49%), those working in the private sector (48%) and respondents with a Ph.D. (46%).

2 It should be noted that CEs actually represent approximately 20 percent of CES membership and that the proportion of CEs participating in this survey is higher.

Incidence of Credentialed EvaluatorsD8a) “Are you a Credentialed Evaluator (CE) with the Canadian Evaluation Society?” (n=405)

0%

10%

20%

30%

40%

No Yes, I am aCredentialed

Evaluator

No but amthinking ofapplying

Currently inthe processof applying

Applicationis currently

pending

Don't Know

33%31%

25%

8%

1% 2%



ONP39TOL-24 Page 13

3.1.10 Anticipated Time Frame for Applying for Credentialed Evaluator Designation

We asked those considering applying for CE designation when they thought they would pursue this and one-half of respondents said in 1 to 2 years (48%). One-quarter said in the next year (23%). The balance either said they would be working towards this designation in more than two years (12%) or did not know (18%) when this would happen.

Figure 10

Anticipated Application Timeframe for Potential Credentialed Evaluators

In next year, 23%

1 to 2 years, 48%

More than 2 years, 12%

Don't know, 18%

Base: Those Considering Becoming Credentialed Evaluators

Q.D8b) “When do you anticipate applying to become a Credentialed Evaluator?” (n=101)


ONP39TOL-24 Page 14

3.2 Professional Development The ability to access different types and sources of professional development can play an instrumental role in attracting individuals to become members of professional organizations or in retaining members. The survey examined CES members’ participation in and preferences for different types of training and professional development opportunities. In addition, the average amount spent on professional development was explored. Research participants were also asked to indicate their preferred delivery mode and to identify professional development topic areas that would be of interest to them.

3.2.1 Average Time Spent on Evaluation-Related Professional Development

In a typical year, over one-half of CES members surveyed (54%) spend five days or more on professional development. One-quarter spend three to four days (25%) while one-in-seven (15%) spend one to two days on training or professional development. Only three percent of respondents spend less than one day on professional development. One of the suggestions put forth at the end of the survey was to increase the range of options for this question to over five days because some felt this ceiling was too low.

Figure 11

• Two-thirds of respondents who reported that virtually all their work is evaluation-related (68%) and a similar proportion of survey participants with Ph.D.s (70%) said they spend five days or more each year on professional development.

• Survey participants who were International members (85%), Manitobans (76%) and/or Credentialed Evaluators (66%) were also more likely to spend five days or more annually on professional development compared to those who were users of evaluation results (48%).

Average Time Spent on Evaluation-Related Professional Development

None, 2%Less than one day,

3%

1 to 2 days, 15%

3 to 4 days, 25%

5 days or more, 54%

Don't know, 1%


Q.B1. “In a typical year, approximately how much time do you spend on evaluation-related professional development (including participation in courses and conferences, and self-directed activities like reading professional journals)? (n= 405)


ONP39TOL-24 Page 15

3.2.2 Appeal of CES Training Formats

The most popular training format among over three-quarters of CES members participating in the survey was face-to-face workshops and courses (78%) followed by online webinars or moderated courses (72%). Online self-directed courses were appealing for around three-in-five respondents (62%) as were blended online and face-to-face courses (60%).

Figure 12

• Less than one-half of International members participating in the research found face-to-face workshops and courses appealing (46%).

• Only one-half of Quebec respondents found the online webinars or moderated courses appealing (53%) calling into question whether or not this is a language-related issue.

Appeal of CES Training Formats

0%

20%

40%

60%

80%

100%

Face-to-faceworkshops and

courses

Online webinars ormoderated courses

(group work)

Online self-directedcourses

Blended online andface-to-face

courses

48% 43%35%

25%

30%29%

27%35%

78%

72%

62% 60% SomewhatAppealing

VeryAppealing

B2. “How appealing are the following CES training formats for you?” (n=405)

* Base: Total respondents


ONP39TOL-24 Page 16

3.2.3 Members’ Interest in Pursuing Selected CES Courses Over the Next Two Years

The survey examined the appeal of specific CES courses among respondents and found relatively high levels of interest in pursuing a number of these courses, particularly those scheduled to be available online soon.

Figure 13

• Respondents who have worked in evaluation for five years or less expressed above-average levels of interest in pursuing a number of these evaluation courses, particularly CES’s Evaluation Theories and Models course scheduled to be available online soon (80%).

Members’InterestinPursuingSelectedCESCoursesOvertheNextTwoYears

Q.B3. “CES has developed a few of its own evaluation courses which are available for Chapters to deliver, and other will soon be available online. Please indicate if you are interested in taking any of the following courses over the next two years?” (n= 405)

0% 20% 40% 60% 80%

CES' Data Quality Management (Onlinesoon)

CES' Evaluation Theories and Models(Online soon)

CES' Interpreting & Using QuantitativeResults (Online soon)

CES' Evaluability Assessment (Onlinesoon)

CES' Mentoring & Coaching forEvaluators (Online soon)

CES' Survey Research (In-persontraining)

CES' Logic Models (In-person training)

CES' Essential Skills Series: Blendedonline and in-person)

CES' Essential Skills Series: Introductionto Evaluation (In-person training)

63%

62%

60%

58%

47%

36%

26%

16%

14%



ONP39TOL-24 Page 17

3.2.4 Appeal of Specific Topic Areas for Training - Highest Levels of Appeal

We also looked at the relative appeal of different topic areas that members might be interested in learning more about over the next two years and found that around one-half of members surveyed expressed an interest in several topics such as performance measurement system design and measurement (47%) while around a third were interested in topics like triangulation/combined analysis of various data sources (36%).

Figure 14

• Respondents who have worked in evaluation for ten years or more expressed below-average levels of interest in learning more about a number of topic areas presented.

AppealofSpecificTopicAreasforTraining- HighestLevelsofAppeal-

Q.B4. “Which of the following topics are you most interested in learning more about over the next two years?”(n=405 )

0% 10% 20% 30% 40% 50%

Performance measurement system design &measurement

Cost benefit analysis, efficiency and cost-effectiveness

Rapid assessment methodologies

Developmental evaluation, systems, complexity theory

Theory of change

Evaluation culture and organizational change

Report writing, data visualization & presentingevaluation results

Qualitative research methods and analysis

Triangulation/Combined analysis of various datasources

Project management, planning and consulting skills

47%

46%

44%

42%

37%

37%

37%

36%

36%

30%



ONP39TOL-24 Page 18

3.2.5 Appeal of Specific Topic Areas for Training -Lower Levels of Appeal

Around three-in-ten survey participants expressed an interest in pursuing training such as case studies in evaluation (29%), qualitative research methods and analysis (28%) and/or formative, summative and process assessments (27%). Other topic areas that were appealing for around a quarter of respondents each included subject areas like communication skills (26%) or cross-cultural effectiveness (23%). Around one-fifth of members surveyed expressed an interest in learning more about gender-based analysis (21%) while slightly fewer were keen to learn more about ethics and professional evaluation standards (17%) or literature reviews and secondary data sources (15%).

Figure 15

AppealofSpecificTopicAreasforTraining-LowerLevelsofAppeal-

Q.B4. “Which of the following topics are you most interested in learning more about over the next two years?”(n=405)

0% 10% 20% 30% 40%

Case studies in evaluation

Qualitative research methods and analysis

Formative, summative & process assessments

Communication skills**

Cross cultural effectiveness

Gender-based analysis

Ethics and professional evaluation standards

Literature reviews & secondary data sources

Other

(None of these)

Don't know

29%

28%

27%

26%

23%

21%

17%

15%

3%

5%

5%

Base: Total Respondents** (Includes: communication, negotiation, conflict resolution, collaboration, diversity, cultural sensitivity)


ONP39TOL-24 Page 19

3.2.6 Average Annual Expenditure on Evaluation-Related Professional Development

A look at average annual expenditure on evaluation-related professional development reveals that around one-third of CES members surveyed spend less than $1,000 annually (32%), one-third spend more than $1,000 annually (32%) while the balance either don’t know (20%) or declined to comment (17%).

Figure 16

• A look at average spending on professional development among research participants offering a response to the question reveals that private sector ($1,368) and self-employed members ($1,308) spent more than members in general ($1,181) whereas those employed by the government ($1,073) reported the lowest average annual spending on professional development.

• Credentialed Evaluators participating in the research were among those most likely to spend $2,000 or more annually on professional development (26%).

AverageAnnualExpenditureonEvaluation-RelatedProfessionalDevelopment

Q. B6. “In a typical year, approximately how much in total is spent by you, your employer and/or other funding sources on course or conference costs related to your own personal evaluation-related professional development?” (n=405)


0%

10%

20%

30%

Zero Less than$500

$500 to$999

$1,000 to$1,999

$2,000 ormore

Don't know Refused/Notstated

8%

11%13%

16% 16%

20%

17%


ONP39TOL-24 Page 20

3.2.7 Sources of Funding for Evaluation-Related Professional Development

Two-in-five CES members participating in the survey (40%) pay for their own professional development and/or conference costs while around one-third of respondents say their employer covers all these expense (31%). One-fifth of CES members surveyed reported that a portion of their upgrading costs are covered by their employer (24%).

Figure 17

• One-half of respondents working for the government (47%) reported that their employer covers all evaluation-related professional development and/or conference costs.

• Four-in-five self-employed survey participants said they cover all their own professional development and conference costs (83%).

Sources of Funding for Evaluation-Related Professional Development

I cover all my own costs, 40%

My employer covers all costs, 31%

My employer covers a portion of my costs,

24%

Don't know, 4%


Q.B5. “Do you pay for your evaluation-related professional development courses and/or conference costs yourself or are these costs covered by your employer?” (n= 405)


ONP39TOL-24 Page 21

3.2.8 Attendance at CES National Conferences

Attendance at past CES conferences was highest when the event was held in Ottawa in 2014 when three-in-ten survey participants were in attendance (30%). Around one-quarter of members surveyed attended the CES conferences hosted in Toronto in 2013 and/or in Montreal in 2015 (24% each) while just slightly fewer went to the conferences held in the Maritimes including this year’s conference in St. John’s (22%) and the CES national conference held in Halifax in 2012 (15%).

Figure 18

AttendanceatCESNationalConferencesQ.B7 “The CES national conference is an opportunity for professional development and networking. Which of the

following CES national conferences did you attend in the past five years?” (n=405)

0% 10% 20% 30% 40% 50%

Halifax 2012

Toronto 2013

Ottawa 2014

Montreal 2015

St. John's 2016

None of these

Don't know

15%

24%

30%

24%

22%

44%

3%



ONP39TOL-24 Page 22

3.2.9 Likelihood of Attending 2017 CES Annual Conference in Vancouver

Close to one-half of CES members participating in the survey (47%) say it is likely they will be going to the 2017 Annual CES Conference to be held in Vancouver from April 30 to May 3, 2017.

Figure 19

• Close to three-in-five Credentialed Evaluators participating in the survey said it was likely they would be attending the 2017 Annual CES Conference (58% including 33% who said it was very likely they would be there).

• Among those surveyed, over two-thirds of members with a multi-sector focus (68%) anticipate attending the 2017 Annual CES Conference in Vancouver, B.C. compared to just two-in-five respondents working for the government (38%).

Likelihood of Attending 2017 CES Annual Conference in Vancouver

0%

20%

40%

60%

Likely Unlikely Don't know

21% 19%13%

26%

20%

47%

39%Somewhat

Very

B8. “How likely is it that you will attend the 2017 Annual CES Conference in Vancouver, BC (April 30 to May 3, 2017)?” (n=405)

* Base: Total respondents


ONP39TOL-24 Page 23

3.3 Appeal and Usage of CES Member Benefits Respondents were asked to gauge the relative appeal of various product and service offerings and to indicate whether or not they had ever accessed these member benefits.

3.3.1 Appeal and Usage of CES Member Benefits - Knowledge and Research Resources

The knowledge and research resources offered by CES were generally well-regarded by over one-half to three-quarters of those members participating in the survey. Usage of these resources was slightly lower, however, with close to two-thirds of respondents (63%) accessing the Journal of Canadian Program Evaluation and one-half using the grey literature database (52%). Only one-quarter of members surveyed, on the other hand, had ever used the online evaluation sourcebooks (27%).

Figure 20

• Knowledge and research resource offerings of CES were generally well-used by respondents who have been CES members for 10 years or more but not as well-accessed by respondents who have been members for less than three years (86% versus 62% respectively).

AppealandUsageofCESMemberBenefits- KnowledgeandResearchResources-

Q. A 13/14. “CES provides members with a variety of programs, benefits and service perks (eg. Group-rate discounts). Please let us know which of the following benefits and services available to CES members are valuable for you. (CHECK ALL THAT APPLY) Also indicate which of these you have ever used.(CHECK ALL THAT APPLY)” (n=405)


0%

20%

40%

60%

80%

Immediate access tothe Journal of

Canadian ProgramEvaluation

Grey literaturedatabase

Online access toevaluation

sourcebooks

None of theseknowledge and

research resources

77%71%

57%

6%

63%

52%

27%21%

Valuable Used


ONP39TOL-24 Page 24

3.3.2 Appeal and Usage of CES Member Benefits - Professional Development

Close to three-quarters of survey participants found professional development offerings like the Chapter Learning Events (77%), Annual CES Conference and Webinars (74% each) to be valuable and over half of members surveyed had accessed these three professional development offerings in the past. Access to the roster of Credentialed Evaluators and CES Mentoring Initiative were somewhat less popular (41% each) and less well-used (29% and 7% respectively).

Figure 21

• Usage of several professional development member benefits was significantly higher among survey participants who have been CES members for 10 years or more compared to those who have been members for less than three years. This was particularly evident for the Annual CES Conference (86% versus 34%) and for chapter learning events (75% versus 45%).

AppealandUsageofCESMemberBenefits- ProfessionalDevelopment-



0%

20%

40%

60%

80%

Chapter learningevents

Annual CESconference

CES Webinars Roster ofCredentialedEvaluators

CES MentoringInitiative

None of theseprofessionaldevelopment

offerings

77% 74% 74%

41% 41%

1%

62% 60% 58%

29%

7% 8%

Valuable Used


ONP39TOL-24 Page 25

3.3.3 Appeal and Usage of CES Member Benefits - Perks and Group Member Discounts

When we look at the perks and group member discounts, what is striking is that even though one-in-seven to one-half of members surveyed find some of these perks to be valuable, they are generally not very well-used at all. In fact, one-half of respondents (53%) reported that they had never used any of these perks and group member discounts.

Figure 22

• Two-thirds of respondents who expressed low levels of satisfaction with CES (68%) and a similar proportion who have been CES members for less than three years (64%) indicated that they have never used any of these perks and member discounts.

AppealandUsageofCESMemberBenefits- PerksandGroupMemberDiscounts-



0%

20%

40%

60%

Reducedrates for

membershipwith AEA or

AES

Discountsfrom

publishers

DiscountsthroughStaples

Discountrentals withEnterprise &

National

Liabilityinsurancewith Hub

International

Discounts forProvalis

Researchproducts

Home & autoinsurance

withT.D./Meloche

Monnex

None ofthese perksand groupmember

discountsoffered

52%

38% 36%

27%

18%15% 14%

21%

30%

9%4% 4% 4%

0%3%

53%

Valuable Used


ONP39TOL-24 Page 26

3.3.4 Suggested Improvements to CES National Membership Perks, Products or Services

One-half of members participating in the survey did not know how the member perks and services could be improved (50%) and one-quarter did not think any changes were needed (24%). However, among those who offered up suggestions for improvement, the most common comment was to promote these more (13%). Less than one-in-ten respondents felt it would be valuable to have discounts for Professional Development resources or software (7%), discounts on membership fees (3%), more perks for Credentialed Evaluators (2%) or travel discounts (1%).

Figure 23

• Two-thirds of respondents expressing low levels of satisfaction with CES (65%) were unsure what improvements could be made to enhance the perks, products or services available to CES members. This suggests that further research such as focus groups might be valuable to gain insights into those membership services deemed valuable by this cohort.

Yes, 26%

No, 24%

Don't know, 50%

Q.A15. “Are there any improvements that could be made or additional national membership perks, products or services that you think would be valuable for CES members?” (n=405)


Suggested Improvements to CES National Membership Perks, Products or Services

0% 5% 10% 15%

Travel discounts

More perks forCE's

Discounts forfees/memberships

Discounts for PDresources/software

Promotemore/better

awareness of perks

1%

2%

3%

7%

13%


ONP39TOL-24 Page 27

3.4 Attitudes Towards Being a Member of the CES The following section examines members’ attitudes towards the CES and assesses the value they receive from being a member of their professional organization. The following graphs only include those with opinions (ie. “don’t know” and “not stated” have been removed). The graphs are presented in order from highest levels of support to lowest levels of support.

3.4.1 Attitudes Towards Being a Member of the CES – Overall Participation

Around three-quarters of members surveyed agreed that they would encourage others to join CES (74%) while just slightly fewer agreed that they were aware of ways and opportunities to get involved (71%).

Figure 24

AttitudesTowardsBeingaMemberoftheCES– OverallParticipation-

Base: Among those with opinions

0% 20% 40% 60% 80% 100%

I would encourage others to joinCES. (n=378)

I am aware of ways andopportunities to get involved.

(n=389)

74%

71%

20%

14%

7%

14%

Agree Neutral Disagree

Q. C3. “Thinking about your involvement with CES, please indicate if you agree or disagree with the following statements. Let’s use a 5-point scale where a “1” means you “strongly disagree” and a “5” means you “strongly agree” with the statement.”


ONP39TOL-24 Page 28

3.4.2 Attitudes Towards Being a Member of CES - Perceived Value and Benefits

Around three-in-five research participants agreed that being involved with CES has helped them achieve some of their professional goals (61%) while just slightly fewer agreed that CES gives members good value for the dues they pay (55%).

Figure 25

• Only one-half of respondents aged 36 to 45 years who expressed an opinion on the subject agreed that being involved with CES has helped them achieve some of their professional goals (50%).

• Though relatively small in number, International members participating in the study (92%) were significantly more likely to concur that being involved with CES has helped them achieve some of their professional goals than individual members (58%).

• The view that CES gives members good value for the dues they pay was more prevalent among senior members than students participating in the survey (81% versus 30% respectively).

AttitudesTowardsBeingaMemberoftheCES– PerceivedValueandBenefits-


0% 20% 40% 60% 80% 100%

Being involved with the CES hashelped me achieve some of my

professional goals. (n=372)

CES gives members good valuefor the dues they pay. (n=374)

61%

55%

21%

28%

18%

17%


Q. C3. “Thinking about your involvment with CES, please indicate if you agree or disagree with the following statements. Let’s use a 5-point scale where a “1” means you “strongly disagree” and a “5” means you “strongly agree” with the statement.” e


ONP39TOL-24 Page 29

3.4.3 Attitudes towards CES Membership– Networking and Connections to CES

Over one-half of CES members surveyed indicated that they have made valuable connections through CES (54%) and that CES welcomes their contribution and listens to their ideas (52%).

Figure 26

• Among research participants with an opinion, a significant proportion of those who have been CES members for ten years or more (69%) and/or Credentialed Evaluators (68%) agreed that they have made valuable connections through the CES.

• Only two-in-five respondents who have worked in evaluation for five years or less agreed that CES welcomes my contribution and listens to my ideas (40%).

AttitudesTowardsBeingaMemberoftheCES– NetworkingandConnections-


0% 20% 40% 60% 80% 100%

I have made valuable connectionsthrough the CES. (n=372)

CES welcomes my contributionand listens to my ideas. (n=268)

54%

52%

24%

37%

22%

12%




ONP39TOL-24 Page 30

3.4.4 Attitudes Towards CES Membership- Making a Difference

Less than one-half of members surveyed agreed that CES makes them feel valued and appreciated for their efforts (43%) and even fewer concurred that their contribution to CES is making an impact on evaluation (28%).

Figure 27

• Research participants working in the private sector were more likely to agree that CES makes them feel valued and appreciated for their efforts compared to those working for the government (59% versus 33% respectively).

• Survey participants who are Credentialed Evaluators and those who have been CES members for over 10 years (62% each) were also more apt to feel that CES makes them feel valued and appreciated for their efforts compared to just one-quarter of those who have been CES members for less than three years (26%).

• Members working in the private sector who participated in the survey were more likely to agree that their contribution to CES is making an impact on evaluation than those working for the government (45% versus 22% respectively).

AttitudesTowardsBeingaMemberoftheCES–MakingaDifference-


0% 20% 40% 60% 80% 100%

The CES makes me feel valuedand appreciated for my efforts.

(n=315)

My contribution to the CES ismaking an impact on evaluation.

(n=291)

43%

28%

38%

38%

19%

34%




ONP39TOL-24 Page 31

3.5 Communications Popular methods for obtaining information about CES and satisfaction with current communications from CES are examined in this section of the report.

3.5.1 Sources of Information about CES Activities

We asked survey participants how they typically find out about what is going on at CES and found four-in-five respondents rely on the weekly “E-blast” newsletter (80%) to keep up-to-date about the organization. Roughly three-in-five get emails from their chapter (62%) while two-in-five visit the CES national website (37%) to obtain information about the organization. Around one-in-eight survey participants said they rely either on their Chapter website or on Chapter events to stay informed (13% and 12% respectively). Only a small proportion of respondents get information through word-of-mouth/colleagues (4%) or social media (1%).

Figure 28

• Though usage of communication vehicles such as the “E-Blast” newsletter, emails and the chapter website was below average for less engaged members, the three most popular sources of information about CES among this cohort were the weekly “E-Blast” newsletter (74%), emails from their Chapter (53%) and the national website (33%).

SourceofInformationAboutCESActivitiesQ.A2. “Thinking about how CES communicates with its members, how do you typically find out about what is

going on at the CES? (CHECK ALL THAT APPLY)” (n=405)

0% 20% 40% 60% 80% 100%

Weekly emails on CES nationalnewsletters ("E-Blast")

Emails from my chapter

CES national websitehttp://www.evaluationcanada.ca

Chapter website (Eg. CES-PEI orCESBCY etc)

At Chapter events and activities

Word ofmouth/Colleagues/Members

Social media

80%

62%

37%

13%

12%

4%

1%



ONP39TOL-24 Page 32

3.5.2 “E-Blast” - CES National Weekly Newsletter Readership

Three-in-five survey participants read the CES National newsletter every week (58%) while one-quarter of respondents said they read it at least once a month (25%) for a total of 83% that are reading it at least monthly.

Figure 29

• Less than three-in-ten CES members surveyed with less than a Master’s degree read the “E-Blast” newsletter on a weekly basis (28%).

“E-Blast”- CESNationalWeeklyNewsletterReadership

Q.A3a) “The CES publishes a weekly national newsletter (“E-Blast”) for members every Sunday which is distributed electronically to share news, announcements and articles on evaluation. How often, if ever, do you read the Sunday newsletter?” (n=405)

0% 20% 40% 60%

Every week

At least once a month

At least once every six months

At least once a year

Rarely or never

Don't know

58%

25%

5%

1%

8%

3%



ONP39TOL-24 Page 33

3.5.3 Suggested Improvements to “E-Blast” Newsletter

Over three-quarters of those surveyed said that either the newsletter did not need to be changed (41%) or they did not know (37%) what changes should be made. Among the roughly one-fifth of survey participants (22%) who felt changes would be welcome, several suggestions were put forth on an open-ended basis. The most popular request was to make the content more relevant (7%) followed by a similar proportion of survey participants who wanted the newsletter to be easier to access (6%). Four percent of respondents wanted to see it sent around less often while an equal proportion wanted the format improved (4% each). A small number of respondents felt it would be useful to offer highlights (2%).

Figure 30

Yes, 22%

No improvement needed, 41%

Don't know, 37%

Q.A3b) “How could this “E-Blast” newsletter be improved, if at all?” (n=405)


Suggested Improvements for “E-Blast” Newsletter

0% 5% 10%

Offer highlights

Improve format/Ease ofnavigation

Send less often

Make it easier toaccess/log in

Improve content/Makeit more relevant

2%

4%

4%

6%

7%


ONP39TOL-24 Page 34

3.5.4 Perceived Importance of Emails on CES Employment-Related Opportunities

Three-quarters of survey participants felt the employment-related emails sent out by CES were important (75% including 35% who thought they were “very important”). Only thirteen percent felt they were not important while a similar proportion offered a more neutral view (11%).

Figure 31

• Nine-in-ten research participants earning less than $75K annually said the employment-related emails sent out by CES were important (90% including 55% who said they were “very important”).

PerceivedImportanceofEmailsonCESEmployment-RelatedOpportunities

Q. A3c) “CES also sends out regular emails with employment-related opportunities in evaluation (ie. Daily, upon submission by anyone and moderated). How important is this information for you?” (n=393)

Base: Among Applicable Respondents

0%

20%

40%

60%

Very important(helps me stay

on top ofopportunities)

Somewhatimportant (evenif I do not always

use it)

Neitherimportant norunimportant

Not veryimportant (I can

find this infoelsewhere)

Not at allimportant (Donot read them

and am unlikelyto)

35%40%

11%6% 7%


ONP39TOL-24 Page 35

3.5.5 Visitation to CES National Website

Around one-half of those surveyed visit the CES National Website at least once a month (46%) while two-in-five survey participants access the national website at least once a year (39%). One-in-ten members check out the national website on a weekly basis (11%).

Figure 32

• Three-in-five highly-satisfied survey participants visit the CES national website at least once a month (61%) and an additional 15 percent access the site on a weekly basis.

• Conversely, a significant proportion of respondents expressing lower levels of satisfaction with CES access the national website infrequently (48% “at least once a year”).

VisitationtoCESNationalWebsiteQ.A4. “CES has recently rebranded and updated its website to include information for the general public + an

intranet with information only available to members (jobs, grey literature, etc.) + a new membership management platform which members can access to update their profile or apply for and maintain their CE designation. How often, if ever, do you visit the CES national website? (n=405)


0%

20%

40%

60%

At least once amonth

At least once ayear

At least once aweek

Rarely or never Don't know

46%

39%

11%

4%1%


ONP39TOL-24 Page 36

3.5.6 Main Uses of CES National Website

Two-thirds of CES members participating in the survey reported that they visit the national website for information/literature/resources (65%) while three-in-five accessed the site for the calendar of events/activities/webinars (57%). One-half of respondents were seeking out Annual CES conference information (51%) while the fourth most popular reason for visiting the CES national website was to find out about the Credentialed Evaluator designation (28%). Close to one-quarter of members were either updating their member profiles (25%), seeking information on evaluation competencies (24%) or seeking information about CES as an organization (24%). Around one-fifth of survey participants said they were either entering their maintenance hours or accessing member services or discount programs (18% each). Less than one-in-ten members surveyed used the website to contact the CES Secretariat (6%).

Figure 33

MainUsesofCESNationalWebsiteQ.A5. “What are your main uses of the CES national website? (CHECK ALL THAT APPLY)” (n=390)

0% 20% 40% 60% 80%

Evaluation-related info/Resources

Calendar of events/webinars

Annual CES conference information

CE Program information

Updating my member profile

Evaluation competencies info

Info about CES as an organization

Entering my CE maintenance hours

Member services and discounts

Contacting the CES Secretariat

65%

57%

51%

28%

25%

24%

24%

18%

18%

6%

Base: Website Visitors *Multiple mentions accepted


ONP39TOL-24 Page 37

3.5.7 Suggested Improvements for CES National Website

One-half of website visitors surveyed said they did not know how the website could be improved (49%) while one-third said that no improvement was needed (32%). For the roughly one-fifth of website visitors who offered suggestions for improving the website, the most common recommendation among one-in-ten website visitors was to make it easier to log in or access information (11%). Other suggestions offered by a small number of respondents each were to improve the format (4%), improve the content (3%), have more or better links (2%) or include more contact information (1%).

Figure 34

Yes, 19%


Don't know, 49%

Q.A6. “How could the CES national website be improved, if at all?” (n=390)

Base: Website Visitors

Suggested Improvements for CES National Website

0% 5% 10% 15%

More contact info

More links/Betterlinks

Improvecontent/Make it more

relevant

Improve format/Easeof navigation

Make easier to login/access info

1%

2%

3%

4%

11%


ONP39TOL-24 Page 38

3.5.8 Perceived Adequacy of Current Communications from CES

When we asked survey participants if they were satisfied with the amount of information they received from their professional society, four-in-five members felt they received the right amount of information from CES (83%). One-tenth of members surveyed felt it was not enough (10%) while five percent thought it was excessive.

Figure 35

• Respondents expressing lower levels of overall satisfaction with CES were more likely to complain that they were not receiving enough information from CES (23%) compared to survey participants who were “somewhat satisfied” (6%) or “very satisfied” (4%) with the organization who shared this same view.

Perceived Adequacy of Current Communications from CES


A7. “Overall, when thinking about the information you receive from CES national and your local chapter, would you say the amount of information you receive is…? (n=405)

0%

20%

40%

60%

80%

100%

Too much The right amount Not enough Don't know

5%

83%

10%3%


ONP39TOL-24 Page 39

3.6 Performance Appraisal of CES in Selected Areas The performance of CES in terms of achieving its stated mandate and providing high quality member services is examined in this section of the report. Only those with opinions are included in the calculations.

3.6.1 Performance Appraisal of CES in Selected Areas – Highest Scores

CES received highest scores for its performance in the following areas: providing professional development opportunities for those involved in evaluation practice (78% of research participants said CES was “performing well”); developing a community of practice for those involved in evaluation practice (74% of members surveyed said they were “performing well”); advocating for increased awareness, understanding and use of evaluation (74% of research participants said CES was “performing well”) and developing networking opportunities (73% of research participants said CES was “performing well” in this area).

Figure 36

• The views of disgruntled CES members participating in the survey varied significantly from those expressing high levels of satisfaction with their professional association. For example, nine-in-ten satisfied members felt CES was doing a good job at providing professional development opportunities compared to less than one-half of dissatisfied respondents (95% versus 47% respectively).

• Virtually all satisfied members surveyed praised CES’s performance in terms of developing a community of practice for those involved in evaluation practice (97%) whereas only two-in-five less satisfied members (38%) were similarly impressed with the performance of CES on this front.

PerformanceAppraisalofCESinSelectedAreas– HighestScores

Q. A8. “How well do you think CES, as a national professional organization, is performing in each of the following areas? Let’s use a 1-5 scale, where “1” means CES is performing “very poorly” and “5” means they are performing “very well” in this area.” (n=405*)

0% 20% 40% 60% 80% 100%

Providing professional developmentopportunities for those involved in

evaluation practice (n=391)

Developing a community of practice forthose involved in evaluation practice

(n=375)

Advocating for increased awareness,understanding and use of evaluation

(n=328)

Developing networking opportunities(n=380)

78%

74%

74%

73%

11%

18%

16%

16%

11%

9%

10%

12%

Performing well (4-5) Neutral (3) Not performing well (1-2)

* Base: Those with opinions. Respondents indicating “don’t know” have been removed from the calculations


ONP39TOL-24 Page 40

3.6.2 Performance Appraisal of CES in Selected Areas – Lower Scores

Areas where CES received lower ratings from survey participants are shown in the graph below. This included providing high quality programs and services to members: (70% of members surveyed said they were “performing well”) and promoting evaluation theory and practice to potential users (66% of respondents said they were doing a good job of this). Less than half of members participating in the survey felt CES was doing a good job of expanding and diversifying membership (42% said they were “performing well”) while just over one-third of respondents said they were happy with the way CES was encouraging Chapters to collaborate with each other and with the national organization (36% said CES were “performing well” at this while 29% felt they were “not performing well” in this area).

Figure 37

PerformanceAppraisalofCESinSelectedAreas– LowerScores

Q. A8. “How well do you think CES, as a national professional organization, is performing in each of the following areas? Let’s use a 1-5 scale, where “1” means CES is performing “very poorly” and “5” means they are performing “very well” in this area.” (n=405*)

0% 20% 40% 60% 80% 100%

Providing high quality programs andservices to members (n=358)

Promoting evaluation theory andpractice to potential users (n=323)

Expanding and diversifyingmembership (n=228)

Encouraging Chapters to collaboratewith each other/with the national org

(n=206)

70%

66%

42%

36%

21%

20%

41%

34%

10%

14%

17%

29%

Performing well (4-5) Neutral (3) Not performing well (1-2)



ONP39TOL-24 Page 41

3.6.3 Incidence of Member Contacts with CES Representatives

When we asked research participants if they had contacted any of the following CES representatives over the past twelve months for information, help or guidance, over one-half said they had not done so (52%).

Figure 38

IncidenceofMemberContactswithCESRepresentatives

Q.A16. “Over the past twelve months, have you contacted any of the following CES representatives for information, help or guidance?” (n=405)

0% 20% 40% 60%

Your local CES chapter

The CES National Secretariat

A CES Board member (Chapter ornational)

The CES Executive Director

No, I have not reached out to anyCES representatives

Don't know

27%

26%

19%

9%

52%

1%



ONP39TOL-24 Page 42

3.6.4 Satisfaction with Interactions with CES Representatives

Overall, members participating in the survey expressed high levels of satisfaction with the interactions they had with CES representatives.

Figure 39

SatisfactionwithInteractionswithCESRepresentatives

Q. A17. “(ONLY FOR THOSE SELECTED ASK*:) Please tell me how satisfied you were with the interaction. Let’s use a 1-5 scale, where “1” means you were “very dissatisfied” and “5” means you were “very satisfied”.”

0% 20% 40% 60% 80% 100%

A CES Board member (Chapter orNational) (n=75)

The CES Executive Director (n=38)

Your local CES chapter (n=109)

CES National Secretariat (n=107)

88%

84%

80%

76%

4%

11%

10%

7%

8%

5%

10%

18%

Satisfied (4-5) Neutral (3) Dissatisfied (1-2)

* Base: Those who interacted with these CES representatives


ONP39TOL-24 Page 43

3.6.5 Suggested Improvements for Better Responding to CES Member Queries

When survey participants were asked if they had suggestions for better responding to CES member queries, over one-half of said they didn’t know what could be improved (55%) and one-third said no changes were needed (33%). Those offering suggestions thought it would be better if CES responded in a timelier manner (5%), had more integrated systems between departments (4%), or provided greater clarity on who to contact (3%).

Figure 40

Yes, 12%

No changes needed, 33%

Don't know, 55%

Q. A18. “Do you have any suggestions on how CES can better respond to members’ queries? Please include any specifics as relevant (ie. Secretariat versus Chapter, etc.) (n=405)


Suggested Improvements for Better Responding to CES Member Queries

0% 5% 10%

Clarity re: Who tocontact?

Integrateinfo/dept/systems

More timelyresponses

3%

4%

5%


ONP39TOL-24 Page 44

3.6.6 Overall Member Satisfaction with CES

Three-quarters of CES members surveyed (76%) said they were satisfied with CES including one-quarter who said they were very satisfied (26%). One-tenth of respondents were dissatisfied with the organization (11%) with only two percent expressing strong levels of discontent. Roughly one-in-eight members offered a more neutral view (13%).

Figure 41

Overall Member Satisfaction with CES

0%

20%

40%

60%

80%

100%

Satisfied Neither Satisfied Nor Dissatisfied Dissatisfied

26%13%

2%

50%

9%

76%

11%

Somewhat

Very

A9. “Thinking about why you joined the CES and what you hope and expect to gain from your ongoing membership, please indicate how satisfied you are overall with the CES. This refers to all aspects of the Society and its overall value to you. Are you:” (n=404*)



ONP39TOL-24 Page 45

3.6.7 Opportunities for CES to Better Meet Members’ Professional Needs

One-half of members surveyed offered suggestions on ways CES could better meet their professional needs. The most common suggestions put forth were professional development-related including offering more online workshops and/or courses (11%), offering more professional development (8%) and organizing more advanced courses and professional development opportunities (5%). A small proportion of respondents also thought the organization should be advocating more (5%) and offering more opportunities for mentoring and/or networking (4%). In addition, close to one-fifth of respondents offered other suggestions on how CES could better meet their professional needs (18%).

Figure 42

Yes, 51%


Don't know, 29%

Q.A10. “How can CES better meet your professional needs? Please feel free to suggest any specific services, programs or supports you would like to receive:” (n=405)


Opportunities for CES to Better Meet Members’ Professional Needs

0% 10% 20%

Other

More networking &mentoring…

More advocacy

More advancedcourses & PD

More PD/coursesoffered

More onlineworkshops/courses

18%

4%

5%

5%

8%

11%


ONP39TOL-24 Page 46

3.6.8 Member Satisfaction with Local CES Chapters

Two thirds of members surveyed with opinions were satisfied with their local CES chapters (64%) including one-quarter who were very satisfied (24%). One-in-seven respondents were dissatisfied with their local CES Chapters (14%) while one-fifth offered a more neutral view (22%).

Figure 43

Member Satisfaction with Local CES Chapters

0%

20%

40%

60%

80%

100%

Satisfied Neither Satisfied Nor Dissatisfied Dissatisfied

24% 22%

2%

40%

12%

64%

14%

Somewhat

Very

A11 “And how satisfied are you with your local CES Chapter overall? Are you:” (n=361*)



ONP39TOL-24 Page 47

3.6.9 Member Satisfaction with Local CES Chapters – By Region

A look at member satisfaction across different Chapters reveals significant variations including higher levels of satisfaction among respondents in Alberta & NWT (78%) and Quebec (77%) and lower levels of satisfaction among those in Manitoba (53%), Saskatchewan (52%) and in the National Capital Region and Nunavut (49%). Caution should be taken in interpreting results at the regional level due to small bases.

Figure 44

• Three-quarters of respondents working in the private sector were satisfied with their local CES Chapter (76% including 47% who were “very satisfied”).

• On the other hand, only one-half of survey participants from the National Capital Region & Nunavut (49%) expressed satisfaction with their local CES Chapter.

• One-half of less engaged members participating in the survey were satisfied with their local CES Chapter compared to three-quarters of engaged members (50% versus 73% respectively).

Member Satisfaction with Local CES Chapters - By Region -

0%

20%

40%

60%

80%

TotalCanada(n=361)

BC &Yukon(n=45)

AB &NWT

(n=46)

MB(n=15)

SK(n=21)

ON(n=66)

Nat. CapRegion &Nunavut(n=83)

Quebec(n=26)

Atlantic(n=45)

64%

71%

78%

53% 52%

70%

49%

77%

69%

Base: Those with opinions

A11 “And how satisfied are you with your local CES Chapter overall? Are you:..” *

*Caution: Small bases


ONP39TOL-24 Page 48

3.6.10 Opportunities for Local Chapters to Better Meet Members’ Needs

When asked if they had any suggestions on how their local chapter could improve their services or better meet their needs, two-in-five respondents said they could not think of any ways to achieve this (36%) while one-fifth of those surveyed thought that no changes were needed (22%). One-in-seven respondents would like to see better information or notice of events and courses available for them (14%) while one-in-ten survey participants wanted to see more events or workshops (11%). Less than one-tenth of members surveyed also mentioned they would appreciate more networking opportunities (7%), easier access for remote members (4%), a more active chapter (4%) and/or more advocacy (3%) from their local chapters.

Figure 45

Yes, 42%

No changes needed, 22%

Don't know, 36%

Q.A12. “Do you have any suggestions on how your local chapter could improve its services, or what it should continue doing to meet your professional needs?” (n=405)


Opportunities for Local CES Chapters to Better Meet Members’ Professional Needs

0% 5% 10% 15%

Other

More advocacy

More activechapter

Easier access forremote members

More networkingopportunities

Moreevents/Workshops

Better info/Noticeof events/courses

1%

3%

4%

4%

7%

11%

14%


ONP39TOL-24 Page 49

3.6.11 Intentions Regarding CES Membership Renewals

Close to nine-in-ten members indicated that they intend to renew their membership with CES (87%) while one-tenth of members expressed uncertainty in this regard (10%). Only a very small proportion of members indicated that they do not intend to join CES next year (2%).

Figure 46

Intentions Regarding CES Membership Renewals

Yes, 87%

Don't know, 10% No, 2%


QD 2/D2a) “Are you planning to renew your CES membership next year? Please tell us why not, if you wish.” (n=405)


ONP39TOL-24 Page 50

3.7 Recommendations This final section of the report provides a summary of recommendations put forth by CES members for better promoting membership with the organization, enhancing levels of participation of under-represented groups and general comments regarding the organization.

3.7.1 Suggestions for Promoting Membership with CES

We asked members who would not currently encourage others to join CES what could be done to turn this around. Three-in-five of these disgruntled individuals said they don’t know (61%) while around one-in-ten mentioned that CES could be more welcoming and inclusive (12%), offer more discounts or reduced fees (11%), provide better training and professional development opportunities (9%) and/or demonstrate the value members receive for their membership (8%). A smaller number of respondents suggested additional ways CES could enhance its image such as by offering more/better networking opportunities (5%), being less centralized (4%), doing more marketing/better marketing (4%), or making CE designation maintenance easier (3%).

Figure 47

SuggestionsforPromotingMembershipwithCES

Q.C4. “What could CES do that would make you encourage others to join this professional association?” (n=100)

0% 10% 20% 30% 40% 50% 60% 70% 80%

Be more welcoming and inclusive

Reduced fees/More affordable

Better training and PD resources

Demonstrate value to members/More value formembership

Better/More networking opportunities

Less centralized offerings

More/Better marketing & messages

Make CE maintainence easier

Other

Don't know

12%

11%

9%

8%

5%

4%

4%

3%

4%

61%

Base: Respondents who would not currently encourage others to join CES


ONP39TOL-24 Page 51

3.7.2 Recommendations for Better Engaging Under-Represented Groups

When we asked all survey participants if they had any ideas on how CES could better engage under-represented groups, close to two-thirds of them said they didn’t know how this could be achieved (63%). The most common response was through better communications and outreach (17%). One-tenth of members felt offering discounts/reduced membership fees (9%) would be a step in the right direction while other suggestions included hosting targeted events and activities (6%), being more inclusive and less focused on Credentialed Evaluators (4%), being more accessible (4%) and/or ensuring relevance/value in the activities they offered (3%). Three percent of respondents suggested that CES consult directly with these groups (eg. through focus groups or other methods) to find out what would help them to become more engaged with the organization. One percent of respondents did not think this was necessary. “Why bother” they questioned.

Figure 48

RecommendationsforBetterEngagingUnder-RepresentedGroups

Q.C5. “How can CES better engage under-represented groups (eg. Youth/Students, Francophones, Aboriginal), new partners (eg. Not-for-profit sector, those involved in community planning etc) and others to be more involved with the organization?” (n=405)

0% 20% 40% 60% 80%

Better communications/outreach

Reduce costs/offer discounts

Host targeted events & activities

More inclusive/less CE-focused

Greater accessibility

Ensure relevance/value

Consult with these groups

Unnecessary

Other

Don't know

17%

9%

6%

4%

4%

3%

3%

1%

0%

63%



ONP39TOL-24 Page 52

3.7.3 Final Recommendations and Comments

Three-quarters of members did not offer any final comments while one-quarter of respondents offered comments in a range of areas including those pertaining to CES as an organization (10%), professional development-related comments (4%), suggestions regarding outreach/promotions (4%), survey-related (3%), Credentialed Evaluator-related (2%) or cost-related (2%) comments. One percent of comments praised the efforts of volunteers with the organization.

Figure 49

FinalRecommendationsandCommentsAboutCES

Q.C7. “Do you have any final comments about CES that you would like to share?” (n=405)

0% 10% 20% 30% 40% 50% 60% 70% 80%

No comments

CES as an organization

Prof. development-related

Outreach/Promotions-related

Survey comments

CE program-related

Cost-related

Thanks to volunteers

Other

Don't know

73%

10%

4%

4%

3%

2%

2%

1%

0%

3%


General National Survey of CES Membership · ever General National Survey of CES Membership to...

Documents

Transcript of General National Survey of CES Membership · ever General National Survey of CES Membership to...