An Analysis of Perceptions of Learning Outcome Competencies

of College Graduates in ABET and non-ABET Accredited

Occupational Safety and Health Programs

Darryl C. Hill, Ph.D., CSP

Oakland University

April 28, 2011

Introduction - Definitions

• ABET – The recognized accreditor for college and university programs in applied science, computing, engineering, and technology, is a federation of 28 professional and technical societies.

• Educational Objectives – Broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve.

• Learning Outcomes – Narrow statements that describe what students are expected to know and be able to do by the time of graduation.

• Occupational Safety & Health – The discipline concerned with preserving and protecting human and facility resources in the workplace (Friend & Kohn, 2001).

Introduction – Background

• There has been a shift in higher education institutional culture from an

emphasis on assessing teaching methodology to assessing student

learning (Allen, 2006) as a way of measuring educational quality.

• This change of paradigm will also make it difficult to assess the quality

because it will be necessary to assess not only the quality of teaching,

but also the quality of learning, which means assessing the performance

of graduates in the world of work. (Hirsch & Weber, 1999, p. 9)

• True assessment of student competence must come from not only the

student’s performance in the undergraduate academic setting; but it

must also come from the program graduates extended experiences

after leaving the undergraduate program. (Palomba & Banta, 2001, p.

156)

Study Hypothesis

• Premise that the skills, knowledge, and behaviors

acquired by graduates from ABET accredited

occupational safety and health (OSH) programs

contribute to a higher degree of learning outcome

perceptions than graduates from non-ABET

accredited OSH programs.

Introduction – Significance

• This research study may assist in the decision-making by the

approximately 30 non-ABET accredited occupational safety and health

programs to actively pursue ABET accreditation.

• The American Society of Safety Engineers (ASSE) may evaluate the

study results to assess learning outcomes within the accreditation

continuous improvement process.

• To assist occupational safety and health professionals in today’s

dynamic workplace, it is necessary to examine accreditation criteria and

standards (Institute of Medicine, 2000) to facilitate student learning.

Literature Review

• Several recently completed studies have addressed assessment of

higher education learning outcomes (Anastasio, 2004; Ferrara, 2007;

Turansky, 2003; Waite, 2004). These studies addressed the success of

student learning outcomes in specific institutions.

• Additional literature exists related to accountability, assessment, and the

impact of accreditation on student learning (Banta & Associates, 2002;

Burke & Associates, 2005; Palomba & Banta, 2001; Tanner, 2005).

These studies focused on the effect of the accreditation process on the

learning parameters of the students in educational institutions.

Literature Review – Theoretical Bases of

Curriculum and Learning Outcomes

• Curriculum theory is the interdisciplinary study of educational

experience.

• Curriculum theory seeks to provide meaning to the overall educational

significance of the curriculum (Pinar, 2004).

• Catherine Cornbleth (1990) sees curriculum as a particular type of

process. Curriculum is what happens in the classrooms, which is further

defined by Cornbleth as “an ongoing social process comprised of the

interactions of students, teachers, and knowledge” (p. 5).

• To better understand curriculum theory and its application to

accreditation, one must understand curriculum ideologies. Schiro (2008)

identifies four curricular ideologies: Social Efficiency, Scholar Academic,

Learner Centered, and Social Reconstruction

Literature Review – Theoretical Bases of

Curriculum and Learning Outcomes

• Social Efficiency ideology promotes the belief that the purpose of

schooling is to efficiently meet the needs of society of training students

to function as future contributing members of the society (Schiro, 2008).

• Scholar Academic ideology believes that history has acquired

knowledge for all cultures which is conveniently organized into

academic disciplines and found at colleges and universities (Schiro,

2008).

• Learner Centered ideology focuses not on the needs of the academic

disciplines or society, but on the needs and concerns of individuals

(Schiro, 2008)

• Social Reconstruction ideology brings awareness to the problems of

society and the injustices done to its citizens, such as those originating

from gender, racial, social, and economic inequalities (Schiro, 2008).

Literature Review

• ABET conducted a study for engineering degree programs to determine the importance of learning outcomes (ABET, 2006). The study titled, Engineering Change (EC2000), indicated that the implementation of the EC2000 accreditation criteria has had a positive impact on engineering programs, student experiences, and student learning.

• Comparison of 1994 and 2004 graduates’ self-reported outcomes show 2004 graduates as measurably better prepared than their counterparts in all nine learning areas assessed.

• Two of the largest three increases – in group skills and societal and global issues awareness – are among the skills frequently cited as critical to the future of engineering and to the future engineers’ success (National Academy of Engineering, 2004).

• The study demonstrated that the accreditation criteria and the continuous improvement process enhanced student learning.

Literature Review: Occupational Safety &

Health Education • Rapid growth in the number of safety curricula, as well as various

influences on them, has caused diversity in curriculum content (Brauer,

1992).

• This diversity creates concern among educators and practitioners

regarding the quality of safety education (Simon & Simon, 1995).

• Safety educators face a dilemma of balancing the scope of curriculum

content with the constraints of a four-year baccalaureate degree

program, especially when one considers that the safety professional’s

role has expanded into areas such as risk management, ergonomics,

and environmental safety (Academic Accreditation Council, 1991).

• To perform their professional functions, the safety professional must

have education, training and experience in a common body of

knowledge. Safety professionals need to have fundamental knowledge

of physics, chemistry, biology, physiology, statistics, mathematics,

industrial processes, business, communication and psychology.

(Adams, 2001, p. 27)

Research Questions

• Question 1: Is there a difference in perceptions of learning outcomes?

• Question 2: Is there a difference in feedback about the programs?

• Question 3: Is there a difference in advice to other students?

• Question 4: Is there a difference with regard to professional

development as determined by the questionnaire survey?

• Question 5: Is there a difference in the perceptions of curriculum theory

used in their programs of study?

Methodology – Approach and Design

• This study used a combined research method, mixing quantitative and

qualitative approaches. This approach is recommended by Coll and

Chapman (2000). A combined quantitative and qualitative research

method would complement each other.

• A survey questionnaire online link was sent to occupational safety and

health program directors who agreed to participate in the study.

• Thirteen colleges and universities agreed to participate in the study (5

ABET accredited and 8 non-ABET accredited). A total of 323 survey

questionnaires (169 – ABET, 154 – non-ABET) were received from the

804 sent by the May 31, 2009 deadline.

• The total surveys returned included 52.3% ABET accredited graduates

and 47.7% non-ABET accredited graduates.

Methodology – Population and Sample

• There are 13 undergraduate and graduate occupational safety and

health programs accredited by ABET. There are 30 non-ABET

accredited occupational safety and health programs.

• Each of the schools in this study is a public university with an average

student population of 15,000. There is an average of 60 students

currently studying in the occupational safety and health programs for the

schools participating in this study.

• Approximately, 2750 students have graduated from the occupational

safety and health programs participating in this study.

Methodology – Instrument Used

• The research instrument used in this study was a 37-item questionnaire

and divided into 5 parts.

• Parts 1-4 of the survey questionnaire were constructed using a four-

point Likert-type scale. The range of the scale is “completely untrue”

represented by 1 to “completely true” represented by 4.

• Part 5 included four open-ended questions to allow respondents to

communicate opinions and engage in personal reflection.

Methodology – Data Collection

• A Survey Monkey link, along with introductory letter, consent form, and

IRB approval, was sent to occupational safety and health program

directors.

• All data was collected in a 6-week timeframe. To prevent multiple

survey submissions, the IP address was identified.

• The response rate for this survey research study was 40.2% based

upon 804 surveys distributed and 323 surveys collected.

• Graduates from an ABET accredited occupational safety and health

program had a mean of 5.52 years experience, while graduates from a

non-ABET accredited program had a mean of 11.06 years experience.

The years of experience difference was statistically significant

(***p<.001).

Research Question 1

• Is there a difference in perceptions of

learning outcomes?

Q1: Mean Comparison of ABET vs. non-ABET by Student

Learning Outcomes

Student Learning

Outcomes

ABET

Mean Rating

non-ABET

Mean Rating

t-test

Function on

multidisciplinary teams

Design and conduct

experiments

Taught math and

science skills

Solve applied science

problems

Understanding of ethics

Use of techniques for

practice

Apply standards and

regulations

Conduct accident

investigation

3.71

3.39

3.57

3.57

3.79

3.56

3.82

3.51

3.66

3.36

3.24

3.34

3.64

3.54

3.68

3.41

.939

.391

3.73***

3.06**

2.37*

.173

1.98*

1.14

Q1: Mean Comparison of ABET vs. non-ABET by Student

Learning Outcomes

Student Learning

Outcomes

ABET

Mean Rating

non-ABET

Mean Rating

t-test

Effective writing skills

Engage in life-long

learning

Apply business skills

Design and evaluate

safety programs

Apply adult learning to

training

Apply principles

through internship

3.66

3.68

3.45

3.56

3.28

3.88

3.46

3.60

3.35

3.51

3.02

3.46

2.62**

1.27

1.21

.664

2.70**

4.76***

*p<.05**p<.01 ***p<.001

Q1: ANOVA Analysis of ABET Program by Student Learning

Outcomes

Learning Outcomes F

Function on multidisciplinary teams .204

Design and conduct experiments .183

Taught math and science skills 10.887**

Solve applied science problems 5.642*

Understanding of ethics 2.685

Use of techniques for practice .011

Apply standards and regulations

*p<.05**p<.01 ***p<.001

3.041

Q1: ANOVA Analysis of ABET Program by Student Learning

Outcomes

Learning Outcomes F

Conduct accident investigation .374

Effective writing skills 3.458

Engage in life-long learning .034

Apply business concepts .646

Design and evaluate safety

programs

.005

Apply adult learning to training 2.203

Apply principles through internship

*p<.05**p<.01 ***p<.001

4.806

Q1 Results: Is there a difference in

perceptions of learning outcomes?

• The means for each of the 14 ABET student learning questions were

higher for graduates of ABET accredited programs than for graduates

from non-ABET accredited programs.

• Seven of the 14 items showed a significant difference (p<.05).

• Results are clearly suggestive of an advantage for ABET accreditation,

but do not provide clear evidence relative to years of experience.

• Applying safety principles through a supervised, cooperative, or

internship experience with a t-test value of t = 4.76, was a significant

relationship (p<.001)

Research Question 2

• Is there a difference in feedback about

the programs?

Q2: What would you do differently about your safety degree

program learning experience (Before) if given the opportunity

to do it over based upon your job performance?

Participants

Responses

ABET

N=94

non-ABET

N=86

Total

N=180

Nothing different

Start in safety career

sooner/research the

safety field sooner

as a career field

Take more business

courses

Take more science

and math courses

Take more

engineering courses

29%

26%

17%

7%

6%

27%

21%

2%

10%

7%

28%

24%

10%

9%

7%

Q2: What would you do differently about your safety degree

program learning experience (During) if given the opportunity

to do it over based upon your job performance?

Participant

Responses

ABET

N=115

non-ABET

N=105

Total

N=220

Take a/more

internships

More involvement in

safety student

organization(s)

/networking

More hands-on (lab)

experience/plant

tours

Take more business

courses

Take more

environmental

courses

19%

7%

11%

6%

8%

39%

23%

13%

13%

10%

29%

14%

12%

9%

9%

Q2: What would you do differently about your safety degree

program learning experience (After) if given the opportunity to

do it over based upon your job performance?

Participant

Responses

ABET

N=95

non-ABET

N=90

Total

N=185

Complete

professional

development

courses

Nothing different

Networking/ASSE

involvement

Acquire safety

certification (CSP)

sooner

19%

18%

13%

7%

37%

29%

19%

22%

27%

23%

16%

14%

Q2 Results: Is there a difference in feedback

about the programs?

• The responses to this research question showed there was a high

satisfaction among occupational safety and health graduates.

• Internships were an important component of the student learning

experience.

• An understanding of business principles and the need for professional

was a high response of occupational safety and health program

graduates. Professional development included professional safety

certifications, safety organization membership, and networking.

Research Question 3

• Is there a difference in advice to other

students?

Q3: Graduates’ Advice to Occupational Safety and Health Students

Participants’

Advice

ABET

N=138

non-ABET

N=126

Total

N=264

Gain

internship/practical

work experience

Engage in life-long

learning

Acquire proficiency

in communication

skills

Recognize

importance in

networking

Acquire business

and management

skills

More environmental

and science courses

48%

9%

10%

11%

12%

5%

41%

13%

10%

8%

4%

4%

45%

11%

10%

9%

8%

4%

Q3 Results: Is there a difference in advice to other

students?

• It was observed that graduates responded that completing an internship

or practical work experience as the most important advice to students

currently enrolled in occupational safety and health programs. This

finding was consistent with research questions one and two that showed

the importance for a supervised, cooperative, or internship experience.

• Another finding was that although most of the general themes response

rates did not differ by much, graduates educated in ABET accredited

occupational safety and health programs advised current students to

acquire business and management skills. This response was higher

than non-ABET accredited graduates.

Research Question 4

• Is there a difference with regard to

professional development as determined

by the questionnaire survey?

Q4: Difference in ASSE Membership between ABET and non-

ABET Graduates

Student Learning

Outcomes

ABET

non-ABET Chi-Sq

Value

Member

Non Member

*p<.05**p<.01

***p<.001

72.2%

27.8%

50.5%

49.5%

24.5***

Q4: Differences in Mean Counts of Certification, Professional

Development Courses, and Articles between ABET and non-

ABET Graduates

Student Learning

Outcomes

ABET

Mean Rating

non-ABET

Mean Rating

t-test

Certifications

Courses

Articles

*p<.05**p<.01

***p<.001

0.94

1.65

0.68

0.63

1.22

0.14

2.71**

1.96*

0.987

Q4 Results: Is there a difference with regard to

professional development as determined by the

questionnaire survey?

• The fourth research question showed the importance for professional

development based upon the quantitative analysis. Professional

development included professional certifications, professional

membership, professional development course attendance, and

published articles.

• Of these four professional development areas, the only difference that

did not achieve statistical significance was the difference in the number

of published articles.

• Professional development supports the ABET accreditation criteria that

graduates have a recognition of the need for and the ability to engage in

life-long learning.

Research Question 5

• Is there a difference in the perceptions

of curriculum theory used in their

programs of study?

Q5: Mean Comparison of ABET accredited graduates and non-

ABET accredited graduates by Curriculum Theory

Student Learning

Outcomes

ABET

Mean Rating

non-ABET

Mean Rating

t-test

Students function as

contributing

members of society

Acquiring

knowledge of

various cultures

Instructors focused

more on my needs

as a student

Program provided

awareness of

problems of society

*p<.05**p<.01

***p<.001

3.41

3.22

3.08

2.68

3.28

3.26

3.14

2.71

1.814

2.469**

2.370**

1.253

Q5 Results: Is there a difference in the perceptions of

curriculum theory used in their programs of study?

• Of the four curriculum theory ideologies – acquiring knowledge of

various cultures and instructors focused on my needs focused on my

needs – were significantly higher among non-ABET accredited

graduates than ABET accredited graduates. These curriculum theory

ideologies included scholar academic and learner centered ideologies.

• This finding is possibly the result of non-ABET accredited occupational

safety and health programs having additional time to focus on

curriculum topics. This finding requires further study.

Findings and Conclusions

• This study showed the need for occupational safety and health

programs to partner with various stakeholders who provide

internship and life-learning opportunities.

• Also, the Schools of Business may partner with occupational

safety and health programs to offer business courses or a

business track.

Limitations

• One of the limitations in this study is the relatively small sample

of occupational safety and health programs.

• A second limitation was the graduate population databases for

the participating schools. The 13 schools had different graduate

database retention times.

Recommendations

• Occupational safety and health programs should encourage students to understand the importance of professional development. Professional development includes membership in professional safety organizations, seeking accredited professional certifications, attendance at professional development courses, and writing articles for professional safety magazines and journals.

• The American Society of Safety Engineers (ASSE), lead society for ABET accredited occupational safety and health programs, should ensure faculty demonstrate competence.

• The findings in this study suggest that there is an opportunity for occupational safety and health programs to partner with the School of Business and that faculty offer business courses for occupational safety and health students.

• Occupational safety and health organizations should increase their marketing campaigns to promote the profession. Targeted marketing and promotions should include high school students and college and university students who are undecided for their program of study.

• This study revealed that occupational safety and health programs should consider ABET accreditation.

Future Studies

• A future study might extend the current study by increasing the number

of schools participating and enlarging the graduate respondents

encompassed by the study.

• A second area might examine the employer’s perceptions of graduate

learning outcomes.

• A third area might examine the analysis of perceptions of educational

objectives of occupational safety and health program graduates and

conduct a similar comparative study.

• A fourth possible study is to conduct a qualitative research at one or

more of the occupational safety and health programs to develop a

greater, in-depth assessment of learning outcomes based upon

graduate perceptions. The qualitative study may include focus groups

and interviews.