1

An Evaluation of e-Learning Program: A Case Study at Institute of

Education Development

By

Syed Jamal Abdul Nasir, Mohd Rashidee Alwi ,Jamaliah Said

Institute of Education Development (InED)

Universiti Teknologi MARA, Shah Alam.

ssyed_itm@yahoo.com

Abstract This paper reveals the result of 322 distance learners’ perception towards e-learning program conducted by UiTM. Generally the respondents rated above average for all aspects of distance learning program irrespective of gender, program of studies, income and occupation. Students’ gender also did not show any difference in their perception. Similarly, semester of studies too, did not indicate any significant difference except their perception towards lecturer. However students’ semester of studies showed significant difference towards lecturer, module and physical. Gender, income and semester of studies did not show any relationship to students’ perception towards all aspects of distance learning program. However students’ program of studies showed significant relationship towards their perceptions of the program. Students’ CGPA showed negative relationship with all aspects of distance learning program. Keywords: e-Learning and evaluation

Introduction

UiTM being one of the established public universities in Malaysia was formed in 1956 which was known as Dewan Latehan RIDA. After a few stages of restructuring, on 26 August 1999 finally it was upgraded into a university status and was known as Universiti Teknologi MARA. The vision is to become a distinguished university aspiring to promote academic excellence in higher education. In its effort to assist those who do not have the opportunity to study fultime, UiTM under one of its department known as Institute of Education Development (InED) offers a variety of programs conducted via different modes. One of the major modes is distance education which first started the program in 1990. Inline with the government vision in encouraging and promoting IT society especially when the Prime Minister launched Multimedia Super Corridor (MSC) in 1996, UiTM took the challenge by shifting its distance education program to electronic distance Education Program which known as Flexible Learning Program (FLP) in January 1998. The Ministry of Education officially launched the FLP program on 26 April 1999. In December 1998 there were only three (3) subjects being offered with seven lecturers conducted the programs with only 72 registered students. Today, the enrollment has increased to 5300 students. E-Learning Model of UiTM The e-PJJ/FLP is designed to take advantage of the developments and advancements in distance learning and education with Information and Communication Technology. These developments and advancements are harnessed into forms of support for students to enhance their learning. Thus InED uses the web as an adjunct of on-line learning which consists of eight main components that are:

2

Figure 1: The Eight Components of Flexible Learning Program (FLP)

Problem Statement UiTM has conducted the e-learning for almost a decade. It has gone into vast development, issues and critics. Thus, it leads to the need for in-depth study towards its perception from the views of adult learners. These perceptions gathered based on feedback derived from studies conducted are crucial to the enhancement and improvement of e-learning. Guidelines and suggestion for improvement resulted from empirical study, will be able to be tailored the needs and interests of the institutions, system providers (servers) and the learners. Therefore several questions need to be answered such as:-

1. What is the student’s perception towards the program?

2. Are there any differences in perception between genders, program of studies, and semester of studies?

3. Is there any correlation between students’ perception and genders, program of studies, semester of studies, age and income?

Research Objectives The specific objectives of the study are as follows:

1. To determine distance learner’s perception towards e-learning programs. 2. To determine differences in perception between genders, program of studies, and

semester of studies.

Self-Instructional

material (SIM)

Support Services

Face to face

Facilitators Learning Center

Library Services

Online discussion and

Forum,

COMPONENTS OF FLP /

e-PJJ UiTM

Online and Supplementary

Learning Materials

3

3. To determine correlation between students’ perception and genders, program of studies, semester of studies, age and income.

Methodology

Research Design This study is a descriptive research carried out among distance learners in the Institute of Education Development (InED), UiTM. A Self developed questionnaire was used to evaluate students’ perceptions towards e-learning programs. The data was gathered just once, making the research a cross sectional study. The secondary data was collected from reviews of prior researchers, journals, reports, articles relating to this study. The factors being evaluated are lecturers, the course, modules, IT infrastructure, physical facilities and support services. Theoretical Framework The theoretical frameworks explain the relationship between gender, programmes of studies, semester of studies, income and academic achievement to students perception towards course, lecturer, module, physical facilities IT infrastructure and support services. The above framework indicates that the focus of the study is to measure student perceptions towards six aspects of e- learning programmes those are:- Lecture. level of lecturers’ commitment, competency and their quality of teaching delivery. Course: Respondents were asked to evaluate on the quality of the course content and to what extent they perceived the syllabus content can help them to improve communication skills, creative skill, problems solving and other knowledge. THEORETICAL FRAMEWORK Figure 1: Theoretical Framework Module: The respondents were asked to evaluate their perception towards quality of module in term of usefulness, organization, clarity, contents, accessibility and the price charge. IT infrastructure: Refer to IT training sufficiency, accessibility of information, virtual library service and accessibility to web-pages. Physical facilities: Refer to respondent’s satisfaction toward the quality of library facility, classroom environment, and teaching equipment.

Perception towards: • Lecturers • Course Content • Modules, • IT Infrastructure

• Physical Facilities Support Services

Gander

Semester of studies

Income

Programme of studies

Academic achievement

4

Support service: Students were asked to assess their perception towards the quality of services provided by the institute. Among the support service covered in the questionnaire were enrolment process, quality of services provided by academic staff, administration of examination and handling of students complaints. Population and Sample A total of 500 questionnaires were distributed to the distance learners of InED, UiTM. The distance learners were from diploma and bachelor level.. The students were randomly selected. Three hundreds and twenty two returned the questionnaires, which was 64.4 percent. Reliability and Validity

The Cronbach Alpha values showed more than .07 which means the reliability of the instrument for each aspect is considered high.

Data Analysis Three statistical procedures were involved in the data analysis in answering the research objectives as follow. 1. To determine distance learner’s perception towards e-learning program. Means scores were used to determine the students’ perception towards six aspects of e- learning programs. 2. Independent t-Test was executed to determine differences in perception between genders, program of studies, and ANOVA for semester of studies.

3 To examine the correlation between students’ perception and genders, program of studies, semester of studies, age and income, Spearman Rank was used.

Research Findings Distances’ Learners Perception Towards e-Learning Program Table 1: Students’ Perception in General

Mean_Lectur

er Mean_CourseMean_Modul

e Mean_IT Mean_facilities Mean_ServiceMean 3.7867 3.8994 3.6372 3.4607 3.34163 3.6810 Rank 2 1 4 5 6 3

Table 1 shows in general, respondents ranked highest for the quality of course content, followed by lecturer, support services, module and IT facilities. The finding shows that respondents agreed quality of physical facilities (such as classroom condition, teaching aid and library facilities) is the lowest in the rank with mean score of 3.34. This perhaps due to unavailability of own building whereby InED keeps shifting their classroom from one place to another. However the mean score for every factor indicates that respondents in general were satisfied with the e-learning program.

5

Students’ Perception by Gender Table 2: Students’ Perception by Gender Gender Lecturer Course Module IT Physical Service male Mean 3.8348 3.9607 3.7281 3.5225 3.4363 3.7155

female Mean 3.7548 3.8619 3.5841 3.4198 3.3978 3.6559

Table 2 exhibits students’ perception by gender. Both groups indicate above average means score for every aspects of distance learning program. However it is interesting to note that male respondents were more satisfied as compared to female counterparts as indicated by higher mean score for every item measured in this study. This may be due to different level of expectation of the two groups as argued by Gurian (2001) that male and female have different set of expectation in mind. Table 3 shows students’ perception by program of studies. The mean score clearly highlight that, degree students showed lower mean scores for every factor measured in this study as compared to the diploma students. This indicates that higher level students rated lower the services provided by the institution as compared to the diploma students. This may be due to higher expectation of bachelor students as the may be holding better position at their work place. Unlike the diploma students who are generally the supporting staff. Schedul 3: Students’ Perception by Program of Studies program Lecturer course module IT Physical Service Diploma

Mean

3.8289 3.9138 3.7104 3.4769

3.4639 3.6800

Mean 3.6884 3.8615 3.4793 3.4129 3.3033 3.6703degree

Table 4: Students’ Perception by Semester of Studies Semester Lecturer course module IT Physical Service 1-2 Mean 4.0159 3.9423 3.7142 3.6438 3.6125 3.73133-4 Mean 3.7670 3.9145 3.6122 3.4299 3.3986 3.61585-6 Mean 3.8248 3.9462 3.6748 3.4509 3.3659 3.64087-8 Mean 3.7124 3.8389 3.6146 3.4522 3.4413 3.7820>9 Mean 3.4400 3.6154 3.4200 3.2000 2.9600 3.6000

Table 4 shows students’ perception by semester of study. Regardless of which semester they were, respondents indicated above average mean scores for all aspects of distance learning program. It is interesting to highlight that junior students seemed to perceive the quality of

6

distance learning program better as compared to their senior in semester 9. Again this may be to higher expectation of senior students as compared to their junior. All students, irrespective of their semester of studies ranked course content higher as compared to other factors. The high quality syllabus offered by UiTM, could explain this phenomenon. Table 5: Students’ Perception by Income income Lecturer course module IT Physical Service

3.7618 3.8601 3.6907 3.4241 3.3773 3.5410

3.7995 3.8853 3.6111 3.4757 3.4714 3.6962

3.7812 3.9678 3.6793 3.4427 3.3284 3.7395

3.7412 3.9690 3.4500 3.5364 3.3455 3.6673

<1000

1001-2000

2001-3500

3501-5000

>5000

3.7778 3.8547 3.6667 3.3889 3.3111 3.9011

Table 5 shows the students’ perception by income. The students from various income backgrounds showed average perception towards all the six aspects of distance learning program. The lower income students perceived better towards physical facilities as compared to higher income students. The higher income would have higher expectation as compared to lower income group that willing to accept lower quality services. Table 6: Students’ Perception by Occupations occupation Lecturer course module IT Physical Service

3.8186 3.8907 3.6880 3.4643 3.4569 3.7021

3.7664 3.8814 3.5504 3.4195 3.3413 3.6221

3.5950 4.1703 3.7929 3.5429 3.3857 3.8255

3.9068 3.9316 3.9667 3.7444 3.7333 3.9259

3.7944 3.8718 3.7167 3.7833 3.9000 3.9333

Public sector

private sector

own business

not working

others

Irrespective of their type of occupations, the students showed above average mean score for every service offered to them. This indicates that they are satisfied with the service provided by InED. The above findings showed that physical facilities were ranked lower as compared to other factors. This implies that, in order to improve image, number of enrollment and meeting learner’s expectation, InED needs to carefully investigate the classroom condition, teaching facilities, library facilities and other physical facilities. One of the factors contribute to this poor physical facilities perhaps is due to unavailability of own building. It is important to highlight that respondents agreed that the course contents is high quality as they strongly felt that the course content is able to enhance their skills at the work place. This could be the quality of syllabus as UiTM keeps reviewing the syllabus to meet current needs of workforce market.

7

In general, respondents agreed that the e-learning program is at satisfactory level as the lowest means score is 3.3413. Likert scale of 1 indicate highly not satisfied and 5 for strongly dissatisfied. Score of 3 point or above indicate that respondents are satisfied with the services. Differences in Perception between genders, program of studies, and semester of studies Table 7: Differences in Perception between genders, program of studies, and semester of studies Sig.(2 tailed) Lecturer Course Module IT Physical

Facilities Service

Gender (t-test)

.161 .084 .060 .190 .630 .425

Program of studies (t-test)

.015 .370 .003 .423 .048 .899

Semester of Studies ( ANOVA)

.015 .388 .825 .502 .246 .407

Table 8: Multiple Comparisons Tukey HSD

Dependent Variable (I) semester (J) semester

Mean Difference

(I-J) Std. Error Sig.

Lecturer 1-2 3-4 .24886 .09763 .083 5-6 .19103 .10131 .327 7-8 .30342(*) .09975 .021 >9 .57587 .23374 .102 3-4 1-2 -.24886 .09763 .083 5-6 -.05782 .07091 .926 7-8 .05456 .06867 .932 >9 .32701 .22226 .582 5-6 1-2 -.19103 .10131 .327 3-4 .05782 .07091 .926 7-8 .11238 .07382 .549 >9 .38483 .22390 .424 7-8 1-2 -.30342(*) .09975 .021 3-4 -.05456 .06867 .932 5-6 -.11238 .07382 .549 >9 .27245 .22320 .739 >9 1-2 -.57587 .23374 .102 3-4 -.32701 .22226 .582 5-6 -.38483 .22390 .424 7-8 -.27245 .22320 .739

* The mean difference is significant at the .05 level. Table 7 shows that male and female students have no significant difference in their perception towards lecturer, course, module, IT, physical and service. The students’ program of studies indicated significant difference in their perception towards lecturer, module and physical facilities

8

but not towards course, IT infrastructure and service. The ANOVA shows that students from different semester of studies showed no significant differences in their perception towards course, module, IT, Physical facilities, and service except towards lecturer. The Tukey HSD shows that students between semester 1 - 2 and 7 – 8 have significant difference in their perception towards lecturer. The students from other semester have not showed any significant difference. Ccorrelation between students’ perception and genders, program of studies, semester of studies, CGPA and income. Table 9: Results of Spearman’s Rho Spearman's rho Sig (2 tailed)

Gender Perception .252 Program of Studies .038 income .877 Semester of Studies

CGPA .176

.023 The result of Spearman’s Rho reveals that there is no significant relationship between students’ gender, income and semester of studies; and perception. However there are significant relationship between students’ program of studies and CGPA; and perception. The result shows that students’ perception depends on their program of studies and CGPA. However the Spearman’s rho does not provide which program and CGPA level have significant relationship to students’ perception. The students’ perception towards distance program was found to have negative relationship to the students’ CGPA. This indicates that the higher achiever students lower perception towards the distance program.

Conclusion

From the findings of the study three conclusions can be made that are:- 1. Generally the students of InED perceive the e-learning program above average standard. 2. Students’ perception independent of gender, income and semester of studies. 3. Students’ perception depend on their CGPA

1. Generally the students of InED perceive the e-learning program at above average standard Obviously all the six elements of the e-learning program rated average by all the respondents. Adult learners as been mentioned by Roger (1981) go back to school or university with high expectation. They expect everything is to be taught by the lecturers or everything is nicely in place. InED with all the limitations such as lack of facilities, like class room, library, computer labs, etc just couldn’t provide all the students’ expectation as they expected. As consequences InED receives a lot of complaints from the students with regards to the improper facilities set up. This could be one of the reasons why the students rated slightly above average of InED e-learning program.

9

2. Students’ perception independent of gender, income and semester of studies. Though gender always been claimed to differentiate between male and female perception on something, but the study found that it did not in line with the claim. This could be the result of an equality of opportunity or treatment by the Malaysian government to the women. In Malaysia gender is not a factor of discrimination in education as in some developing countries. Thus, they have freedom to evaluate things as the male do. The students from different income background too did not show any significant difference in their perception. It means that the six aspects of distance learning program have been genuinely evaluated based on academic perspective regardless of socio-economy background. Seniority as a student also did not indicate any difference in their perception as compared to the junior students. Once again the evaluation was done purely based on academic and quality perspective. 3. Students’ perception depends on their CGPA level Unlike gender, income and semester of studies, the students’ CGPA level did influence the students’ perception. It means the students’ academic achievement differentiates their perceptions. The high achievers showed higher expectation towards the programs as compared to the lower achievers. Logically, the smart students expect more than less smart students.

Recommendations Based on the study, several recommendations can be made as follow:-

1. Recommendation for practice a. Facilities such as classroom, students centers, quality of module, support

service and even the commitment on the part of teaching need to look upon. b. To improve the Learning Management System (LMS) environment to

provide more structure and guidance to students in learning from interaction and group activities. Further research and development are needed to refine the LMS to support and enhance students’ learning using a problem-based model.

2. Recommendation for further research

a. To use multiple methods of study such as qualitative and quantitative research to get genuine input.

c. To involve all students from branches in order to get representative findings.

10

REFERENCES Abdullah, Szarina. (2003). Understanding Adult Learners. Proceeding on the "Seminar

Memahami & Memudahcara Pelajar-Pelajar dewasa e-PJJ UiTM". Kuala Lumpur

Bourne, J. R., McMaster, E., Rieger, J., & Campbell, J. O. (1998). Paradigms for online learning: a case study in the design andimplementation of an asynchronous learning networks course,http://www.aln.org/alnWeb/journal/issue2/assee.htm.

Carswell, L. (2000). Distance education via the Internet: The student experience. British Journal of Educational Technology, 31 (1), 29-46.

Collins, M. (2000).Comparing Web, correspondence and lecture versions of a second- year non-major biology course. British Journal of Educational Technology, 31 (1), 21- 27.

Corrent-Agostinho, S., Hedberg, J., & Lefoe, G. (1998).Constructing problems in a Web- based environment. Educational Media International, 35 (3), 173- 180.

Delling, R.M. (1985, August). Towards a theory of distance education. Paper presented at the ICDE Thirteenth World Conference in Melbourne, Australia.

Fredericksen, E., Pickett, A., Pelz, W., Shea, P., & Swan, K. (2000). Student satisfaction and

perceived learning with online courses: Principles and examples from the SUNY Learning Network. Journal of Asynchronous Learning Network, 14 (2), http://www.aln.org/alnweb/journal/Vol14_issue2/le/Fredericksen/LE- fredericksen.htm

Hislop, G. (2000). Working professional as part-time online learners.Journal of Asynchronous

Learning Networks, 4 (2), http://www.aln.org/alnweb/journal/vol4_issue2/le/hislop/LE-hislop.htm

Hong, K.-S., Ridzuan, A. A., & Kuek, M.-K. (2003). Students' attitudes toward the use of

the Internet for learning: A study at a university in Malaysia. Educational Technology & Society, 6(2), 45-49

Ismail, Yusof. (2002). Evaluation of an Internet-Based, Distance Learning: Undergraduate

Business Program. Paper presented on the Seminar on E-Learning Initiatives in Malaysia, Hotel Pan Pasific, Kuala Lumpur.

Jiang, M., & Ting, E. (1998). Course design, instruction, and students' online behaviors: A study of instructional variables and student perceptions of online learning. Paper presented at the Annual Meeting of the American Educational Research Association, April 13-17, 1998, San Diego, CA.

Jurich, Sonia.(2003).Virtual Education: Trends and Potential Uses. http://www.techknowlogia.org/TKL_active_pages2/SearchCurrent/main.asp?Issu eNumber=19

Kearsley, G. (2000). Online education: Learning and teaching in cyberspace, Belmont, California: Wadsworth.

Klco, Bill & Munson, Kevin. (2000). e-Learning: Revolution or Evolution? www.kmgpinc.com/downloads/kmswp.pdf

11

Kulik, C., & Kulik, J. (1991). Effectiveness of computer-based instruction: An updated

analysis. Computers and Human Behaviour, 7.

Matuga, J. M. (2001). Electronic pedagogical practice: The art and science of teaching and learning online. Educational Technology & Society, 4 (3), http://ifets.massey.ac.nz/periodical/vol_3_2001/matuga.html.

Mokhtar, M., Nordin, N. & Ibrahim, M.K. (2002) The Academic Performance of Electronic Distance Learners: An empirical Evidance from Malaysia. Paper presented at the Seminar on E-Learning Initiatives in Malaysia, Hotel Pan Pasific, Kuala Lumpur.

Motiwalla, L., & Tello, S. (2000). Distance learning on the Internet: An exploratory study. The Internet and Higher Education, 2 (4), 253-264.

Muhd Kamil. (2001). How to Study the e-PJJ Way. Proceeding on the Workshop on e- Learning & e-PJJ, Selangor.

Nakabayashi, Kiyoshi. (2003). E-Learning Initiatives in Japan. Proceeding on the

Seminar on E-Learning Initiatives in Malaysia, Hotel Pan Pasific, Kuala Lumpur.

Oliver, R., & Omari, A. (2001).Student responses to collaborating and learning in a Web- based environment. Journal of Computer-Assisted Learning, 17 (1), 34-47.

Yunus, A.S.(2003). Emergence of Web-Based Teaching and Learning Culture in Universiti Putra Malaysia. www.homestead.com/peoplelearn/WebResearch.html

Rosenberg, Marc. (2002). The DNA of e-Learning. www.internettime.com/store/DNA%20- -%20DRAFT.pdf

Shireen Haron. (2002). Profile of Web-based Learners and Learning Environments in Universiti Teknologi MARA. Paper presented on the Seminar on E- Learning Initiatives in Malaysia, Hotel Pan Pasific, Kuala Lumpur

Speck, M. (1996, Spring). Best practice in professional development for sustained educational change. ERS Spectrum, 33-41.

Syed Othman. (2002). e-LEARNING: A Malaysian Case Study. Paper presented at the Africa- Asia Workshop on Promoting Co-operation in Information and Communication Technologies Development, Kuala Lumpur.

Swan, K., Shea, P., Fredericksen, E., Pickett, A., Pelz, W., & Maher, G. 2000).Building knowledge building communities: Consistencies, contact and communication in the virtual classroom. Journal of Educational Computing Research, 23 (4), 359-383.

Wegner, S. B., Holloway, K. C., & Gordon, E. M. (1999).The effects of Internet-based instruction on student learning. Journal of Asynchronous Learning Networks, 3 (2),http://www.aln.org/alnweb/journal/vol3_issue2/wegner.htm.

12

.

.

13