Building, Testing and Assessing a Learning Management …

Journal of Computing and Information Technology - CIT 13, 2005, 1, 25–41 25

Building, Testing and Assessinga Learning Management System

Luigi Colazzo and Andrea MolinariUniversity of Trento – Department of Computer and Management Science

This paper summarizes the experiences of our researchgroup in managing the full process of building, testingand assessing a Learning Management System for an ed-ucational institution. We will show how we moved acrossthe “make or buy” dilemma that normally educationalinstitutions have to face when deciding to implement asupport for online and distance learning activities. Wewill deal with this theme by referring to systems and pro-totypes developed by our research group. Specifically,we will describe our experiences in realizing systemsfor making didactic material available on the Internet,such as in the Learning Management Systems – LMS.This experience is based on what has been developed,tested, and put to use in some faculties of the Universityof Trento, where non-traditional learning environmentsin which students require more than the traditional face-to-face lessons are becoming ever more important. Theprototypes produced are based on platform-independenttechnologies, and they make it possible to broaden one’shorizons, even for those students, such as lifelong learn-ers, who want freedom from time constraints and alsofreedom from some technological constraints that areimposed by some forms of distance education currentlyavailable on the market. Some extensions to mobile toolslike PDA, tablet PC, cellular phones etc. are presented,with regard to the impact of their introduction on theLMS and, in general, on the information system of theinstitution.

Keywords: Learning Management System, online learn-ing, mobile education.

1. Introduction

The point of departure of our argument is the re-ality, often overlooked by authors, that to teachin a university context it is necessary to usedifferent approaches, depending on the subjectarea covered. For example, teaching history ofphilosophy in a Humanities program is vastlydifferent from teaching a technological disci-pline such as database management in a com-puter science course, without even considering

the content itself. In a history of philosophycourse, the knowledge to transmit is generallyof little importance �rarely more than biograph-ical information on the authors and their rela-tionship to the culture of the time�: what reallycounts is equipping the students to engage incritical discussion on the questions raised in aphilosophical debate of the past, and also to beable to recognize the influence of that debateon issues currently under discussion in philos-ophy. Diagrams and visual material are essen-tially nonexistent, while lectures are fundamen-tal, as are reading and writing.

On the other hand, in a database course the ideasto transmit are numerous and complicated; toknow the essential facts and to understand thetechniques of organizing a database are muchmore important than to develop the capacity todiscuss the current research issues in databasemanagement �these are the typical topics cov-ered in a PhD course�. In this type of courseit is necessary to teach students how to inter-act with complex systems �DBMS� and withprogramming languages. Consequently, somespecific tools which can be used in both teach-ing environments will be of greater or lesserimportance, depending on the subject taught.For example, an electronic discussion forum inthe first case would be essential, while the man-agement of FAQ would be of little importance.In the second teaching situation, a discussionforum could be useful, but a high number ofFAQ is indispensable. Superficially, these di-vergences can be seen merely as different ap-proaches �lectures vs. constructivism�, but it isnot so simple; in reality, teachers in both disci-plines probably adopt a constructivist approach,but the real difference lies in the nature of the

26 Building, Testing and Assessing a Learning Management System

knowledge area. A discussion forum is an ex-cellent tool within a constructivist approach tothe study of a philosophical discipline, and, in-deed, this constructivist approach was alreadyused long before now: the Socratic maieuticwas, in fact, a constructivist approach.

A second issue regards the reasons why it isnecessary to use electronic support tools in uni-versity teaching. The usefulness of using Inter-net to disseminate didactic material stems notonly from the fact that students can take ad-vantage of this learning opportunity from a dis-tance, but also from the fact that this methodworks particularly well with different teach-ing and study schedules. In other words, withthe exception of systems dedicated to distancelearning, where flexibility in organizing studyand other commitments is important for the stu-dent, it is undesirable to maintain distance be-tween teachers and students. This aspect is offundamental importance for lifelong learning,where the educational experience goes beyond�in time and place� the physical educational in-stitution, in both work and personal life. In thissense, the possibilities take on a new form, thestudent-institution relationship changes consid-erably, the interests of the user �student� aremore diversified and less focused on the physi-cal presence of the teacher at a lesson — even adistance lesson — and are more focused insteadon having a well-organized course with easyaccess to and retrieval of the course material.Mobile technologies like cellular phones, PDA,tablet PCs, smartphones, UMTS etc. indicatethat mobile users will need different models oflearning that adapt to their being in movementand to their fragmented use of time.

The validity of distance learning has beenwidelydebated for some time, particularly within theuniversity context �1� �2�, starting from the earlyexperiences with CBL systems �3� until the ad-vent of Internet �4� and the modern integrationsamong various media in real-time during thelesson �5�. This use of e-learning systems dif-fers notably from the typical distance learning�DL� situation. As a consequence, even if thetechnologies used may be identical, the aimsand objectives are significantly different. Theunderlying idea of DL is to compensate for theabsence of the instructorwho, for reasons of dis-tance or time, cannot be in the physical presenceof the learner. Systems of DL aim to adapt tothis reality by working towards four objectives:

� To “virtualize” the instructor’s function �thetraditional lesson of the teacher thus be-comes a registration�;

� To “virtualize” the class �the students are a“class” only in a virtual sense�;

� To prepare learning objects adapted for self-study and for different supports with differ-ent screen resolution, eventually incorporat-ing the standards for creation of didactic ma-terial, such as the IMS project �6�, the LOM�7� and SCORM �8� models, etc.

� To create a feedback channel �negative �corrections, positive � encouragement�.

The issues are inherently different in traditionallearning environments. The instructors and stu-dents enjoy a real and tangible relationship;technology is not used as a surrogate, but ratheras an extension of communication already tak-ing place — it “overflows” into the virtualworld. The situation in this case is substan-tially different from that of distance learning,for several reasons:

� the instructors are “real”;

� the students are “real”;

� study is guided and self-study representsonly one portion of the overall learning ex-perience;

� feedback channels are more frequent and canbe both official and less formal, non-officialin form and content.

In these situations an academic institution suchas a university faculty is represented as an aggre-gate of the didactic processes organized by anacademic institution. Wedealwith this aspect ofthe problem in this paper. We present the largertechnological scenario being referred to in thepaper, briefly presenting learning communitiesand varying communication situations in whichcommunication technologies are merely an ex-tension of natural communication. A course,for example, constitutes a small real commu-nity that gathers for a designated period of timeduring which a teacher, a specific number oftutors and the learners embark on a learningprocess. Also regarding this notion, see �9�,where the idea of community as an outcome ofworking within an online environment is dis-cussed. If everything works according to plan

Building, Testing and Assessing a Learning Management System 27

and schedule, the outcome of the process is anincrease in knowledge for everyone involved. Inmodern terms, it is no accident that the wordscommunity and communication have their rootsin the Latin word communis, meaning to worktogether �com� for a public good �munus�. Acommunity that fails to communicate ceases toexist as a community. In other words, the learn-ing communities that an academic institutionorganizes use technology to broaden their pos-sibilities for communication and, as such, thesystem of e-learning becomes the means to agreater end.

To discuss these issues more deeply, we presentthe research and experiments conducted at theUniversity of Trento in the last four years. Firstwe will look at the characteristics of a client-server service, typical of the Learning Manage-ment Systems previously mentioned, which iscreated from scratch as an alternative to acquir-ing a system already on the market. In this con-text we analyse the importance of the project“Faculty on-line”, specifically noting the or-ganizational peculiarities that lead it to decidewhether to “make” or “buy” the service. In sec-tion three, we demonstrate the necessity for thistype of system to have personalized commu-nication channels for every actor in the educa-tional process. These tools have been developedprincipally to maintain contact with the studentsdeciding to extend their study programme be-yond the standard time period, and who perhapscannot participate actively in the daily life of thefaculty, or for those students having interests orgoals different from those of the traditional stu-dent. In the fourth section we present the sys-tem developed at the Faculty of Economics atthe University of Trento as an example of whatis required in building this type of educationalsystems. The fifth section shows the results ofthe assessment procedure we built in order tounderstand the level of satisfaction in using thesystem reached by all the kinds of users. Lastsection will be devoted to explain the exten-sions toward mobility we are implementing inthe system, thus integrating a traditional LMSwith all devices, tools, models and problemsrelated with people on the move.

2. The Faculties, Central Institutions inUniversity Education

University studies in Italy take place predomi-nantly in public institutions whose characteris-tics and organization are defined by federal law.The study programs offered by the universitiesare grouped by faculties, and the faculties are,therefore, the organizational structures that ma-terially combine educational offer into homo-geneous disciplines �Science, Humanities, En-gineering, Economics, etc.�. It is worth notingthat in this system the faculties are responsibleonly for teaching, while scientific research isorganized by other entities, notably the depart-ments. The departments are designed to be thecenters of research only; therefore, they have noinput in the organization of courses, just as thefaculties have no power to intervene in researchissues and activities. In short, a person in Italyis a member of a faculty only in the role of ateacher and is a member of a department in therole of a researcher. On one hand, this situationintersects with the teaching process �a personcan run different courses in different faculties�while on the other hand it has a definite influ-ence on the administrative management of thefaculty, and, therefore, on what can be called the“back office” of a university faculty. The edu-cational system of a faculty is roughly divisibleinto three areas of management, these being:

� management of the teaching staff,

� administrative area,

� area of didactic management �for example,a LMS�.

The instructional information system based onfaculties assumes that these three areas are heav-ily intertwined, as shown in Figure 1.

Fig. 1. The different aspects of a Faculty-basedInformation System.

The material organization of courses is han-dled by the staff of the dean �back office�; the


staff executes the decisions of the faculty com-mittee regarding educational programs, specificcourses, and teachers. The personnel of the fac-ulty also take care of marketing, as well as ofhandling feedback information on the didacticactivity so as to help the teachers to conductself-evaluations, and also to evaluate the facultyin general. The principal responsibilities of theteachers are coursework and thesis guidance, inaddition to the ongoing work of evaluating stu-dent performance �exams�. It is obvious, then,that what is traditionally considered to be didac-tic material in electronic form �Courseware� isreally part of a much larger instructional systeminto which it becomes integrated.

In order to solve the above problems, the Fac-ulty of Economics of the University of Trentodecided to start the implementation of part of theoverall picture of a Faculty-based InformationSystem. The project took off in late 1999: wefirstly established which functionalities shouldhave been provided and if the system could havebeen found on the market or should have beenbuilt ex-novo. Now we will briefly discuss thislast issue. At the evaluation stage, it was im-portant to investigate costs of building a systemusing internal resources. This was because, al-though there were strong reasons for purchasingan off-the-shelf Learning Management System,such as TopClassTM, Lotus Learning SpaceTM

or WebCTTM, there were many successful in-stances where universities developed a systemby themselves. It was evident that homegrownsystems were often more tailored to a univer-sity’s specific needs. For an analysis of this,see �10� �11�. LMS are being promoted as ameans of simplifying the creation and manage-ment of instructional websites. There are manystudies that seek to compare between specificcourseware management tools �e.g. �12�; �13��,but it seems that the efficiency and effectivenessof these tools with respect to customized Webbased systems is less easy to define. Summariz-ing, we decided for the “make” alternative withthe following motivations.

� The cost: the systems thatmanage, distributeand maintain the on-line learning materialsare high-cost systems that imply also othersecondary costs, like adequate hardware andassistance.

� The skills: LMS are complex to understandand manage, so normally some personnel

should be devoted to their management. Wewere not in the position to use resources forthis task: the Faculty has no personnel to beused and instructed on such environments.

� Some specific end-user needs, in particu-lar the integration with administrative andbureaucratic functions in the perspective ofan integrated Faculty-based Information sys-tem: many of the functionalities that nor-mally relate with the didactic activity of theteacher are closely related with administra-tive tasks that should be carried out in spe-cific way and time, thus constituting a bigconstraint for packaged systems that haveno personalization features or that need adeep knowledge of personalize. Moreover,in a highly competitive market like the oneinvolving educational institutions, having apersonalized system on which we are able toperform every change and improvement weprefer, is undoubtedly a great competitiveadvantage.

� The time: a prototype should have beenready for the second part of the academicyear 2000 �march 2001� with a pilot groupof teachers and students, and a beta versionof the system was to be used by all theteachers and students in the first semesterof 2001. This schedule was not compatiblewith our implementation of an LMS like theones cited above.

For all these reasons, we decided to build thesystem internally, creating onlywhatwas needed.The main objectives of the project were the fol-lowing:

� to allow the teacher to keep a constant re-lationship with his�her students, in the dif-ferent ways that will be explained in the fol-lowing sections;

� to allow the teacher to distribute educationalmaterial to students;

� to keep the data for administrative and re-porting tasks to be performed by the teacherand by the secretary of the Faculty.

3. Functional Requirements for On-LineCourses

The use of the Web as a tool for didactic ac-tivity has proved particularly useful in recent


years �14�. Courses offered through the Web area very common practice now in teaching �15�,offering different opportunities for interactionbetween teacher and student. These systemsare often used to support distance learning, butin this case the objective is different. The aim,in fact, is to use electronic communication toimprove the traditional way of instruction. Thisgoal makes the following assumptions:

� there is a real course that prescribes a sched-ule, meeting place, lectures or other activity�exercises, project work, drafting and com-pleting papers, experiments, etc.�,

� teachers, tutors and students interact regard-ing the coursework,

� all courses have rules regarding how course-work activities must be carried out and re-garding the assessment of results,

� students attend more than one course at atime and, therefore, are doing coursework indifferent disciplines,

� teachers give lessons in different topics andat different levels, possibly in the same aca-demic period,

� teachers can be assisted by a certain numberof tutors.

A system that makes it possible to combine thenormal channels of communication �their exis-tence is presupposed by the system described�with opportunities for electronic communica-tion must respect some general requirements toguarantee a certain level of effectiveness. Theprincipal requirements are configurability, scal-ability, modularity, accessibility, usability andfreedom of choice. We now offer a brief dis-cussion of each requirement.

a) Configurability: Each edition of a coursehas its own history. In fact, courses change inrelation to the classes, time, developments in thefield of study, interests of the teacher as well asthe labormarket. It follows that in a support sys-tem based on electronic communication, eachsingle edition of a course should have its ownvirtual space �for example, the Home page ofa Database course, second semester, academicyear 2001-2002�. The teacher working with thesystem should be able to configure every edi-tion of a course with appropriate communica-tionmechanisms. Some of these exist by default

�course syllabus, lesson schedule, etc.� whileothers can be used by choice, if deemed nec-essary or useful �chat forums, video conferenc-ing, message board, newsgroups, etc.�. Also,students need to configure their own virtual en-vironment, because not all students follow thesame courses at the same time. In the end, thenumber of students per course is not uniform�in Italy some university courses have hundredsof students and others very few�: a significantchange in course enrollment has important con-sequences on how electronic communicationcan be organized. In practical terms, for ex-ample, a teacher of 200 students would be wiseto limit the use of e-mail so as to avoid spend-ing an inordinate amount of time responding tostudent enquiries.

b) Scalability. In the real university contextthere are small faculties �with few study pro-grams and students� and large faculties �withseveral study programs and large numbers ofstudents�. In some cases the technologicalsupport infrastructure that we have briefly de-scribed is not completely present, or the com-munication channels within the university net-work differ from those available privately to thestudents.

c) Modularity. A network at the universitylevel is almost never constructed with the samecomponents: it is more likely built over time,in layers, and with varying growth patternsand, therefore, varying capabilities. To havea monolithic system with a uniform techno-logical infrastructure is not a realistic expecta-tion, at least in the majority of cases in Europe.Moreover, the naturally occurring differencesin teaching methods among diverse faculties inthe same university suggest architectures con-structed with models that are duplicated on dif-ferent machines, and also distributed databases.It is worth remembering that faculties are almostalways located in different buildings, and inmany cases there may be more than one branchof the university or faculty, perhaps located sev-eral kilometers away.

d) Accessibility. If possible, the system shouldbe accessible from every Internet access point.This means not only from computers availablein university buildings but also from any otherlocation and from mobile devices. In fact, uni-versity teachers often travel for research, per-


sonal study, participation in scientific meetings,and, in addition, they obviously do not have ac-cess to the internal campus network from theirhome. The same is true for students and teach-ers, albeit for different reasons. Furthermore,electronic communication opens the door tonew learning experiences, such as a course inwhich academics from all over the world con-tribute in different ways �through video con-ferences, with on-line lectures in real time, withasynchronous contributions to discussions, etc�.A good system will guarantee all these at thelowest cost possible.

e) Usability. A system is obviously a failure ifit is deemed unusable by the same people whoshould use it. The problemwith these systems isparticularly complex because it involves �a� thenature of theLearningObjects used in electroniccommunication; �b� the most general commu-nication environment inwhich the LearningOb-jects are used; and �c� an organization of activi-ties that guarantees correct Learning Objects atcorrect time for a specific learning task. Theterm “Learning Object” has been defined bymany researchers, but the most convincing def-inition is that proposed by the IEEE LearningandTechnology StandardsCommittee: A learn-ing object is any digital or non-digital entity thatcan be used, reused or referenced in technology-based teaching �16�. Differing degrees amongusers of awareness and familiarity with the sys-tem, not to mention access issues, complicatethe issue further. The advent of mobile devicessuch as PDA and cellular phones add new prob-lems to usability, due to the limitation of screensize and memory of these devices.

f ) Freedom of choice. The system must pro-vide varying degrees of freedom at different lev-els. The requirements for configuration, scal-ability, and modularity partly contribute to thisneed, but by “freedom of choice” we refer tosomething more all-encompassing. Using thesystem must not be an obligation but ratheran opportunity, both for teachers and students.Moreover, technological limits cannot be im-posed on the nature and complexity of the learn-ing objects that a teacher considers useful for acourse. In this case the question is no longerideological, but also leans toward a pragmaticevaluation of academic teaching. A look at theItalian academic system reveals that the averageage of the three types of teacher, foreseen by the

national system is over 55 for full professors,around 50 for associated professors, and over40 for research assistants. This means that onlya small percentage of these teachers attendedcomputer science courses during their own uni-versity studies. In fact, in Italy just 30 yearsago, computer science courses were offered ex-clusively at the faculties of science and engi-neering. Consequently, everything that the ma-jority of Italian university teachers know aboutcomputer science has been acquired through on-going study �often self-study� during their pro-fessional academic career. Similar observationscan be made about other European realities, asmade in �17�, and perhaps for a large part ofhigher learning, even on a global scale. Also,a Learning Object is by nature variable, rang-ing from a simple electronic text to a complexrole-play in real time; this is discussed furtherin �18�, �19�, �20�. Therefore, no limits maybe imposed on the technological realization ofLearning Objects.

In the next section we describe the functionsof interactive On-Line Teaching System, devel-oped by our group at the Faculty of Economics,University of Trento.

4. An Interactive On-Line Teaching System

The system we are presenting has been oper-ational for about three years at the Faculty ofEconomics of the University of Trento. Thefaculty has approximately 3000 students and afew hundred teachers. From an architecturalperspective, the system is an application of dy-namic Web pages: in other words a DBMS andother generalized systems are integrated so asto maintain an environment favoring communi-cation and self-organization of the work using abrowser. The system offers its services princi-pally to three different types of actors: teachers,students and administrative staff. All actors arerecognized by the system and for each actorthe system offers a personalized environmentto access the selected functions. A difficultchoice, nowadays still under discussion, regardsthe fact that the system should be open to ev-eryone or accessible only to registered students,i.e., only to the students that regularly pay thefees. Some teachers believe that the educational


material should be freely distributed to every-one in the world, while others prefer that onlytheir students �freely� download the educationalmaterial. At the moment, the more restrictivevision has been adopted: the students must reg-ister in the system with their personal data andtheir faculty registration numbers. A new loginand password is released for the registered stu-dents. The same registration procedure is madefor the teacher, and it is more obvious in orderto avoid that inappropriate material is loadedinside the system. There are many functionsusable for each of the three actors �teachers,students and administrative personnel�. We willbriefly present the main functions.

The teacher can make use of several functionsof the system. Having gained access with thelogin and password, the teacher selects a coursein which to work. To avoid any confusion thecourses are distinguished by year. After choos-ing the course, the teacher is offered differenttools through which it is possible to have allthe functionalities necessary to complete thetasks desired: to insert information, to update,to eliminate or to add material. Specifically,these instruments are:

� the Bulletin Board, where the teacher caninsert and modify information to provide no-tices and information to students enrolled inthe course;

� Syllabus �Figure 2�, a system summarizingthe content of the lesson and allowing stu-dents to identify the material relevant for agiven lesson. From this page the teacher canalso add a file to the lesson �uploading it onthe server�.

� Materials, the teacher can manage any typeof material in electronic format to make it

Fig. 2. Framework of the syllabus.

available to students in the course. Theteacher, following a prescribed procedure,can upload files from the teacher’s computerto the server. The material is subdividedby category in a way that permits the userto find quickly the desired type of mate-rial; some typologies available are: notes,bibliographies, exercises, lists of links, for-mulas, graphics, suggested readings, com-mentaries, manuals, assignments, assign-ment solutions, etc; once an element fromthe list has been chosen, the user has ac-cess to the actual list of materials for thatcategory. From here it is possible to insert,modify, or eliminate a file from that list. Theteacher also has the possibility to make a filevisible or not; this functionality is useful forthe teacher who may or may not want tomake a specific file available at any giventime, as necessity dictates. �Figure 3�

Fig. 3. List of materials.

� Timetable, the teacher can specify the les-son times and any changes to them, as wellas office hours for meeting students;

� Information on the course, it is possible tospecify the course program, the exam proce-dures, the list of texts used or suggested forthe course;

� Student Mailing List, the complete list ofstudents enrolled in the course. It is possibleto use this list to send mail to some or all ofthe students. �Figure 4�

� Diary of the lessons �Figure 5�, which dif-fers from the syllabus in that it permits ad-ministrative functions, while with the syl-labus it is possible only to upload or down-load files. The diary makes it possible to


Fig. 4. Teacher Area: student list, functionalities ofmailing list.

generate lessons, specifying date, time, a de-scription of the lesson, and related notes;

Fig. 5. The teacher’s diary.

� Video-conference �Figure 6�, it is possi-ble to have a video conference, integrat-ing a personalized version of the softwareNetmeetingTM internallywith theWebBrow-ser. Therefore, when the teacher accessesthe video conference page, the students thatare connected to the principal page of thesystem will automatically see a green lightindicating that the teacher is on line �Figure

Fig. 6. A video conference session.

7 n.1� and available for a video conference.The system automatically memorizes the IPadress of the teacher, thus helping the stu-dents and teacher to overcome the technicaldifficulties of configurating and setting up avideo conference via Internet.

� Newsgroups can be created by the teacher:this function will be described in greater de-tail in the following paragraphs.

The tools available to the students are generallysimilar to those used by the teacher, the onlydifference being that those available to studentsare usually “read-only”. This means that thesyllabus is usable only to download materialfrom the server, and the course program is onlyvisible to students who cannot modify it in anyway. To have access to the course page, thestudent must register, that is, the student mustchoose the preferred courses from those on thelist of the courses offered. These choices de-fine the study area of the student, and as suchdetermine the courses to which the student haselectronic access �Figure 7 n.3�. One of themost important features available to the studentis the area called “News” �Figure 7 n.2�.

Fig. 7. The principal page of the student.

This function is made available to the studentevery time he or she connects to the system; itshows all the documents that have been placedon the server since the last time the studentconnected, for all courses in which the studentis enrolled �visible, as described above, in theleft-hand frame�. It also shows other informa-tion placed on the bulletin board by the teacher.The student, therefore, before entering a course,can select directly a document present on the“News” list, independently of the course andwithout having to “dig around” the course page.This functionality helps the student greatly tobe updated on events or changes taking placeduring the course, especially if the student is


unable to attend a given lesson. In the near fu-ture, this function will be implemented usingmobile telephones to interact with students.

Another important characteristic of the systemis the Newsgroup. The validity of this tool inteaching had been widely proven. �21� �22� �23��24� �25�. By selecting a newsgroup, the user hasaccess to the list of related messages; the usercan decide to insert a new message, read one,enroll in the preferred group, or simply viewthe messages by list rather than by thread. Eachcourse can have more than one newsgroup. Theteacher, once entered in a newsgroup that he orshe has defined, is given the possibility to inter-act with the messages in different ways. As wecan see �Figure 8�, the messages inserted by theteacher are shown in a different color; moreoversome icons can be used to point out particularlyinteresting messages �FAQ, censured messages,etc.�. Other possibilities offered to the teacherare to cancel, modify, and grade messages, aswell as to moderate the entire newsgroup. Theuser can decide to activate the service of pre-ferred messages �Figure 9�. This service makesit possible to periodically receive at the user’se-mail address all new messages inserted in thenewsgroup.

Fig. 8. The view of the messages present in anewsgroup.

Each message, whether shown in the principal,general view �Figure 8� or in a detailed view,shows the number of times it has been read �Fig-ure 10 n.1�. In addition, students can participatein the discussions by giving a sort of grade toeach message �Figure 10 n.2�; this grade can beseen by anybody, so that the importance of a sin-gle message can be weighed. As one can note

from Figure 10 n.3 the typical “thread” struc-ture is present also in the message compositionphase, so as to give the user greater flexibilityin interacting with other users. All messagesare memorized in text files and not in databasestructures.

Fig. 9. The preferred service.

Fig. 10. A single message with the options provided bythe system.

In the area of maintenance there are severalfunctions which are indispensable for the fac-ulty secretaries. One of these is linked to thechange in academic year; each academic year,in the month of September, in fact, the coursescan be changed or their titles can be changed,as well as the teachers and tutors associatedwith the courses to mention just a few possi-ble changes. With the system, the secretary hasa tool that can be used to redefine and updateall the information related to the courses andteachers. Other instruments can be used to cre-ate new courses and provide information aboutthese teachers, tutors, or staff related to eachcourse. Apart from these functions, there arealso tools allowing for e-mail communicationwith some or all of the teachers and students.Yet other functions regard the administrationand management of the newsgroups created bythe teachers.


5. The Results of Our Experiment

As mentioned, one of the main purposes of theproject was to evaluate user satisfaction in orderto understand the impact of the system on theinteraction between the students and the insti-tution. To this end, we used a self-administered“Web questionnaire” with closed answers thatwas directed to all students enrolled in the sys-tem. Subdivided into three sections, the ques-tionnaire has 24 questions. Besides personaldata �gender, year of university enrollment, de-gree programme�, the survey brought to lightsome important user feedback, organized as fol-lows:

� in the first section the assessment is on a10-point scale, measuring simplicity, clarity,efficiency, and power of the systems com-pared;

� in the second section the general assessmentis again on a 10-point scale, measuring ma-terials and information available online, forevery course the user has been enrolled in;

� the third and final section measures satis-faction with the efficacy and efficiency ofthe system as a tool supporting teaching andlearning.

The survey was carried out between June andJuly 2002, duringwhich 574 students completedthe questionnaire. Before describing the resultsit is useful to underline that the group surveyedis a “narrow slice” of the university population,and not representative of the larger student body.This is true, primarily because the sample sur-veyed is from the population that voluntarilydecided to enroll in the system and, secondly,because nearly 98% of this group is enrolled inthe Faculty of Economics — a logical outcome,given that almost all of the e-learning coursesare offered by this faculty. We cannot assertthat the group surveyed is a representative sam-ple of students enrolled in or attending coursesat the Faculty of Economics, because not allprofessors participated in the system, therebynullifying the need for the e-learning option fortheir students.

We can now describe the characteristics of re-spondents,which are relatively evenly distributedbetween male �52.3%� and female �47.7%�.

About four fifths �81.0%� attended courses reg-ularly and nearly all �85.1%� had already en-rolled in the system the semester before the sur-vey. Half of the survey group �50.4%� indi-cated that they were employed, which is a rel-atively low number when compared with grad-uates from the same faculty in 2001 �84.0%source AlmaLaurea�, who declared that theyhad worked while completing their universitydegree.

Students consulted the system frequently, atleast 35% doing so at least once a day and 60%one or more times a week. The adoption of thesystem also led to changes in the use of Inter-net as a didactic tool. Before using our LMS,less than one third �31.5%� of respondents in-dicated regularly using the computer to searchfor information for personal study, while nowover four fifths �86.6%� also consult Web pagesmade available by professors. The result is afull-time student in every sense of the word,who is active and takes advantage of a varietyof study tools.

Unfortunately, this type of system is found lack-ing in another equally important aspect. TheItalian university system is among the moreflexible ones in industrialized countries, withlowenrollment costs, comparedwith other coun-tries; moreover, attendance at lessons is notobligatory, nor are there constraints on the ex-ams methods. These facts mean that a largesegment of the student population can enrollin the courses and take exams without havingto attend lessons, thus being allowed to holdfull-time jobs. Therefore e-learning systems areincreasingly perceived as a valid alternative tosupport individual study for those who are un-able to attend lessons regularly, or those whosestudies are to some degree penalized by workcommitments. Contrary to our expectations, thenumber of students in this category is decidedlycontained. As noted, the percentage of work-ing students is smaller than expected; moreover,only 6.5% of those who rarely or never attendlessons use the e-learning system.

We are required to look into the causes of thisunderuse, which, to our view, is not related to thetool itself but to how it is used. If the instructorsuse the system merely as an electronic noticeboard for distributing copies and bibliographicalreferences, the students similarly make limited


use of the tool as a means to support their stud-ies. This can depend on different factors: eachcommunication system must overcome its phys-ical limits and create its own linguistic codesthat make it recognizable and meaningful. Asyet, we are not able to determine to what degreethis has been accomplished, however, a notabledifference exists between users and non-users,where the latter group would appear to be ex-cluded from this generation of linguistic codes,hence from a form of communication. In otherwords, we are dealing with a system that fostersa form of social inequality in terms of access toresources.

On the other hand, we cannot in all truth assertthat those using the system have made the ef-fort necessary to learn to take full advantage ofwhat the tool offers. An interesting statistic inthis regard derives from the use of the discus-sion forum, a feature offered by the tool. Whileone sixth of the students �18.5%� declared thatthe instructor of the course�courses in whichthey were enrolled had not used the discussionforum feature, only one fifth �20.2%� of the re-maining students had used the forum, one third�33.6%� indicated no interest in the feature andjust over one fourth �27.7%� expressed otherreasons for not using the tool. Poor ability tocooperate or communicate, or even inability touse the system because of the uncertainties re-garding its purpose: all could work togetherto explain these statistics. Despite these num-bers, the fact remains that as many as 82.2%of users indicated that the system was highlyuseful. Four fifths �81.5%� reported that thesystem had helped them to clarify topics beingstudied; three fourths �72.0%� mentioned be-ing better informed of the opinions of their fel-low students, and, similarly, two thirds �68.7%�indicated that they had gained a better under-standing of the instructor’s position and 63%were more able to follow the course.

Why, then, do such a small number of studentsmake use of the tool? We could assert thatthere are courses that, by their nature, simplyleave little room for discussion; a formula foran econometrics function is invariable, the argu-ment would follow. But can we be sure? Whilethe roles of teacher and student are yet to bedefined, it is clear that the communication andtransfer of knowledge cannot go in only onedirection. Never before has the success of a

work project hinged so much on the ability ofthe group to work together and circulate ideas.Poor communication skills, coupled with the in-ability to take full advantage of these types oftools, in the end means exclusion.

The system passed with top marks. On an as-sessment scale from 1 �poor� to 10 �excellent�,the average score was 8.1. The highest scoreswere given to the system architecture for itssimplicity of use �average score 8.5� and forthe capacity of the system �average score 8.4�.This also shows that the decision to “make” dur-ing the LMS acquisition phase did not prejudicethe quality of the product or the satisfaction ofthe user. A score of 8.0 was given to the clar-ity of content, the responsibility which lay inthe hands of individual professors. The systemwas found to be largely efficient �7.6� but notwithout some criticism; the fact that the eval-uator may well have included the entire uni-versity network, which, while quite good, doesnonetheless reveal inefficiencies, in the assess-ment could account for this somewhat lowerscore.

On thewhole, positive scoreswere also recordedfor the more technical aspects of system archi-tecture. Nearly all users �97.0%� declared thatit was quite, or even very easy to locate informa-tion, and nine tenths �91.0%� indicate that thebuttons were clear and thorough. Slightly lowerresults were shown for the efficiency of thecommunication system, problems which are,as mentioned, to a larger extent due to inef-ficiencies in the university system than to thee-learning tool itself. In any case, four fifths�81%� of respondents said that the connectionwas fast and that it was easy to download mate-rial.

A different type of assessment takes place whenconsidering the system as a tool to support di-dactic activities. While the tool proves highlysatisfactory for self-study, its role as a means ofcommunication among different actors is lessclear. Four fifths �83.0%� maintain that the useof the system led to improved individual study,even nine tenths �86.6%� affirmed that it helpedthem to pass an exam, while over nine tenths�91.9%� of respondents found the system use-ful when gathering information about courses.

The communicative aspects of the system re-ceived lower scores. Only 58.3% of studentsfound that the system had helped then improve


their communication with the instructor, whileone sixth �16.1%� noted improved communica-tion with other students �referring only to thosewho joined a discussion forum�. This brings usback to the earlier problem, wherein the use ofthe system as a didactic support tool is highlyeffective, but as a system to support commu-nication is found wanting. This gap has moreto do with the users than with the system it-self. On one hand, the professor may not knowhow to take full advantage of the tool, or isforced by other commitments to reduce the useof the system as a means of communication. Atthe same time, students may be overly focusedon exams and perhaps unable or unwilling towork in groups, thereby not seeing the poten-tial that communication among fellow studentscan have, even if it is only to pass an exam. Inshort, the system offers numerous opportunitiesto those who choose to take advantage of them,as confirmed by the positive scores received. Itspotential is limited only to the degree that theuser fails to take advantage of it, an issue to beexplored elsewhere.

6. The Future: Mobile Extensions to theSystem

Mobile learning, or M-learning, has been con-sidered as the future of learning or as an integralpart of any other form of educational process inthe future. Mobile learning is a fieldwhich com-bines two very promising areas — mobile com-puting and e-learning. Mobile learning couldbe any form of learning �studying� and teachingthat occurs in a mobile environment or througha mobile device, like cellular phones, PersonalDigital Assistants �PDA�, smartphones, tabletPC etc.

Some studies show that introducing new formsof teaching �even if this means just using a stan-dard tool for drawing on a PDA� make studentsspend more time in working on that subject,comparing to other subjects. �26� The currentevolution and analyses of m-learning projectsshow many positive results. On the other hand,there are some doubts if this excitement is, oris not, a temporary side effect. Most of the re-searchers think ��27��28�� that PDAs and othermobile devices should be seen like an exten-sion to, rather than replacement of, the exist-ing learning tools. Moreover, not all kinds of

learning content and�or learning activities areappropriate for mobile devices �29�.

In order to support the experimentation of anytool or technique of m-learning, a rather com-plex information system is necessary. Its role in-cludes distributing didactic material, user iden-tification and authorization, gathering of thedata relative to the user-system interaction, pro-visioning of mobile services etc. The plat-form necessary for this objective is already inplace, and we have all the skills and informationneeded to extend it with mobile functionalities.In this regard, e-learning systems in general, andmore specifically Learning Management Sys-tem, are a vital component in the distance ed-ucational field. We have to integrate our LMSwith two different classes of processes:

� on one hand, processes connected with theadministrative �back-office� activity of a fac-ulty �like registering exams, programmingthe teaching activity, theses management,bookkeeping of lecture hours, faculty mar-keting etc.: all such processes show impor-tant overlaps with processes managed by anLMS.

� on the other hand, technology evolution haspushed toward a strongmobility of all the ac-tors, and has provided mobile devices �PDA,pocketPC, cell-phones, smart-phones, tablet-pc� that accompany the user in his�her ev-eryday life. Such tools can become addi-tional terminals for a LMS, because theyallow all actors �students, teachers, secre-taries, dean, tutors, administrative personneletc.� to stay in touch with the LMS whereverthey are.

The number of possible applications is huge:for instance, the possibility for the administra-tion to communicate in real time with studentsequipped with such devices, new forms of col-laboration among students and teachers withina university course, the chance for the studentsto interact among them regarding the coursesetc. The focus moves, therefore, from a sys-tem that is based on “offering courses” into asystem based on the idea of “virtual commu-nity”. A virtual community is a highly gen-eralized collaboration space. In such a way, acourse given by a teacher, a seminar, a groupof students preparing their thesis with the sameteacher, students working together on a project,


etc. are all instances of virtual communities. ALMS becomes a computer-based tool that ren-ders services to virtual communities, and mustbe adapted to specific needs of each particu-lar community. We already built, over sev-eral years, a community-oriented learning por-tal. Starting from this existing background, weintend to experiment with various ways to sup-port collaboration among users interconnectedby mobile technologies through the already ac-tive portal based on our LMS.

Adaptation of the Learning Management Sys-tem to infomobility will need different steps:

a� Extension of the traditional functions of alearning management system to the mobile-computing needs required by the project.Thiswill imply the creation of teacher-system-student interaction tools mainly based onSMS messages concerning the activities ofthese actors in the system. Moreover, theportal will provide an access point to thesystem’s actors, in order to download theeducational material and the self-evaluationtests produced according to the objectivesof the project. Besides, different structureswill be created to support research activitiessuch as forums usable via mobile technolo-gies, mailing lists for various users, manage-ment of some virtual communities �studentsenrolled in a course, participants to lab exer-cises etc.�.

b� Distribution of educational material specif-ically created for the use on mobile equip-ment. This will regard both the educationalmaterial and the self-evaluation tests createdin point c�.

c� Integration of the self-evaluation system intothe LMS. This system will allow conductingtests on the main platforms that currentlyequip the most widespread PDAs on the mar-ket. These platforms will be the ones basedonSymbianOSonone side �thismeans to in-volve the whole cellular phones market withthe biggest world producers�, and, on theother side, the platforms equipped with Win-dowsCE, i.e. the PDAs that present points ofcontact with the Windows desktop environ-ment in terms of applications and workingenvironment. The choice of producing self-evaluation applications for both the PDAsenvironments is the result of our wish to ex-tend as much as possible the experiment, and

most of all to create a self-assessment mech-anism that must be generalized as much aspossible with respect to technological plat-forms, due to the extreme volatility of themarket.

As regards the development of the systems, wedecided on which devices to concentrate ourdevelopment. This is a very important issue,as the market is continuously changing, withnew products emerging every day. So, it ispractically impossible to have a general mecha-nism for involving all possible devices currentlyavailable. We found the following devices use-ful for our experiments:

� GSM�GPRS cellular phones

� PDA

� Smartphones

� UMTS telephones

� Tablet PCs

Fig. 11. Mobile extensions to the LMS platform.

The platforms have been already found in theirmain components. We will also experimentwith the Palm OS, so that our experiment willcover a very large share of the market. In thefirst step of the project, however, the choicemade on some MicrosoftTM-dependent PDAsis related mainly to the fact that most of theeducational material is currently published inMicrosoftTM software tools, especially Pow-erPoint and Word. In this sense, a deviceequipped with MicrosoftTM operating systemswill facilitate exchange of educational materialalready available. However, modular structureof the approach followed in building web ser-vices based on XML and SOAP will provide


a sufficient grade of extensibility of our mo-bile platform to other PDAs, like those that areequipped with Symbian OS.

The test of the system will consist of somelessons conducted by Learning objects distri-buted by the LMS and by students and teachersutilizing PDAs, traditional viewers �like Power-Point and Acrobat Reader� and other availablemobile devices. Part of these educational mate-rials will be available only through mobile de-vices: students will have to learn to study onlyon PDAs. In this way, different groups thathave studied on different devices with differentapproaches will be available for our research:those who followed face-to-face lessons, thosewho studied on learning objects without fol-lowing the lessons and those who studied onmobile devices. By creating a specific and cal-ibrated set of tests, we want to verify the levelof learning of the single groups, by analyzingthe differences and relative motivations. Theresults of these tests will be matched with theresults of the self-evaluation tests distributed tothe students, in order to thoroughly verify thelevel of learning reached by the students. Re-actions of the students will also be analyzed,especially those related to problems in study-ing with a new, but limited tool like a portabledevice. For this purpose, a forum on the webwill be specifically activated, and some tutorswill be available in order to help students withpractical or technical problems.

Regarding the use of specific tools availablewith mobile technology, the most evident prob-lem we faced in the design phase was the choiceof the technology to be used to build the toolsprovided to the client in order to use our ser-vices. The current project provides ten differ-ent classes of services to mobile users, but inorder to simplify the choice, we decided to con-centrate initially on two different services formobile devices:

� Management of SMSs sent by teachers tostudents or by administrative staff to teach-ers and students when particular events hap-pen �meetings, reminder of expiration datesetc.�

� Consultation of a common agenda �we callit organizer� that will be available on themobile device and will keep all the impor-tant dates for the actor �mainly students andteachers�

The first service is quite simple to build, but notso easy to manage, if the LMS that operates be-hind the scenes does not have all the informationneeded. The main problem was found in allow-ing the right person to send and receive SMSs,and in granting this permission inside correctboundaries, in terms of the number of SMSssendable by the user. The second service is un-der development and is more complicated, as itinvolves one of the most difficult tasks to man-age inside an LMS, i.e., time management. Weare currently building a system that allows stu-dents and teachers to connect with their mobiledevice and consult their agenda, dynamicallybuilt with all the events that could happen dur-ing a normal university activity. This impliesa great effort of abstraction and integration be-tween theLMSplatformand themobile devices.We have evaluated five different alternatives tobuild the interaction between the PDA �the plat-form chosen for the experiment� and the centraldatabase. The problem is related to the way theclient �the PDA� interrogates the remote servermodule, requesting the update of events sincelast connection. These are the alternatives weevaluated and tested, from the simplest to themost complicated ones:

� Using the embedded browser of the PDAto navigate through the web pages that webusers will see using the traditional browseravailable for desktop PCs. This is the sim-plest solution, both for the users and forthe development team. Only a particularattention to screen adaptation is necessary inorder to concentrate the most important in-formation on the left-uppermost part of thescreen and to avoid the necessity of frequentscrolling. The web page will be created us-ing device-specific tags and languages, likethe .NETTM mobile toolkit, in order to navi-gate through the data available on the server.However, we decided not to follow this so-lution as the primary one, because of thenecessity for the user to be constantly con-nected to the Internet to navigate throughthe organizer, thus requiring permanent con-nections �like WI-FI settings� or significantexpense for both students and teachers whenconnected to the net using GPRS technol-ogy. In Italy, this solution is very costlyat the moment, and WI-FI technology withwireless LAN is still in its infancy. Other


short-range connection solutions have beenabandoned, as we want this service to beused outside the campus.

� Using a client database application built spe-cifically for mobile devices, that interrogatesthe server DB through the Internet, synchro-nizing the data on the mobile device. This isa proprietary solution bounded to the back-end DB used and the availability of an In-ternet connection on the PDA that requiresalso quite complicated settings from the end-user perspective. However, from our tests,this solution has the advantage of dramat-ically boosting performance, thus reducingthe connection times.

� Synchronizing the PDAwith the central data-base and the agenda of the user by usingcradles and database synchronization: thissolution will solve a lot of issues, but cre-ates a problem in terms of cradle availabil-ity around the campus, and especially theproblem of supporting different cradles fordifferent models of PDA.

� Building a client�server application inwhichthe client �on thePDA�uses traditionalRPC�RMI mechanisms to invoke server methodsin order to receive the data. This has the ad-vantage of requiring short-time connectionto the central system and could be personal-ized to the PDA device. The disadvantage ofthis solution is the proprietary mechanism ofcommunication between server and client,and also the necessity of using particularTCP�IP – UDP ports that could complicatemanagement of the security on the serverside due to firewalls.

� Building a web application that requests aweb service through the use of XML�SOAPmessages to the server. This is the bestsolution we have found, as it provides theaccess in short time to the central databasethrough the use of an open technology likeXML�SOAP, will use a port that is alreadyopened for web access, and, finally, willguarantee extension of the client part to otherPDAs, simply by creating a new client inter-face to the web service. We will, therefore,provide the agenda synchronization througha web service that will recognize the user,verify the state of his�her agenda, and willsend an XML-formatted packet of data re-garding last events in the system. The client

side of the application, specific for the de-vice, will format this data for the display:after that, the connection with the server willbe closed and the navigation on the agendawill be completely off-line.

7. Conclusions and Lessons Learned

Summarizing a four years project involvingmore than twenty developers and analysts, andan entire faculty with more than 3.000 studentsand 200 teachers is quite difficult. In this paperwe presented our experience in building whatwe, after four years, like to call a “LearningInformation System” rather than a more tradi-tional “Learning Management System”. This isdue to our most evident finding of the project:introducing a system that manages the educa-tional process �as a whole� of an educationalinstitution does not limit its effects to the mostspecific didactical aspects of the institution. In-troducing self-made, personalized, customizededucational management system could in suchsituations lead to a radical change of the wholeinformation system of the institution, whichenormously amplifies the possibility of inter-action with different actors �students, teachers,staff, tutors etc.�. We followed this approachby building from scratch our Learning Manage-ment System and, after that, we improved it byadding different functionalities and tools basedon a spiral development model and on subse-quent refinements of the system suggested di-rectly by the users. Thisway of building, testingand assessing the system has proved successful,also allowing us to provide users with interest-ing management features on the processes thathappen inside an educational institution For ex-ample, one of themost highly appreciated “side-effects” is the ability to monitor the system fromdifferent viewpoints, thereby providing interest-ing and useful information regarding the learn-ing process and the behaviour of actors engagedin the system — all viewed from within an ex-tended version of the system itself. This opensnumerous opportunities for future developmentof the system, extension of virtual communitiesat varying levels among the actors involved inthe system — teachers, students and adminis-trative personnel.


8. Acknowledgment

We would like to thank all the collaboratorsand students in the Laboratory of Maieutics forhelping us start and conclude this huge workdespite all the difficulties of different types weencountered. Thank you once again.

References

�1� ANDERSON, T., Integrating Lectures and ElectronicCourse Materials, Innovations in Education andTraining International, 34�1�, pp. 24–31, 1997.

�2� BROOK, C., OLIVER, R., Supporting The Develop-ment Of Learning Communities In Online Settings,in Proc. ED-Media 2002, Denver, CO, USA, June24–29, 2002.

�3� HELIC, D., MAURER, H., SCERBAKOV, N., Web-Based Training: What Do We Expect from theSystem, Proceedings of ICCE 2000, Taiwan, pp.1689–1694, 2000.

�4� GAINES, B. R. AND SHAW, M., Institutional Trans-formations to a Learning Web, in Proceedings ofED-Media/ED-Telecom, the World Conference onEducational Multimedia/Hypermedia, T. Muldnerand T.C. Reeves �Eds.�, Association for the Ad-vancement of Computing in Education, 1997.

�5� IEEE P1484.12 Learning Object Metadata WorkingGroup �http��ltsc�ieee�org�wg��

�6� IMS Global Learning Consortium, Inc.�http��www�imsproject�org��

�7� MORRISON, G. AND MCPHEE, S., The ChangingRoles of Faculty in the Course Development Pro-cess, inProc. of ED-Media/ED-Telecom, T.Muldnerand T. C. Reeves �Eds.�, AACE, Calgary, Canada,June 1997.

�8� SMEATON, A. F., Using Hypertext for ComputerBased Learning, Computers and Education, 17�3�,pp. 173–179, 1991.

�9� ADL SCORM �2001�, Sharable Content Ob-ject Reference Model Version 1.2 Specifica-tion, Advanced Distributed Learning, available atwww�adlnet�org

�10� AGGARWAL, A., �Ed.� Web-based learning andteaching technologies: Opportunities and chal-lenges, Hershey, PA, USA, Idea Group Publishing,2000.

�11� MCMAHON M., LUCA J, Courseware ManagementTools and Customised Web Pages: Rationale, Com-parisons and Evaluation, in Proc. ED-Media 2001,World Conference on Educational Multimedia, Hy-permedia and Telecommunication, Tampere, Fin-land, June 25–30, 2001.

�12� Infoworld Publishing Company �1998�, Web -basedTraining Solutions, http��wwwinfoworld�com�cgibin�displayTC�pl��comp�htm

�13� PC Week �1997�, Computer-based training onthe Web, http��www��zdnet�com�pcweek�reviews��ibt�html

�14� TEARE, RICHARD, DAVIES, D. & SANDELANDS, E.,The Virtual University: An Action Paradigm andProcess for Workplace Learning, London, UK,Cassell Academic, 1999.

�15� A’HERRAN A., Integrating a course delivery platformwith information, student management and admin-istrative systems, in Proc. ED-Media 2001, WorldConference on Educational Multimedia, Hyperme-dia and Telecommunication,Tampere, Finland, June25–30, 2001.

�16� IEEE Learning and Technology Sandards Com-mittee, URL: ��last visit 26-02-2002� http��ltsc�ieee�org�

�17� AUFENAGER, S., Teacher Education by HypermediaLearning Enviroments – An Example from Ger-many, Ed-Media, 2000.

�18� KOPPI, T., HODGSON, L., BAYLY, J., The Often Miss-ing but Essential Component for Online Learning:a Learning Resource Catalogue, ED-Media, 2000.

�19� HODGSON, L., KOPPI, L., Universitas 21 LearningResource Catalogue using IMS Metadata and NewClassification of LearningObjects,Ed-Media, 2001.

�20� WILEY, D. A., NELSON, L. M., �1998� The fun-damental object, URL ��last visit 26–02–2002�http��wiley�ed�usu�edu�docs�fundamental�html�

�21� COLLIS, B., KNEZEK,G., �1997�Teaching and learn-ing in the digital age-research into practice withtelecommunications in educational settings Eugene,OR: International Society for Technology in Educa-tion.

�22� EASTMOND, D. V., Alone but together: Adult dis-tance study through computer conferencing, NewYork, USA, Hampton Press, 1995.

�23� HANNA, D. E., GLOWACKI-DUDKA, M., CONCEI-CAO-RUNLEE, C., 147 Practical tips for teachingonline groups: Essentials of Web-based Education,Madison, WI, USA, Atwood Publishing, 2000.

�24� HORTON, S., Web teaching guide: A practical ap-proach to creating course Web sites, Yale Univ.Press, USA, 2000.

�25� PALLOFF, R. M., PRATT, K., Building learning com-munities in cyberspace: Effective strategies forthe online classroom, San Francisco, CA, USA,Jossey-Bass, 1999.

�26� DVORAK, J. D., BURCHANAN, K., Using Technol-ogy to Create and Enhance Collaborative Learning,Proc. of ED-Media, 14th World Conference on Edu-cational Multimedia, Hypermedia and Telecommu-nications, Denver, CO, USA, June 2002.


�27� KUKULSKA-HULME, A., Cognitive, Ergonomic andAffective Aspects of PDA Use for Learning, Pro-ceedings of the European Workshop on Mobile andContextual Learning, pp. 32–33, Birmingham, UK,June 2002.

�28� WAYCOTT, J., SCANLON, E., JONES, A., Evaluatingthe Use of PDAs as Learning and Workplace Tools:An Activity Theory Perspective, Proceedings ofthe European Workshop on Mobile and ContextualLearning, pp. 34–35, Birmingham, UK, June 2002.

�29� KEEGAN, D., The future of learning: FromeLearning to mLearning, available online athttp��learning�ericsson�net�leonardo�thebook�book�html

Received: June, 2003Accepted: September, 2004

Contact address:

Luigi ColazzoUniversity of Trento

Department of Computer and Management SciencesVia Inama, 5

TrentoItaly

e-mail: colazzo�cs�unitn�it

Andrea MolinariUniversity of Trento

Department of Computer and Management SciencesVia Inama, 5

TrentoItaly

e-mail: amolinar�cs�unitn�it

LUIGI COLAZZO is a professor of database management, and multime-dia at the Faculty of Economics. His scientific and research interestshave been in the areas of systems for information retrieval, software ar-chitectures for knowledge discovery in data base, information systemsand systems to support cooperation �CSCW�. Since 1989 his work hasfocused on the use of hypermedia systems and Web-based systems tosupport the didactic activities. He is member of several professionalassociations including the IEEE, ACM, AACE, and AICA, Doctor Co-lazzo has authored numerous papers.

ANDREA MOLINARI has worked as a consultant in the field of analysisand development of information systems and database since 1980. Hehas been member of the research group in the field of multimedia andhypermedia systems at the Department of Computer and ManagementSciences in Trento, Italy since 1988 and is responsible for distancelearning unit at the Laboratory of Maieutics of the University of Trento.

Building, Testing and Assessing a Learning Management …

Documents

Transcript of Building, Testing and Assessing a Learning Management …