Assessment and evaluation at a distance: project CAMEL

Abstract Students in distance education at the KATHO (University college in Flanders, Belgium, www.katho.be) only need to come to the campus for assessment. Since September 2008, the center of excellence in e-learning, MAD (Mad About Distance) of KATHO organized a small project with 5 students located in five different countries all over the world. One student nearby the desert in Dubai inspired us to call this research “the Camel project”. The people involved in the project explored new methodologies and new technical tools to enable teachers to assess those students online. The project is still running and inspiring teachers of the KATHO to develop particular settings about online assessment, but also about online coaching. This paper is meant to share some of our experiences and conclusions concerning the online assessment and evaluation.

Filip Vervenne, KATHO University college Belgium

Introduction

In the specific area of teacher education, KATHO has now been organizing distance courses for 13 years. Actually, with over 2000 students KATHO became a leading institution in distance education in Flanders. The organization of the distance courses, called “open higher education”, is student centered. This is illustrated with the choice for a personal learning route, the use of adapted methodology,… While a regular F2F course takes 3 years and 60 study points each year, the distance course takes, depending on accreditation of prior experiential learning of the students, between 2 and 4 years. The digital learning environment is based on Blackboard and consists of digital courses, assignments, discussion boards,… Although students can study at home and enjoy their practice in schools within their own environment, they still need to come to the university campus when it comes to assessment, oral or written tests. Since 2007, some Belgian students living abroad, showed their interest in becoming primary school teachers by means of the distance course. Those candidates could not be accepted due to practical problems concerning assessment and follow up of their practical training.

Background

The Expertise Centre distance learning of KATHO-Tielt (MAD or Mad About Distance), started up a small research project on online assessing. In September 2008, the Camel project was born and 5 students signed a letter in which they agreed to take part in this experiment, with all due risks. The concrete challenges of this project: translating and adapting documents concerning their teaching practice, development of good instructions and instruction movies, and most of all, a good scenario for the assessment, both for the teachers and the students involved.

About the challenges

The students in this program generally get the same study program as their fellow students in Belgium: they participate in the online courses, make the assignments and participate in the discussion board on Blackboard. On top of that, these students also get access to a Blackboard community with specific information. Anyhow, the project is very demanding for the study coach, who is – regardless of the specific disciplines - responsible for all the students, but spends more hours on the students of this particular project…

When it comes to assessment and evaluation at a distance…creating a setting with real reliability and objectivity is a basic condition. First, we recommend open book exam as a first option to avoid problems with

cheating. But that is not enough. We need to be able to create an environment for assessment and evaluation, that takes the criteria of the basic condition of objectivity very seriously on the one hand and is flexible enough on the other hand to use a wide range of question types and different ways of support and feedback from the teacher instructor. We need to be able to deal with different types of assessment.

The scenario

There are hundreds of possibilities when it comes to meeting online. However, videoconferencing programs are obviously created to establish online conferences and many programs never where mentioned for cases of assessment and evaluation… After trying out several videoconferencing tools and comparing possibilities versus prices, we chose Adobe Connect (http://www.adobe.com/products/acrobatconnectpro/) to work out the best setting for an exam on distance. This videoconferencing program allows us to talk and see, to share documents, to chat and to take over the screen of the student’s computer.

Prior to the first exam by means of Adobe Connect, both student and teacher can learn to work with the program by watching a screen casted instruction movie on the specific community in Blackboard.

After the first six months, as a result of trial and error, we summarized a brief description of 5 formats within the scenario for assessment and evaluation at a distance. Formats that both students and teachers experienced as rather comfortable and reliable settings for assessment. In what follows, we give a brief description of 5 successful formats within the scenario of a test (summative, assessment of learning) at a distance.

1. The pure oral exam

As if it was a face2face setting, the teacher and the student plan an individual meeting for the exam. The teacher is asking the question, the student responds in real time. In this case, the quality of webcam and headset is very important. When it comes to a good sound, sometimes Skype revealed to be the most efficient tool for VOIP (Voice Over Internet Protocol).

2. Written exam, digital and synchronous

Again basically a digital face2face setting. In this format, the teacher shares the test as an online document in Adobe Connect. The student can write ‘live’ in the document. Sometimes there is a little delay, but the result is very satisfying and makes full profit of the extra value of the videoconferencing tool. The shared document can be an Office Word document but also an Excel or a Powerpoint. This makes a huge difference with ordinary chat!

3. Written exam in a traditional way

The student prints the test, fills it out, scans it and sends it back. In case you want to avoid any technical difficulties or e.g. when students need to make complicated drawing. Math exams are solved this way.

4. Written exam in the learning management system

The teacher makes the test available in his course within Blackboard. The advantages of a pool with questions and automatic correction will facilitate the task of the teacher in the long run.

5. Written exam with the use of assessment software.

The teacher makes the test available in Edumatic, comparable to the internationally well known authoring tool Questionmark (www.questionmark.com). This tool allows the use of a wide range of question types.

Description of the practical measures

In each of those formats, the online connection, the webcam and headset are the basic tools in the setting. To initiate a new exam, the student and the teacher make an individual appointment to meet online. An important practical measure: the student must install a webcam behind his back, in an angle that shows the room and the computer screen. The teacher must be able to confirm that the student is acting trustworthy.Recently, computers tend to have an integrated webcam. If the webcam is integrated in the PC, and a second webcam cannot be installed for some reason, we recommend the student to install a mirror behind his back.

The quality of the connection is a threat for a reliable assessment environment. There must be a permanent alternative to continue the exam in case of a technical failure. Therefore the teacher is advised to open Skype (www.skype.com) and Teamviewer (www.teamviewer.com) as backup in case of a technical failure in the Adobe Connect environment.

Anyway, there is always technical support for the teachers during the exam. This expert also prepares the online meeting space for the exam.

Findings and implications for future application.

Teachers of the distance education currently use the five described formats of the scenario. They are used to teach and coach students in an online environment.

Once both student and teacher are familiar with the setting, the preparation takes less and less time. However, the very individual process of assessing on a distance is very time consuming for the teacher...

Among the concrete objectives for 2010, the detailed description of the scenarios is a priority as well as more tutorials and instructional screen casts for both teachers and students involved in assessment and evaluation at a distance. In the mean time, based on literature on identical experiences and new research in this area, we need to refresh our TPACK (Technological Pedagogical and Content Knowledge) to refine and improve the project.

Conclusions

As this is a work-in-progress project, we hope to learn from our experiences and implement some of the new techniques in the “normal” distance program. Since October 2009, we have 9 students in the project, of whom 3 are living in Belgium. They participate in the project because they need to work during the weekend and in the evenings… and are not able to participate in the regular programs. Yes, it is clear that this is a project with lots of potential and without any doubt, lots of candidates. On the other hand, it is clear that the actual approach is too time consuming to maintain if we accept a larger group of students…

The most important conclusion is that the whole project helps us to reflect about the meaning of assessment. It leads us to a critical approach and even towards questioning traditional forms of assessment. It challenges us to find out, to invent completely new forms of assessment… Perhaps the biggest challenge is to involve the students in this urgent construction of new assessment and the design of the distance education of the future with richer and more accurate forms of assessment.

References

Salmon, G. (2000). E-Moderating: The Key to Teaching and Learning Online. London: Kogan PageKnight, L. A. (2002). The role of assessment in library user education. Reference Services Review, 30(1), 15-24.

Dochy F., Schelfhout W., Janssens S. (2003). Anders evalueren. Lannoo CampusJuwah, C. (2006). Interactions in Online Education. Implications for theory and practice. New York, USA