Greek Evaluation

31
User-Centered Evaluation of an On- Line Modern Greek Language Course Panayiotis Zaphiris The Centre for HCI Design City University Giorgos Zacharia Center for Biological and Computational Learning Massachusetts Institute of Technology

TAGS:

description

 

Transcript of Greek Evaluation

Page 1: Greek Evaluation

User-Centered Evaluation of an On-Line Modern Greek

Language Course

Panayiotis ZaphirisThe Centre for HCI Design

City University

Giorgos ZachariaCenter for Biological and Computational Learning

Massachusetts Institute of Technology

Page 2: Greek Evaluation

Kypros-Net Inc. ( http://kypros.org/)

Formation– By a group of volunteers in 1996.– Composed mainly of Computer Science,

Information Technology and Human-Computer Interaction students.

Purpose:– “to utilize the Internet and particularly the

World Wide Web (WWW) system for the promotion of the world of Cyprus“

Page 3: Greek Evaluation

Learn Greek Online (http://kypros.org/Greek/)

Course Description– Independent study course to learn Greek

Online• 105 audio lessons, transcriptions,

vocabulary, spell checker, dictionary– Objectives

• Assist people in learning Greek language• Develop site content using Participatory

Design

Page 4: Greek Evaluation

Learn Greek Online Home Page

Page 5: Greek Evaluation

Participatory Design Methodology

(a) Building bridges with the intended users

(b) Map user needs and suggestions to the system

(c) Develop a prototype (d) Integrate feedback and continue the

iteration.

Page 6: Greek Evaluation

Design Methodology

Building bridges with the intended users

(Determine objectives, alternatives, constraints)

Mapping user needs and suggestions to the system

(Evaluate alternatives identify, resolve risks)

Developing a prototype (Develop, verify next level

product)

Integrating feedback & continuing the iteration

(Plan next phases)

Page 7: Greek Evaluation

STEP 1: Building bridges with the intended users Open lines of communication between

intended users and the development team.

– Initiate multidisciplinary development team

– Identify key groups of end users

– Create new methods of communication with users.

Page 8: Greek Evaluation

Intended Users

Bridges with users build through years of work at providing information about Cyprus through the web pages of Kypros-Net, Inc. who primarily attracts the same user population as the Greek language online course.– People of the Greek Diaspora– Travelers to Cyprus and Greece and other Greek

speaking areas – People who are generally interested in the Greek

culture and language or languages in general.

Page 9: Greek Evaluation

STEP 2: Map user needs and suggestions to the system Conceptual design model:

– “to design an online effective Greek language course that can build and sustain an online learning community of students”.

Tried to match user needs with our conceptual design model:– Easy to follow– Both Elementary and Advanced– Could attend at their own pace

Page 10: Greek Evaluation

STEP 3: Develop a prototype

The project consists of 105 audio files, which were originally recorded as Radio lessons in Modern Greek for English speakers back in 1960's.

The lessons were retrieved from the archives of the Cyprus Broadcasting Corporation, digitized in Real Audio 5.0 format and published on-line.

Page 11: Greek Evaluation

Prototype (Continued)

Although, an optional textbook accompanied the original Radio lessons, the lessons were designed as a complete standalone course.

We used several tools to assist the students of the lessons, including an online English-Greek-English dictionary, a Greek spell checker and a web-based discussion board for the students of the lessons.

Page 12: Greek Evaluation

STEP 4: Integrate feedback and continue the iteration. Feedback from our users and suggestions are

continuously incorporated into our design through a series of additions and corrections.

An important element in the participatory design methodology is the direct involvement of the user in all stages of the design process (including the evaluation).

Page 13: Greek Evaluation

Student Involvement

Users started exchanging through email written notes taken by the advanced users. – We provided the users with the capability to post their

notes on the project's site. Two months later, the students of the Audio lessons

have managed to transcribe 81 out of the 105 lessons, correct them through a peer review process among themselves and post them on the project's website.

Six months after that, the students had transcribed and peer reviewed all 105 lessons.

Page 14: Greek Evaluation

Evaluation Stages

1. Analysis of server logs 2. Analysis of discussion board postings 3. Questionnaires 4. Usability Testing

Page 15: Greek Evaluation

Key Evaluation Questions

How useful is the course? How functional is the course? How well does it meet on-line course

standards? What can be done to improve the course?

Page 16: Greek Evaluation

Analysis of server logsAccess Statistics

R2 = 0.7776

R2 = 0.6711

R2 = 0.606

0

5000

10000

15000

20000

25000

1 6 11 16 21 26 31 36 41 46

# of weeks

# o

f p

age

view

s

Total accesses

Audio accesses

Notes accesses

Board accesses

Boards+Notes

Expon. (Total accesses)

Expon. (Audioaccesses)

Expon. (Boards+Notes)

Page 17: Greek Evaluation

Questionnaires

Page 18: Greek Evaluation

Questionnaire 1 - ResultsQuestion Rating Q1 Overall, I am satisfied with how easy it is to use this online course. 1.40 (1.14) Q2 It was simple to use this online course. 1.40 (1.13) Q3 I can effectively complete my work using this online course. 1.20 (1.22) Q4 I am able to complete my work quickly using this online course. 1.05 (1.27) Q5 I am able to efficiently complete my work using this online course. 1.10 (1.24) Q6 I feel comfortable using this online course. 1.49 (1.07) Q7 It was easy to learn to use this online course. 1.37 (1.15) Q8 I believe I became productive quickly using this online course. 1.19 (1.17) Q9 The online course gives error messages that clearly tell me how to fix problems. 0.13 (1.60) Q10 Whenever I make a mistake using the online course, I recover easily and quickly. 0.81 (1.29) Q11 The information (such as help, on screen messages and other documentation) provided with

this online course is clear. 0.89 (1.34)

Q12 It is easy to find the information I needed. 0.83 (1.37) Q13 The information provided for the online course is easy to understand. 1.24 (1.07) Q14 The information is effective in helping me complete the tasks and scenarios. 1.19 (1.06) Q15 The organization of information on the online course screens is clear. 1.20 (1.23) Q16 The interface of this online course is pleasant. 1.30 (1.12) Q17 I like using the interface of this online course. 1.23 (1.17) Q18 This online course has all the functions and capabilities I expect it to have. 0.99 (1.30) Q19 Overall, I am satisfied with this online course. 1.40 (1.06)

Page 19: Greek Evaluation

Questionnaire 1: User Satisfaction

1.5 1.5 1.5 1.4

1.1 1.10.9 0.9

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Overall Ease ofuse

Overallsatisfaction

Simple to use Easy to Learn Info Effective toComplete Tasks

Complete WorkEfficiently

Easy to Find Info Information Clear

Sca

le fr

om -2

to +

2

Page 20: Greek Evaluation

Questionnaire 1: Findings

User Expectations

– High satisfaction

– Positive comments• Easy to understand

• Fun

• It helped refresh my Greek

• Finally after surfing for a long time I found that you could learn Greek from the internet

Page 21: Greek Evaluation

Questionnaire 2: (Demographics)

Total of 16,848 registered students (6,098 completed the demographics questionnaire) – 49% Male, 51% Female– 65% found the page through a link from an

other page– Age 15-30 (44%), 31-50 (35%), 51- (10%)– Internet usage: 1-3 years (34%), 4-6 years

(37%), 7+ (16%)– Primary language: 72% English

Page 22: Greek Evaluation

Questionnaire 2: (Demographics)

– Education: High School (16%), Some College (20%), College Graduate (22%), Master (10%), PhD (3%)

– Location: Europe (38%), USA (43%)– Computer Platform: Windows 98 (54%),

Windows 2000 (20%), MAC (2%)

Page 23: Greek Evaluation

Analysis of Discussion Board (371 postings)

Category/Subject # % Total %TechnicalProblems with needing/installing keyboard fonts 56 15.1%Problems with/wanting to download lessons 49 13.2%Problems with installing/using RealPlayer 40 10.8% 39.1%

ContentPostings of on-line notes 73 19.7%Questions about availability of text, CDs or

including links to web sites 37 10.0% 29.7%

ComprehensionQuestions about vocabulary 28 7.5%Questions about grammar/spelling 17 4.6%Questions about dialect 2 5.4% 17.5%

Miscellaneous 69 18.6% 18.6%

Page 24: Greek Evaluation

Formal Usability Test

How functional is the course?– Do the operations work?– Is the architecture successful?

Findings– Site is subject to typical Internet problems– Learn Greek! architecture can be improved

Page 25: Greek Evaluation

Formal Usability Test - Findings Operations

– The site generally works (Translations, lessons, etc. function)

– Web operations less dependable (Server downtime, Slow lesson downloads, Intermittent transmission)

Architecture– Labeling

(Incorrect, Confusing, Out-of-order)

– Content (No directions, No descriptions, No “Help” or FAQ, Multiple translations)

Page 26: Greek Evaluation

Formal Usability Test - Site Architecture-Labeling

Page 27: Greek Evaluation

Formal Usability Test - Site Architecture-Formats

Page 28: Greek Evaluation

Discussion

Students created an open online community whose collaboration has boosted the learning experience of the whole community.

The web-based discussion board has proven to be the most constructive tool for the students learning experience and the main source of feedback for the maintainers of the project.

Page 29: Greek Evaluation

Discussion

The experiences shared on the discussion board include tricks and tips on how to record the audio files, installation of Greek fonts, learning methodologies and questions about the Greek language itself that arise from the lessons.

Also used for announcing the availability of their personal notes.

Page 30: Greek Evaluation

User Leaders

The advanced users (some of them retired teachers of foreign languages) have taken a lead role in the vast majority of the threads on the discussion board

Answering most of the questions and encouraging the beginners to study the lessons further.

Communication interface between the maintainers of the project and the community's requests. Maintain the online notes pages

Page 31: Greek Evaluation

Discussion

The accesses the audio lessons, the language tools, and the total access of the message board and the notes pages, all kept increasing exponentially.

Once we allowed our users to publish their own notes, there was a dramatic shift of traffic from the message board to the notes pages. Probably– Because it was no longer necessary to visit the message

board to find out where other users had posted their notes, since all were aggregated in a central location.


Introduction to the Greek New Testament Manuscripts...In Part I, Manuscripts and Text Evaluation, an introduction to using the Greek New Testament evidence is followed by examples

Introduction to the Greek New Testament Manuscripts...In Part I, Manuscripts and Text Evaluation, an introduction to using the Greek New Testament evidence is followed by examples

Ancient Greek to Modern Architecture with Greek Influence ... · Ancient Greek to Modern Architecture with Greek Influence - News - Greek Gateway Wednesday, 05 October 2011 22:20

Ancient Greek to Modern Architecture with Greek Influence ... · Ancient Greek to Modern Architecture with Greek Influence - News - Greek Gateway Wednesday, 05 October 2011 22:20

Communicate in Greek Greek Edition

Communicate in Greek Greek Edition

Museum Entrance Welcome to the Lobby Greek Olympics 1 Greek Olympics 2 Greek Olympics 4 Greek Olympics 3 Greek Olympics Visit the Author First Winner Of.

Museum Entrance Welcome to the Lobby Greek Olympics 1 Greek Olympics 2 Greek Olympics 4 Greek Olympics 3 Greek Olympics Visit the Author First Winner Of.

Ancient Greek Religion. Agenda Ancient Greek Civilization – key characteristics What is Greek Mythology? Greek Mythology in Modern Culture The Origin.

Ancient Greek Religion. Agenda Ancient Greek Civilization – key characteristics What is Greek Mythology? Greek Mythology in Modern Culture The Origin.

Greek Mythology Jeopardy. Greek Gods Greek Goddesses TitansCreatures The Lightning Thief 100 200 300 400 500 Greek Mythology Jeopardy.

Greek Mythology Jeopardy. Greek Gods Greek Goddesses TitansCreatures The Lightning Thief 100 200 300 400 500 Greek Mythology Jeopardy.

Homeric Greek - Textkit | Greek and Latin Learning Toolscdn.textkit.net/CP_Homeric_Greek_AR5.pdf · Greek and Latin Learning Tools Learn Greek and Latin at ... Title: Homeric Greek

Homeric Greek - Textkit | Greek and Latin Learning Toolscdn.textkit.net/CP_Homeric_Greek_AR5.pdf · Greek and Latin Learning Tools Learn Greek and Latin at ... Title: Homeric Greek

Identification and social evaluation of dialectal Greek intonation

Identification and social evaluation of dialectal Greek intonation

EVALUATION OF GREEK PUBLIC HOSPITAL WEBSITES

EVALUATION OF GREEK PUBLIC HOSPITAL WEBSITES

Analysis and Evaluation of English 6th Grade Textbook Used in Greek Public Schools-libre

Analysis and Evaluation of English 6th Grade Textbook Used in Greek Public Schools-libre

ANCIENT GREEK ARTWORKS - joslyn.org Plan - Greek...ANCIENT GREEK ARTWORKS ... and ... BC, black-figure, commerce, functional art, Greek mythology terms, Greek vase shapes ...

ANCIENT GREEK ARTWORKS - joslyn.org Plan - Greek...ANCIENT GREEK ARTWORKS ... and ... BC, black-figure, commerce, functional art, Greek mythology terms, Greek vase shapes ...

storage.googleapis.com  · Web viewphobia (Greek: “fear”) philia (Greek: “affection”) mania (Greek: “madness; frenzy”) biblio (Greek: biblion “paper” or “book as

storage.googleapis.com  · Web viewphobia (Greek: “fear”) philia (Greek: “affection”) mania (Greek: “madness; frenzy”) biblio (Greek: biblion “paper” or “book as

The sound of Greek A critique of Greek · PDF fileThe sound of Greek A critique of Greek phonology. ... attacked Modern Greek as a degenerated language. ... ideal Greek at an ideal

The sound of Greek A critique of Greek · PDF fileThe sound of Greek A critique of Greek phonology. ... attacked Modern Greek as a degenerated language. ... ideal Greek at an ideal

eBook Ancient Language Greek Wwg Greek Grammar

eBook Ancient Language Greek Wwg Greek Grammar

Greek salads - Kerby's Koney Island · Greek salads Served with grilled pita and your choice of KERBY’s Original Greek, Creamy Greek or Lo-Cal Greek Dressings Greek Salad small

Greek salads - Kerby's Koney Island · Greek salads Served with grilled pita and your choice of KERBY’s Original Greek, Creamy Greek or Lo-Cal Greek Dressings Greek Salad small

Greek vs. Greek Athens and Sparta - Greek Play

Greek vs. Greek Athens and Sparta - Greek Play

Minotaur, Labyrinth, Greek Tragedy and Greek Architecture

Minotaur, Labyrinth, Greek Tragedy and Greek Architecture

Top Six Sweet Greek Treats - Greek Recipes - Greek …greekgateway.com/greek-recipes/top-six-sweet-greek...Top Six Sweet Greek Treats - Greek Recipes - Greek Gateway It’s pretty

Top Six Sweet Greek Treats - Greek Recipes - Greek …greekgateway.com/greek-recipes/top-six-sweet-greek...Top Six Sweet Greek Treats - Greek Recipes - Greek Gateway It’s pretty

The Greek Revolution/ Greek War of Independence

The Greek Revolution/ Greek War of Independence

The Early Greek world and Greek Myths - latrobe.edu.au · The Early Greek world and Greek Myths Chris&Mackie& &

The Early Greek world and Greek Myths - latrobe.edu.au · The Early Greek world and Greek Myths Chris&Mackie& &