JAG Second Language Software Evaluation Model

8/6/2019 JAG Second Language Software Evaluation Model

1/15

The JAG Second Language Software Evaluation Model


Submitted by Amy Barrieau, Gina Boudreau and Joan Cunningham

A group project submitted to Dr. D Lloyd,

in partial fulfillment of the requirements of

EDUC 531: Assessment of Software & Information Tech Applications for Education

Faculty of Education

Cape Breton University

Sydney

Nova Scotia


2/15

The JAG Second Language Software Evaluation Model 2

July 25th, 2011


3/15


Table of Contents

Part A- Evaluators and Scenarios for Implementing JAG Software Testing...1

Part B- The 9 Steps of the JAG Second Language Software Evaluation Model.1

Part C- Rationale..5

Part D-Checklist and Quick Guide..7

References..12


4/15


Part A- Evaluators and Scenarios for Implementing JAG Software Testing

The JAG Second Language Software Evaluation Model is designed to be employed by

various groups, from a single second-language classroom teacher, to specialized personnel

interested in widespread software use. Depending on the scope of the intended software

implementation and the allotted resources for the evaluation process, combinations of steps 1 to

9 of the JAG model can be applied to fit multiple evaluation scenarios.

All evaluators should complete Steps 1 to 5 of the JAG checklist, which involves

exploring need, general features of the software, instructional objectives, evaluation methods and

student attitude checks. Those who are working within a budget and/or limited timeframe may

then proceed to Step 9 which suggests researching post-user feedback and constructing a final

recommendation of the software in question.

For the JAG model to be most effective and thorough, Steps 6 to 8 should also be

completed by the evaluator. These steps include initial observations and student pre-testing, one-

on-one student evaluation of software-use and small group evaluation. The JAG model supports

the belief that, as Komoski and Plotnick (1995) purport the most effective way to judge whether

software is appropriate or not is to observe students as they interact with the program. We feel

this is an essential component, especially when assessing second language learning.

Some evaluators may choose to complete the one-on-one student evaluation and leave out

the small group evaluation, or vice-versa. Again, completing all steps is best practice; however

sound software recommendations can be achieved using the combinations suggested above.

Part B- The 9 Steps of the JAG Second Language Software Evaluation Model

In our mission to help teachers select appropriate software to fulfill the needs of second

language learners, we have created the following 9 steps to achieve effective software


5/15


evaluation. This model is based primarily on that of Reiser and Dick (1990), which also places

great value on student input and field testing. Other models, including that of Squires and Preece

(1999) and Evalutech (2009) were essential in informing the criteria put forth in the JAG model.

Essential language outcomes addressed via effective software, as outlined by the aggregator site

Top Ten Reviews (2011), were also considered in our review of learning objectives.

Step 1: Exploring Software Need and Premise

This initial step involves using a checklist (See Part D of this document). This checklist

is important in identifying the need for the software in question before proceeding with the rest

of the evaluation. Evaluators must reflect on the intended use of the software and whether such

software is the appropriate medium for representing and measuring the anticipated language

learning outcomes. Also, to ensure credibility, the evaluator must attempt to find out whether the

software is upheld by a notable source or linguistic society and judge at first glance, perhaps

from the packaging or alternative reviews or advertisements, whether the software claims to

coincide with applicable goals of language education.

Step 2: Exploring General Features of the Software

This step is also completed by use of a checklist. It requires the evaluator to begin

working through the software. An evaluator should first decipher if the software comes with a

free demo to test before purchase. In second language software, it is also important to determine

who the intended users are, whether the software assesses previous capabilities and whether it is

able to interact with students. When correcting or guiding students, the software may also need

to address the student in their native tongue, highlight or underline text, give feedback, and track

student progress.

Step 3: Exploring and Developing Instructional Objectives


6/15


Teachers are often given opportunities to purchase software specific to certain subject

areas. It is important that teachers acting as evaluators consider whether the lessons offered via

certain software coincide with the intended classroom outcomes. Evaluators using the JAG

model are encouraged to assess both basic-level second language learning objectives, like

questioning techniques, speaking and writing in short sentences and identifying errors, in

addition to activities that prompt higher-order thinking. In speaking of the latter, such features

more easily motivate and engage students and allow them to flex their repertoire of multiple

intelligences.

Step 4: Self-evaluation and Self-monitoring of Progress

In second language learning, students need opportunities to listen and improve their

comprehension and opportunities to speak, hear themselves speak and have their speech

corrected and improved. This can be done in a limited fashion in a classroom of 20-30 students.

Although language labs are traditionally not part of a junior high or high school set up, a good

piece of software will offer students a similar experience. Software allows all students to be

working on their own individual progress in the areas of speaking and listening comprehension.

Selected software for a second language classroom needs excellent audio production and

playback in order to facilitate this process. As the website, Top Ten Reviews (2011), points out:

Top learn-French software includes speech recognition tools, which "listen" to your French

pronunciation, pinpoint mistakes and give corrections. Students should also have the

opportunity to track their progress in speaking and listening comprehension. A top piece of

software for second language instruction will allow students to keep records and recordings and

give teachers the opportunity to review the records and test items.

Step 5: Attitude Check


7/15


If students will be working independently on their language skills, instructors should

have the opportunity to review not only their learning records but their attitude while working.

Some students may not enjoy trying to produce language in a one-way setting and the instructor

needs this feedback.

Step 6: Initial Field Test Observations, Pre-testing and Selection of Candidates for One-on-

one Evaluation of Software;

Step 7: Conduct One-on-one Evaluation and Student Evaluations;

Step 8: Conduct small group evaluation

A key requirement of the JAG Software Evaluation Model is to have students intimately

participate in the evaluation of software for the second language classroom. Steps 6, 7 and 8 of

the model allow this to occur. What may appear on the surface to meet curriculum guidelines,

different learning styles and technology requirements may still not be comfortable for the

second language student if the language (speed, intonation, accent) is not appropriate for the

students. Students in a second language classroom become very accustomed to the accent of

their particular teacher and the speed with which the teacher speaks. In exposing students to

second language software, the instructor wants the students to have the advantage of hearing

other people speak the language and yet, the language included in the software needs to be clear

enough for the learner to comprehend. The main goal of second language teachers is to have

their students speak the language authentically in the classroom (Claes, 2003). If students are

not comfortable with the tools provided to them to assist them to that end, it will undermine the

goals of the second language classroom. As Jan Claes (2003), a leader in Core French reform in

the HRSB said, Core French students do not use their French, except under duress or lure of


8/15


candy. It is important that software be appealing to students and incite them to speak the

language more readily.

Step 9: Write evaluation report using findings and outside sources

This final step involves reviewing notes, observations and checklist items and drawing

conclusions based on the data accrued. Also included in this step is a research component, where

the evaluator is encouraged to seek literature and post-user feedback to substantiate their ultimate

recommendation. Such research may link to the data initially retrieved in Step 1 that required

the evaluator to seek out recommendations from a notable source or linguistic society. Within

this final report, the evaluator should summarize the review process and articulate

recommendations regarding the use of the tested software in a second language setting.

Part C- Rationale

As mentioned, the JAG Second Language Software Evaluation Model is based on

thorough research completed by Reiser and Dick (1990), Squires and Preece (1999), and

Evalutech (2009). The JAG model borrows from all three of these very comprehensive albeit

differing models of software evaluation.

As Second Language instruction insists that students be exposed to different texts, audio,

and visuals, and that they also experience authentic language (Cleas, 2003), the quality of

resources used in the second language classroom must be examined not only by experts, but also

by students. Therefore, JAG borrows from software evaluation models like Reiser and Dick

(2009) where the student is involved in the evaluation process at some point. Experts can

examine and evaluate the authenticity of the language used in the software and the heuristic

techniques put forth in the software. However, if students are not comfortable with, for example,


9/15


the accents or the speed and intonation of the voices on the software, it will be difficult to

encourage them to take full advantage of the opportunity that a language software tool offers.

Authenticity and comprehension of the language spoken on the software are important,

but do not stand alone in the Second Language classroom. Students also require relatable topics

and scenarios and age-appropriate information to help them engage and to learn a language based

on their ability. They also require feedback and support to encourage them to go forward when

they are unsure. Therefore, second language software must follow many of the items mentioned

in the Evalutech model (2009) developed for schools in North Carolina. In addition, it is of

utmost importance that the second language student can bring their personal experience to the

language classroom or to a piece of software through some creative problem solving of scenarios

that they find realistic and relatable. Two sections of the Evalutech (2009) checklist model are

highly correlated to the needs of the second language classroom:

Presentation:

Information presented in a manner to stimulate imagination and curiosity

Activities that provide opportunities for creative problem solving

Use of appropriate and supportive feedback

Options for help, tutorial segments

Uncluttered screen displays

Captions, labels, or legends for visuals

Legible text and print size that is appropriate for the intended audience

Quality:

Visuals relevant to the content

Sound that is clearly understandable and consistent in quality and volume


10/15


Sound and music that is relevant to screen displays

If software is heuristically-based, as Squires and Preece (1999) recommend, then the frustration

level for a second language learner can be drastically reduced. A student who is trying to use

their personal experience to complete second language work should be able to manipulate the

software intuitively and logically. Students should also be able to use their intuition and personal

experience to work through the language aspect of the software as well. Therefore, the

importance of heuristics in second language software are twofold, the intuition used to navigate

the software and the experience and intuition the student can draw from his/her life to solve the

language problems presented. If the stories and scenarios are not relatable, or are not age-

appropriate, there is little motivating the student to become involved in learning a second

language.

Finally, there are several other basics of software evaluation items that must also be

present in the software evaluation tool for second language. Information and test questions must

be error free and bias free and must embrace a wide range of abilities and cultures. Expert

opinion can answer most of these criteria. However, a student evaluation in addition to expert

opinion will help answer all of the questions as they present themselves in practice, in a second

language classroom prior to purchasing software.

Part D-Checklist and Quick Guide

The following is both a checklist and quick guide for the JAG Second Language Software

Evaluation Model. Items included in the checklist are considered valuable attributes to the

software in question. Other italicized items or steps are included in this table to create a

complete image, or quick reference guide of our model. Please refer to Part A for suggestions on

how to best use this guide, depending on your evaluation needs.


11/15


Preliminary Information

About the Evaluator

Name:

Current Title:

Credentials Applicable to Second Language Learning:

Name of School and District:

Contact Information:

Date of Evaluation:

About the Software

Title:

Publication Date:

System Requirements:

Cost:

Step 1: Exploring the Software Need and Premise

Is using computer software the appropriate medium for the intended language learningoutcome(s)?

Is this software recommended by a notable source or linguistic society?

Does this software claim to coincide with one or more general goals of language education (i.e.:

oral comprehension/production, written comprehension/production)? Notes:

Step 2: Exploring General Features of the Software

Is there a free demo available to test before purchase?


12/15


What is the intended grade level?

Does this software take into account previous second language education?

Does this software have bilingual capabilities for aiding second language students in their nativelanguage?

Does this software highlight written errors and give students explanations and other

improvement options? Are there oral correction and/or interaction capabilities between the student and this software?

Does this software track student progress?

Notes:

Step 3: Exploring and Developing Instructional Objectives

Based on the language lessons offered, do the specific objectives put forth in this software

coincide with the intended classroom outcome(s)?

Evaluators may also consider whether the following basic second language learning objectivesare addressed in the software:

- Students can speak in short sentences. - Students are able to ask simple questions. - Students are able to understand basic commands. - Students begin to write in short sentences. - Students are able to identify simple errors within their own writing. - Students are able to identify simple errors within their speaking. - Students are able to identify errors while listening to the second language.

21st Century Learning Check:- Are varied instructional techniques employed?

- Are critical thinking skills encouraged? - Are there opportunities for constructivist learning? - Are learning styles and/or multiple intelligences taken into consideration?

Notes:

Step 4: Self-evaluation and Self-monitoring of Progress

In considering test items

Are there test items included?

If yes, do they coincide with the prescribed outcomes of the second language class? Is there logical progression in the test questions?

Are test questions 100% free from error?

Are test questions bias free?

Do test questions require some higher-order thinking?

Can test records be retained?

Can students compare future test scores for improvements in learning?

Can instructor access test records and progress records?


13/15


Can instructor access oral responses given by students?

Do test items need to be changed?

Can test items be changed?

Who has authority/access to change them?

Notes:

Step 5: Attitude Check

Is student attitude assessed for teacher review purposes?

Notes:

Step 6: Initial Field Test Observations, Pre-testing and Selection of Candidates for One-on-one Evaluation of Software

Observations: Can student navigate a new piece of software with some comfort?

Conduct pre-test and choose 3-4 students by ability. Consider the following:

Do the selected students represent a range of second language ability in the classroom?

Allow students to preview the questions in Step 7 to ensure selected students are comfortable

with the language of the questions.

Notes:

Step 7: Conduct One-on-one Evaluation and Student Evaluations

Instructor is present for this single interactive one-on-one evaluation session and gathers

information by observation notes and questioning during the one-on-one evaluation. Studentsalso should rate the software during the session. Following a scale of 1 to 5, 1 being the lowest

score and 5 being the highest or strongest score they can give, students should rate the

following:

During instruction/learning:

Are the screens/visual presentation of the screens stimulating?

Are the screens and the presentation of material confusing?

Are the images and interface age-appropriate?Does the work progress logically?

How easy is it to navigate this program?

Does this program offer an appropriate challenge to you?Was there enough variety of activities?

After instruction/learning:

When indicated, students take a teacher or software created assessment that is based onthe intended learning objective(s). Each student also responds to an attitude


14/15


questionnaire, describing their satisfaction with the experience. Questions evaluators

may use include:Rate your overall experience on a Likert scale of 1 to 10, 10 being the top rating.

Describe your satisfaction with this software.

Discuss the pros and cons of using this product.

Notes:

Step 8: Conduct Small Group Evaluation

Return to findings in Step 6 and now select 8 to 20 students whose capabilities, as measured by

the pre-test, represent a range of language abilities. Students should study using the softwarefor a prescribed amount of time, as would be allotted in a typical class or lab setting at their

school. Then, students are given summative post-test to see how much they have learned. They

also respond to a questionnaire that assesses attitude.

Notes:

Step 9: Write Evaluation Report using Findings and Outside Sources

Is there post-user feedback available and/or research to support your findings?

Review the information collected throughout this evaluation process and prepare a report,including a summary and recommendations regarding the use of this particular software in a

second language setting.

Notes:


15/15


References

Claes, J. (2003). Kiss my donkey or raising the level of authentic communication in my core frenchclassroom. An Online Journal for Teacher Research. Retrieved from:http://journals.library.wisc.edu/index.php/networks/article/view/104/105

Department of Education, Nova Scotia (1998). Orientation document for atlantic canada core

French curriculum.Acadian and French Language Services Branch. Retrieved from:

http://www.ednet.ns.ca/pdfdocs/curriculum/core_french.pdf

Evalutech (2009). Criteria for evaluating computer courseware. SREB. Retrieved from:

http://courseware.cbu.ca/moodle/file.php?file=%2F458%2FEvalutech.pdf

Komoski, P., Plotnick, E. (1995). Seven steps to responsible software selection.ERIC

Clearinghouse on Information and Technology Syracuse NY. Retrieved from:https://reader009.{domain}/reader009/html5/0516/5afb12b73e59a/5afb12beb8216./ERIC/Softwa

Reiser, R., Dick, W. (1990). The model of Reiser and Dick. Evaluating Instructional Software.Retrieved from: http://tecfa.unige.ch/staf/staf9698/mullerc/3/evalRD.html

Top Ten Reviews. (2011). Learn French Software Review. Retrieved from:http://learn-french-software-review.toptenreviews.com/

Squires, D., Preece, J. (1999). Predicting quality in educational software: Evaluating for learning,

usability and the synergy between them. Interacting with Computers. Retrieved

from:http://web.archive.org/web/20080529070706/http://www.ifsm.umbc

.edu/communities/Heur2.html
http://journals.library.wisc.edu/index.php/networks/article/view/104/105http://www.ednet.ns.ca/pdfdocs/curriculum/core_french.pdfhttp://courseware.cbu.ca/moodle/file.php?file=%2F458%2FEvalutech.pdfhttp://www.vtaide.com/png/ERIC/Software-Selection.htmhttp://tecfa.unige.ch/staf/staf9698/mullerc/3/evalRD.htmlhttp://learn-french-software-review.toptenreviews.com/http://web.archive.org/web/20080529070706/http://www.ifsm.umbc%09%09%09%09%09.edu/communities/Heur2.htmlhttp://web.archive.org/web/20080529070706/http://www.ifsm.umbc%09%09%09%09%09.edu/communities/Heur2.htmlhttp://journals.library.wisc.edu/index.php/networks/article/view/104/105http://www.ednet.ns.ca/pdfdocs/curriculum/core_french.pdfhttp://courseware.cbu.ca/moodle/file.php?file=%2F458%2FEvalutech.pdfhttp://www.vtaide.com/png/ERIC/Software-Selection.htmhttp://tecfa.unige.ch/staf/staf9698/mullerc/3/evalRD.htmlhttp://learn-french-software-review.toptenreviews.com/http://web.archive.org/web/20080529070706/http://www.ifsm.umbc%09%09%09%09%09.edu/communities/Heur2.htmlhttp://web.archive.org/web/20080529070706/http://www.ifsm.umbc%09%09%09%09%09.edu/communities/Heur2.html

JAG Second Language Software Evaluation Model

Documents

Transcript of JAG Second Language Software Evaluation Model