Online Survey and Assessment Techniques

Dr. Curtis J. Bonk Associate Professor, Indiana

UniversityPresident, CourseShare.comhttp://php.indiana.edu/~cjbonk,

cjbonk@indiana.edu

Pause…What tools do you use

or know about?

Collecting Evaluation Data

& Online Evaluation Tools

Types of Data Collection

• Direct Observation in Work Setting– By supervisor, co-workers, subordinates,

clients

• Collect Data By Surveys, Interviews, Focus Groups– Supervisors, Co-workers, Subordinates,

Clients

• Self-Report by learners or teams• Email and Chat• Online Tests• E-Portfolios or Learning Logs

I. Online Testing Tools

(see: http://www.indiana.edu/~best/)

Choice: Select companies that

specialize in online assessment.

Or: Use what the courseware package gives ya…

Some things in life are free…!

Test Selection Criteria

(Hezel, 1999)• Easy to Configure Items and Test• Handle Symbols• Scheduling of Feedback

(immediate?)• Provides Clear Input of Exam

Dates• Easy to Pick Items for Randomizing• Randomize Answers Within a

Question• Weighting of Answer Options

More Test Selection Criteria

• Recording of Multiple Submissions

• Timed Tests• Comprehensive Statistics• Summarize in Portfolio and/or

Gradebook• Confirmation of Test

Submission

More Test Selection Criteria

(Perry & Colon, 2001)• Supports multiple items types—multiple

choice, true-false, essay, keyword• Can easily modify or delete items• Incorporate graphic or audio elements?• Control over number of times students

can submit an activity or test• Provides feedback for each response

• Flexible scoring—score first, last, or average submission

• Flexible reporting—by individual or by item and cross tabulations.

• Outputs data for further analysis• Provides item analysis statistics

(e.g., Test Item Frequency Distributions).

More Test Selection Criteria

(Perry & Colon, 2001)

Web Resource: http://www.indiana.edu/~best/

II. Electronic Gradebooks (Vockell & Fiore, 1993)

• Calculate scores, store info• Weight scores• Flag students with certain

characteristics• Print reports by individual or

group• Provide prompt feedback• But inflexible, impersonal, &

can be incorrect

III. Online Survey Tools for

Assessment

Perhaps you have heard of my survey

techniques?

Survey Tool Features

• Maintain email lists and email invitations• Conduct polls• Adaptive branching and cross tabulations• Modifiable templates• Maintain library of past surveys• Publish reports• Technical support, chat advice• Different types of accounts—hosted,

corporate, professional, etc.

Still More Online Survey Features

• Online help, FAQs, demos, tutorials, white papers, glossaries, site maps, tips, overviews

• Website evaluation tools or options• Supports different types of items

(Likert, multiple choice, forced ranking, paired comparisons, etc.)

• Customizable audience lists• Launch and close surveys

More Online Survey Features

• Online viewing of assessment results and question analysis (including various graphical displays)

• Easy to author surveys• Instant feedback to participants• Custom reporting, editable images

and background, personalized greetings

• Storage of results in Access or SQL

Still More Features

• Ability to start and stop a survey• Reports on subgroups• Randomize q’s to reduce bias• Easy email reminders• Personalized banner or logo• Expanded question options• Ability to copy surveys

Web-Based Survey Advantages

• Faster collection of data• Standardized collection format• Computer controlled branching

and skip sections• Graphics can reduce fatigue• Easy to answer clicking• Wider distribution of

respondents

Web-Based Survey Problems: Why Lower

Response Rates?• Low response rate• Lack of time• Unclear instructions• Too lengthy• Too many steps• Can’t find URL• Perceived as aggressive

Web-Based Survey Solutions: Some Tips…

• Send second request• Make URL link prominent• Offer incentives near top of request• Shorten survey, make attractive,

easy to read• Credible sponsorship—e.g.,

university• Disclose purpose, use, and privacy• E-mail cover letters• Prenotify of intent to survey

Tips on Authentification

• Check e-mail access against list• Use password access• Provide keycode, PIN, or ID #• (Futuristic Other: Palm Print,

fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID)

Sample Survey Tools• Zoomerang

(http://www.zoomerang.com)• IOTA Solutions

(http://www.iotasolutions.com)• QuestionMark

(http://www.questionmark.com/home.html)

• Survey Solutions from Perseus (http://www.perseusdevelopment.com/fromsurv.htm)

• Infopoll (http://www.infopoll.com)

Sample Survey Tools

• Active Feedback • (http://www.activefeedback.com/af)• Majon International • (http://www.coolsurveys.com)• EZSurvey from Raosoft • (http://www.raosoft.com/)• SurveyShare

(http://SurveyShare.com; from Courseshare.com)

Zoomerang Steps

1. Register: Sign Up for Basic Membership (SKIP FOR NOW!)

2. Click on new survey.

3. Click on survey template (e.g., “Education” and Professor)

4. Delete, modify, or add 1-2 items.

5. Submit survey.

Zoomerang Steps

6. Modify Web greeting and submit.

7. Pick option #1 for submission.

8. Type Survey invitation & email message

9. Enter email addresses for potential survey respondents

10. Decide on how share results

Zoomerang Steps

11. Launch survey

12. Click on “My Surveys” and browse

13. Click on “Email List” and browse

14. Click on “Account Info”

15. Other explorations…

SurveyShare Steps

1. Register: Sign Up (perhaps take Quick Tour, Flash version)

2. Login3. Click on Create Survey4. Check out existing templates5. Select one of the templates6. Add one item, delete one item, modify

one item

SurveyShare Steps

7. Edit Survey Introduction and Title

8. Edit Closing and Submit

9. Select Survey Invitation Via Email

10. Type emails to complete the survey

11. Click add (to move emails to right)

12. Give email subject and message

SurveyShare Steps

13. Enter email subject line and text

14. Preview survey

15. Activate survey

16. Go to MySurveys

17. Click on Survey Info

18. Go to mailing lists

SurveyShare Steps

19. Add a person

20. Add list

21. Check out resources (books)

22. Check out help, quick tour, and sitemap

23. Perhaps create a survey from scratch

24. Perhaps create a survey from “My Previous Surveys”

Any Reactions to Online

Surveys???

Fitting Surveys into Evaluations

Plans

Bonk’s Evaluation Plan…

Considerations in Evaluation Plan

1. Student

2. Instructor

3. Training

4. Task5. Tech Tool

6. Course

7. Program

8. University or

Organization

What to Evaluate?

1.Learner—attitudes, learning, use, performance.2.Instructor—popularity, course enrollments.3.Training—internal and external components.4.Task--relevance, interactivity, collaborative.5.Tool--usable, learner-centered, friendly,

supportive.6.Course—interactivity, participation, completion.7.Program—growth, long-range plans.8.Organization—cost-benefit, policies, vision.

The 4 Levels

•Reaction•Learning•Behavior•Results

Figure 26. How Respondent Organizations Measure Success of Web-Based Learning

0102030405060708090

Learner satisfaction Change inknowledge, skill,

atttitude

Job performance ROI

Kirkpatrick's Evaluation Level

Pe

rce

nt

of

Re

spo

nd

en

ts

Ziegler, April 2002, e-Learning

“…the key is not to measure every possible angle, but rather to focus on metrics that are pragmatic and relevant to both human and business performance at the same time.”

Some Final Advice…

Or Maybe Some Questions???