Instructional Design Project - University of...

Instructional Design Project

LLDT Collective

Laura Lucas

Lauren Butcher

Dawn Cancelliere

Tom Hammond

Instructional Design

The University of Akron

Dr. Cheryl Ward

April 25, 2012

Instructional Design Project 2

Table of Contents

Introduction .............................................................................................................................. 3

Analysis Phase ......................................................................................................................... 3

Design Phase ............................................................................................................................ 6

Task Analysis .............................................................................................................. 6

Learning Theory .......................................................................................................... 6

Sequencing .................................................................................................................. 7

Instructional Objectives ............................................................................................... 8

Module 1: Learning in the 21st Century ....................................................................... 8

Module 3: Evaluating Websites & Resources.............................................................. 9

Development Phase ................................................................................................................ 10

Organization & Presentation for both Modules ......................................................... 10

A Closer Look at Module 1: Learning in the 21st Century ......................................... 12

Module 1: Added and Created Media/Informational Resources with Rationale ........ 13

A Closer Look at Module 3: Evaluating Websites & Resources ............................... 14

Module 3: Added and Created Media/Informational Resources with Rationale ........ 15

Extra Module that was Developed: Help and Tutorials for Online Learners ............. 16

Implementation Phase ............................................................................................................ 16

Implementation Action Plan ...................................................................................... 17

Alignment with the CLER Model.............................................................................. 18

Timeline .................................................................................................................... 20

Evaluation Phase .................................................................................................................... 20

Evaluation Method .................................................................................................... 20

Validity ..................................................................................................................... 22

Assessing Knowledge................................................................................................ 22

Evaluation Standards ................................................................................................. 23

Evaluation Results ..................................................................................................... 23

Conclusion ............................................................................................................................. 27

References .............................................................................................................................. 28


Introduction

The instructional design problem presented to the LLDT Collective was to move a current course,

Educational Technology, from a face-to-face format to a fully online course. The team was responsible

for creating materials for two modules: Learning in the 21st Century and Evaluating Websites and

Resources. The team completed the ADDIE model in order to ensure that course objectives were met, the

needs of the learners were met, the materials aligned with course objectives and evaluations, and that

proper implementation and evaluation phases were conducted.

Analysis Phase

Dr. Kline is a professor at the University of Akron in the department of Curriculum and

Instructional Strategies. She teaches a course titled Educational Technology. Recently, the course was

updated to address the need to have pre-service teachers prepared for 21st century teaching and learning.

The course is currently designed and delivered as a face-to-face course and contains web enhanced

learning components. Although it is organized in modules, the modules need revised so that students are

able to obtain the knowledge, concepts, skills, and abilities to complete projects through a program that is

entirely online. According to ISTE’s National Educational Technology Standards for Teachers (NETS-

T), teachers should be able to "Facilitate and inspire student learning and creativity, design and develop

digital-age learning experiences and assessments, model digital-age work and learning, promote and

model digital citizenship and responsibility, and engage in professional growth and leadership" (NETS

Project, 2007, p. 9). What better way to teach students how much technology can benefit themselves and

their careers than by creating a fully online class where students are required to do all of the above?

The learners are undergraduate students. This course is open to students prior to entering the

College of Education. Some of the students enrolled in this course may be out of high school, whereas

others may be adult learners who are going back to college to make a career change. The course is

designed to accommodate diversity amongst its students. The LLDT Collective will ensure that the

recommended online materials will accommodate this diversity in regards to characteristics, lifestyles,


and histories. Online courses appeal to students because of their flexible nature with time and travel

constraints. It is important to keep in mind that students who enroll in online courses often have busy

lives outside of academia with families, jobs, etc.

“One of the key steps of the design process is defining the content needed to address the

instructional need or problem. This content is then used to identify the objectives, design the instructional

strategies, develop test items, and create the instruction” (Morrison, Ross, & Kemp, 2007, p. 94). The

problem is that the current face-to-face instruction needs modified in order to move to an online course.

The problem presented to the LLDT Collective is specific to developing two learning modules.

The first module concerns "Learning in the 21st Century." The instructor would like to see

learners investigate skills that they can use when teaching in the 21st century classroom. Overall, she

would like them to understand what 21st century teaching and learning is all about. There were not

specific skills listed in the course syllabus, but the course objectives stated that students will develop the

basic understanding of using existing and emergent educational technologies in achieving curricular

goals. This includes classroom management, curriculum design, and instructional strategies. The team

believes that this objective is not concise enough to be measurable using the ADDIE process (Analysis,

Design, Development, Implementation, and Evaluation). To this end, the team clarified the objective,

stating that students will demonstrate an understanding of 21st century teaching and learning skills.

The second module concerns "Evaluating Websites & Resources." Throughout this module, the

instructor requires that learners become discriminating consumers when using websites and resources.

Students need to learn how to evaluate information and make good choices when locating and using

information on the web (for example, how to check the validity of a website). This is important because

they, in turn, need to be able to teach their future students to be discriminating consumers as well. There

was no clear objective pertaining to this module. Dr. Kline expressed her expectations in terms of what

she would like her students to be able to do at the end of this module: 1) Students will differentiate

between reliable web resources and unreliable web resources, and 2) Students will create techniques used

to teach others how to evaluate web resources.


After interviewing the subject-matter expert (SME), Dr. Kline, the team decided that the first

need was to rewrite objectives for the modules to be more specific and thus more measurable. The team

also decided that students need access to course materials, projects, and assessments online. Virtual

learning communities must be formed to promote interaction and strengthening of skills through

teamwork, discussion, and debate. Course module assignments must be online experiences to encourage

self-directed learning. The materials must be accessible to students of varied technological

skills/competencies, disabilities, and diversity.

After completing a topic analysis, the team decided that it would be beneficial to add online

learning components to the current topics in both modules to enhance student learning. The topics

covered in the "Learning in the 21st Century" module include exploring 21st century learning,

understanding 21st century learners, and achieving 21

st century learning. The topic covered in the

"Evaluating Websites & Resources" module is preparing pre-service teachers to identify reliable and

unreliable websites and online resources. This is currently accomplished in the course using WebQuests.

A completed procedural analysis allowed the LLDT Collective to determine the steps needed to

create the online learning components for the current topics in the modules. These steps included:

1. Evaluate current course materials and content (topic analysis).

2. Decide which topics would benefit from added learning components.

3. Develop learning components for each module.

4. In a fully online course, provide methodologies for the completion of assignments

virtually instead of traditional classroom instruction.

5. Evaluate new learning components in regards to alignment with current materials and

topics.

As the team moved through the ADDIE model, the LLDT Collective strove for alignment with

the College of Education's theme of being Educators as Decision Makers, and developed strategies to

include components of knowledge, technology, diversity, and ethics.


Design Phase

Task Analysis

The LLDT Collective identified the following instructional problem: the Educational Technology

course needs to transfer from face-to-face instruction to a completely online format. The team was asked

to address this problem in reference to two out of the six learning modules that the course covers:

“Learning in the 21st Century” and “Evaluating Websites & Resources”. After completing a topic

analysis, the team decided to add online learning resources and assessments to the current modules that

were created by Dr. Kline. The topics covered in “Learning in the 21st Century” include exploring,

understanding, and achieving 21st century learning, while the topics covered in “Evaluating Websites &

Resources” include evaluating websites for validity and reliability.

Learning Theory

When designing the instructional plan for the Educational Technology course, the LLDT

Collective focused on behavioral and cognitive learning theories. One tenet of the behavioral learning

theory is the use of instructional objectives to measure mastery of content. Morrison et al. (2007)

asserted, “Such objectives clearly identify the overt responses reflecting mastery of the content, the level

of achievement indicating acceptable performance and the conditions under which the evaluation takes

place” (p. 348). Cognitive learning theory (or more specifically, constructivist theory) is also represented

because students will be active participants in their learning. Through online experiences such as

discussion forums, blogs, video presentations, etc., students will analyze and synthesize information while

building upon their prior knowledge. Morrison et al. (2007) explained that “social interactions can

facilitate learners’ progression to more advanced ways of thinking and problem solving” (p. 355). As

self-directed learners, students will work independently and collaboratively with their classmates to learn

new concepts and to solve problems.


Sequencing

Through the needs assessment and learner analysis, the LLDT Collective determined that students

in the Educational Technology course will have a variety of technological skills. Before entering the

course, however, students will have completed a computer skills assessment that prepares them for the

remainder of the online course. This assessment is the first step in learning related instructional

sequencing, the identifiable prerequisite. As students become acquainted with the learning management

system (Springboard), the team would like to appeal to students’ interests by providing videos, articles,

and thought-provoking online discussions about real-world issues. These types of online components are

the most familiar and easiest to navigate and learn about. Eventually, content resources will gradually

become more difficult as students continue to develop technological competencies. The less difficult

material will be taught before the most difficult, and learners’ competencies will be assessed before

moving onto a new topic.

Instructional Objectives

After analyzing the instructional problem, needs assessment, learner analysis, task analysis, and

learning theories, the LLDT Collective determined the overarching learning goal of the two modules:

Learning in the 21st Century and Evaluating Websites & Resources

Students will develop basic understanding of using existing and emergent educational

technologies in achieving curricular goals including classroom management, curriculum

design, and instructional strategies.

In order to reach this goal, the following instructional objectives will be put in-place for the

modules:

Learning in the 21st Century

Using the “Learning in the 21st Century” online resources, examine the characteristics of

21st century learners meeting the criteria of the Springboard discussion board and

VoiceThread response rubrics.


Evaluating Websites & Resources

Based on the “Evaluating Websites & Resources” online resources, differentiate between

reliable and unreliable websites meeting the criteria of the Wikispace website evaluation

rubric.

Module 1: Learning in the 21st Century

Objective Using the “Learning in the 21st Century” online resources, examine the

characteristics of 21st century learners meeting the criteria of the Springboard

discussion board and VoiceThread response rubrics.

Pre-instructional strategies Prior to beginning the Module 1 content, students will complete the

SurveyMonkey Pretest in order to determine their current knowledge of 21st

century teaching and learning. Morrison et al. (2007) asserted that “a pretest

is designed to heighten the student’s awareness of the content by serving as

cues to the key points” (p. 171). The short pretest contains five true/false

questions that include factual evidence and real-world scenarios about 21st

century teaching and learning. The pretest is meant to “get their wheels

turning”, so it is not for an official grade. The results of this pretest will be

shared with the instructor so that he/she can initiate a discussion board for

students if necessary.

Instructional strategies The initial presentation strategy focuses on defining the concept of 21st

century learners through videos, articles, and Internet resources on

Springboard. The organizational generative strategy will follow, whereby

learners analyze key ideas in order to identify the features that define the

concept (Morrison et al., 2007). First, students will participate in a

discussion board with the following prompt:

Now that you're aware of what 21st century teaching and learning skills are

all about, think about how you would ensure that your students learn these

skills. Many schools lack the tools and funding needed to integrate

technology into their classrooms. As a result, we, as educators, need to

become extremely resourceful in our efforts to make sure our students obtain

these skills. How would you do this in a school that lacks technology?

Put your creative thinking caps on! Assume that you teach in a school that

has only three computers with internet access per classroom. Share some

ideas about how you'd make sure that your students obtain the 21st century

skills that they need for the future.

Next, students will use VoiceThread to post a response to the following:

Reflect on the articles that you read and the videos you watched in module 1:

Learning in the 21st Century. Think about what you learned and why 21st

century skills are so important in education today. You will use this


VoiceThread to demonstrate your understanding of the content (see link to

VoiceThread activity for further instructions).

Finally, students will participate in the following discussion board once they

have viewed all the postings on the VoiceThread:

To be prepared for this discussion, you must have first listened to (or read)

each of your peers’ VoiceThread summaries about 21st century learning.

Your task for this discussion board is to choose two of your peers'

VoiceThread summaries that really stood out to you or gave you a new

perspective. You are to create discussion posts about these two particular

VoiceThread summaries here. These should be very reflective posts that

explain why you chose to respond to each particular summary. So, instead of

one "initial post" in this discussion, you will create two separate "initial

posts" that discuss the VoiceThread summaries of two of your peers.

You might discuss things that your peer mentioned that you agree/disagree

with, and then state the reasons why you feel that way. You could also

discuss something they said that struck you as interesting or maybe

something you never thought of before, and then state why. Maybe you even

have a personal experience that relates to what they discussed in their

summary. Be reflective!

Module 3: Evaluating Websites & Resources

Objective Based on the “Evaluating Websites & Resources” online resources,

differentiate between reliable and unreliable websites meeting the

“Wikispace Website Evaluation Activity Rubric” criteria.

Pre-instructional strategies Prior to beginning Module 3 content, students will view a Voki introduction

that provides the learning objective for this module. One reason for initially

presenting the learning objective is that “Learners were better able to

interpret and understand the implications of the learning directions and

questions because they were presented in simpler sentences” (Morrison et al.,

2007, p. 172). Additionally, students will complete the “Website Evaluation

Pretest”. This true/false quiz has 9 questions that will activate any prior

knowledge about the characteristics of valid and reliable websites.

Instructional strategies The initial presentation strategy teaches principles and rules for determining

reliable websites through articles and Internet resources on Springboard. The

elaboration generative strategy will be used when students evaluate two

websites and reflect on their reliability on the Wikispace Website Evaluation

Activity. The following is the task for this assignment (which is found on the

wiki):

There are two different links listed below (Website #1 and Website #2). One

of the links will direct you to a website that can be described as a "reliable"

source of information based on what you've learned about evaluating

websites. The other link will direct you to a website that can be described as

"unreliable". The question is: which one is the reliable website?


1) Click on each link below and view the two different websites. As you take

a look at each website, think about everything you've learned about website

evaluation.

2) After you've viewed each website, your assignment is to use the criteria

listed in the 5 W's of website evaluation to evaluate each website. So, revisit

Website #1, and provide a detailed evaluation of Website #1 based on each

of the 5 W's. Pay attention to the questions listed under each of the 5 W's and

try to answer them all in your evaluation! Simply stating that the website is/is

not a reliable source of information does not showcase critical thinking

about what you've learned! When you're finished with your evaluation of

Website #1, use the same evaluation process to provide a descriptive

evaluation of Website #2!

3) When you are finished evaluating each site, decide which website can be

considered the “reliable” site. Was it Website #1 or Website #2? Why do you

feel that way?

4) Make sure that you reference the rubric before you begin! Good luck!

Development Phase

The LLDT Collective took many factors into consideration while developing instruction for both

learning modules. The team reflected on what would be appropriate and effective for a diverse group of

learners with varying degrees of technological skills and learning styles. Thoroughly reviewing the team’s

established goals, strategies, and objectives helped the team stay focused on solving the instructional

problem: moving instruction from a face-to-face format to one that is fully online. The team determined

that the objectives it created support the resolution of the instructional problem and that the learning

resources it developed are conducive to an online learning environment.

Organization & Presentation for both Modules

As a part of this phase, a detailed storyboard was developed. This helped to keep the team

organized in terms of how it decided to reconstruct its modules. It provided a comparison of what the

original modules looked like and had to offer versus the team’s reconstructed modules (what the team

replaced, added, and created in its new modules). A module “key” that describes the types of learning

resources used is displayed at the bottom of the map. This helped the team stay mindful of using a variety

of learning resources to accommodate different learning styles. It also helped the team organize its


materials appropriately. The storyboard can be viewed on the LLDT Collective Wikispace presentation:

http://instructionaldesignspring11.wikispaces.com/LLDT+INCORPORATED.

The LLDT Collective cautiously organized the learning resources in a sequence that supported

student ease of use. The team suggested that the students view the resources in the order in which they

are presented because they are structured to introduce materials that are less difficult (videos) before those

that are more challenging (reading materials that the students have to process on their own). Furthermore,

each module begins with a pretest that is designed to stimulate interest and provide students with the "big

idea" before they dive into the content. Similarly, both modules end with one or more assessment

activities that are designed to showcase knowledge and skills that have been acquired, determining if the

objectives have been met.

According to Morrison et al. (2007), research suggests that "a mix of abstract (text that is not

represented by a single mental image) and concrete (text that allows the learner to easily invoke mental

images) information increases both the learner's interest and recall of the information" (p. 200). For this

reason, the team designed modules to include reading materials that provide students with both concrete

and abstract text; some of the required readings are very straightforward and comprehensible while others

are more conceptual. Additionally, supplying students with concrete references such as visual aids,

simple definitions, stories of personal experiences, and videos helped to reduce what Morrison et al.

(2007) referred to as the "extraneous cognitive load" that they have to exert (p. 207).

This course is an undergraduate course that does not require prerequisites. Therefore, assuming

that all students have familiarity with the content in each module would have led to poor design.

Consequently, the LLDT Collective developed learning resources, exercises, and interactions that are all

interrelated and expand upon the main idea. This will decrease what Morrison et al. (2007) referred to as

"step size" or large jumps between ideas (p. 201). Making precise references back to what students

learned when providing them with instructions for the assessment activities was another strategy the team

used to control the step size (p. 201). All activities that require directions for completion offer extremely

detailed instructions that are stated simply so that confusion about the requirements is minimized.

http://instructionaldesignspring11.wikispaces.com/LLDT+INCORPORATED












The team’s systematic approach also aspired to reduce what Morrison et al. (2007) referred to as

pacing. "Pacing is a function of the number of examples, problems, interactions, or exercises presented

with an idea" (p. 202). Each module was designed to control the pace and give students enough time and

support needed to grasp and understand the main idea. Every resource had been developed with the

intention to illustrate the objectives in an informing and engaging fashion.

Everything that was developed provided specific, detailed, and step-by-step instructions for how

to either navigate content or complete assignments. Since the students are not in a face-to-face

environment where they can instantly receive feedback, this was extremely important! Numerous

revisions concerning content, activities, instructions, rubrics, website design, etc. were made throughout

the entire project to include precise information for students. The instructions appeared to be more

redundant than they probably typically would. However, for new online students, it is important to be

repetitive and offer this type of support. Every single word that was typed and re-typed was taken into

consideration. This attention to detail is crucial to the success of the team’s module reconstruction.

A Closer Look at Module 1: Learning in the 21st Century

Objective Using the “Learning in the 21st Century” online resources, examine the

characteristics of 21st century learners meeting the criteria of the Discussion

board and VoiceThread response rubrics.

Pre-instructional Strategy This module opens with a pretest that was developed using SurveyMonkey.

As previously stated, the pretest is designed to stimulate interest and provide

students with the "big idea" before they dive into the module. Since the

results are sent directly to the instructor (Dr. Kline), she is instantly provided

with data about her students’ prior knowledge and can gauge their learning

needs. The instructor will also post answers to the quiz after the students

complete all of the activities in Module 1, so that they are provided with

feedback about their initial responses.

Note: after the team is finished with this project, Dr. Kline’s email address

will be added to the account so that she will receive the results.

Initial Presentation As stated above, this module is structured to introduce materials and

activities that are less difficult before those that are more challenging. All of

the learning resources have descriptive titles which provide the students with

cues as to what they’ll be learning about as they progress through the

module.


Generative Strategy The students will be actively engaged in the materials as they participate in

the pretest, watch the videos, study definitions and graphics, and read the

articles. As students partake in the discussion posts, the instructor will

involve herself in the discussion and play an active role in providing

feedback. Morrison et al. (2007) suggested that “feedback should include

directions to the learners that they should compare their responses to that of

the expert to identify differences” (p. 204). Therefore, after students

complete their VoiceThread summaries, the instructor will create her own

summary based on her expertise of the content so that students can compare

their responses to hers.

Module 1: Added and Created Media/Informational Resources with Rationale

Checklist of

Activities for

Module 1

This was created to serve as visual representation of all learning resources and

assignments for Module 1. It serves as a checklist that is viewed and printed

before the students begin their work. It is designed to help them stay on track and

ensure that they’re completing all necessary tasks. SurveyMonkey

Pretest This seven-question pretest was developed using SurveyMonkey. Bits and pieces

of information were taken from all of the resources throughout the module and

integrated into the pretest (rationale stated under the pre-instructional strategy).

Since this survey tool does not offer an option for immediate feedback, the team

configured it so that students are provided with data that shows how their answers

rank (percentage) in comparison with others who have taken the pretest. The

questions were not developed with the intention of being too difficult. Rather, they

are meant to “get the students’ wheels turning” about the content that lies ahead.

The team used SurveyMonkey because it wanted to expose the students to a useful

way of collecting data. 21

st Century Skills

Diploma News

Story

This YouTube video of a local news story was added because it puts the

importance of learning 21st century skills into a meaningful, real-world perspective.

Students may hear about the importance of these skills in different classes, but

seeing it on the news and hearing that employers see a major lack of these skills

during interviews will be a real eye opener. This will stimulate interest for the

following materials in the module. A Vision of 21

st

Century Teachers This YouTube video was added to expand upon the message presented in the video

that precedes it. The first video is about 21st century students, while the video the

team added is about 21st century teachers. The videos are a perfect match because

both follow the same visual layout. This will provide students with a glimpse of

what 21st century students and teachers look like; not just students.

A Definition of 21st

Century Skills This online Wikipedia resource was added to help students understand what 21

st

century skills actually are. It presents them with factual, straightforward

information about 21st century skills. The definition also illustrates a few graphics

for the students to study and view which will aid their comprehension of this

material. This also offers an alternative to the more typical video or article.


Discussion Topic

#1 (Becoming

Resourceful)

This topic prompts students to use what they have learned and demonstrates their

resourcefulness as educators in a rapidly changing world (see storyboard for

prompt). This was something that the SME stated that she wanted the students to

be able to explain. Students will be expected to share ideas for promoting 21st

century skills in classrooms that may not have the funding or tools to support

technology integration. This discussion also fosters personal reflection, peer

interaction, and idea-sharing. An explanation of how to create a post/reply is

offered right in the discussion board and a rubric is provided as well. Discussion Board

Rubric This rubric is located in the Content area of Springboard.

VoiceThread

Summary

Assignment

This tool was developed as a method to assess knowledge that students gained as a

result of this learning module. They will be expected to provide an overall

summary of what they have learned, including critical questions presented to them

in the VoiceThread activity description. This will give the instructor insight as to

whether or not the objective has been met. Detailed instructions for the task and

how to create the finished product are embedded in the activity and a rubric is also

provided. The team wanted to showcase VoiceThread as a purposeful tool that can

be used by anyone.

VoiceThread

Rubric This rubric is located in the Content area of Springboard. It is not as detailed as

the other rubrics, because the criteria is straightforward and specifically laid-out

for the students. Discussion Topic

#2 (VoiceThread

Response)

This discussion serves as a response to the VoiceThread assignment (see link in

Springboard for complete prompt). It is intended to initiate a reflective discussion

between students and their peers. Students will have the opportunity to comment

on their peers' VoiceThread submissions. The goal is for students to think about

the perspectives of others and reflect upon concepts or ideas that gave them a new

perspective on 21st century learning. An explanation of how to create a post/reply

is offered in the discussion board area and a rubric is provided as well.

Note: The LLDT Collective restructured the SME’s module to exclude the PowerPoint presentations. The

PowerPoint slides correspond with a book that is no longer a requirement. The SME stated that the book

is very expensive and will become a suggested reading material for her students. The team determined

that removing the slides and replacing them with more direct and engaging learning resources will better

suit the objective.

A Closer Look at Module 3: Evaluating Websites & Resources

Objective Based on the “Evaluating Internet Information” online resources,

differentiate between reliable and unreliable websites meeting the criteria of

the Wikispace website evaluation rubric.

Pre-instructional Strategy Students will be introduced to this module by a speaking avatar created on

www.voki.com. The avatar will describe the objective, content, and

assessment activity. This will provide learners with simple dialogue about

what they are going to learn, why it is important, and what is expected of

them as an outcome. They will also complete a pretest created on

www.quizz.biz.


Initial Presentation The initial presentation teaches students how to evaluate websites for

validity. Many materials were developed and collected to help students

achieve the objective. Resources are organized in a sequential manner that

will help students learn the process of website evaluation. All of the learning

resources have descriptive titles which provide students with cues as to what

they will be learning as they progress through the module.

Generative Strategy The generative strategy is used when the students demonstrate the knowledge

and skills they have learned and put them to-the-test in the module’s class

Wikispace assignment. They are challenged to look at two websites that

contain similar content, use specific evaluation criteria to evaluate both sites,

then determine which site can be described as reliable and which one cannot.

The instructor posts the correct answer at the end of the discussion board

after all students have completed the module. The instructor will also be

actively involved in the discussion.

Module 3: Added and Created Media/Informational Resources with Rationale

Checklist of

Activities for

Module 3

This was created to serve as a visual representation of all learning resources and

assignments for Module 3. It serves as a checklist that is viewed and printed before

the students begin their work. It is designed to help them stay on track and ensure

that they are completing all necessary tasks.

Voki

Introduction This speaking avatar was developed on www.voki.com (see pre-instructional

strategy). This is a unique way to begin a new module; a good idea for capturing the

attention of students! Website

Evaluation

Pretest

This website evaluation pretest was created using www.quizz.biz. The questions

were not developed with the intention to be difficult. Rather, they are meant to “get

the students’ wheels turning” about the content that lies ahead.

This pretest is designed provide students with immediate feedback about what they

already know and/or what they need to learn more about in terms of website and web

resource evaluation. Quizz.biz was chosen for a specific reason. It is a free site for

anyone to use, and one does not have to register or pay to use it. Saved quizzes can

be accessed at any time. This is a good tool for anyone to use!

Evaluating

Websites Using

the 5 W's

The team added this YouTube video because it demonstrates a technique that students

can use when evaluating websites and resources. It provides step-by-step instructions

for them to follow, and describes what to look for when determining if a website

should be considered reliable/valid.

The 5 W's of

Website

Evaluation

This PDF was also added because it uses the same language as the above mentioned

YouTube video. It illustrates short, simple sentences that reiterate what the video

teaches students about evaluation techniques. With these connected learning

resources, students will be exposed to both an auditory and visual representation of

the content. Also, they will use this in the Wikispace assessment activity at the end of

the module.


The ABC’s of

Website

Evaluation

This informational resource was added to serve as an extension of the knowledge

gained in the preceding resources. More detailed information is available through

this site. This site provides students with explanations that describe why website “A”

and website “B” are reliable/unreliable. Links to the sites are provided so that

students can see what the author is describing as reliable/unreliable information; it is

an interactive experience.

Wikispace

Website

Evaluation

Activity

This wiki was created to serve as an assessment tool. It is designed to test the

students’ website evaluation skills; they will use what they have learned throughout

this module and apply it in this evaluation activity. This assessment activity provides

the instructor with evidence that supports whether or not the objective has been met.

Also, it exposes the students to a "wiki page" and gives them a new way to reflect

upon their learning. Detailed instructions for the task and how to create the finished

product are included on the page. A rubric is also provided. Wikispace

Website

Evaluation

Activity Rubric

This rubric is located in the Content area of Springboard.

Note: The LLDT Collective restructured the current module to exclude the WebQuest materials as the

SME stated (in the initial interview and through a phone conversation) that she would like students to

focus on website evaluation.

Extra Module that was Developed: Help and Tutorials for Online Learners

As the development phase was being edited, the LLDT Collective brought attention to the fact

that the existence of a module which offered assistance to online learners was necessary. Since there is a

possibility that students who enroll in this class may have limited experience in an online learning

environment, a module was developed to address this need. Resources to help students with Springboard,

as well as the Wikispace and VoiceThread activities, were added to this module (see storyboard for a list

of added resources). This module is titled “Help and Tutorials for Online Learners.”

Implementation Phase

Now that the content has been analyzed, designed, and developed, the next phase of the project is

where the "rubber meets the road", the implementation. One cannot assume that presenting the team's

content to a select audience will automatically be understood and integrated; the content must be prepared

and presented in such a way as to "sell" it to the client, to make it something that they need and find value

in.


The innovation of the content proposed by the LLDT Collective must have perceived value. In

other words, when the team suggests a new way of doing something, how does the client trust this new

idea of bringing about a better way to learn the content? The innovative aspect of the team's proposed

content focuses upon online discussions through the Springboard classroom management system.

Students read articles, watch videos, complete quizzes, and then come together to the discussion board

"watering hole" to share what they have learned. During implementation, the team wants students to

realize the worthwhile learning that takes place through these online discussions and the value of learning

communities that are formed.

Successful implementation also requires that the team's goals align with that of the client, Dr.

Kline. Her goals and values with content presentation must be compatible with the team's offering. In

this particular case, Dr. Kline wants to transition her semi-online class to being completely online. The

team’s goals are compatible with hers. The real issue is to ensure that the modes of delivery that the team

chose are compatible with how readily they are embraced by the real clients, the students. The LLDT

Collective needs to ensure that the team’s content matches their values, needs, and experiences as well.

Through the design and development phases, the team established that the content supports proper pacing

and sequencing. Content difficulty starts with simpler concepts which gradually increase in difficulty and

complexity. Pacing is carefully set so that plenty of time and examples are presented before moving to

the next concept. This idea of complexity is important for content acceptance and has been addressed by

the team.

Implementation Action Plan

To implement the content, the LLDT Collective decided upon a twofold approach using different

audiences. The first audience consists of students enrolled in Timothy Sisson's Educational Technology

class. After reviewing the team's content, he realized that it "would be a valuable learning experience for

them to not only develop this content knowledge, but also to think about the learning process and what

that means through online delivery." The team also sought professional involvement during the


Implementation Phase. To this end, the following educational professionals agreed to implement and

evaluate the content as well:

Betty Rogge An instructor who currently teaches Educational Technology

Denise Uitto Another instructor involved with Educational Technology teaching

Lori Brinker Coordinator of the Computer Based Training program (CBT)

Patsy Malavite Computer based Applications instructor for the CBT program

Jackie Ashbaugh Received her Master's in Instructional Technology two years ago

The involvement of these two groups (students and educational professionals) during

implementation provided valuable feedback from two perspectives. Although both will complete the

content as students, the professionals may identify strengths and weaknesses in the content that students

may not and vice versa. Thus, the team meets the goal of successful implementation by piloting the

content in these small groups before its adoption into future classes. It also assures observable results in

that the educational professionals listed above, in addition to Mr. Sisson, can attest to the content's

viability and can encourage others to adopt it as well.

Alignment with the CLER Model

To execute a successful implementation, it is important to specify "whom and how to

communicate information about an instructional product" (Morrison et al., 2007, p. 393). The team chose

the standard CLER model for this purpose, developed by H. S. Bhola (1982, 1988-1989). CLER involves

identifying the project's various constituents (configuration), how these participants interact with each

other (linkages), how the environment must be configured for effective implementation, and the proper

selection and application of educational resources.

With this particular project, the designer team consists of the four members of the LLDT

Collective as mentioned above. Each team member is specialized for each phase of the ADDIE model,

though all members collaborated together during each phase. The team received instructional supervision

and guidance through Dr. Ward. The client system consists of Dr. Kline, Mr. Sisson, Ms. Rogge, the


students in both of their Educational Technology classes, and the four educational professionals who also

implemented the material. The linkages between these groups of people revolve around the Instructional

Technology and Educational Technology programs, with Dr. Ward managing the former and Dr. Kline

the latter. LLDT Collective formed a working relationship with Dr. Ward for overall project direction,

while the team worked with Dr. Kline to assess her needs and goals for the actual content. The team's

relationship with Betty Rogge is two-fold. Working under Dr. Kline as a senior lecturer, she offered first-

hand experience with the class' current content offerings. She also served to implement the content as a

student, providing valuable feedback as she would evaluate her own content.

The environment for the implementation phase is online, requiring computer resources with an

Internet connection. Mr. Sisson and Ms. Rogge provided these resources to their classes via a computer

lab, while the educational professionals obtained these resources on their own through office computers,

home computers, and access to computer labs at Wayne College. LLDT Collective organized and posted

all of this content into a sample Springboard classroom, making it centralized and easy to access by

clients, students, educational professionals, and the LLDT design team. Technical support to ensure

proper delivery through Springboard was provided by Chris Dreher, the designated Springboard support

specialist at Wayne College. There were several influence resources (motivations) working in tandem

during this project. From the students' aspect, Mr. Sisson and Ms. Rogge guided their students through

the team's materials and activities during their scheduled class sessions. The educational professionals

completed the materials as a form of evaluation to assess their viability to possibly be adopted into their

own classes and programs.

Financial backing was not needed to implement this project with these pilot groups, as the

facility, technical equipment, and support personnel were available. Should the project be adopted into

the mainstream program, accommodations may be needed to properly compensate for increased use of

these resources. Transportation to/from the educational setting is also a given, as the students are already

at the University for Mr. Sisson's regular class content and at Wayne College for Ms. Rogge’s. The

educational professionals can complete the content while at Wayne College for their regular duties or at-


home using their own Internet connection to access everything located on Springboard. Other resource

accommodations such as housing and food are not necessary as students are already present at the

University for Mr. Sisson's class and Wayne College for Ms. Rogge’s class; the same goes for the

educational professionals. The team’s goal for this project implementation was to be minimally invasive

and incur the least possible cost. For these reasons, the team chose to implement the content using a

centrally located resource (Springboard) during times when students and professionals were already on-

campus to access the content. Because all of the materials are online, there were no costs for packaging,

duplication, warehousing, and shipping of traditional paper-based materials and video media.

Timeline

From a scheduling standpoint, the team worked with Mr. Sisson to determine the best class

session that he could task his students with completing the project's assignments. Mr. Sisson was flexible

in this respect, agreeing upon Wednesday evening, April 4th, as the assigned class session. The

educational professionals’ involvement during implementation was even more flexible. Because the

materials were accessible online, they had an entire week to complete the activities at their leisure from

wherever they had access to a computer with an Internet connection. During this week, representatives

from the team directed the project's implementation, acting as instructors to guide the participants through

the activities and discussions that ensued. The team did not require much training on managing a class

through Springboard, as they had ample experience from taking classes involving this system in the past.

Still, Chris Dreher provided support for any higher-level questions that the team had about managing the

class during the implementation phase.

Evaluation Phase

Evaluation Method

The LLDT Collective performed an extensive evaluation of the content materials and delivery

methods. Because the team enhanced the content of two modules within an existing class, formative


evaluation methods were chosen, which allows evaluation of instructional processes and outcomes during

development and small group trials. A summative evaluation was not appropriate because the subset of

learning modules did not run the full course of the class; the team's proposed content would occupy two

weeks (one week for each module) of learning during a 16-week class. Similarly, a confirmative

evaluation was not appropriate for this study because this was the first time the materials were used, thus

long-term benefits could not yet be assessed.

The purpose of the formative evaluation is to not only realize the strengths and weaknesses of the

learning materials & methods, but more so to use these results to revise and improve the materials for

future learning sessions. The team received feedback from two different Educational Technology classes

spaced seven days apart. The first class was hosted by Tim Sisson at The University of Akron, and the

second class was hosted by Betty Rogge at The University of Akron Wayne College. The evaluations

from the first class allowed the LLDT Collective to revise the materials to be used for the second class.

The team also requested formative evaluations from six professional educators. Out of these, Jackie

Ashbaugh, Lori Brinker, Patsy Malavite, Betty Rogge, and Denise Uitto provided valuable feedback from

an instructor's viewpoint. Thus, the audience for this project comprised of two "live" classes of students

and five educational professionals.

During staging, all LLDT team members tested the content individually. Some of its members

conducted one-to-one trials with friends, family members, and their own classroom students to ensure that

the technology worked correctly with different computer configurations. Small group trials were not used

as testing was done as field trials during actual class sessions.

The team decided to use paper based evaluations to minimize technology barriers, promoting a

larger set of responses. The student evaluations were decision oriented, contained a single page of yes/no

questions with short essays to describe responses further if desired. The educational professional

evaluations were connoisseur-based, two pages in length, and requested elaboration. Evaluations were

returned to the LLDT Collective, scanned as PDFs, then distributed to team members for analysis,

reporting, and subsequent content revisions. Four rounds of revisions were made (during one-to-one


trials, after each class session, and after reviewing the professional evaluations). The end result is a new,

online learning experience that can be applied to future classes with effective learning efficacy.

Validity

In order for formative evaluations to be effective, there "must be a direct relationship between

instructional objectives and assessment measures" (Morrison et al., 2007, p. 258). If the instructional

objectives are supported by the evaluation questions and responses, then the instrument has a high degree

of validity. To this end, the team included short essay responses to support each yes/no question in the

evaluation. Validity was ensured by the quantity of short essays provided, indicating feedback on a

deeper learning level. If the evaluations simply had yes/no responses, students could answer questions

without really thinking about how it applied to the learning objectives and their own learning experience.

The team received a short-answer response rate of 20%, responding deeply to 55 out of 279 questions.

Thus, one out of every five questions provided the team with valuable feedback. Educational

professionals provided short-answer essays to almost every question.

Another sign of evaluation test validity is the use of multiple data sources and multiple

instruments. As mentioned above, the LLDT Collective used two data sources (two class sessions and the

feedback of five educational professionals), allowing feedback from 36 individuals. Two evaluation

instruments were used; one for the learners and a different evaluation for the educational professionals.

The diversity of the data sources and instruments ensured minimal bias on the feedback.

Assessing Knowledge

To determine program effectiveness, the LLDT Collective assessed knowledge from two

perspectives. The first assessment was through student and educational professional evaluations that

occurred after the course materials were implemented. The second assessment involved providing

mechanisms for grading the materials on Springboard as if students would receive an actual grade for

their work. While the latter assessment did not contribute to the learners' grades, creating grading

instruments in the Springboard system enabled the team to connect them with the learning objectives.


The student evaluations were objective tests that were comprised of nine yes/no questions. While

this type of test is fast and easy to complete (ensuring a high response rate), responses reveal only lower

levels of learning. To address this deficiency, the team included constructive elements to questions by

offering space for short essay responses. This way, learners could explain why they chose their yes/no

response, if desired. This type of response assesses a higher level of learning. Performance testing was

also utilized via the course assignments, the "Wikispace Website Evaluation Activity" in particular. For

this assignment, learners were evaluated on how they determined the validity of two websites.

From an attitudinal viewpoint, instructors Tim Sisson and Betty Rogge observed the learning

sessions while learners completed the team's materials. They addressed any questions that arose, taking

note of positive or negative comments and behaviors that came to surface. The team recognized attitude

related responses in the short answer responses as well, though these are more prevalent verbally and

visually during implementation. The professional educators provided attitudinal feedback data that the

team also used to revise the learning materials and methods.

Evaluation Standards

The LLDT Collective used pretesting to not only indicate learner preparedness for the learning

activities, but also to serve as a baseline performance to judge how much was learned at the end of

training. Some reported that the pretest questions were too easy and/or realized that what they learned

from the team's materials awoke what they already knew. Relative evaluation standards were not used for

these evaluations; there was no need to compare learners' performance with each other. The team chose

standards based assessments that involve real-world tasks, such as the “Wikispace Website Evaluation

Activity”. The responses received on the evaluation instruments themselves provided feedback from

these real-world tasks. Student self-testing was not utilized, although progress checklists were provided

for each learning module so that students could self-track their progress.


Evaluation Results

The team summarized all of the short essay responses to evaluate the effectiveness of the learning

materials and delivery methods. From the educational professionals' perspective, the overriding theme of

responses centered upon the appropriateness of content and how well it aligned with the course

objectives. Professionals stated that the "level of learning matched higher education" and that the

"activities and quizzes matched instructional objectives". The detail and conciseness of the assignment

grading rubrics was positively commented upon as well. Teaching methods were appropriate to the

content and the time commitment of one week was sufficient to complete all materials and activities.

Given that this feedback came from experienced fellow teachers, the team ensured that it was valid.

In terms of content, both learners and educational professionals valued the readings, videos, and

diversity of activities. Students (for the most part) liked the use of discussion boards and wished that

more classes would use them to exchange ideas. The team’s liberal inclusion of learning materials were

"a good reference tool to find information and answer questions." The activities themselves were "cool,

but challenging." Some students even wanted to know if this particular class would be offered fully

online in the future and stated that they would “prefer to take this course online instead of in-class.”

Lastly, both audiences complimented the team on the organization and layout of the learning

materials on Springboard. Students commented that the content was "easy to navigate; everything was

explained clearly" and that the "website was easy to follow." Educational professionals appreciated that

the content was "very organized”. Both audiences appreciated how the learning activities were explained

well.

The following are comments made by the educational professionals:

Jackie Ashbaugh: "I think you did an outstanding job on this project."

Lori Brinker: "Great work! Articles were informative. Would meet needs of a

variety of learning styles."

Patsy Malavite: "Great work on this!!!! Impressed!!"

Betty Rogge: “It is a very good start.”


Denise Uitto: “Meets different learning styles.”

Constructive criticism was provided in five major categories: organization, content, instructions,

pacing, and technical mechanisms. What the team found interesting was that the positive feedback

received above included just as much criticism below. While initially conflicting, this feedback is useful

as each learner has a diverse learning style which needs to be accommodated. The team took all of the

feedback seriously and revised the learning materials and methods to address the widest range of learning

styles as feasibly possible. Evaluation results showed more constructive criticism after the first class

piloted the modules. After numerous revisions and a second class piloting, there was much less

constructive criticism.

In terms of organization, the overriding criticism was that there were "too many steps to navigate

the required content." Learners had to jump back & forth between pages to read the instructions, find the

content, and perform the activities. While the team did its best to minimize the location of materials, it is

the overall layout of Springboard that contributes to this "organizational flow" problem. The best effort

was made to minimize this confusion with navigational tips throughout the materials, but mechanics of

the learning management system could only be simplified so much. A few learners also noticed that the

"instructions were all over the place," indicating that in the team's effort to explain the navigation of

Springboard, there was too much explanation. Learners suggested combining some of the content

together so that the initial presentation is not overwhelming. The LLDT Collective found it interesting

that the majority of learners with little to no online experience were the ones who found it easiest to

navigate Springboard. Since the majority of constructive criticism revolved around navigating

Springboard, the team added another module that includes tutorials for navigating Springboard,

VoiceThread, and Wikispace.

From a content perspective, while learners appreciated the varied reading materials and videos, a

few thought that some of the content "may be too advanced for the students’ levels." This opinion

conflicted with most of the educational professionals’ viewpoints, as they stated that the content matched

the instructional objectives quite well and was at the appropriate learning level. Learners also suggested


providing multiple discussion topics to allow choice in responding to what they were most interested in.

One student noted that the pretest was repetitive, and one professional noted that it was too easy.

However, these instruments were intended as pre-assessments of learning before instruction and were not

intended to be difficult nor challenging. That was the intended design of those activities. Morrison et al.

(2007) asserted that “a pretest is designed to heighten the student’s awareness of the content by serving as

cues to the key points” (p. 171).

The instructions for using the content received some criticism, but also received plenty of

suggestions for improving its presentation and usefulness. Some of the feedback concerns revolved

around using the discussion board and navigating Springboard in general. The team provided more

supplemental resources and instructions to get them started as quickly as possible without detracting from

the assignments at-hand. While one learner stated that the rubrics "were vague", the educational

professionals stated they were very concise. Again, the difference could be how they are interpreted (the

viewpoint), which must be taken seriously. Some learners stated that overall, the directions were "hard to

follow", "hard to figure-out what [I] was supposed to do", and that the "assignment directions [were]

confusing". The team went over the instructions several times, trying to optimize them before the second

class implemented the learning materials. Again, there was far less confusion after the second class

piloted the materials. The educational professionals’ feedback provided the most value as their criticisms

were followed by suggestions to improve these areas. Patsy Malavite's contributions were especially

helpful. A LLDT team member met with Patsy after the pilot to fully understand and realize the

numerous suggestions that she made to improve the learning materials and delivery methods.

Pacing was also criticized in the evaluations, especially during the first class session. Learners

only had an hour to complete the materials that were designed for two weeks; one week for each module.

This frustrated a number of learners because they weren’t aware that the content they were trying to

complete in one class session was designed to be completed over a two week period. Looking back, the

team should have adjusted the quantity of materials to allot for the limited time. This adjustment was

made for the second class, which completed a minimal set of learning materials in the time allotted.


Educational professionals agreed that each module could, in fact, be completed in a week's time, which

meets the design goals of this project.

Again, the LLDT Collective took all of the compliments and criticisms seriously and made

numerous revisions to the learning materials, instructions, and delivery methods to optimize the

experience as much as possible. Although the second class had far less constructive criticism, the team

would have liked to implement a third pilot to assess the effectiveness of these revisions had the team

received additional time. Based on the improvements and consistent positive feedback by the educational

professionals as well as the majority of students, the team feels that the team’s goals have been met, the

team’s problem has been solved, and materials are ready to be included in future classes that cover this

content.

Conclusion

Using the ADDIE model, the LLDT Collective created two online instructional modules for the

Educational Technology course. The team defined the learners, analyzed the course’s current content,

developed learning objectives, and designed relevant assignments/assessments. The team implemented

the learning modules with undergraduate students and educational professionals. Through evaluations, the

team learned of ways to improve the modules. Transferring the face-to-face Educational Technology class

to fully online has had a successful start, and hopefully the educational professionals will implement the

LLDT Collective’s instructional modules within their future courses.


References

Bhola, H. S. (1982). Planning change in education and development: The CLER model in the context of a

mega model. Viewpoints in Teaching and Learning, 58, 1-35.

Bhola, H. S. (1988-1989). The CLER model of innovation diffusion, planned change, and development:

A conceptual update and applications. Knowledge in Society: The International Journal of

Knowledge Transfer, 1, 56-66.

Morrison, G. R., Ross, S. M., & Kemp, H. K. (2007). Designing Effective Instruction (5th ed.). Hoboken

NJ: John Wiley & Sons, Inc.

NETS Project (2007). National Educational Technology Standards for Teachers: Second Edition.

Washington, DC.: ISTE.

Instructional Design Project - University of...

Documents

Transcript of Instructional Design Project - University of...