1

Final PresentationSpring 2006 T/TAC Online Project Virginia Assessments

May 18, 2006

Matt HumphreyAala’a MashaalPamela SharpeTim SmithMaria Washington

Graduate School of Education George Mason University

2

T/TAC Online Regions

3

Our Faculty and Facilitators

Dr. Michael Behrmann Dr. Nada Dabbagh Dr. Kevin Clark Anya Evmenova Gary Dickelman Jocelyn DelSignore

4

Clients

John Eisenberg Severe Disabilities/Assistive Technology Specialist,

Virginia Department of Education

Mary Wilds Statewide Coordinator of Distance Education, Old

Dominion University

Dr. Michael Behrmann Professor, Director of Helen A. Kellar Institute for Human

disAbilities (KIHd), George Mason University

Nathan Sparks Coordinator, Division of Assessment and Reporting,

Virginia Department of Education

5

Presentation OverviewPart One

Part Two

Part Three

Part Four

Part Five

Instructional Design Process

Background & Introduction Mission and Goal of Project

Analyses Design Development

Formative Evaluation Prototype Demonstration

Future T/TAC Site Recommendations

Questions and Answers

6

Project Background

No Child Left Behind Act (NCLB) 2002 Every child must be assessed according to the

state standards Individuals with Disabilities Education

Improvement Act (IDEA) 2004 Accommodations for all children with

disabilities

7

Project Background

Assessment options for students who are exempt from traditional assessment programs Virginia Alternate Assessment Program (VAAP)

• Students assessed at grade levels K-5

Virginia Grade Level Assessment (VGLA) • Students assessed at grade levels 3-8

Virginia Substitute Evaluation Program (VSEP)• Students assessed at grade levels 9-12

8

Project Problem StatementVA Assessments

Students with severe disabilities were only assessed on life skills prior to new mandates

Transition to cognitive standards based assessments

Communication skills are now considered in the assessment process

New process for teachers Our project concentrates on the VAAP

assessments

9

Instructional Design ProcessThe ADDIE Model

Analysis Design Development Implementation Evaluation

10

The Instructional Design Process

Who Learner

What Learner needs Instructional Strategies

How Evaluation

11

AnalysisAnalysis

Performance Analysis

Needs Analysis

Learner Analysis

Task Analysis

Analysis Design Development Implementation Evaluation

12

AnalysisPerformance Analysis

Determine: Goals and Objectives Drivers and Barriers Actuals and Optimals

13

AnalysisPerformance Analysis

Problem Life skills in addition to Cognitive and

Communication skill assessments State mandated assessments Online Manual Teacher schedules Structure for assessment Public tool One stop shop

14

AnalysisNeeds Analysis

Determine the best solution Generate supporting data Priorities for action Provide specific recommendations

15

AnalysisNeeds Analysis

Teachers are not comfortable with the new process of aligning their lesson plans to ASOLs

Teachers would welcome a job-aid Walk through the lesson plan process Provide resources

Teachers need examples of good lesson plans

16

AnalysisLearner Analysis

Primary user Environment Varied experience levels Users:

Novice Teacher Intermediate Teacher

Expert Teacher

Reluctant Teacher T/TAC Online Administrator

17

AnalysisTask Analysis

To determine the goals and objectives of learning: User tasks Skills and knowledge Task sequence Strategies, media, environment

18

AnalysisTask Analysis

Lesson Plan Creation

Old NewLife Skills Cognitive skills

Communication skillsLife skills

19

AnalysisTask Analysis

Three instructional strategies: Direct instruction Group instruction Self directed instruction

Procedural and cognitive tasks Eight step process Support those processes in the

solution system Existing lesson plan document

20

AnalysisInstructional Objectives

Facilitate and support lesson plan creation process Guide teachers through each step of creating a

lesson plan Additional supports to assist teachers throughout

the process Encourage knowledge sharing

Lesson plan database• Allows teachers to search and download

exemplary lesson plans Search VAAP Manual

21

Design Design

Analysis Design Development Implementation Evaluation

Design Approach

22

DesignDesign Approach

Support the process of creating lesson plans aligned to new standards and allow knowledge sharing

Develop an electronic performance support system (EPSS)

EPSS should include guides and necessary supports that will: Guide and scaffold the lesson plan creation

process Allow special education teachers the ability to

develop, share, and download lesson plans

23

DesignDesign Approach Rationale

Supporting an existing process Teachers know how to create lesson

plans New constraints on that process

Teacher experience levels vary Just-in-time support in the classroom

24

DesignEPSS Characteristics

Electronic Performance Support System (EPSS)

Computer software program that improves performance by: Reducing complexity Providing the performance information to

perform a task Providing a decision support system that

enables the user to identify appropriate actions

25

DesignEPSS Characteristics

Job aid Controlled by the user, used on

demand Reduce the need for prior training in

order to accomplish the task Supports new constraints on an

existing process

26

Development

Analysis Design Development Implementation Evaluation

Flowcharts

Prototype

Usability Testing

27

DevelopmentPrototype

Operationalize instructional objectives VAAP section will include three main

sections:1. Create Lesson Plans

Teachers will be guided through the lesson process with additional just-in-time supports

2. Search the VAAP Manual Enable teachers to find VAAP information more

efficiently3. Search for Created Lesson Plans

Teachers can search and download exemplary lesson plans created by other teachers

28

Flowchart 1:Flowchart 1: Searching the VAAP Searching the VAAP ManualManual

29

Flowchart 2: Lesson Plan DocumentFlowchart 2: Lesson Plan Document

30

Flowchart 3: VA Assessments Site FlowFlowchart 3: VA Assessments Site Flow

31

DevelopmentInitial Prototype

32

DevelopmentFinal Prototype

33

Evaluation

Analysis Design Development Implementation Evaluation

Formative Evaluation

Summative Evaluation

34

EvaluationFormative Evaluation

The formative evaluation consisted of three stages of usability testing:

Stage one - novice users One on one, think aloud protocol

Stage two - experienced users. One on one, think aloud protocol

Stage three - expert users Small group, expert review

35

EvaluationResults

Positive Aspects of the System as Identified by the Testers:

Users could easily sign in with their existing accounts and new users could create an account easily

The navigation is clear and consistent with lesson plan objectives of the teachers simulating the real process for this task

The supports are appropriate for each section of the system with adequate information to support a teacher in creating a great lesson plan

The system is easy and clear in choosing organizing topics and having the standards auto populate dynamically is a necessary and strong feature

36

EvaluationResults

System Improvements Usability Instructional Content Interface Design

37

EvaluationUsability

Add a descriptor to the lesson plan generator system so users are aware of what they can do (This is an area where you can create your own lesson plan, ideas are provided through links on the right)

Make text throughout the site bolder with a sharper font for varying visual needs of users

Create a flashing button for “select” next to the organizing topic to assure that the users get the correct set of standard

38

Evaluation Instructional Content

Create documents defining the standards and organizing topics

Reword the directions for the activities and strategies section to be clearer to the users

Add scoring rubrics to the evidence collection page to identify significance of peer interaction

Add examples and clarification documents to the evidence page

Remove the list of AT Vendors as it will not assist the user with the task at hand.

39

EvaluationInterface Design

Made the Navigation bar have actual section titles as well as highlight where the user is in the system.

Made the logon screen bigger and the word “new user” larger to accommodate different visual needs

Made the user name and date auto populate based on logon user information.

40

Next Steps - Recommendations1. Add a spell check feature.2. Create a flashing button for “select” next to the

organizing topic to assure that the users get the correct set of standard

3. Convert help files to PDF instead of Microsoft Word documents.

4. Add a Help button on Preview screen. Also an enlarge font button.

5. The age appropriate materials document should be changed to a pop-up box instead of a word document since the content is short to make loading it quicker.

6. Add visuals for better comprehension.7. System implementation – experienced teachers entering

lesson plans into the Lesson Plan Generator8. Make the lesson plan search feature functional9. Summative evaluation

41

Acknowledgements Shuangbao Wang – T/TAC Online Programmer Seunghun Ok – T/TAC Online Site Administrator MSHA Immersion Team Kilmer Center - Vienna, VA Courthouse Road Elementary – Spotsylvania County,

VA Vienna Elementary School – Vienna, VA Assistive Technology Taskforce T/TAC Staff Brenda Mueller

42

Thank You for Listening

Are there any questions?