Case Study #9Jackie Adams: Evaluating a Federally Funded Faculty Training Program

Melody Murphy AIL 606The University of Alabama

Part One: Creating an Evaluation Plan• Jackie Adams has a instructional

design position with a large university for a federally funded project.– Her main responsibilities were to work

with technical subject matter experts in the development, delivery, and evaluation of in-service faculty education.

• The name of the project Jackie was working on was the Advanced Manufacturing Technology Education (AMTE).

Part One: Continued• The goal of the AMTE project was one

of many projects funded by the Advanced Technology Education (ATE) program. – Its goal was to improve science and

engineering technician education at the undergraduate and secondary school levels.

– Also, it focused on curriculum development, instructional materials development, teacher/faculty enhancement, and/or student recruitment.

Main Goal• Its main goal was to focus on teacher

enhancement as a means of advancing technology education.– The rationale behind AMTE was to provide

educators with state-of-the-art knowledge in the technical disciplines, so that their students would in turn benefit from their teachings.

• Over a three-year grant, the AMTE project was to provide faculty development to 100 science, math, and engineering technology educators each year.

Progress• Jackie had been working on the

project for about four months when Ray DeMilo (her boss) asked her to develop an evaluation plan for the project.

• In order for them to receive funding for the project an evaluation plan was going to be required to assess the performance of their project.

Jackie’s Plan for the Evaluation Plan1. Give it to her boss.2. Send it to the funding agency.3. File it with the university grants

office.4. Use it to conduct the evaluation.

Jackie’s Planning Evaluation Notes1. Why Evaluate?2. Evaluate What?3. How should the evaluation be

conducted?4. Who should be involved?5. When should it be done?

Preliminary Analysis Questionsfor Part One – Question #1

Based on Jackie’s questions, what does she need to do to complete the evaluation.

First of all Jackie decided that on why the program must be evaluated. It should be evaluated for the ongoing funding of the program and to measure whether or not learning is taking place in the workshops

Preliminary Analysis Questionsfor Part One – Question #2

How can Jackie determine whether short- and long- term outcomes are being achieved? What

evaluation data does she need to collect? Short-term• Jackie can conclude by receiving the

surveys from the faculty about how they are using what they have learned in the workshops to conclude that by what they have learned they can use in their classrooms.

Long-term• Jackie can conclude from the surveys that

if the faculty can continue or use the information now that they can use then the faculty can use it for long range planning in their classes.

Preliminary Analysis Questionsfor Part One – Question #3

How should Jackie design the instruments for this evaluation?

Based on the material taught, she wants to know what the learners have gained from the workshops in order to report to the team that is reviewing her project in order to keep the grant money.

Part Two: Conducting a Meta-Evaluation

Conducting the Evaluation• The funding agency sent out a six-

member team to review the project’s progress and impact.

• The team would report the success of the project to key administrators at the university and the funding agency.– The team included four post-secondary

educators with a background in science, engineering, and technology and two engineering practitioners.

Team Review• During the two-day meeting, Hank

spent the entire first day with Jackie.• He reviewed the progress and impact

of the project.• At the end of their meeting, Hank felt

that confident that he had enough information to start the review.

Preliminary Analysis QuestionsQuestion #1Part A: Questions Related to the Evaluation Plan

What are the main evaluationquestions Jackie appears to be addressing? Why – why evaluate the project?What – what to evaluate in the project? What should

be measured?How – How should it be evaluated?Who – Who should be evaluated?When – When should it be evaluated? Periodically?

Preliminary Analysis QuestionsQuestion #2

What are the strengths and weaknesses of the evaluation design for answering the questions?Strengths – the evaluation plan answers why

evaluate the objectives (reason for evaluation), purpose, when it should be done which is periodically.

Weaknesses – How it should be done. It says documentation but it does not specify the

documentation used or the learners in Jackie’s evaluation plan.

Preliminary Analysis QuestionsQuestion #3

Given the goals stated in the ATME proposal, are these questions sufficient?Yes, it does give the basic why, what, how, who, and

when.

Preliminary Analysis QuestionsQuestion #1Part B: Questions Related to Instrumentation

How appropriate are the instrumentsfor the goals of the evaluation?I question the validity. This is a lot of Jackie’s opinion on the evaluation plan. I would probably recommend incorporating interviews and surveys to the evaluation plan.

Preliminary Analysis QuestionsQuestion #2

Are the evaluation questions answered in enough detail, or are other instruments needed?• I felt like there could have been more detail, for

example, when stated “determine how disciplined and effective the organization’s operations are.” How will these be determined?

Preliminary Analysis QuestionsQuestion #3

What, if anything, would you change about the instruments and why?

I would add a survey to be given to the faculty/staff. For reinforcement for what is stated in the evaluation plan.

What are the implications for instructional design practice in this case? • The instructional design of high-tech

areas using computers.– For example computer numerical control,

programmable logic controllers, robotics, electromechanical controls, lasers, solid modeling, and rapid prototyping in the workshops conducted by Jackie.

As an instructional designer, what did I learn? • In order to keep your

program or project funded it is important to regularly evaluate it.– Ask questions such as:

– Why, what, how, who, and when.

My Questions1. How important do you think

evaluation plans are to instructional design projects?

2. What types of instruments would you recommend for Jackie to use in helping with the evaluation of her project?