Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online...
Transcript of Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online...
![Page 1: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/1.jpg)
1
Geoscience Projects and The Online Evaluation Resource Library
(OERL)
Daniel R. Zalles, Ph.DSRI International
Funded by the Research, Evaluation, and Communication (REC) Division of the National Science Foundation
Conrad Katzenmeyer, Program Officer
: Online Evaluation Resource Library
![Page 2: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/2.jpg)
2
: Online Evaluation Resource Library
A Web-Based Tool to Support the Professional Development of Evaluators
![Page 3: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/3.jpg)
3
: Online Evaluation Resource Library
Reviewed collection of plans, instruments, and reports, complete or excerpted
![Page 4: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/4.jpg)
4
: Online Evaluation Resource Library
Reviewed collection of plans, instruments, and reports, complete or excerpted
![Page 5: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/5.jpg)
5
: Online Evaluation Resource Library
Reviewed collection of plans, instruments, and reports, complete or excerpted
![Page 6: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/6.jpg)
6
: Online Evaluation Resource Library
Submission Process
NSF solicits from targeted project areas
SRI reviews and catalogs
Current and pending solicitations:
· Advanced Technology Education
· Course, Curriculum, and Laboratory Improvement
· Programs for Gender Equity
![Page 7: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/7.jpg)
7
: Online Evaluation Resource Library
Current Representation of Geoscience Projects in OERLPROJECT AREAS
Curriculum Development
Teacher Education
Faculty Development
Under-Represented Populations
Complete Plans 1 1 1
Plan Excerpts 4
Instruments 3 3 11 4
Complete Reports
1
Report Excerpts 5 7 1
![Page 8: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/8.jpg)
8
: Online Evaluation Resource Library
Plan Excerpts – Integrated Math-Physics Project
![Page 9: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/9.jpg)
9
: Online Evaluation Resource Library
Curriculum Evaluation Instrument
![Page 10: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/10.jpg)
10
: Online Evaluation Resource Library
Student Assessment Instrument
![Page 11: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/11.jpg)
11
: Online Evaluation Resource Library
Professional Development Follow-Up Instrument
![Page 12: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/12.jpg)
12
: Online Evaluation Resource Library
Report Excerpts – Environmental Science Teacher Education Project
![Page 13: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/13.jpg)
13
: Online Evaluation Resource Library
Complete Report – Professional Development for Physics Instructors
![Page 14: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/14.jpg)
14
Alignment Table Example – Glossary, Quality Criteria, Standards
![Page 15: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/15.jpg)
15
Alignment Table with Pop-Up Definitions
![Page 16: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/16.jpg)
16
Professional Development Module on Methodological Approach and Sampling: Step by Step Introduction to the Strategy
![Page 17: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/17.jpg)
17
Professional Development Module on Methodological Approach and Sampling: Details about Step 1 of the Strategy
![Page 18: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/18.jpg)
18
Professional Development Module on Methodological Approach and Sampling: Introduction to the Scenario (model application of strategy)
![Page 19: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/19.jpg)
19
Professional Development Module on Methodological Approach and Sampling: Introduction to Case Study
![Page 20: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/20.jpg)
20
Professional Development Module on Methodological Approach and Sampling: Interactive Exercise in Case Study
![Page 21: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/21.jpg)
21
Professional Development Module on Methodological Approach and Sampling: Feedback on Case Study User Response
![Page 22: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/22.jpg)
22
: Online Evaluation Resource Library
Evaluation Design StrategiesMETHODOLOGY
Identify the project's assumptions about change.
Decide how much you should be looking at implementation and how much at outcomes.
Determine what types of data to collect, and who to collect it from.
If examining outcomes, determine where the desired outcomes fit on a proximal-to-distal continuum and how the evaluation will address them.
Decide what, if any, comparison outcome data will be collected.
![Page 23: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/23.jpg)
23
: Online Evaluation Resource Library
Evaluation Design Strategies – Links to the Collection
Link phrase: assumptions about change1. briefly describes project goals and how the evaluation was designed to see if they were met (click to see report)
2. describes what the evaluators expected to find as an indirect result of the intervention (click to see report)
3. states the theories of change that were behind the intervention (click to see report)
![Page 24: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/24.jpg)
24
: Online Evaluation Resource Library
Evaluation Design Strategies (cont.)SAMPLING
Determine if data should be collected from an entire population or from a sample. If sampling, randomly select the sample from the population.
Identify which attributes need special attention in the design, and determine how they should be attended to in the sampling process to avoid bias.
Minimize the effects of other sources of bias besides sampling bias.
Determine the optimal sample size.
Maximize the efficiency of your sample.
![Page 25: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/25.jpg)
25
: Online Evaluation Resource Library
Questionnaire Strategies - Appropriateness1. Identify the goals and broad questions that underly of your evaluation
2. Identify the groups from which you need to get information
3. Justify the use of a questionnaire in the evaluation. Condition A: You must be able to say “yes” to one or more of these questions about the population.
- Does it have opinions on the issues that interest you?
- Is it qualified to give you facts or informed judgments on the issues?
Is it the best or sole source of background information about itself (e.g., of background information that is either not available or not as desirable from alternative sources, such as test scores or other records)
Condition B: You must have a clear idea of what variables you want to gather data about, and you need to be reasonably sure you will get that data by posing the same (e.g., standardized) questions.
Condition C: You must believe that the respondents can be relied on to give you the information you need, perhaps with incentives. This means that they will be proficient enough in the language to comprehend the questions and repond properly, truthful, and motivated enough to complete the questionnaire and respond carefully
![Page 26: Geoscience Projects and The Online Evaluation Resource Library … · 2014-09-04 · 23: Online Evaluation Resource Library Evaluation Design Strategies – Links to the Collection](https://reader033.fdocuments.us/reader033/viewer/2022042420/5f371c9b25fcd60be21eabda/html5/thumbnails/26.jpg)
26
: Online Evaluation Resource Library
Questionnaire Strategies (cont.) – Writing Questions
Generate a list of purposes for your questions that address yourevaluation goals.
Compose first drafts of the questions that correspond to your list of goals, and identify what they are supposed to indicate about the phenomena being investigated.
Determine whether each question should be open response or closed response.
Tailor closed response questions to provide maximum benefit.
Word all questions clearly so that respondents understand what you are trying to learn from them.