W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator:...
-
Upload
mavis-stevens -
Category
Documents
-
view
225 -
download
0
Transcript of W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator:...
![Page 1: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/1.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Planning Engineering Education Research
Facilitator: Matthew W. Ohland
![Page 2: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/2.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Workshop Objective
Describe steps in the planning and assessment of engineering education research
Give examples of each step
Small-group discussion to implement the step using a common example
![Page 3: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/3.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Workshop Agenda
4:00-4:05 Introduction4:05-4:20 Defining the Purpose of the Evaluation
(What questions are we asking?)4:20-4:40 Clarify Project Objectives (What do we expect?)4:40-5:00 Create a Model of Change
(How will our efforts lead to the objectives?)5:00-5:15 Select Criteria and Indicators
(What data do we need to measure our progress?)5:15-5:25 Identify Data Sources (Where will we get that data?)5:25-5:45 Design Evaluation Research
(How will we analyze that data?)5:45-5:50 Monitor and Evaluate (What actually happened?)
Use and Report Results (Who needs to know?)5:50-6:00 Wrap-up and evaluations
![Page 4: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/4.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 1: Purpose of the Evaluation
FormativeSummative
EfficacyEffectivenessCost
Examples• Is there a benefit to doing
this?• Can we improve it?• Does that benefit lead to
better retention / grades?• Do the benefits justify the
program cost?• Can we achieve the same
benefits on a larger scale?
![Page 5: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/5.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 1 Action Steps
Decide purpose(s)
Primary questions? Order?
Primary audience(s)?
How will the results be used?
![Page 6: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/6.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 1: Purpose of the Evaluation
Do we have a learning community?
Is it leading to better course retention / grades?
Can we achieve the same benefits cost-effectively on a larger scale?
![Page 7: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/7.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 2: Project Objectives
Impact
Outcome
Process
Example• Improve graduation rate
by some %
• Increase course passing rate by some %
• Student or faculty opinions / behaviors change
• Reserve any facilities needed
![Page 8: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/8.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 2 Action Steps
Write down project objectives• Impact• Outcome• Process
![Page 9: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/9.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 2: Project Objectives
Increase graduation rate from 50% to 75%
Increase first-time Calculus passing rate from 50% to 70%
Students have positive opinion of group work
Secure dormitory and classroom space
![Page 10: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/10.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 3: Create a Model of Change
Identify assumptions you can assessChoose relationships to test based on• Resources• Where you anticipate
problems• Where you have
control / can make improvements
Link what you do to what you expect to happenExample• Common residence /
classes • in between, we need a
sound theory of why this will happen
more will graduate
![Page 11: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/11.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 3 Action Steps
Model of change• as specific / complete as needed
Review model assumptions
Use criteria to prioritize• Resources• Relevance• Control
![Page 12: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/12.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 3: Create a Model of Change
Common residence / classes affiliation
Affiliation support
Support performance
Performance future performance
Future performance more will graduate
![Page 13: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/13.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 4: Criteria and Indicators
Validity
Reliability
Sensitivity
Ease of interpretation
Usefulness
Example: define• Graduation rate
• Course pass rate
• Affiliation
• Life-long learning
• Enrollment
![Page 14: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/14.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 4 Action Steps
Define a set of indicators and criteria • Impact• Outcome • Process
![Page 15: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/15.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 4: Criteria and Indicators
Graduation rate• # graduated / # in original cohort
Calculus pass rate• # A, B, C / # in original cohort• D’s are no good – must be retaken
Affiliation • # in study group at end of sophomore year /
# students
Number of students enrolled in program
![Page 16: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/16.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 5: Data Sources
ExamsSurveysObservationsStudent recordsSAT scores
Frequency• Resources• Guidance• Change
Example• Institutional Research
office (annual)• Course records (each
semester)• Registrar• Admissions• Assessment office• Process - monitor until
achieved
![Page 17: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/17.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 5 Action Steps
Define data sources
Define frequency of measurement
![Page 18: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/18.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 5: Data Sources
Grad / retention rate – Inst. Res. – AnnualCalculus performance – Math dept. – includes course grades and common exam results –end of semesterDorm space allocated – Housing – monitor until achievedSurvey of program participantsObserver evaluations of class interactionSAT scores—Admissions
![Page 19: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/19.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 6: Design Evaluation Research
Selection
Mortality
Placebo is not an issue
Qualitative• Interviews
• Focus groups
• Systematic observation
Quantitative• Non-experimental
- Posttest only
- Pretest-Posttest
• Quasi-experimental- Time Series
- Nonequivalent control
• Experimental- Pretest-Posttest Control
- Multiple Intervention
![Page 20: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/20.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 6 Action Steps
Design evaluation research studies for key questions
![Page 21: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/21.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 6: Example Designs
Example-Interviews• Student expectations—
invitation process modified
• Resource utilization—approaches to motivate attendance
Example-Efficacy• Non-equivalent control
group
• Test for selection bias using baseline measures
• Matched pairs / groups
• Calculus grades and overall GPR
![Page 22: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/22.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 7: Monitor and Evaluate
Establish project information system
Budget for evaluation
Evaluation meetings
Review and revise evaluation plan
Carry out studies
![Page 23: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/23.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Step 8: Use and Report Results
Report results – to everyone
Use results to make improvements
![Page 24: W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.](https://reader030.fdocuments.us/reader030/viewer/2022033102/5697bf821a28abf838c85f8e/html5/thumbnails/24.jpg)
W W W . C E S . C L E M S O N . E D U / G E /
Conclusions
Planning and assessment is essential
Start small if resources are limited
Develop a plan before starting
NSF and other agencies support well-designed educational research• CCLI—EMD / A&I, ASA, and other programs
Seek appropriate partners from education, psychology, sociology, statistics, etc.