Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based...
-
Upload
claire-wright -
Category
Documents
-
view
214 -
download
0
Transcript of Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based...
![Page 1: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/1.jpg)
Getting Started With Your ATE Evaluation
ATE PI ConferenceOctober 24, 2012
This material is based upon work supported by the National Science Foundation under Grant Nos. 0802245 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.
![Page 2: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/2.jpg)
To promote the goals of the ATE program by- partnering with ATE projects and centers to
strengthen the program's evaluation knowledge base
- expand the use of exemplary evaluation practices
- support the continuous improvement of technician education throughout the nation
’s Mission
![Page 3: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/3.jpg)
- Understand the role evaluation should play in your ATE project
- Have clear expectations for your evaluator
- Know where to turn if you need more guidance about your ATE evaluation
Session Objectives
![Page 4: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/4.jpg)
Formative EvaluationLearn how the project could be improved as it is implemented
Summative EvaluationProvide evidence of the quality and impact of your achievements
AccountabilityDocument what you did with the grant money
Evaluation Purposes
![Page 5: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/5.jpg)
The most important purpose of evaluation is not to prove, but to improve.
―Daniel Stufflebeam
Formative Evaluation
![Page 6: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/6.jpg)
Project staff…- make time for evaluation- are open to hearing
negative findings - use feedback to inform
decision making
Formative Evaluation
![Page 7: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/7.jpg)
Evaluator…- learns about the project,
its context, and stakeholders
- gathers data in sound, efficient ways
- provides timely feedback- guides project staff in
understanding and using results
Formative Evaluation
![Page 8: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/8.jpg)
What is the project’s quality? What are the project’s impacts?
Summative Evaluation
![Page 9: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/9.jpg)
What is the project’s quality?What are the project’s impacts?- changes in knowledge, skills, abilities, attitudes,
performance, practices, or policies- show up at the individual, program, institution,
organization, regional, or national level- occur in the short term, long term, or in between
Summative Evaluation
![Page 10: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/10.jpg)
Document… - What you did- Who you reached- Who you worked with - What you produced
(outputs)
Accountability
![Page 11: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/11.jpg)
Accountability
www.evalu-ate.org/about_us
Create a project vita- Mission- Goals- Funding- Activities- Products- Staff- Students- Contributors
& collaborators- Consultants
![Page 12: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/12.jpg)
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
![Page 13: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/13.jpg)
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
- Feedback to internal stakeholders
- Reflection and redirection
![Page 14: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/14.jpg)
Formative Evaluation
How is the project doing and how can
it improve?
AccountabilityWhat was done, who was served, who contributed,
what are the outputs?
- Summative report to key stakeholders- FastLane/annual report to NSF - ATE Impacts book - Annual ATE survey - New proposals- Promotion and dissemination
Summative EvaluationWhat is the quality and
impact of the project?
Evaluation Purposes & Uses
- Feedback to internal stakeholders
- Reflection and redirection
![Page 15: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/15.jpg)
Report Sections- Participants- Activities & findings- Publications & products- Contributions
Annual Reports to NSF
www.fastlane.nsf.gov
![Page 16: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/16.jpg)
See also: EvaluATE’s summary of FastLane report components
evalu-ate.org/resources Keyword: FastLane
Annual Reports to NSF
![Page 17: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/17.jpg)
For example, # of students and their
demographic characteristics
# of professional development participants
# of materials developed
www.evalu-ate.org/annual_survey
Annual ATE Survey
![Page 18: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/18.jpg)
Student impact data
Workforce impact data
www.atecenters.orgwww.ateprojectimpact.org
ATE Projects/Centers Impact Book & Website
![Page 19: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/19.jpg)
“The Project Description must begin with the subsection on Results of Prior Support…. This subsection must contain specific outcomes and results including metrics to demonstrate the impact of the activities undertaken including evidence of the quality and effectiveness of the project's deliverables.
—ATE Program Solicitation
“
Results of Prior NSF Support
![Page 20: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/20.jpg)
Be an Informed ConsumerJoint Committee on Standards for
Educational EvaluationThe Program Evaluation Standards
www.jcsee.orgAmerican Evaluation Association Guiding Principles for Evaluators
www.eval.org
Canadian Evaluation SocietyCompetencies for Evaluation Practice
www.evaluationcanada.ca
![Page 21: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/21.jpg)
What should we expect to receive from our evaluator?
― Participant in the 2010 ATE PI Conference
Getting Started Workshop
![Page 22: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/22.jpg)
- Detailed evaluation plan- Opportunities to review
and provide feedback on draft plans and materials
- Interim reports- Annual reports- Knowledge and insights
you could not have obtained on your own
Evaluation Deliverables
![Page 23: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/23.jpg)
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
![Page 24: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/24.jpg)
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
![Page 25: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/25.jpg)
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
![Page 26: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/26.jpg)
EVAL REPORT
EVALUATOR PI
PROGRAM OFFICER
OTHER PROJECT STAKEHOLDERS, E.G.:partners, advisors, participants,
host administrators
Reporting
![Page 27: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/27.jpg)
EVAL REPORT
Ways of reporting evaluation results to your program officer
PI PROGRAM OFFICER
Ask your program officer how he/she prefers the evaluation results to be conveyed.
Complete original report
Originalexecutive summary
Evaluators’ summary in
annual report
Reporting
![Page 28: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/28.jpg)
EVAL REPORT
Ways of reporting evaluation results to your program officer
PI PROGRAM OFFICER
Ask your program officer how he/she prefers the evaluation results to be conveyed.
Complete original report
Originalexecutive summary
Evaluators’ summary in
annual report
ASK!
Reporting
![Page 29: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/29.jpg)
Using EvaluATE
www.evalu-ate.org
![Page 30: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/30.jpg)
NewslettersFeature articles by experienced ATE PIs and evaluators
Q & A
Definitions & discussion of evaluation terms
Resource highlights
![Page 31: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/31.jpg)
Resource Library
Search by - Evaluation task- ATE topic- Keyword- Author
![Page 32: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/32.jpg)
Evaluator Directory
Search by - location- discipline- project type- keyword
![Page 33: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/33.jpg)
Annual Survey
Survey form FAQsReports
![Page 34: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/34.jpg)
Webinars
Upcoming eventsMaterials from past events:- Recordings- Slides- Handouts
![Page 35: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/35.jpg)
Next Webinar
Register at www.evalu-ate.org/events
![Page 36: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/36.jpg)
Conference ActivitiesConcurrent with this sessionPI and evaluator workshop on evaluationTonight Showcase #209-210 Tomorrow Afternoon Birds of a Feather Session:Beyond Satisfaction and Short-Term Self-Reports: Evaluating the Impact of Your ATE Grant
![Page 37: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/37.jpg)
Thank You!
![Page 38: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/38.jpg)
Advice from the Trenches Elaine’s Golden Rule:
The one who has the gold rules!i.e., don’t disperse grant funds to contractors or partners until obligations have been met
“ “
Elaine CraftATE PI & Evaluator
![Page 39: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/39.jpg)
Advice from the Trenches Set up regular data
collection milestones or checkpoints so that data is being collected as part of the normal process of carrying out the grant and doesn't have to be ‘bolted’ onto the process of the grant after it is underway.
““
Karl KappATE Evaluator
![Page 40: Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation.](https://reader030.fdocuments.us/reader030/viewer/2022032414/56649efa5503460f94c0c192/html5/thumbnails/40.jpg)
Advice from the Trenches The evaluator is the expert
in evaluation but you are the expert on your project. If you feel that the evalua-tion is not providing the information you need, then talk to your evaluator. Do it immediately. … You can change evaluators but first, communicate, communicate. communicate.
““Jane Ostrander
ATE PI & National Visiting Committee chairperson