Post on 12-Jan-2016
Joint Committee on Standards for Educational Evaluation
Evaluation Standards
Joan Kruger
Spring 2008
Joint Committee on Standards for Educational Evaluation
http:/www.wmich.edu/ evalctr/jc
SPONSORING ORGANIZATIONS:American Association of School AdministratorsAmerican Counseling AssociationAmerican Evaluation AssociationAmerican Educational Research Association
American Indian Higher Education Consortium American Psychological AssociationCanadian Evaluation Society
Canadian Society for the Study of Education
Consortium for Research on Educational Accountability and Teacher Evaluation
Council of Chief State School Officer
Council of the Great City Schools
National Association of Elementary School Principals
National Association of School Psychologists
National Association of Secondary School Principals
National Council on Measurement in Education
National Education Association
National Legislative Program Evaluation Society
National Rural Education Association
What is it?
Created in 1975, the Joint Committee (JC) is a coalition of major professional associations concerned with the quality of education
It is incorporated in the US as a private non-profit organization
It is accredited by the American National Standards Institute (ANSI) and all standards become certified by ANSI
What is its mission?
To promote concern for evaluations of high quality based on sound evaluation practices and procedures
To meet existing and emerging needs in the field of evaluation
Where is it located and what is the role of CES?
The Joint Committee is housed at the Evaluation Center, Western Michigan University
Canadian Evaluation Society has been a Sponsoring Organization since 1994 and participates as a voting member, participates in special projects, and promotes all Standards (Program, Personnel and Student) to CES members.
Direct Products of the JC
Publications: 3 sets of evaluation standards Personnel Evaluation Standards. 2nd ed. 2008.
Corwin Press Program Evaluation Standards. 2nd ed. 1994.
3rd in testing. Sage Student Evaluation Standards. 2003. Corwin
Press.
Associated Accomplishments
Adoption and adaptation by schools, states, and organizations nationally and internationally.
Numerous associated materials (e.g., Checklists for metaevaluation).
http://www.wmich.edu/evalctr/checklists/
Widespread translation and/or dissemination internationally, for use in Europe (especially Germany and Switzerland, Latin America, Africa and others)
Pervasive Nature of Program Evaluations
Primary tool for guiding program evaluation, crossing all academic disciplines and courses
Evaluations regularly clients and stakeholders in a wide variety of decision situations.
Impact of Program Evaluations
Evaluations done well can be of significant service.
Clients are victims of and harmed by poor evaluations in both high- and low-stake evaluations
Barriers to Sound Program Evaluations
Inadequate preparation Inadequate administrative and technical
support Professional disagreements Lack of shared language Inadequate policy
Why do we need Program Evaluation Standards?
What is available? Assessment &
measurement design Performance
management systems
Standards for student and personnel evaluation
What are the gaps? No standards for the
programs evaluations across disciplines.
No desktop reference to ensure quality
What is a Standard?
A principle commonly agreed upon by experts in the conduct and use of evaluation for the measurement of the value or quality of an evaluation. A standard is seen as a principle governing
good practice vs. a rule Technical guide to conduct evaluations Criteria of good practice for those who receive
and are affected by an evaluation
What is a Standard?
A guideline may provide the evaluator with a recommendation to help ensure basic quality evaluation. Other evaluators, in the same situation in the same or similar institutions following the guidelines, may also ensure basic quality.
The CES felt standards used as guidelines would help evaluators make more consistent quality judgements over time.
What is a Standard is not.
Rules are much more restrictive. The evaluator must or should conform to rules. This may lead to a narrower rating of a program or project by not allowing the evaluator to adapt the standard to the norms of the evaluation setting.
Rules are usually policed by a professional body, and offenders may be reprimanded.
How are the Standards Organized?
All three sets of Standards are organized into four main categories:
Utility Propriety Feasibility Accuracy
Utility
To ensure that an evaluation will serve the information needs of intended users.
Propriety
To ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation as well as those affected by its results.
Additional resource: Canadian Evaluation Society Ethical Guidelines
http://www.evaluationcanada.ca/site.cgi?s=5&ss=4&_lang=EN
Feasibility
To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.
Accuracy
To ensure that an evaluation will reveal and convey technically adequate information bout the features that determine worth and merit of the program being evaluated.
How are the Standards Presented in the Book?
A uniform pattern is followed: Standard Statement Overview (Explanation, Rationale, Caveats) Guidelines Common Errors Two illustrative case studies Supporting documentation (references)
Number of Standards(Program: proposed 3rd ed)
Category Program Personnel Student
Utility 8 5 7
Propriety 7 5 7
Feasibility 4 3 3
Accuracy 8 8 12
TOTAL 27 21 29
National Public Hearings
National Public Hearings: The Joint Committee will hold open forums at meetings of each of the sponsoring organizations to gain additional feedback from members of these organization (See 5.3.5 of the Operating Procedures-).
5.3.5 National Public Hearings National Public Hearings shall be held to provide an open forum for discussion and
critique of the standards. Hearing schedules shall be publicized and all interested parties will be encouraged to participate. In addition, each Sponsoring Organization will be asked to select members of their organization and encourage them to respond to the draft through the vehicle of the hearings. (NOTE: If the standards under consideration are to be submitted to the American National Standards Institute for approval as American National Standards, the Joint Committee shall also arrange for announcement of the draft in ANSI's STANDARDS ACTION for comment, in accordance with Section 1.2.6 of the ANSI Procedures for the Development and Coordination of American National Standards.)
Process for Hearing
Overview of the Standards Comments from CES members and
others (criticisms, observations, recommendations)
Comments from Field test participants
Further Communication
Visit the JC web site at
http://www.wmich.edu/evalctr/jc/ and enter ES review page.
Contact Arlen Gullickson arlen.gullickson@wmich.edu
Next Steps
Complete field tests Complete National Hearings Revise Final Approval by the Joint Committee Submission to American National
Standards Institute for certification Publish
Evaluation Web Sites
Canadian Evaluation Society (CES)
http://www.evaluationcanada.ca/
Canadian Evaluation Society Educational Fund (CESEF) – for students and those new to eval.
http://www.evaluation-education.org/index.html
Saskatchewan Chapter, CES
http://sk.evaluationcanada.ca/
American Evaluation Association (AEA)
http://www.eval.org/