11

Power of Reviewing StandardsPower of Reviewing StandardsPower of Reviewing StandardsPower of Reviewing Standards

Consortia Conference CallConsortia Conference Call

March 15, 2005March 15, 2005

Susan PimentelSusan Pimentel

Consortia Conference CallConsortia Conference Call

March 15, 2005March 15, 2005

Susan PimentelSusan Pimentel

22

Welcome Evaluation!Welcome Evaluation!

Benefits:Benefits:• Ensures standards are appropriate, clear, Ensures standards are appropriate, clear,

accurate and completeaccurate and complete• Builds ownership and awareness Builds ownership and awareness • Provides ideas about rollout of standards. Provides ideas about rollout of standards.

professional development, etc.professional development, etc.

33

Make it Multi-Faceted!Make it Multi-Faceted!1. 1. Internal ReviewInternal Review: Determine how the standards drafts align to : Determine how the standards drafts align to

the research and other documents as required (e.g., GED, K-12 the research and other documents as required (e.g., GED, K-12 standards, NCTM) {Alignment}standards, NCTM) {Alignment}

2.2. Field ReviewField Review: Ensure standards are understandable and : Ensure standards are understandable and acceptable to field who will be charged with standards’ acceptable to field who will be charged with standards’ implementation. {Validity}implementation. {Validity}

3.3. External Expert ReviewExternal Expert Review: Focus here is primarily on the : Focus here is primarily on the sufficiency and understandability of the content.sufficiency and understandability of the content.

{Validity}{Validity}

44

Tips for Internal ReviewTips for Internal Review

1. Ask members of writing teams to conduct. 1. Ask members of writing teams to conduct.

2. Get specific about how standards align to research or other docs. Are 2. Get specific about how standards align to research or other docs. Are they:they:• Congruent?Congruent?• Aligned?Aligned?• Contradictory?Contradictory?

3. Document extent of alignment (develop written rationale or correlation 3. Document extent of alignment (develop written rationale or correlation chart).chart).

55

Tips for Field Review: Tips for Field Review: Focus GroupsFocus Groups

1.1. Focus Group:Focus Group:• Involve instructors, program directors, etc.Involve instructors, program directors, etc.• Keep groups small (10-15)Keep groups small (10-15)• Plan on 3 hoursPlan on 3 hours• Be prepared to walk through the docsBe prepared to walk through the docs• Employ members of writing teams to runEmploy members of writing teams to run• Send invitations and drafts well in advance of Send invitations and drafts well in advance of

sessionsession• Hold sessions around key regions of the stateHold sessions around key regions of the state

66

Focus Groups, cont’d.Focus Groups, cont’d.

• No defensiveness allowed!! Time to No defensiveness allowed!! Time to listen and inquirelisten and inquire

• Develop feedback forms/surveysDevelop feedback forms/surveys• Document conversations/feedbackDocument conversations/feedback• Report back to focus groups (meet again Report back to focus groups (meet again

if possible)if possible)

77

Tips for Field Review:Tips for Field Review:SurveySurvey

• Test survey with members of your writing teamTest survey with members of your writing team• Mail out to programs w/personal appeal to Mail out to programs w/personal appeal to

directorsdirectors• Make it web-based as wellMake it web-based as well• Collect quantitative and qualitative informationCollect quantitative and qualitative information• Ask about more than just the quality of the Ask about more than just the quality of the

standards (e.g., roll out ideas, prof. dev.)standards (e.g., roll out ideas, prof. dev.)• Consider using a general question survey and Consider using a general question survey and

standards specific surveystandards specific surveySee attached draft survey.See attached draft survey.

88

Combo is Best!Combo is Best!

1.1. Pros of Focus Group: Personal and in-depth inquiry possible (but time Pros of Focus Group: Personal and in-depth inquiry possible (but time consuming and can only accommodate small numbers)consuming and can only accommodate small numbers)

2.2. Pros of Surveys: Able to reach large numbers and time and cost efficient (but Pros of Surveys: Able to reach large numbers and time and cost efficient (but can’t inquire, explain, discuss and promote greater understanding).can’t inquire, explain, discuss and promote greater understanding).

Combo allows gives you depth and breadth; allows you to gather rich qualitative Combo allows gives you depth and breadth; allows you to gather rich qualitative and quantitative information.and quantitative information.

99

Tips for External Expert Tips for External Expert ReviewReview

1.1. Develop a set of questions for the expert reviewer. Develop a set of questions for the expert reviewer. • Include key aspects of the academic domain? Any gaps? Include key aspects of the academic domain? Any gaps? • Clear and specific enough to inform instruction?Clear and specific enough to inform instruction?• Adherence to research? Adherence to research? • Appropriate rigor?Appropriate rigor?• Reasonable and manageable?Reasonable and manageable?• Proper sequence or progression of skills?Proper sequence or progression of skills?

2.2. Be clear about how specific you want the reviewer to be (e.g., general evaluation and Be clear about how specific you want the reviewer to be (e.g., general evaluation and recommendations or specific suggestions for wording changes and refinements).recommendations or specific suggestions for wording changes and refinements).

3.3. Need to give “experts” a month or more lead-time.Need to give “experts” a month or more lead-time.

1010

Pros of Expert ReviewPros of Expert Review

They are: They are: • Current on researchCurrent on research

• Familiar with expectations across the nation (and perhaps the world!) so they offer a Familiar with expectations across the nation (and perhaps the world!) so they offer a broad perspectivebroad perspective

• IndependentIndependent

• Able to recognize common pitfallsAble to recognize common pitfalls

• Offer specific recommendations about how to strengthen your standardsOffer specific recommendations about how to strengthen your standards

• Able to provide standards with a “stamp of approval” that you can present to legislators, Able to provide standards with a “stamp of approval” that you can present to legislators, policy makers, etc.policy makers, etc.

1111

Prepare Teams for ReviewPrepare Teams for Review1.1. Reviewers tend to tell you what they don’t like rather than what Reviewers tend to tell you what they don’t like rather than what

they do like.they do like.

2. Read between the lines: Standards mean change and change can 2. Read between the lines: Standards mean change and change can be scary so important to hold on to your principles (e.g., must be scary so important to hold on to your principles (e.g., must align to the research, prepare adults for success).align to the research, prepare adults for success).

3. Prepare for conflicts in the reviews and determine means to 3. Prepare for conflicts in the reviews and determine means to grapple with the contradictions (go back to why you asked a grapple with the contradictions (go back to why you asked a group or individual to give you feedback).group or individual to give you feedback).

1212

Prepare Teams, cont’d.Prepare Teams, cont’d.

4. Let teams know that feedback offers a set of suggestions, not mandates.4. Let teams know that feedback offers a set of suggestions, not mandates.

• Having said that, let teams know when it is just a matter of style or the change Having said that, let teams know when it is just a matter of style or the change won’t diminish the draft, make the change even if it isn’t their preferred choice. won’t diminish the draft, make the change even if it isn’t their preferred choice. It promotes ownership.It promotes ownership.

6. Set the expectation that teams will review every suggestion carefully and either adjust the standards or note a rationale for not accepting the feedback.

The EndThe End