Standard 6 Pilot of the Analysis of Student Work (ASW) Process
description
Transcript of Standard 6 Pilot of the Analysis of Student Work (ASW) Process
![Page 1: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/1.jpg)
Standard 6 Pilot of the Analysis of
Student Work (ASW) Process
May 28, 2013
NCDPI Reflection & Planning Retreat
![Page 2: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/2.jpg)
Welcome & Overview
![Page 3: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/3.jpg)
NC Student Growth Portfolio Training wikipage
All of today’s materials are posted online and can be downloaded for notetaking
http://wlnces.ncdpi.wikispaces.net/ncsgptraining
![Page 4: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/4.jpg)
Comfort & Considerations
• Restrooms & Breaks
• Wireless Network
• Electronic Device
• Power Strips & Extension Cords
• NCDPI Retreat Planning Dochttp://bit.ly/NCDPIRetreat For going green . . .
![Page 5: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/5.jpg)
Agenda for May 28th
• Welcome & Overview• Reflection on Standard 6 Pilot
Feedback • Planning Groups
– Form & Prioritize Year 1 Implementation Needs
– Conduct Work Sessions
• Debrief & Next Steps
![Page 6: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/6.jpg)
NCDPI Retreat Objectives
• Reflect on the Standard 6 Pilot Feedback – Listen carefully– Discuss critically– Think creatively
• Help plan and create professional development materials for Year 1 implementation:
July 1, 2013 – June 30, 2014
![Page 7: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/7.jpg)
Introductions
With the group, please share:
• Name & Title(s)
• Division
• A decision, project, idea, etc. that you got a “green light” on this year
(in 12 words or less)
![Page 8: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/8.jpg)
Overview of Spring 2013 Pilot
• 100+ educators with 30-35 in each content area
• 2/3 Submitters & 1/3 Evaluators
• Over 80% of piloters . . . – Submitted work samples– Attended the webinar– Completed the evaluation survey
(60 Submitters & 27 Evaluators)
![Page 9: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/9.jpg)
Pilot Progress & Planning Group → Analysis Teams
Submission Process• Slater Mapp• Carmella Fair• Julie Malcolm• Cathy Mathews
Evaluation Process• Christie Lynch Ebert• Mike Martin• Nadine McBride• Tara Patterson
Online Platform• Les Spell• Randy Craven• Dru Davison• Helga Fasciano• Nadine McBride
Future Planning• Ann Marie Gunter• Donna Albaugh• Jen DeNeal• Jennifer LaGarde
![Page 10: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/10.jpg)
Going Green . . . from a mind map
![Page 11: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/11.jpg)
Going Green – Jigsaw Approach
1. Each Analysis Team will present a summary of the feedback on their topic from the pilot, with recommendations.
2. Individually, we will consider the information just presented and brainstorm ways to Go Green.
![Page 12: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/12.jpg)
Going Green – Jigsaw Approach
3. After the last presentation and reflection, we will take a break and then use the NCDPI Retreat Planning Doc (http://bit.ly/NCDPIRetreat) for Going Green.
4. As we work, please join a planning group and start prioritizing for Year 1.
![Page 13: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/13.jpg)
Submission Survey Questions 1-7
Submission Process
![Page 14: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/14.jpg)
![Page 15: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/15.jpg)
![Page 16: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/16.jpg)
![Page 17: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/17.jpg)
![Page 18: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/18.jpg)
![Page 19: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/19.jpg)
![Page 20: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/20.jpg)
7. Now that you have completed the pilot, what would you do differently in preparing submissions to improve the process? (Follow this link to see actual responses.)
Planning 24%
Time24%
Objectives16 %
DigitalFormat
16%
Nothing 9%
Pre-Post 7%
would have planned for the assessment better, chosen different CO's for submission, collected more students samples, and perhaps chosen specific students as opposed to submitting work from classes.
spend more time with the SCOS to choose a variety of objectives that would work well for showing student growth and think more about comparing similar tasks in two points of time.
![Page 21: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/21.jpg)
What we can learn from Venture Labs
Feedback Analysis
![Page 22: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/22.jpg)
![Page 23: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/23.jpg)
![Page 24: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/24.jpg)
![Page 25: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/25.jpg)
![Page 26: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/26.jpg)
![Page 27: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/27.jpg)
![Page 28: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/28.jpg)
![Page 29: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/29.jpg)
What will be Submitted and Evaluated
Matrix
![Page 30: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/30.jpg)
1 2 3 4 5
Teaching LoadRepresentative Courses or Classes
Instructional LevelAt least three of theevidences need to show growth for students at different instructional levels (e.g., students below, at, or above grade or proficiency level at the beginning of instruction).
StudentsAt least one of the evidences needs to represent multiple students. Indicate here which sample/s represents multiple students.
Time Indicate in this row which evidence sample represents growth over an entire course or class.
Instructional Strands Indicate in this row at least two different strands for your licensure area/s.
![Page 31: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/31.jpg)
Going Green – Jigsaw Approach
Individually, consider the feedback and recommendations just presented and brainstorm
ways to Go Green with the Analysis of Student Work
(ASW) Submission Process
![Page 32: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/32.jpg)
EVALUATOR SURVEY AND DESCRIPTIVE FEEDBACK
• Evaluator Survey (1-9)
• Descriptive Feedback (Rounds 1 and 2)
• 27 Respondents for survey
Team members:• Christie Lynch Ebert
(Lead)• Mike Martin• Nadine McBride• Tara Patterson
![Page 33: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/33.jpg)
QUESTION 1 (N=27)
![Page 34: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/34.jpg)
QUESTION 2 (74% SPENT 2-5 HOURS EVALUATING SUBMISSIONS)
![Page 35: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/35.jpg)
QUESTION 3 (85% SPENT 0-3 HOURS)
![Page 36: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/36.jpg)
QUESTION 4 (70.3% CONFIDENT/VERY CONFIDENT)
![Page 37: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/37.jpg)
QUESTION 5
What signs should an Evaluator look for to score a submission as Exceeds Expected Growth? (25 responses) – see narrative
Summary indicators from narrative:
• Exemplary/significant growth in relation to CO
• Demonstration of exceeding standards
• Quality of submission, meaningful tasks, details in submission descriptions
• Concordance between: goals/standards; topic/resources/activities; time and results
• Comparison to unpacking documents
• Connection from teacher to student’s growth very clear (extraordinary and beyond meeting expectation of CO – can not be quantified in data alone)
• Complete analysis of how teaching strategies impacted student growth (rubric or checklist)
• Evidence of above and beyond scope of what was taught in lesson
![Page 38: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/38.jpg)
QUESTION 5
What signs should an Evaluator look for to score a submission as Exceeds Expected Growth? (25 responses) – see narrative
Summary indicators from narrative: (continued)
• National board quality descriptions in writing and significant evidence in student performance
• Aligned instruction and assessment with objective leading to student surpassing normal expectations
• Analysis of: pre-assessment and gaps; teaching strategies and articulation of purposeful use; how teaching strategies impacted student growth (not just end product as evidence without meaningful, authentic analysis)
• Depends on CO – ex. Move from Novice Mid to Novice High – not just wtithin the same proficiency level
• Depth to evaluation
• Goes beyond the norm according to the standard.
• Alignment: all required info., especially narrative included; ID of Clarifying Objective
• Targeted objective was exceeded and thoroughly explained and demonstrated with and without words.
![Page 39: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/39.jpg)
QUESTION 5
What signs should an Evaluator look for to score a submission as Exceeds Expected Growth? (25 responses) – see narrative
Summary indicators from narrative: (continued)
• Videos, pictures, student work, self-assessments, teacher rubrics pre and post
• Comparison between more than one student - how students were pushed to grow; how the middle of the class grew, and how the low student grew. Look at more than 2-3 students for credibility. Would be nice to see a whole class rubric score than pick out 3-4 students in the class to highlight and show work.
• Look for growth over time and compare to context information and work samples; view the submission holistically
• Look at every aspect of evidence 1 and 2, how excellent they are and how clearly the growth can be determined
![Page 40: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/40.jpg)
QUESTION 6 (85% 0-10 HOURS)
![Page 41: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/41.jpg)
QUESTION 7
![Page 42: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/42.jpg)
QUESTION 7 (WHAT ISSUES DID YOU HAVE WITH THE SUBMISSIONS YOU EVALUATED?)
Narrative responses:
• Examples did not adequately address CO
• Too many log-in issues, could not get to submissions
• Everything needs to be on one page – need to be able to see as reading descriptions, flipping back and forth was frustrating
• Student sample had incorrect answers of information as a result of teacher providing incorrect instruction
• Too many Cos identified to focus on growth
• Able to see growth but lack of meaningful analysis of how teacher influenced student’s growth
• Unclear measurement devices
• Submitter not skilled in writing about evidence
• Some CO’s not developmentally appropriate
• Submitter only submitted one evidence which makes it hard to evaluate
![Page 43: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/43.jpg)
QUESTION 8: GUIDELINES
What additional information or materials would have helped you better evaluate the submissions? (23 responses – see narrative)
Guidelines • Complete background information on duration of course, minutes or
hours per day in course, number of days of instruction per week, hours completed prior to collecting pre and post evidence
• Concise instructions on what information and how to submit
• Standard form of questions
• Running monologue is not conducive to evaluation
• Suggest specific questions asked of each teacher and then space for additional info. as needed
![Page 44: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/44.jpg)
QUESTION 8: GUIDELINES
What additional information or materials would have helped you better evaluate the submissions? (23 responses – see narrative)
Guidelines • Training on what to focus on and a checklist for feedback – writing anything
down in feedback allows too much leeway for untrained teachers/evaluators
• More segments of student submissions (beginning, mid-point, end)
• Information asked for is more than sufficient if submitter provides enough data and detail; teachers should not expect to meet expectations based on narrative alone
• Consistent tools to measure every student
• Too vague, too many CO’s – helpful to have continuity of CO’s and proficiency levels
![Page 45: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/45.jpg)
QUESTION 8: GUIDELINES
What additional information or materials would have helped you better evaluate the submissions? (23 responses – see narrative)
Guidelines • Rubric with examples that meet, exceed, or do not meet expectations
• Checklist of things to look for when evaluating
• Clear criteria for submitters – work samples not enough – how did activities lead to student growth? Articulation of learning styles and complete checklist or rubric for post assessment (missing from samples)
• Clearly defined parameters about what “exceed” would look like; more guidelines for “not met”
• Printable guide on evaluations
• Content-specific points related to scoring and agreement on how to score
![Page 46: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/46.jpg)
QUESTION 8: FORMAT
What additional information or materials would have helped you better evaluate the submissions? (23 responses – see narrative)
Format• Samples need common format (jpg, doc, pdf, etc.)
• Consistency in acceptable formats (e.g. length of submission and file formats)
• Need video to see growth in dance (not pictures)
• Easier format to look up submissions and evidence
• All materials in one location
• Samples labeled correctly
• Samples need to be viewed side by side to judge growth
• Accessibility to samples (blocked YouTube site)
• Easier format
![Page 47: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/47.jpg)
QUESTION 9
Now that you have completed the pilot, what would you do differently to improve the evaluation process? (23 responses – see narrative)
• Specific training for evaluators
• Clear, concise instructions on what constitutes evidence; how in-depth should evidence be?
• What duration of time is a fair measuring stick for a WL (Arts Ed, HL) teacher?
• Can evidence of student growth include products with teacher feedback?
• Software – access easier, linear and step-by-step
• Process: clear examples needed throughout process
• Size and validity of evidence in portfolio – even in worst case, isn’t it possible to show 3 or so students have grown?
![Page 48: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/48.jpg)
QUESTION 9 (CONTINUED)
Now that you have completed the pilot, what would you do differently to improve the evaluation process? (23 responses – see narrative)
• Who will train principals/administrators?
• Improve platform, make process anonymous, provide evaluators training; provide specific feedback items
• Utilize trained reviewers for success
• Reviewers must be content specific (e.g. you can not ask a choral music teacher to review a band portfolio)
• I am very concerned about K-8 teachers with heavy loads (700-900 students weekly) – How do we adjust for inequities across the state – student contact time, resources, etc?
![Page 49: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/49.jpg)
QUESTION 9 (CONTINUED)
Now that you have completed the pilot, what would you do differently to improve the evaluation process? (23 responses – see narrative)
• Platform should have been evaluated prior to this pilot – very frustrating and upsetting; improve user-friendliness
• It is all very subjective
• Assign CO’s at the beginning of the year and mandate that all submissions for that level class focus on these CO’s – teacher could pick students and work samples for pre-determined areas.
• Guidelines more in-depth; nice to have specific or several different CO’s to choose from
• Process needs to be clearer and subject-specific
![Page 50: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/50.jpg)
DESCRIPTIVE FEEDBACK (ROUNDS 1 AND 2)• Document created and posted on google site with Scoring
guide and descriptive feedback
• We have received numerous inquiries about piloters desiring feedback on their submissions so one action should be to determine how/what might be shared
• Based on analysis and conversation with team, it is recommended that we either have: (see examples on following slides)
• 1. Optional descriptive feedback • 2. No descriptive feedback (this was advised from legal counsel in
TN)• 3. Pull-down menu of descriptive feedback or rationale (may need
to continue to be built over time) – current data does not support generalized options so will need continued work in this area
• 4. Slider scale with criteria listed for rating.
![Page 51: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/51.jpg)
1. OPTIONAL DESCRIPTIVE FEEDBACK GRID
Specific strengths of the submission.
Things that could be improved.
Things that I don’t understand.
?
New ideas to consider.
This grid may be completed by the evaluator and would be accessible to the submitter. The feedback does not impact the score. [reminder about professional language and clear, objective statements]
![Page 52: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/52.jpg)
2. NO DESCRIPTIVE FEEDBACK
This was an issue discussed with Dru on May 20, 2013.
• TN legal counsel advised against providing descriptive feedback.
• Some effort may be underway to collect feedback to create generalized drop-down or similar means to provide feedback.
![Page 53: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/53.jpg)
3. PULL DOWN MENU: DESCRIPTIVE FEEDBACK/RATIONALE
Does Not Meet Expected Growth
MeetsExpected Growth
ExceedsExpected Growth
The student sample did not provide adequate evidence of student growth in relationship to the identified Clarifying Objective(s).
The student sample provided adequate evidence of student growth in relationship to the identified Clarifying Objective(s).
The student sample significantly exceeded growth expectations in relationship to the identified Clarifying Objective(s).
Rationale:• No CO identified• The pre- and/or post-
work does not evidence growth in relationship to the CO (not aligned)
Rationale:• The evidence clearly
illustrates growth aligned with the RBT verb and intended learning outcome identified.
Rationale:• The evidence clearly
illustrates exceptional growth as indicated by alignment with pre/post work, identified CO, and student work samples and supporting materials.
![Page 54: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/54.jpg)
4. SLIDER WITH CRITERIA
Growth Range
Does Not Meet Expected Growth The student sample did not provide adequate evidence of student growth in relationship to the identified Clarifying Objective(s).
Meets Expected Growth The student sample provided adequate evidence of student growth in relationship to the identified Clarifying Objective(s).
Exceeds Expected Growth The student sample significantly exceeded growth expectations in relationship to the identified Clarifying Objective(s).
Does Not Meet Expected Growth
Meets Expected Growth Exceeds Expected Growth
![Page 55: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/55.jpg)
PROS AND CONS OF PROVIDING FEEDBACK
PROS CONS
• Can inform teacher practice for future submissions
• Legal ramifications/appeals?
• Forces evaluator to rationalize/justify ratings
• Subjective if open-ended/narrative vs. pull down
• Depending on structure, could reduce subjectivity
• Time consuming for evaluators
• Provides more information about how ratings are determined
• Lack of generalized list
![Page 56: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/56.jpg)
TRAINING RECOMMENDATIONS
Submission:
• Recommend using the descriptive feedback to inform training components for developing evidence collections.
• Questions: Is or will this be tied to Pay for Performance? Are we setting our teachers up to not be able to receive additional pay because “exceeds” is so difficult to define? Is it possible to receive an exceeds rating in extremely limited teaching situations?
Evaluation:
• Recommend giving examples for evaluators to score and then compare with an expert rating.
• Include rationale for why score was selected. (Your score, expert’s score, why?)
![Page 57: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/57.jpg)
EXAMPLES WITH EXPERT RATIONALEExperts Explanation
A Does Not Meet
Expected Growth
Meets Expected Growth
B Meets Expected Growth
Exceeds Expected Growth
C Does Not Meet
Expected Growth
Does Not Meet
Expected Growth
![Page 58: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/58.jpg)
RECOMMENDATIONS BASED ON ANALYSISEvaluation:
• 74% of evaluators spent 2-5 hours reviewing submissions during pilot
• Estimate apprx. 10 hours of man-time for each teacher collection of 5 evidences (2 points in time) to be reviewed by 2 separate reviewers X apprx. 8500 Teachers = 85,000 hours of review • Example: If each reviewer reviewed 8 collections, this would = 40 hours
• State or regional review committees will need to take into account proportions for numbers of teachers in each content area and grade span:
1,900 – World Language teachers 1,200 – Healthful Living teachers 5,400 - Arts teachers
![Page 59: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/59.jpg)
RECOMMENDATIONS BASED ON ANALYSISContent Area and Level
Number of Teachers
Est. number of hours for review
Est. number of reviewers
Dance ~200 2000 (if each reviewer reviews 8 collections and each collection is reviewed twice)
Music ~2,600
Theatre Arts ~350
Visual Arts ~2,100
Health ~1,200
Physical Educ.
World Languages
~1,900
![Page 60: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/60.jpg)
EXAMPLE: TN
![Page 61: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/61.jpg)
RECOMMENDATIONS BASED ON ANALYSISNarrative comments identify need for decision making on
various topics, such as:
• Content-specific evaluation
• Consideration for various teaching situations (K-8)
• Clear, precise process (training materials for submitters and evaluators with content-specific examples)
• Functional online platform
• Guidance for selecting/alignment with CO’s
• Guidance and focus of narrative in submission process
• Parameters for rating categories and subject-specific examples of each rating
• Other
![Page 62: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/62.jpg)
Going Green – Jigsaw Approach
Individually, consider the feedback and recommendations just presented and brainstorm
ways to Go Green with the Analysis of Student Work
(ASW) Evaluation Process
![Page 63: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/63.jpg)
ASW Pilot AnalysisOnline Platform and General Feedback
![Page 64: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/64.jpg)
Submitter Survey Question #8
![Page 65: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/65.jpg)
Submitter Survey Question #9
![Page 66: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/66.jpg)
Submitter Survey Question #10
![Page 67: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/67.jpg)
Evaluator Survey Question #10
![Page 68: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/68.jpg)
Evaluator Survey Question #11
![Page 69: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/69.jpg)
Evaluator Survey Question #12
![Page 70: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/70.jpg)
General Feedback
![Page 71: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/71.jpg)
Moving Forward with Online Platform
Non-NegotiableSecureDouble-Blind Evaluation and trigger for 3rd evaluation if
neededUnique Identifier for Submitters and RatersEase of Use
Single page with tabs File type homogenizer which handles uploads File viewer built into system
Clear Directions and Training Submission Process Types of Submissions and limits Evaluation Process
![Page 72: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/72.jpg)
Moving Forward……….. To Be Determined
Which platform moving forward Timeline for development or adaptation Training and materials Size and type of files Who will manage the submissions timeline What is the process if submissions are not complete and therefore not ratable Who will manage the distribution of rater assignments Who will manage the completion of ratings/follow up with assigned raters who
have not completed Will there be an arbitration process and how will this be managed in the
online platform Data Transfer to EVAAS: Format and Timing HelpDesk: technical; policy oriented Which system do we integrate with for the coming year: SchoolNet,
PowerSchool, Identity Access Management, or NCEducation for the 2013-2014 school year
![Page 73: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/73.jpg)
GLADiS
• Evidence Collection Tool
• Allows for Peer Review
• Accepts a variety of formats
• Accessible until portfolio is closed for submission
• Accessible outside of MCS network
GLADiS Software
![Page 74: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/74.jpg)
Going Green – Jigsaw Approach
Individually, consider the feedback and recommendations just presented and brainstorm
ways to Go Green with the Analysis of Student Work (ASW) Online Platform
![Page 75: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/75.jpg)
FUTURE PLANNING
ASW Pilot Analysis
![Page 76: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/76.jpg)
Submitter #11 and #12 Evaluator #13 and #14
1) Anonymous NC teacher2) State committee
3) Self4-5) Principal or Central Office
Others: retired teachers, 10+ years exp. or various years of exp. on cmte., district teams with mix of content teachers or teams from similar-sized districts, 3 ratings per submission with evaluators receiving ratings on how well they do as evaluators
1) State committee2) Anonymous NC teacher
3) Self4) Central Office5) Principal-no 1st choice votes
Others: peer in same school as 2nd evaluator
Who would be best to evaluate submissions?
![Page 77: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/77.jpg)
Evaluator/State Committee Qualifications
![Page 78: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/78.jpg)
Evaluator/State Committee Qualifications
1. Leadership Experience2. Teaching Experience3. Graduate degree in content area4. Mentor training or experience5. College degree in licensure area6. Specialized certification or training7. Recognized contributor to the field
Others: graduate degree in education, student teacher supervisor, methods instructor, educator knowledgeable in NCES, master teacher, professional recommendation
![Page 79: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/79.jpg)
Peer Review
Recommendations
Year 1
• State Committees• Application/selection
process• Training with Certificate
Year 2
• State Committees• 1 Anonymous Peer
Reviewer
Year 3
• 2 Anonymous Peer Reviewers
• State Committees when reviews do not match
![Page 80: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/80.jpg)
Creating a Submission Sharing a Submission
Selecting a single Clarifying Objective (CO)
Selecting student work that demonstrates growth
Clarifying how the CO is connected to the student work sample
Writing about your teaching context and other details of submission with clarity (i.e. use of NCES)
Preparing submissions for uploading
Professional Development Topics
![Page 81: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/81.jpg)
Extension of NCEES Training?
Others . . .
Understanding the category ratings
Content-specific examples of each category rating
How to upload samples
How to pick a representative sample
How to select student work samples for pre/post or two points in time
Professional Development Topics
![Page 82: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/82.jpg)
Dissemination of PD
Blended (70-85%) Online PD materials
with face-to-face training
Train-the-Trainer to bring back locally
Face-to-face regionally (62-82%)
Access materials posted online (52-63%)
Webinars/online only (33-59%)
Others: Online quarterly
updates County-wide f2f Mentoring with local
trainer/facilitator Differentiated based
on years of classroom experience
Options so teachers could choose what works best for them individually
![Page 83: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/83.jpg)
Submitters Evaluators
Overwhelming majority: 82%Peer with similar content experience
Almost 40%1) Coordinator/Central Office2) Regional PD Consultant
25%: Principal
< 15%: Peer with different content experience or AP
< 5%: No one
Majority: 52-63%1) Coordinator/Central Office2) Peer with similar content
experience3) Regional PD Consultant
< 15%: AP or Principal
< 10% Peer with different content
experience No one
Who would you consult with locally?
![Page 84: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/84.jpg)
Going Green – Jigsaw Approach
Individually, consider the feedback and recommendations just presented and brainstorm
ways to Go Green with the Analysis of Student Work (ASW) Future Planning
![Page 85: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/85.jpg)
Break
![Page 86: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/86.jpg)
Planning Groups Form & Prioritize for Year 1
Implementation Needs
![Page 87: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/87.jpg)
Year 1 Implementation
Date Activity
May 289 a.m.-4 p.m.
NCDPI Reflection & Planning Retreat
May 28 – June 12 Plan & Create PD materials for ASW Process for Year 1 Implementation
July 1, 2013 2013-2014 academic year begins
July 8 - 24 SI 2013 Design Studio on ASW Process
August - December Various trainings and initial use of the ASW Process with revisions based on feedback for Year 2 Implementation
January - June
July 1, 2014 2014-2015 academic year begins:Year 2 Implementation
July 1, 2015 2015-2016 academic year begins:Year 3 Implementation
![Page 88: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/88.jpg)
Work Sessions for Planning Groups
Join a group at http://bit.ly/NCDPIRetreat
1:00 – 3:45 p.m.
![Page 89: Standard 6 Pilot of the Analysis of Student Work (ASW) Process](https://reader030.fdocuments.us/reader030/viewer/2022033102/56813e7a550346895da8a107/html5/thumbnails/89.jpg)
Debrief & Next Steps