Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation...
Transcript of Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation...
![Page 1: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/1.jpg)
Orientation Assessments: New and Revised Resources
for the CO Continuum
February 26, 2015
Sanja Bebic and Julie Sugarman
Center for Applied Linguistics
![Page 2: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/2.jpg)
CO Design and Delivery Building Blocs
Overseas and R&P Objectives and Indicators
Welcome set
R&P curriculum
Assessment toolkit
ToT guide and video
2
![Page 3: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/3.jpg)
Online Orientation Assessment Toolkit
3
http://www.culturalorientation.net/
providing-orientation/toolkit/
orientation-assessment-toolkit
![Page 4: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/4.jpg)
Online Orientation Assessment Toolkit
4
![Page 5: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/5.jpg)
Assessment Pilot
Feedback sent by CO focal points; analysis of assessment modifications
Analysis of a sample of assessments given to refugees− 255 assessments from 12 affiliates (7 agencies)
− Sample somewhat over-represented Iraqis, principal applicants, and refugees with U.S. ties
− Individual responses were entered electronically and rescored
− Assessors provided feedback on challenges
5
![Page 6: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/6.jpg)
Average Score of Pilot Participants
6
58%
84%
![Page 7: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/7.jpg)
Question by Question Analysis
Score (full/partial or no credit)− Questions that were too easy or too hard
− Participants didn’t answer the question the way that was intended
Difference in score for different populations
Quality of scoring− Scoring/prompting as intended based on guidance
− Scoring as intended based on indicator
Problems reported by assessors on cover sheet
7
![Page 8: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/8.jpg)
Question by Question Results
8
![Page 9: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/9.jpg)
Assessment Pilot Use
Suggestions for increasing the validity of assessments− Online Assessment Toolkit guidance
− November 2014 webinar
Revision of individual questions and scoring guidance on the Model CO Assessment
Development of written version of Model CO Assessment
9
![Page 10: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/10.jpg)
What’s New with the Model Assessment
Replaced picture-based questions
Revisions based on feedback
Added scoring guidance
Available as MS Word or PDF
10
![Page 11: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/11.jpg)
Revised Model Assessment
11
![Page 12: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/12.jpg)
Revised Model Assessment
12
![Page 13: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/13.jpg)
Revised Model Assessment
13
![Page 14: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/14.jpg)
Revised Model Assessment
14
![Page 15: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/15.jpg)
Revised Model Assessment
15
![Page 16: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/16.jpg)
Revised Model Assessment
16
![Page 17: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/17.jpg)
Revised Model Assessment
17
![Page 18: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/18.jpg)
Revised Model Assessment
18
![Page 19: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/19.jpg)
Revised Model Assessment
19
![Page 20: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/20.jpg)
Revised Model Assessment
20
![Page 21: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/21.jpg)
Revised Model Assessment
21
![Page 22: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/22.jpg)
Written Version of Model Assessment
22
![Page 23: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/23.jpg)
Written Version of Model Assessment
23
![Page 24: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/24.jpg)
Written Version of Model Assessment
24
![Page 25: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/25.jpg)
Written Version of Model Assessment
25
![Page 26: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/26.jpg)
Guidelines for Use of the Written Version
The assessment can be administered in a group, but each individual should fill out her or his own paper
The written Model CO Assessment may be given to refugees who meet the following criteria:− Are literate and have good language comprehension skills in the
language of the assessment
− Have at least some formal secondary education in any country
− Have demonstrated the ability to fill out forms or complete written work independently (excluding assistance with translation to English)
See p. 2 of guidelines for scoring
26
![Page 27: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/27.jpg)
Overseas CO Multiple Choice Question Bank
27
![Page 28: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/28.jpg)
Overseas CO Multiple Choice Question Bank
Questions based on − Overseas Objectives and Indicators
− Existing overseas assessments
− Refugee misconception survey
Organized into topics (travel, role of the resettlement agency, etc.)
Total of 35 questions− May use instead of/in addition to questions currently used on
assessments
− Select questions that align with content taught in CO
28
![Page 29: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/29.jpg)
Multiple Choice Questions - Examples
29
![Page 30: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/30.jpg)
Multiple Choice Questions - Guidance
We are not testing a participant’s ability to take written tests, so we want to do as much as possible to make the questions easy to answer and to avoid tricks or distractions in the way the questions are organized and worded.
Having well-designed questions gives you confidence that people who get them right really know the answer and people who get them wrong really do not.
30
![Page 31: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/31.jpg)
Multiple Choice Questions - Guidance
Guidance topics− Directions
− Length
− Customization
− Order of questions and answer choices
− Selection of questions
− Question types (e.g., true/false, select all)
− Question wording
31
![Page 32: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/32.jpg)
Multiple Choice Questions - Guidance
Key suggestions− You can customize for your population, but you can only
compare scores using the same assessment
− Start with one or two easy questions
− Be sure that questions in the A, B, C, or D spot show up relatively randomly
− Limit the use of true/false or yes/no questions
− Avoid tricky questions (all of the above, negatively phrased, select all that apply)
− The correct answer should not stand out because of how it is worded
− Answer choices should be plausible
32
![Page 33: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/33.jpg)
Assessment Plan Template and Checklist
33
![Page 34: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/34.jpg)
Assessment Plan Template
34
![Page 35: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/35.jpg)
Assessment Plan Checklist
35
![Page 36: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/36.jpg)
Assessment Plan
Topics− Logistics (who, when, where, how to give assessment)
− Follow-up with participants based on performance
− Feedback loop on assessment procedures
− Quality control
− Training
− Uses of data
− New/modified questions
− Scoring guidelines
− Data entry, analysis, and sharing
36
![Page 37: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/37.jpg)
Other Tools
Still in the Online Assessment Toolkit− Overseas and domestic objectives and indicators
− Questions and Answers about the Model CO Assessment (slightly revised)
− Frequently Asked Questions About Assessment and Evaluation
− February 2014 and November 2014 webinars
Still to come!− CO Assessment training module
37
![Page 38: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/38.jpg)
Assessment Training Module
38
![Page 39: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/39.jpg)
Assessment Training Module
39
![Page 40: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/40.jpg)
Online Assessment Toolkit
40
![Page 41: Orientation Assessments: New and Revised Resources for the ...Assessment… · Online Orientation Assessment Toolkit 4. Assessment Pilot Feedback sent by CO focal points; analysis](https://reader033.fdocuments.us/reader033/viewer/2022060519/604dc55abc5af010026330a3/html5/thumbnails/41.jpg)
CAL Resources
41
www.culturalorientation.net