A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS Dale Curry, Kent...
-
Upload
rafe-parker -
Category
Documents
-
view
214 -
download
0
Transcript of A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS Dale Curry, Kent...
A NATIONAL A NATIONAL CERTIFICATION EXAM CERTIFICATION EXAM
FOR CHILD AND YOUTH FOR CHILD AND YOUTH CARE WORKERS: CARE WORKERS:
PRELIMARY RESULTSPRELIMARY RESULTS
Dale Curry, Kent State UniversityDale Curry, Kent State University
Basil Qaqish, University of North Basil Qaqish, University of North Carolina-GreensboroCarolina-Greensboro
Organization of CompetenciesOrganization of Competencies
DomainsDomains
Sub-DomainsSub-Domains
CategoriesCategories
CompetenciesCompetencies
The NACP Competencies
DomainsDomains
I. I. ProfessionalismProfessionalism
II. II. Cultural & Human DiversityCultural & Human Diversity
III. III. Applied Human DevelopmentApplied Human Development
IV.IV. Relationship & CommunicationRelationship & Communication
V. V. Developmental Practice Methods Developmental Practice Methods
Example
from NACP Competencies Document
I. PROFESSIONALISM 6. Advocacy – MEDIUM WEIGHT a. demonstrate knowledge and skills in use of advocacy - SUPERVISOR ASSESSMENT b. access information on the rights of children, youth and families - PORTFOLIO c. describe the rights of children youth and families in relevant setting/s and systems – EXAMINATION d. advocate for the rights of children, youth, and families in relevant setting/s and systems - PORTFOLIO e. describe safeguards for protection from abuse including institutional abuse - EXAMINATION f. advocate for safeguards for protection from abuse including institutional abuse - SUPERVISOR ASSESSMENT g. ensure that children, youth and family views are heard and considered during the decision making processes which directly affect them -EXAMINATION
Situational Judgment ItemsSituational Judgment Items
Can be used to place competencies in a Can be used to place competencies in a context of relationships, events, and context of relationships, events, and settingssettings
Can help establish face validity of test if Can help establish face validity of test if they are based on real case studies they are based on real case studies collected from the fieldcollected from the field
Moves beyond the simple recitation of Moves beyond the simple recitation of learned knowledgelearned knowledge
Situational Judgment ItemsSituational Judgment Items
Case studies of real incidents were Case studies of real incidents were collected from child and youth care collected from child and youth care practitioners in a variety of practice practitioners in a variety of practice settingssettings
Used as the basis for the exam questions.Used as the basis for the exam questions.
Example ofSituational Judgment Item
Example of Situational Judgment Item
Competencies assessed by examination:
I.6.c. Describe the rights of children youth and families in relevant setting/s and systems
I.6.e. Describe safeguards for protection from abuse including institutional abuse
Item 8: Read this case study of an incident and answer the questions at the end:
Lisa (a girl in care) became angry during individual counseling session with Social Service Manager (MH) and cursed at the staff member. Lisa was escorted to “off dorm” time out due to disrupting milieu. While in time out, Lisa continued to use profanity toward staff members, urinated on the floor, and said “maybe I should pimp again and make some money…
After giving her several expectations and after Lisa refused to calm down, staff member (JT) began to physically escort Lisa to seclusion when Lisa became physically assaultive, hitting staff member. Staff members JT and SR placed Lisa in a therapeutic hold and eventually removed her to seclusion room. By 8:15 P.M., Lisa regained physical control of herself and agreed to a plan to maintain the safety of Lisa and the staff members. Lisa cleaned seclusion room and time out room.
Which of the following statements are true? [Circle True or False for each statement]
T/F Lisa’s rights were properly protected in
this situation.
T/F The physical restraint, as explained in
this case study was justified and proper.
T/F An alternative intervention such as
having the staff member (JT) switch off
with another staff member could have
been more effective.
Pilot TestingPilot Testing
STUDY GOALSSTUDY GOALS
Administer exam and get feedback Administer exam and get feedback regarding face validity & ways to improveregarding face validity & ways to improve
Conduct item analysisConduct item analysis
Examine relationship between test scores Examine relationship between test scores and supervisory assessment of worker and supervisory assessment of worker performance (concurrent validity)performance (concurrent validity)
Explore differential performance resultsExplore differential performance results
Determine a cut score (pass/fail)Determine a cut score (pass/fail)
METHODMETHOD
Assessment Work Group developed 100 Assessment Work Group developed 100 item situational judgment exam.item situational judgment exam.
Exam was administered to 775 child and Exam was administered to 775 child and youth care workers in 29 sites.youth care workers in 29 sites.
Sites located in six states & two provinces.Sites located in six states & two provinces.
METHODMETHOD
Scores were statistically compared with Scores were statistically compared with supervisor’s rating of competence.supervisor’s rating of competence.
Supervisor ratings of worker competence Supervisor ratings of worker competence (six items) administered.(six items) administered.
Examinee feedback on exam and face Examinee feedback on exam and face validity solicited through questionnairevalidity solicited through questionnaire
PROCEDURESPROCEDURES
Item analysisItem analysis
Correlation of exam total score with Correlation of exam total score with supervisor composite ratings supervisor composite ratings
Modified Angoff procedure to determine Modified Angoff procedure to determine cut point (pass/fail)cut point (pass/fail)
ITEM ANALYSISITEM ANALYSIS
Reliability analysisReliability analysis
Difficulty analysisDifficulty analysis
Discrimination analysisDiscrimination analysis
Distracter analysisDistracter analysis
Differential Item Analysis (DIF)Differential Item Analysis (DIF)
RELIABILITY ANALYSISRELIABILITY ANALYSIS
Excellent internal reliability (Chronbach’s Excellent internal reliability (Chronbach’s alpha=.90)alpha=.90)
18 items identified for possible deletion or 18 items identified for possible deletion or revisionrevision
DIFFICULTY ANALYSISDIFFICULTY ANALYSIS
Difficulty scores obtained for each item Difficulty scores obtained for each item (acceptable difficulty score range (acceptable difficulty score range from .10-.80)from .10-.80)
Also examined top and bottom 20%Also examined top and bottom 20%
Most items were of moderate difficultyMost items were of moderate difficulty
DISCRIMINATION ANALYSISDISCRIMINATION ANALYSIS
Point Bi-serial correlation between the Point Bi-serial correlation between the item and total test scoreitem and total test score
Also compared top 20% with bottom 20%Also compared top 20% with bottom 20%
.15 minimum discrimination score.15 minimum discrimination score
DISTRACTER ANALYSISDISTRACTER ANALYSIS
0.00
0.20
0.40
0.60
0.80
1.00
upper2nd3rd4thlower
Correct answer=1
0.00
0.20
0.40
0.60
0.80
1.00
1234other
Figure 1
DIFFERENTIAL ITEM DIFFERENTIAL ITEM FUNCTIONING (DIF)FUNCTIONING (DIF)
Exploring differences for race and gender Exploring differences for race and gender for those with similar abilities on the testfor those with similar abilities on the test
Use of computer program SIBTEST and Use of computer program SIBTEST and Mantel Haenszel procedureMantel Haenszel procedure
7 items identified by either SIBTEST or 7 items identified by either SIBTEST or Mantel Haenszel for race and 1 item for Mantel Haenszel for race and 1 item for gender (must give these items more gender (must give these items more scrutiny)scrutiny)
FACE VALIDITY AND OTHER FACE VALIDITY AND OTHER FEEDBACK FROM EXAMINEESFEEDBACK FROM EXAMINEES
90% indicate that the items in the exam 90% indicate that the items in the exam accurately assess important aspects of child and accurately assess important aspects of child and youth care workyouth care work
90% indicate that the case studies provide 90% indicate that the case studies provide realistic samples of child and youth care workrealistic samples of child and youth care work
80% indicate that the exam content is similar to 80% indicate that the exam content is similar to their job dutiestheir job duties
59% indicate that their performance on the exam 59% indicate that their performance on the exam is an excellent indicator of their job performance is an excellent indicator of their job performance (34% neither agreed or disagreed)(34% neither agreed or disagreed)
OTHER FEEDBACKOTHER FEEDBACK
Test is consistent with real situations-Test is consistent with real situations-overall a good and fair testoverall a good and fair test
Motivating and thought provokingMotivating and thought provoking
Test too long – perhaps give a break or do Test too long – perhaps give a break or do it in two partsit in two parts
Not familiar with some of the terminologyNot familiar with some of the terminology
Some of the case studies were hard to Some of the case studies were hard to followfollow
CANADIAN RESULTSCANADIAN RESULTS
Mean total exam score - Mean total exam score - 75.775.7 (U.S. 59.3) (U.S. 59.3)
Mean competency rating (composite of 6 Mean competency rating (composite of 6 items, 5 point scale) – items, 5 point scale) – 28.4 28.4 (U.S. 26.2)(U.S. 26.2)
Mean face validity rating (composite of 4 Mean face validity rating (composite of 4 items, 5 point scale) – items, 5 point scale) – 15.415.4 (no significant (no significant difference with U.S. examinees)difference with U.S. examinees)
COMMENTS CANADIAN COMMENTS CANADIAN EXAMINEESEXAMINEES
Exam was too long, a break was neededExam was too long, a break was neededWell-thought questions with very realistic, Well-thought questions with very realistic, engaging scenarios but a mix in style of engaging scenarios but a mix in style of question type (e.g. true/false) may have question type (e.g. true/false) may have been a welcomebeen a welcomeThe exam did not reflect the cultural The exam did not reflect the cultural aspects of British Columbiaaspects of British ColumbiaKnew about N.A. Code but only familiar Knew about N.A. Code but only familiar with Ontario Code of Ethicswith Ontario Code of Ethics
POST PILOT ACTIVITIESPOST PILOT ACTIVITIES
Explored items for possible deletion to Explored items for possible deletion to improve reliability (18 items)improve reliability (18 items)Examined content of items identified by Examined content of items identified by DIF for gender/racial biasDIF for gender/racial biasExamined examinee feedback pertaining Examined examinee feedback pertaining to each item and regarding the test in to each item and regarding the test in generalgeneralCompleted process of setting cut-off Completed process of setting cut-off (passing scores) for exam(passing scores) for exam
What remains to be done?What remains to be done?
Continue to collect and analyze data to Continue to collect and analyze data to improve the exam and accumulate improve the exam and accumulate evidence of the exam’s validityevidence of the exam’s validity
Continue to develop and research new Continue to develop and research new exam itemsexam items
Continue to explore reliability and validity Continue to explore reliability and validity of exam in future as it is implementedof exam in future as it is implemented
More InformationMore Information
North American Certification ProjectNorth American Certification Project
www.acycp.orgwww.acycp.org