ACVP - c.ymcdn.com · PDF filecompetencies and content of current veterinary pathology...

Special Edition - September 2015

ACVPACVPAn Official Publication of The American College of Veterinary Pathologists

continued on page 2

Background and Summary of ACVP Certifying Examination Redesign Proposal

The Certifying Examination (CE) of ACVP has undergone changes in the last few years in order to align with evolving standards for professional examinations.1 The General Pathology section was separated from the remainder of the CE as a phase I test offered annually in March, with successful completion a prerequisite for eligibility to take the Phase II examination. Equating and standard setting were introduced in 2014.2 Simply put, equating is an effort to assure that test difficulty is similar from year to year, and that differences in candidate performance reflect candidate and not test variability. Standard setting was initiated to define the level of proficiency expected of an entry-level pathologist through estimating the proportion of candidates who would answer test items correctly by a panel of experts. These estimates are then translated into calculation of cut-scores (= passing scores) for each test. Equating was implemented by ACVP through inclusion of a proportion of repeated items whose performance is used to “equate” the difficulty of the test over time.3 The CE is associated with the role delineation study conducted in 2008 to assure that test items reflect competencies and content of current veterinary pathology practice.

The above changes addressed issues related to overall test reliability and validity. In addition, ACVP has addressed the performance of actual test items. For multiple choice questions (MCQ), this information is readily and automatically generated and has always been used to provide a statistical basis to allow multiple correct answers or to eliminate items from final score calculation during quality control procedures.4

In order to get some descriptive information on item performance for the other item types, “difficulty” and “discrimination” for each free text item (essays and short answers) were calculated in 2014. “Difficulty” is a measure of the proportion of candidates who answered the item correctly. “Discrimination” is a measure of candidates’ performance on a given item (correct or incorrect) and the candidates’ score on the overall test. For example, low discrimination scores suggest that overall candidate performance on the test did not relate to getting the right answer on this particular item.4

Additional test statistics such as response frequency and distribution are exceedingly difficult to derive from free text (short answer, essay) items but are readily generated from MCQ. This means that, for example, for free text items (such

as image-based short answer questions) if a majority of candidates uniformly answer something incorrectly, a large proportion will get a “wrong” answer; however, it will not be apparent whether the wrong answers were homogeneous and the image possibly misleading, or if there was a diversity of “wrong” answers suggesting it was a difficult but fair item. Therefore, since short answers and essays are graded simply as “right” or “wrong,” it is impractical to generate the same statistics that are used for MCQs to assess the nature of the wrong answers, except, in the case of narrative responses, through the visual inspection of responses to determine if there are problematic patterns. Calculation of difficulty and discrimination retrospectively for past exams revealed that both could be highly variable for free text sections.

Testing higher levels of competency is thought to be more straightforward with essay type questions that ask candidates to integrate recognition and interpretation of lesions and laboratory data, and to identify additional investigative options.5,6 MCQ offer great advantages from a test assessment perspective, and importantly, MCQ can be constructed that test beyond factual memory and also assess whether candidates have achieved higher levels of learning such as ability to apply, analyze, and synthesize the knowledge required of entry-level pathologists.7 ACVP intends to retain both kinds of test items in the redesigned CE.

Overall structure of the current phase II CE is complex, amenable to limited quality control of item performance, expensive, and involves more than 7,000 volunteer hours per year. For example, the Anatomic Pathology (AP) phase II test comprises 3 sections with choice of 4 among 5 sub-sections in “Veterinary Pathology”. With changes implemented in 2014, for the “Gross Pathology” section, about 10,000, and for the “Microscopic Pathology” section about 2,000 free text responses are manually entered into a database. Then, difficulty and discrimination are calculated, candidate scores are weighted per section, the equating factor is applied, and a final score is determined.

All data then have to be related to R1, R2, and R3 candidate status and a pass/fail decision has to be made. The Clinical Pathology (CP) phase II test consists of 8 different subsections, therefore quality control is more complex but less laborious due to fewer candidates. These quality control processes, while essential by standards for certification, have become

ACVP Newsletter • Special Edition - September 2015

2

exceedingly complex, time consuming, and expensive.1 Since a fraction of items with free text answers have the potential to lack good difficulty and discrimination characteristics, and can pose enormous quality control challenges, we propose to greatly reduce that type of item.8

As a result of thorough investigation of testing principles in light of the current CE format, redesign has evolved. The proposed redesigned CE proposes to simplify the exam structure by increasing the proportion of MCQ, eliminating short answer response questions, retaining comprehensive essay type questions while enabling keyboard entry of answers, and retaining CE scope and increasing rigor. Interpretation of glass slides and laboratory data with essay-type answers will be retained, therefore necessitating use of testing centers that can accommodate computers and microscopes in a suitable environment. The test facility of the American Board of Pathology (ABP) in Tampa, FL, was identified as the most suitable location for administration of the revised CE. The test facility is very close to the airport, and lodging accommodations are available in the same building as the testing center. The needs of ACVP with regard to item bank, microscopes, slide archive, and professional support staff are very similar to those of ABP. Hence, the overall goals of the redesign are to:

1. Use computer terminals to answer MCQ and enter free text interpretations of glass slides or data sets.

2. Reduce the overall testing time from approximately 12.5 hours over two days to 8.5 hours over one day with two break periods.

3. Convert the Phase II CE (Clinical Pathology and Anatomic Pathology) to be graded as a single pass/fail examination. In other words, the Phase II CE will no longer be divided into subsections that can be passed independently over several years.

4. Reduce the number of glass slides, and transfer more image-based “recognize and interpret” type of questions to MC format.

5. Implement greater focus on “core” material, and eliminate subsections in “Veterinary Pathology.”

6. Establish and utilize a database of items characterized over multiple years and aligned with role delineation study topics.

Exact distribution of item types, definition of “core”, procedures for generating and choosing items and other details are being

finalized in consultation with members of the EC and Council, and will be solidified and publicized following final decision regarding redesign. Other aspects of the CE that may be assessed in the future are automated essay scoring, assessing item performance prior to scoring, use of whole slide images and digital pathology, and closeness of alignment with a role delineation study to be repeated in the next few years.9

REFERENCES

1. National Commission for Certifying Agencies (NCCA) Standards for the Accreditation of Certification, Institute for Credentialing Excellence (ICE), 2005. (Update to be released for 2016).

2. Kolen MJ, Brennan RL. Test Equating, Scaling, and Linking. Methods and Practices, 2004.

3. Guilera G, Gómez J. Item response theory test equating in health sciences education. Adv Health Sci Educ Theory Pract. 2008 Mar;13(1):3-10.

4. http://www.washington.edu/oea/services/scanning_scoring/scoring/item_analysis.html; Understanding item analysis reports, Office of Educational Assessments, University of Washington, accessed Aug. 26, 2015.

5. Anderson, LW, et al. (2001). A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s Taxonomy of Educational Objectives. New York: Pearson, Allyn & Bacon.

6. Bjork EL, Soderstrom NC, Little JL. Can Multiple-Choice Testing Induce Desirable Difficulties? Evidence from the Laboratory and the Classroom. Am J Psychol. 2015 Summer;128(2):229-39.

7. Freiwald T, Salimi M, Khaljani E, Harendza S. Pattern recognition as a concept for multiple-choice questions in a national licensing exam. BMC Med Educ. 2014 Nov 14;14:232

8. Ablemann CH, Elmore RF, Even J, Kenyon S, Marshall J. (2004). When accountability knocks, will anyone answer? In Richard F. Elmore (Ed.) School Reform from the Inside Out, 133-199. Cambridge, MA: Harvard Education Press.

9. Gierl MJ, Latifi S, Lai H, Boulais AP, De Champlain A. Automated essay scoring and the future of educational assessment in medical education. Med Educ. 2014 Oct;48(10):950-62.

Continued from Page 1


3

Anatomic Pathology CE Redesign with Examples

1

2

4



5



6



7



8


ACVP Board Certification Exam Redesign Proposal: Frequently Asked Questions

Are psychometricians and a desire to pattern our exam after those in other specialties driving the exam redesign proposal?

No. It has long been recognized that our examination structure is complex as a result of many sections, subsections, and question formats. The amount of data generated to support the equating process strains our capability to manage and maintain good quality assurance (QA).

The Certification Examination Board (CEB), which includes former Examination Committee (EC) members and ACVP Diplomates with no EC experience, has been carefully deliberating about the structure of the examination for 4 years. The current proposal to redesign the Phase II Certifying Examinations in Anatomic and Clinical Pathology is the culmination of numerous thoughtful discussions regarding maintaining the rigor of the examinations, assuring that the examinations are fair and relevant to practice of the profession, and improving compliance with current recommended standards for excellence in professional certification. These discussions have included guidance from psychometricians with experience administering examinations similar to ours, and this input has been very helpful.

Why is there a proposed one-day exam for Phase II? Does this reduce the breadth of content tested?

The proposed 1-day exam is a reduction from 12.5 hours to 8-9 hours of exam contact. This is achievable by reducing handwriting of lengthy essays and by going from 3 question formats (essay, short answer, multiple choice) to 2 question formats (essay, multiple choice). There will be less emphasis on descriptions and greater emphasis on interpretation of glass slides, images, and data. The breadth and depth of exam content is not intended to decrease and may increase by moving more material to pattern recognition, multiple choice question (MCQ) format.

While not a primary consideration, there is also likely a considerable cost saving for candidates and the ACVP by occupying the testing facility for 1 day rather than 2 days.

Will consolidation of Veterinary Pathology modules result in an overwhelming breadth of knowledge in preparation for the exam?

It is proposed that this concern be addressed by defining the proportion of questions related to species’ groups, with continued exam emphasis on companion and production animal pathology. The Examination Committee is in the process of refining guiding criteria for selection of core, entry-level pathologist material for use in test item selection.

In addition, it is proposed that guidelines be developed for preparation related to non-domestic species, and that these are limited in number and scope.

Why is there a proposed change from short answer responses to a MCQ format for some of the images? Does this make the exam too easy?

The rationale for moving some pattern recognition images to MCQ format includes:

• The current number of hand-written short answers requires immense manual grading time and massive data entry with associated QA effort. This is required for equating and item analysis for building a content database. Equating examinations year to year is essential to assure similar difficulty, but is made less manageable with the current large proportion of short answer responses.

• For some questions, short answer responses can be difficult to grade objectively because of unanticipated answers and interpretations. Alternative phraseology can also contribute to grading challenges.

• MCQ format encourages the candidate to answer a focused question about a specific piece of knowledge intended by the Examination Committee.

• The manual procedure to calculate “Difficulty” and “Discrimination” for short answer responses identifies a poorly performing question, but does NOT provide information regarding why the question performed poorly or a basis for accepting multiple correct answers.

• Automated grading of MCQ provides nearly instantaneous score results. More importantly, it provides an automated statistical question performance analysis. It more readily permits identification of poor performing questions for review. The data for each question, in the form of foil analysis, may also be diagnostic for why some questions performed poorly.

• There is ample opportunity in the retained “essay” narrative portion of the exam to subjectively test a candidate’s ability to synthesize knowledge application. Such test items may incorporate glass slides plus gross images and/or laboratory data, and therefore more comprehensively and realistically test a candidate’s ability and performance.

• The range of item difficulty in testing knowledge and pattern recognition is not expected to change.

9


ACVP Board Certification Exam Redesign Proposal: Frequently Asked Questions

Is it true that evaluation of glass slides and complex biochemistry data will be converted to MCQ format testing?

Essays for describing and interpreting glass slides or data are still present in the proposed redesign, but the quantity would be reduced.

The pass rate on the Gross Pathology section of the anatomic pathology exam has traditionally been higher than other exam sections. Will the shift toward the MCQ format make this section easier and inflate the pass rate?

The pass rate in 2014 for the Gross Pathology section was 72.6%. The scores might improve with MCQ format, but standard setting and equating processes can function to adjust the pass point using best practices to help maintain more uniform examination difficulty year to year, thereby modulating concern about the format being “too easy”. The MCQ format still tests the knowledge to correctly identify imagery questions that would be presented on a flat screen display terminal. The MCQ format would be predominantly used for testing pattern recognition in the practice of pathology. More importantly, we need a manageable procedure to evaluate the validity of test item performance.

Why are we moving to single pass/fail Phase II Certifying Examination for Clinical Pathology and Anatomic Pathology?

We are moving to a single pass/fail Phase II exam because we believe with this format a candidate’s ability to integrate information and data from different sources (e.g. for anatomic pathology—histopathology, macroscopic pathology, and clinical pathology) is better assessed. Additionally, a single exam has more data points for use in the standard setting and equating processes compared to shorter multiple section exams and subsection examinations. In effect, numerous fragmented equating exercises can be condensed into a single equating process.

Won’t candidates now be able to pass the Phase II CE by doing well on the MCQ’s while being deficient in their ability to describe microscopic preparations?

We don’t think so for a several reasons: 1) high quality MCQ’s will supplement the written descriptions in evaluation of the candidates’ ability to interpret microscopic preparations/data sets; and 2) the written descriptions will be weighted appropriately relative to the MCQ’s. There is the potential to increase the testing of microscopic diagnostic capability by moving microscopy pattern recognition into the MCQ visual exam; this can be done in less time by not requiring lengthy written descriptions. In addition, there will be continuous evaluation of CE performance allowing us to make changes if we believe there are issues with any component of the examination.

Is implementation of the proposed exam format by 2017 moving too fast?

Converting to the proposed exam format in 2017 is both doable and desirable for the following reasons: • The proposed format should have minimal impact on training

to be a pathologist and on preparation for the exam. The overall subject matter is unchanged, although anatomic candidates will no longer be able to exclude literature-based questions devoted to a class of species.

• Two years provides ample opportunity to practice mock exams based upon sample questions formatted to the proposed changes in the exam.

• This time frame permits the organization to more readily realize significant efficiencies in implementation of technology, exam preparation database advantages, test item analysis improvements, and major cost savings.

• Two years to transition allows for continuity of vision and minimizes diluted efforts resulting from lost momentum.

• There are additional implementation details to resolve, but these will not be addressed until there is organizational approval of the proposal.

What microscopes are available at the American Board of Pathology (APB) testing facility in Tampa, FL?

The ABP testing facility has individual carrels equipped with recent model flat-screen monitors, computers, and Nikon Alphaphot 2 microscopes. The microscopes have a 10x wide field ocular, 4, 10 and 40x dry objectives, and a 100x oil objective. The microscopes are cleaned after the Clinical Pathology examination.

Can I bring my own microscope?

Yes, the carrels can accommodate an additional microscope.

Will there be time to set up my microscope?

Yes, there will be time to set up individual microscopes before the exam starts.

ACVP - c.ymcdn.com · PDF filecompetencies and content of current veterinary pathology...

Documents

Transcript of ACVP - c.ymcdn.com · PDF filecompetencies and content of current veterinary pathology...