7/29/2019 Is the Evaluation of the Personal Statement..(1)

1/4

ORIGINAL REPORTS

Is the Evaluation of the Personal Statement aReliable Component of the General Surgery

Residency Application?

Bobbie Ann Adair White, MA,* Mark Sadoski, PhD, Scott Thomas, MD, andMohsen Shabahang, PhD

*The Office of Faculty Development, Texas A&M Health Science Center, College of Medicine, Temple,Texas; The Office of Educational Development, Texas A&M Health Science Center, College of Medicine,Bryan, Texas; Department of General Surgery, Scott & White Healthcare, Temple, Texas; Department ofGeneral Surgery, Geisinger Medical Center, Danville, Pennsylvania

BACKGROUND: Each year, fourth-year medical studentsspend considerable time writing and rewriting their personalstatements. However, there is little evidence of what role thepersonal statement plays in deciding which applicants will beinvited for an interview.

OBJECTIVE: To evaluate theinter-rater reliability of a surgicalselection committees ratings for both the personal statementand the application summary parts of the residency application.

DESIGN: We completed a retrospective analysis of the 2007

2008 Scott & White surgical residency application pool. Froma total pool of 174 residency applications, we selected 8 (5%)applications randomly to be evaluated by 4 experienced mem-bers of the selection committee. The 4 committee membersrated each personal statement on a 7-point scale, from nega-tivewould not invite for an interview to positivewill in-vite for an interview. They rated respective application sum-maries separately on a similar 7-point scale. Committeemembers also listed their top three reasons for assigning theirscores.

METHODS: Rating scores for the personal statements and theapplications were analyzed for inter-rater correlation. The qual-

itative data (ie, reasons for the scores) were reviewed to help theinvestigators profile the reasons given for very positive and verynegative scores.

RESULTS: For the application summaries, the correlations be-tween each pair of raters ranged from 0.79 to 0.94 with anoverall average of 0.88. For the personal statements, inter-rater

correlations ranged from0.83 to 0.63 with an overall averageof0.09.

CONCLUSION: These results demonstrate that the personalstatements lacked objective criteria for evaluation. (J Surg 69:340-343. 2012 Association of Program Directors in Surgery.Published by Elsevier Inc. All rights reserved.)

KEY WORDS: residency application, match, personal state-ment, application criteria

COMPETENCIES: Interpersonal and Communication Skills,

Professionalism, Systems Based Practice

INTRODUCTION

Each year, fourth-year medical students spend considerabletime writing, rewriting, and seeking editorial advice about thecontent of their personal statements. Students believe that thequality of the personal statement may determine their chancesfor a residency interview. Many medical schools provide a list ofdos and donts, and some distribute guidelines for writingthe personal statement. But the personal statement is ultimatelypersonal, and evaluators are faced with varied styles, format,

and content.It is difficult to measure objectively what makes a personal

statement a good one, although there have been attempts inthe literature to characterize it more objectively.1 Also poorlyunderstood is what role thepersonal statement plays in deciding

which applicants will be invited for an interview. The NationalResident Matching Program (NRMP) attempted to capture therelative contributions of application components through their2010 NRMP Program Director Survey2 (for additional detailsabout their survey, see http://www.nrmp.org).

Sixty-eight percent of responding directors cited the personalstatement was a factor in interview selection. This percentage is

higher than that of grades in required clerkships (62%), the

Correspondence: Inquiries to Bobbie Ann Adair White, MA, The Office of Faculty Devel-

opment, Texas A&MHealthScienceCenter,Collegeof Medicine,2401 South 31stStreet,

Temple, TX 76508; fax: (254) 724-6810; e-mail: bawhite@medicine.tamhsc.edu

Presentedin partat the Association of AmericanMedical Colleges,Joint Student Affairs

and Careers in Medicine Professional Development Conference, June 11, 2011.

Journal of Surgical Education 2012 Association of Program Directors in Surgery 1931-7204/$30.00Published by Elsevier Inc. All rights reserved. doi:10.1016/j.jsurg.2011.12.003

340

http://www.nrmp.org/mailto:bawhite@medicine.tamhsc.edumailto:bawhite@medicine.tamhsc.eduhttp://www.nrmp.org/

7/29/2019 Is the Evaluation of the Personal Statement..(1)

2/4

deans letter (60%), and class ranking (59%). The only appli-cation components that had a higher percentage were letters ofrecommendation in the specialty (71%) and the United StatesMedical Licensing Examination (USMLE) step 1 score (73%).The survey also attempted to quantify the importance of thefactors used in ranking applicants using a 5-point scale. On that

scale, personal statements fell at 3.4, whereas interactions withfaculty and interpersonal skills were of the highest at 4.7.

Although all specialties were surveyed in the NRMP 2010Program Director Survey, for our purposes we will review onlygeneral surgery data. In all, 402 surveys were sent with a 36.8%response rate. Sixty-five percent of the 148 general surgery pro-grams completing the survey indicated that personal statements

were a factor in selecting applicants to interview. The impor-tance of the personal statement in ranking an applicant aver-aged 3.3 on a 5-point scale.

However, in 1998, Joseph T. Crane et al. completed a similarsurvey specifically for emergency medicine resident selection

that had a 79.7% return. They found that personal statementswere the least important factor in resident selection with anaverage of 2.75 on a 5-point scale. A study by Taylor et al. 3

found that personal statements were the last factor consideredfor interview invitations by directors of obstetrics and gynecol-ogy, whereas it was the second most important factor for familypractice residency directors. Similarly, they found that obstet-rics and gynecology directors ranked the personal statementlast, whereas family medicine directors ranked it third for mak-ing decisions about ranking.3 Because of the mixed results thepersonal statement seems to have as a factor in the interviewselection process, we thought we would attempt to get a closer

look at what a selection committee found to be appealing.

METHODS

This study used qualitative and quantitative methods. A total of174 applications were submitted to the surgical residency pro-gram of Scott & White Memorial Hospital in Temple, Texasfor the 2007-2008 academic year. The applications were di-vided into 2 parts: an application summary and the personalstatement. The application summary included a summary ofobjective data (college and medical school attended, scores onUSMLE I and II, class rank, surgery grade, publications, appli-

cant summary/accomplishments found on curriculum vitae,and hobbies), whereas the second part solely consisted of the

students personal statement. Each part was given a differentidentification number known only to the research coordinator.The application summary was given an 8-digit number,

whereas the personal statement was given a number that rangedfrom 1 to 174 to avoid the possibility of raters connecting the 2parts of the application. After deidentification and number as-

signment, 8 application summaries with corresponding per-sonal statements were pulled randomly from the stack of appli-cations by the research coordinator. Although the application

was split in 2 parts and deidentified, each rater received all 8summaries and all 8 personal statements so that later we couldcompare the ratings of the personal statement with that of theapplication summary.

To assess inter-rater reliability, all 4 raters reviewed the com-plete applications in two parts. Each rater was experienced withreviewing applications during interview season. Using 7-pointrating scales (Appendices A and B), each rater rated all theapplication summaries and then rated all the personal state-

ments. Additionally, we asked the raters to specify their top 3reasons for rating the statement or summary sheet the way theydid (see Appendix A). Inter-rater reliability was calculated todetermine consistency across graders, and we computed thedegree of correlation between ratings for the personal state-ments and the application summaries. This study was approvedby Scott & White Healthcares Institutional Review Board.

RESULTS

We correlated the ratings of all possible pairs of raters. For theapplication summaries, the correlations between each pair of

raters ranged from 0.79 to 0.94 with an overall average of 0.88,indicating a high degree of agreement between raters on what isvalued in an applicant or application summary. All these indi-vidual correlations were statistically significant (p 0.05). Forthe personal statements, inter-rater reliability had a range of0.83 to 0.63 with an overall average of0.09 (see Table 1 fora full listing of the personal statement scores). None of theseindividual correlations was statistically significant except one(p 0.05). These results demonstrated that common criteriafor evaluation of the personal statement was lacking.

The qualitative data regarding the personal statements re-flected the inconsistency found in the correlations of the rat-

ings. A single personal statement could receive 4 completelycontradictory comments from separate raters:

TABLE 1. Ratings Listed for Each Personal Statement Provided by the Selection Committee Members (R1R4)*

Faculty and Resident Raters

Personal Statement Scores

PS 1 PS 2 PS 3 PS 4 PS 5 PS 6 PS 7 PS 8

R 1 4 6 3 7 1 4 4 1

R 2 6 6 3 2 2 7 6 3

R 3 4 5 6 6 6 2 2 5

R 4 6 5 4 3 3 3 6 2

*The ratings were based on the scale in Appendix B.

Journal of Surgical Education Volume 69/Number 3 May/June 2012 341

7/29/2019 Is the Evaluation of the Personal Statement..(1)

3/4

1. Mentions CV essentially . . . explaining away some of hisproblems (rating 1)

2. Uses context of surgery to say a lot about themselves (rat-ing 7)

3. No ending . . . hard to get past failure of Step 2; like thepassion, but worry about making right decisions and prior-

ities (rating 3)4. Service to others correlates well to surgical career, Interest-

ing storywanted to meet the candidate . . . poor step 2,prelim, but good explanation (rating 6).

Although it does not seem possible, these comments werereferring to the same statement, where the student explains

what happened with their step 2 score.Another example of comments from separate raters is as

follows:

1. Humble, good quote, succinct . . . (rating 7)2. Too short, uses a family example that is not dramatic, clich

(rating 2)3. Would have given a 7 if grammar error . . . was not

there . . . sounds like someone I would want to train (rating 6)

4. Short, does not say much, does not explain/give exam-ples . . . (rating 3).

The personal statement is described as too short with arating of 2 and then as succinct with a rating of 7, whichmakes these evaluations contradictory. Even for the 1 per-sonal statement that seemed to have consistent ratings, theremarks were different:

1. Good first paragraphattention getting and relevant; de-scribes why surgery, gives a goal that is attainable (rating 6)

2. Eloquent, good examples, could give more insight (rat-ing 6)

3. Like that he/she understand the approach, not overconfi-dent . . . weird comment about consulting other services

was not sure what to think about that (rating 5)4. Clearly shows patient ownership, acknowledges multifac-

eted job of surgeon, need for team, middle of the road, nowow factor (rating 5)

Based on numbers alone, every rater agreed this was a good

personal statement; however, the criteria for which it wasdeemed good were not consistent. This finding illustrates thesubjective evaluation of the personal statement.

DISCUSSION

Our results suggest that application summaries enjoy a substan-tial degree of objectivity, whereas personal statements are amuch more subjective measure, indicating that different thingsappeal to different people. Thus, a personal statement wouldnot be a stable indicator of being granted an interview in ourstudy. If these results are generalizable to other institutions,

then students should know that they cannot count on the per-

sonal statement to be a reliable measure for interview offers andthat different evaluators will judge the same statement differ-ently. Unfortunately, we cannot generalize our results as oursmall sample size was chosen originally to establish inter-raterreliability within a larger study. Thus, this small size was notpowered for generalization and may be vulnerable to errors of

statistical inference.We had hoped to find out how the rating of the personal

statement correlated with the rating of the application sum-mary and how both together predicted the likelihood ofgaining an interview. Because of the high degree of unreli-ability, in the ratings of the personal statements, such anal-yses were precluded.

Although our original objective was not met, our finding thatthe inter-rater reliability of the personal statement was unstable

was significant enough to share with students and faculty thatare engaged in the match process. This finding suggests an issuein much need of continued research: What is expected in a

personal statement? Although it is advisable that students donot write anything offensive or grammatically incorrect in theirpersonal statement, the content can vary and its appeal will bevariable.4 To ensure objectivity in a potentially subjective per-sonal statement evaluation, institutions need to determine whatcharacteristics a personal statement should exhibit to merit aninterview and stay close to those criteria in scoring them. WayneState Universitys Radiology Program began this process, whichis illustrated in Smith et al.4Another program that found suc-cess in standardization of the screening process is the Universityof Pennsylvania. Their surgical program had an organizationalspecialist help standardize the screening and interviewing pro-

cess; the adjustments made improved the programs ability tomatch the candidates fit with faculty expectations.5 Althoughthese are 2 great examples of programs standardizing the appli-cation process, to our knowledge, this is not being accom-plished widely.

ACKNOWLEDGMENTS

A special thank you to Gale G. Hannigan, PhD, MLS, MPHfor her editorial, creative, and supportive contributions and to

June Lubowinski, MLS, MA for her help with literaturereviews.

REFERENCES

1. Crane JT, Ferraro CM. Selection criteria for emergencymedicine residency applicants. Acad Emerg Med. 2000;7:54-60.

2. National Resident Matching Program. Data Release andResearch Committee. Results of the 2010 NRMP ProgramDirector Survey. Washington, DC: National ResidentMatching Program; 2010. Available at: http://www.

nrmp.org/data/programresultsbyspecialty2010v3.pdf.

342 Journal of Surgical Education Volume 69/Number 3 May/June 2012

http://www.nrmp.org/data/programresultsbyspecialty2010v3.pdfhttp://www.nrmp.org/data/programresultsbyspecialty2010v3.pdfhttp://www.nrmp.org/data/programresultsbyspecialty2010v3.pdfhttp://www.nrmp.org/data/programresultsbyspecialty2010v3.pdfhttp://www.nrmp.org/data/programresultsbyspecialty2010v3.pdf

7/29/2019 Is the Evaluation of the Personal Statement..(1)

4/4

3. Taylor CA, Weinstein L, Mayhew HE. The process of res-ident selection: a view from the residency directors desk.Obstet Gynecol. 1995;85:299-303.

4. Smith EA, Weyhing B, Moody Y, Smith WL. A criticalanalysis of personal statements submitted by radiology res-idency applicants. Acad Radiol. 2005;12:1024-1028.

5. Kelz RR, Mullen JL, Kaiser LR, et al. Prevention of surgicalresident attrition by a novel selection strategy. Ann Surg.2010:537-543.

Appendix AApplication Checklist 1-7

Reviewer _______________________________ Applica-tion Summary #__________

Please rate the application and list your top reasons for yourrating (objective application information).

1 ___ Negative: Would not invite for an interview___________________________________________________________________________________________________________________________2 _______________________________________________________________________________________________________________________________________________________________________3 _______________________________________________________________________________________________________________________________________________________________________4 ______________________________________________________________________________________________________________________________5 ______________________________________________________________________________________________________________________________6 ___

___________________________________________________________________________________________________________________________7 ___ Positive: Will invite for an interview__________________________________________________________________________________

_________________________________________

Appendix BPersonal Statement Checklist 1-7

Reviewer _______________________________ PersonalStatement #____________

Please rate the personal statement and list your top threereasons for your rating.

1 ___ Negative: Would not invite for an interview___________________________________________________________________________________________________________________________2 ______________________________________________________________________________________________________________________________3 ______________________________________________________________________________________________________________________________4 ____________________________________________5 ______________________________________________________________________________________________________________________________6 ______________________________________________________________________________________________________________________________7 ___ Positive: Will invite for an interview___________________________________________________________________________________________________________________________

Journal of Surgical Education Volume 69/Number 3 May/June 2012 343