Comparison of ceiling effects

Comparison of ceiling effects between two patient-rating scores and a physician-rating score in the

assessment of outcome after the surgical treatment of distal radial fractures

S-J. Kim, B-G. Lee, C-H. Lee, W-S. Choi, J-H. Kim, K-H. Lee

Hanyang University College of Medicine, Seoul, South Korea

The Bone and Joint Journal

A Journal Club presentation by Dr. Libin Thomas Manathara, Amala Institute of Medical Sciences

Introduction

• Traditionally, the outcome of surgery for patients with a distal radialfracture was assessed clinically using objective physical andradiological parameters

• However, considered to be insufficient

• Do not reveal enough useful information about the function of thewrist

• So, self-administered patient-rating scores have been developed forpatients to provide a more detailed assessment

Introduction

• These scores are now being used increasingly in the evaluation of theoutcome of surgery for distal radial fractures

• The Disability of the Arm, Shoulder and Hand (DASH) score• and

• Patient-Rated Wrist Evaluation (PRWE) score• are the most frequently used in the literature for the evaluation of function

after distal radial fractures, and their validity, reliability, and theresponsiveness have been validated

Introduction

• However, the ceiling effect of these scores has not been investigated

• Uttl reported that the ceiling effects are present when the maximumscores are relatively easily attained by a substantial proportion ofpatients so that the true extent of function cannot be assessed

• The content validity and reliability of the score are reduced whenceiling effects are present as it is not possible to distinguish the truedifference between two different groups

Introduction

• Surgical fixation of unstable distal radial fractures with a volar lockingplate has gained popularity owing to its technical simplicity, and goodoutcomes

• Therefore, the authors aimed to investigate the ceiling effects of twopatient rating outcome scores, the DASH and PRWE, and comparethem with the ceiling effect of a physician-rating outcome score, themodified Mayo wrist score (MMWS) in the evaluation of the outcomeof the surgical treatment of distal radial fractures using a volar lockingplate in elderly female patients

CEILING EFFECT

• The ceiling effect is the level at which an independent variable no longer has an effect on a dependent variable

CEILING EFFECT- example 1• A population survey about lifestyle

variables influencing health outcomesmight include a question about smokinghabits

• To guard against the possibility that arespondent who is a heavy smoker mightdecline to give an accurate response aboutsmoking, the highest level of smokingasked about in the survey instrumentmight be "two packs a day or more"

• This results in a ceiling effect in thatpersons who smoke three packs or more aday are not distinguished from personswho smoke exactly two packs

CEILING EFFECT- example 2

• A population survey about income similarly mighthave a highest response level of "Rs. 10,00,000 peryear or more," rather than including higher incomeranges, as respondents might decline to answer atall if the survey questions identify their income toospecifically

• This too results in a ceiling effect, not distinguishingpersons who have an income of Rs. 50,00,000 peryear or higher from those whose income is exactlyRs. 10,00,000 per year


• A crude example would be measuring theheights of trees with a ruler only 20 metersin length, if it is apparent on the basis ofother evidence that there are trees muchtaller than 20 meters

• Using the 20-meter ruler as the sole meansof measuring trees would impose a ceilingon gathering data about tree height


• Suppose that an IQ test has three subtests:vocabulary, arithmetic, and picture analogies

• Now suppose that Joe obtains the maximumscore of 20 on the arithmetic test, but gets 10out of 20 on the vocabulary and analogies tests

• Is it fair to say that Joe's total score of 20+10+10,or 40, represents his total ability?


• The answer is no, because Joe achieved themaximum possible score of 20 on thearithmetic test

• Had the arithmetic test included additional,more difficult items, Joe might have gotten30 points on that subtest, producing a"true" score of 30+10+10 or 50


• Compare Joe's performance with that of Jim, who scored 15+15+15 =45, without running into any subtest ceilings

• In the original formulation of the test, Jim did better than Joe (45versus 40), whereas it is Joe who actually should have gotten thehigher "total" intelligence score than Jim (score of 50 for Joe versus45 for Jim) using a reformulated test that includes more difficultarithmetic items


• A ceiling effect is said to occur when a high proportion of subjects in astudy have maximum scores on the observed variable

• This makes discrimination among subjects among the top end of thescale impossible

• For example, an examination paper may lead to, say, 50% of thestudents scoring 100%

• While such a paper may serve as a useful threshold test, it does notallow ranking of the top performers

Patients and Methods

• The medical records of 163 consecutive female patients wereretrospectively reviewed with an unstable fracture of the distal radiustreated with a volar locking plate between June 2011 and July 2013

• The inclusion criteria were• patients with this fracture who were treated in this way

• who had completed the questionnaires one year postoperatively


• Exclusion criteria were

• a history of injury to the same hand or wrist (n = 10)

• an accompanying diaphyseal or metaphyseal ulnar fracture (n = 25)

• women aged < 50 years (n=17)

• failure to complete the questionnaire one year post-operatively (n = 23)

• lost to follow-up (n = 11)

• TOTAL- 86 excluded

• The fractures were classified according to the AO/OTA classification

Patients and Methods- Demographic data

• Total- 77

• Mean age- 64.2yrs

• Side- 41 right and 36 left

• Fracture type, AO/ OTA

• 23- A- 29

• 23- B- 14

• 23- C- 34


• The DASH instrument was developed by the Institute for Work andHealth in Ontario and the American Academy of OrthopaedicSurgeons in 1994

• It consists of a 30-item questionnaire that evaluates symptoms andfunction with an ordinal five-grade rating scale for each item

• Zero points indicates no disability of the upper limb and 100 pointsindicate maximum disability


• The PRWE is a patient reported 15 -item instrument that concentratesprimarily on the function of the wrist

• Five items evaluate pain, and ten evaluate specific tasks and commonactivities of living

• Zero points indicate a normal wrist and 150 points is the worstpossible score


• The MMWS (modified Mayo wrist score) is a physician-rating scoringsystem consisting of a total of 100 points; 25 for the assessment ofpain, 25 for the active flexion/extension arc of the wrist, 25 for gripstrength, and 25 for the ability to return to regular activities

• Pain is rated according to the patient’s description

• A score of 90 to 100 points is defined as an excellent result, 80 to 89points as a good result, 65 to 79 points as a fair result, and < 65 pointsas a poor result


• All the fractures were surgically treated using a volar locking plate by the senior author

• Two kinds of volar locking plate were used

• 2.4 mm Variable Angle LCP Two-Column Volar Distal RadiusPlate- Synthes

• Acu-Loc Volar Distal RadiusPlate- Acumed


• A short-arm splint was used post-operatively and changed to a wristbrace two weeks after the operation

• Active exercises of the fingers were allowed on the day of surgery,and wrist exercises were encouraged intermittently two weeks afterthe surgery and increased as tolerated

Detection of the ceiling effect

• A review of the literature revealed two different definitions for thethreshold of the ceiling effect and one other method for assessing it

• The first was described by McHorney and Tarlov who noted thatceiling effects are present if > 15% of patients achieved the maximumpossible score

• According to the second definition described by Wang et al,researchers should be cautious of their conclusions if the proportionof patients reaching the ceiling is > 20%

Detection of the ceiling effect

• The third approach for assessing the ceiling effect was described byUttl, using ‘standardised distances’

• This distance was calculated by subtracting the mean from themaximum score and dividing the result by the standard deviation (SD)

Statistical analysis

•Cronbach’s alpha was estimated for each score to determine theinternal consistency and test reliability

•Descriptive statistics were used for the mean and median, SD andinterquartile range, and 95% confidence interval of each variable

•Spearman’s rho was used to analyse the correlation between theDASH, PRWE scores, and MMWS

Results

• A significantly high correlation between the DASH and PRWE scoreswas confirmed

• In contrast, no significant correlation was found between the patientrating DASH score and the physician-rating MMWS score or betweenthe PRWE and MMWS

Results

• The skewness of the data assessed with the DASH and PRWE scoreswere 2.42 and 2.34, respectively, whereas that of the data assessedwith MMWS score was -O.96.

Results

• The number of patients with the maximum possible score was 18 ofthe 77 patients (23.4%) with both the DASH and PRWE

• On the other hand, only eight (10.4%) of the 77 patients had themaximum possible score with the MMWS

Results

• Based on the definitions of ceiling effect described bv McHorney andTarlov and Wang, the data assessed with both patient-rating scores,the DASH and PRWE, had a substantial ceiling effect, and the dataassessed with the physician-rating score, the MMWS, had no ceilingeffect

Discussion

• DASH and PRWE scores had a substantial ceiling effect and no ceilingeffect in the outcome assessed with the MMWS

• Patient-rating outcome measures that quantify perceived disabilityhave been developed to provide a more comprehensive assessmentof outcome

Discussion

• However, Hsu et al noted that the DASH values in a group ofintercollegiate athletes were of limited use because of its substantialceiling effect

• They assumed that various outcome measures used may be limitedby a ceiling effect and that their appropriateness should be assessedbefore use

Discussion

• If there is a ceiling effect in the data, one can only observe itsthreshold, which may misrepresent the true scores of the patients

• It can cause biased results

• The reliability is reduced as the patients with the maximum possiblescore cannot be differentiated from one another

Discussion

• Another example of a ceiling effect related to the use of patient-ratingscores in orthopaedic research was described by Shauver and Chung

• They noted that the ceiling effect was observed in the outcome datafor distal radial fractures assessed using the Michigan Hand OutcomeQuestionnaire (MHQ)

Discussion

• The clinically important minimal difference of the MHQ when used forthe assessment of outcome of distal radial fractures could not beestablished because of the ceiling effect in the third month of follow-up

• Elsewhere, where such concern has emerged, it has been proposedthat the problem of the ceiling effect should be addressed whenevaluating the outcome of surgery

Discussion

• Here, it was found that the MMWS had no ceiling effect one yearpost-operatively

• This may be because half of the total score of the MMWS iscomprised of objective measures: 25 points for range of movementand 25 for grip strength

• The strength of the current study lies in it being the first to comparethe ceiling effect between the outcome assessed using patient-ratingscores and a physician-rating score in the surgical treatment of distalradial fractures

Discussion

• As the use of patient-rating scores is increasing in orthopaedicresearch, these results emphasise the importance of selectingappropriate outcome assessment tools

• Care should be taken when interpreting results

• Beware of the possibility of finding an absence of a significantdifference between two groups when differences are actually present

• When designing a study using a patient-rating score the researchershould be careful to choose an appropriate length of follow-up

Discussion

• If the study requires at least one year of follow-up, a physician-ratingscore should be included in addition to a patient-rating score to avoida false negative result

• The present study only included women aged > 50 years

• The ceiling effect in the outcome data might have been higher if menor younger patients had been included

Discussion

• A patient-rating outcome measure that includes more difficult tasksfor the patients might be needed to obtain the true score in thesepatients

Discussion- LIMITATIONS

• They evaluated all patients at one year after surgical treatment

• If those with a shorter follow-up had been included, and theoutcomes had been assessed several times during the post-operativeperiod, they might have identified the exact point at which the ceilingeffect emerged

Discussion- LIMITATIONS

• A second limitation is that they could not estimate a proper samplesize before the study was conducted

• They could not find any statistics to calculate proper sample size indealing with the ceiling effect

• Terwee et al noted that if no ceiling effects are present in > 50patients, a positive rating could be given for the absence of ceilingeffects

• Therefore, 77 patients in the current study seems to be enough toanalyse the ceiling effect

SUMMARY

• The outcome of surgery for unstable distal radial fractures assessedby patient-rating scores such as the DASH and PRWE was found to becompromised by substantial ceiling effects one year post-operatively

• In contrast, a physician-rating score, the MMWS, did not suffer fromthe same deficiency

• Researchers should bear in mind the possibility of a ceiling effectwhen assessing the outcome of treatment in these patients usingpatient-rating scores

THANK YOU

Comparison of ceiling effects

Health & Medicine

Transcript of Comparison of ceiling effects