Moon Seok Park, MD Seoul National University Bundang Hospital Testing reliability and validity in...
-
Upload
janice-green -
Category
Documents
-
view
220 -
download
3
Transcript of Moon Seok Park, MD Seoul National University Bundang Hospital Testing reliability and validity in...
Moon Seok Park, MD
Seoul National University Bundang Hospital
Testing reliability and validity in medical research
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Reliability
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
• 1 년 차 때 , 교수님이 “ 내일까지 X-ray 1000 장 재 봐서 결론 내 !!” 고 오더를 내리셔서 .
• 처음 재보는 각도 , 밤새 측정을 했다 . 힘들어서 인턴도 시켰다 . 제대로 했는지도 잘 모르겠다 .
• 그런데 , 결과는 의미 있게 나왔다 . OK!!
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
• 두 개의 다른 방법으로 측정을 했을 때 , 신뢰도를 알아
보려면 paired t-test 로 하면 안 되는가 ?
• Paired t-test 는 어떨 때 쓰는 방법일까 ?
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Reliability
• Extent to which scale items measure the same construct, with freedom of random error
• 신뢰도• 측정 시 마다 측정치가 비슷한가 ?• Test-retest reliability, Inter-rater reliability,
Intra-rater reliability, Alternative form reliability, Internal consistency.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Test-retest reliability
• 주로 Psychometric analysis : 인터뷰 , 설문지… .
• 일정한 시간 간격을 두고 , 같은 검사를 시행 .• Cohen’s kappa, weighted kappa, Pearson’s
correlation, Intraclass correlation coefficient(ICC).
• Cf) Intra-rater(observer reliability) : 방사선 검사 계측… .
• Memory contamination
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Inter-rater reliability
• 전문가에 의한 인터뷰 , scoring, 신체 계측 , 방사선 계측 .
• 여러 명이 한 객체를 계측하여 , 비슷한가 비교 .• Cf) Agreement : 혼용되어 사용되지만 , 특히 다른
기구를 이용한 측정 , 예를 들어 MRI 와 CT 의 비교 등…
• 방사선 계측 등에서는 intra- and inter-observer(rater) reliability 를 set 로 .
• Cohen’s kappa, weighted kappa, Pearson’s correlation, Intraclass correlation coefficient(ICC)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Internal consistency
• 이전의 reliability 와는 조금 다른 의미 . Psychometric analysis ( 설문지 , 인터뷰 ) 등에 주로 국한 되어 사용 .
• Homogeneity • 가령 10 개의 문항이 있다고 하면 , 각각의
문항이 서로 비슷 .• Item to item, Item to total, Cronbach’s
alpha• Too high internal consistency = Item
redundancy.• Cf) Uni-dimensionality, Item response
theory, Rasch analysis(INFIT statistics)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Question: which is reliable?
1 2
3 4
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
What are the main measures of reliability?
• What if the data are dichotomous or polychotomous?– Kappa coefficient
• What if the data are quantitative (interval or ratio scale?– Intraclass Correlation Coefficient (ICC)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
ICC
• Intraclass correlation coefficient
• Reliability test for quantitative data
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Models of ICC
• One-way random effect model– Raters: a random effect
• Two-way random effect model– Raters: a random effect– Subjects: a random effect
• Two-way mixed effect model– Raters: a fixed effect– Subjects: a random effect
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Types of ICC
• Absolute agreement– Measures if raters assign the same
absolute score
• Consistency– Measures if raters’ scores are highly
correlated even if they are not identical in absolute terms
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Measures of ICC
• Single measures– Individual ratings constitute the unit of
analysis
• Average measures– The mean of all ratings is the unit of
analysis
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
ICC
• Affected by true subject variability as well as measurement error
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Example
• Measurement error– Data 1 = Data 2
• Subject variability– Data 1 < Data 2
Data 1 Data 2
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
ICCs for sample data 1 and 2
models Sample data 1 Sample data 2
1 way random-0.059 (-
0.308~0.407)0.922 (0.799~0.978)
2 way random0.217
(0.007~0.614)0.924 (0.237~0.986)
2 way mixed0.217
(0.007~0.614)0.924 (0.237~0.986)
ICC values were calculated with the assumption of absolute agreement and single measurementData are presented as ICC (95% confidence interval)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
• Propose 6 ICC types:
ICC(1,1) ICC(2,1) ICC(3,1) ICC(1,k) ICC(2,k) ICC(3,k)
Shrout and Fleiss, 1979
Expected Reliability of a Single Rater’s Rating
Expected Reliability of the Mean of a set ofk Raters
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
k (no.of observers), n (no.of targets)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
between-target mean square (BMS); within-target mean square(WMS); BMS represents true subject variability, and WMS represents measurement error
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Shrout and Fleiss, 1979
• Important issue in the choice of an appropriate index– Whether the ANOVA design should be
one way or two way– Whether raters are considered fixed
or random effects– Whether the unit of analysis is a
single rater or the mean of several raters
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Pitfalls and important issues in
testing reliability using ICC in
orthopaedic research
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Literature review
• Pubmed database
• Orthopaedic articles that used ICC
• Of the 92 articles identified, 58 (63%) did not clarify the ICC model used.
• The model, types, and measures used were clearly declared in only 5 (5%)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
ICC of physical examinations
• 30 patients with CP• Interobserver reliability of physical
examinations using ICC– Popliteal angle– Thomas test– Staheli test
Same dimension !! (joint angle)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Reliability of physical examinations evaluated by various statistical methods
Popliteal angle Thomas test Staheli testMean (°) 47.6 4.7 2.5SD (°) 15.2 5.9 8.8Range (°) 8~80 0~20 -17~28ICC2 way randomconsistency/average 0.881 (0.794~0.936) 0.742 (0.552~0.860) 0.463
(0.067~0.708)consistency/single 0.713 (0.562~0.829) 0.490 (0.291~0.672) 0.224
(0.023~0.447)absolute/average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464
(0.070~0.708)absolute/single 0.710 (0.560~0.826) 0.490 (0.292~0.671) 0.224
(0.024~0.447)2 way mixedconsistency/average 0.881 (0.794~0.936) 0.742 (0.552~0.860) 0.463
(0.067~0.708)consistency/single 0.713 (0.562~0.829) 0.490 (0.291~0.672) 0.224
(0.023~0.447)absolute/average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464
(0.070~0.708)absolute/single 0.710 (0.560~0.826) 0.490 (0.292~0.671) 0.224
(0.024~0.447)1 way random average 0.880 (0.792~0.935) 0.742 (0.553~0.860) 0.464
(0.072~0.708) single 0.709 (0.559~0.826) 0.489 (0.292~0.671) 0.224
(0.025~0.447)SEM (SDx√(1-reliability)
0.112~0.175 0.590~0.830 2.02~2.43
MAD 9.4 3.6 6.1CV (SD/mean) 0.32 1.16 2.76ICC, intraclass correlation coefficient; SEM, standard error of measurement; MAD, mean absolute difference; CV, coefficient of variation
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Simulated data
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Conclusion
• ICC value could represent the opposite tendency to true measurement error (mean absolute difference) even when measuring similar dimension
• ICC could be variable depending on the model used.
• ICC value was affected by measurement error, subject variability, and slopes.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
결론적으로 이렇게 해야 ..
• ICC values were large when measurement errors were small, subject variability large, and slopes parallel.
• Clinical context need to be considered when interpreting ICC.
• ICC setting should be declared.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Validity
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Validity
• Extent to which instruments is really measuring what it purpose to measures.
• 보통 internal validity 라고 이야기 한다 .
• Cf) external validity = generalisability
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Validity
• Face validity
• Content validity
• Criterion(concurrent, predictive) validity
• Construct(convergent, discriminant) validity
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Face validity
• 안면 타당도 ( 액면 타당도 )
• Content validity 와 혼동될 수 있지만 , 좀 더 추상적임 .
• 예를 들어 영어 시험의 문항에 수학 문제가 있으면 , face validity 에 문제가 있는 것 .
• 대게 저자들이 screening 하는 정도로 표현 .
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Content validity
• 내용 타당도
• Face validity 와 비슷하지만 , 좀 더 systematic 하게 분석 .
• 일정 수의 panel 이 모여서 content validity를 scoring 하여 , 점수화 하고 , 평균 점수가 미달이면 기각 .
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Criterion validity
• Concurrent validity : gold standard 와 얼마나 비슷한가 ?
• 방사선 지표를 측정한다 . Gold standard 로 생각하는 CT 측정치와 비교 .
• Cf) convergent validity.
• Predictive validity
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Construct validity
• 구인 타당도• Convergent validity : 비슷한 지표 (gold
standard 는 아님 ) 와 상관관계가 있는가 ?• TEPS 라는 영어시험을 만들었다 . 타당도를
보려고 , TOFLE 과 상관관계를 보았다 . (영어실력의 gold standard 는 ?)
• 사람이 측정한 방법과 컴퓨터가 측정한 방법에 상관 관계가 있는가 ?
• Pearson correlation.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Construct validity
• Discriminant validity : 전혀 다른 것을 측정하는 지표와 상관 관계가 있는가 ?
• 인성검사와 지능검사의 상관관계
• Cf) Known group validity : 확실히 다른 집단에서 다른 점수가 나오는가 ?
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Others
• Precision• Responsiveness• Sensitivity• Specificity• Sensitivity analysis• Item response
theory• Rasch analysis
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Introduction
• Increased femoral anteversion and coxa
valga are common deformities associated
with intoeing gait and unstable hips in CP,
which need surgical correction.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Introduction
• Physical examination and neck shaft angle
measured on hip radiographs are primary
tools evaluating femoral anteversion and
coxa valga.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Introduction
• Physical examinations measuring femoral
anteversion include
– Trochanteric prominence angle test (TPAT)
– Hip internal rotation (IR)
– Hip external rotation (ER)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Introduction
• CT measurement is accurate, but
expensive and involves radiation
exposure.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Purpose of Study
• To assess the validity and reliability of physical exams measuring femoral anteversion and neck shaft angle on hip X-ray– Concurrent validity
– Intra- and interobserver reliability
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Reliable and valid Not reliable but valid
Reliable but not valid Not reliable and not valid
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Materials and Methods
• Prospective study approved by IRB
• 36 consecutive patients with CP– Mean age 11.0 years (SD 1.3)
– M : F = 26 : 10
– GMFCS I / II / III / IV / V 5 / 11 / 11 / 7 / 2
• Exclusion– Previous Op, trauma, infection, etc.
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Hip Internal Rotation
• Prone position
• Angle between vertical line & long axis of the leg– legs are rotated
outward maximally
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Hip External Rotation
• Prone position
• Angle between vertical line & long axis of the leg– leg is rotated inward
maximally
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Trochanteric Prominence Angle Test
• Prone position
• Palpate G. trochanter
• External rotate limb until G. T. reaches most lateral
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
NSA on x-ray
• AP hip X-ray with hips 20°-30° internally rotated
• Angle : a line through midpoint of shaft & line through head and neck center
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Femoral anteversion on 2D CT
• Standard method
• Radiation hazard
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
NSA on 3D MRP Image
Standard method for concurrent validity of NSA on X-ray
MRP: multiplanar reformatted
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Valdity
• Physical exam measuring femoral AV– Correlation with femoral anteversion
measured on 2D CT
• NSA measured on X ray– Correlation with NSA measured on 3D MPR CT
image
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Reliability
• Interobserver reliability of physical exam using three orthopaedic surgeons on a single day
• Intra- and interobserver reliability of NSA on X-ray– Repeated measurements with an interval of 3
wks
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Statistics
• Validity– Pearson’s correlation coefficients
• Reliability– Intraclass correlation coefficients (ICCs)– 2 way random effects, single measurement &
absolute agreement• Multiple regression test
– To predict the accurate femoral anteversion (CT) from physical exam
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Results
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Femoval AV on CT= 0.92 x TPAT - 3.2 (R2=0.829)
SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL ◦ SEOUL NATIONAL UNIVERSITY BUNDANG HOSPITAL
Conclusions
• TPAT and NSA on X ray showed clinically
relevant validity and reliability compared
with CT measurement.
• CT evaluating proximal femoral geometry
could be replaced by physical exam and
X-ray in patients with CP, avoiding
unnecessary radiation exposure.