Agreement between Methods

Karlijn J. van Stralen¹, Kitty J. Jager¹, Carmine Zoccali², and Friedo W. Dekker1,3

1 ERA–EDTA Registry, Dept. of Medical Informatics, Academic Medical Center, Amsterdam, The Netherlands

2 CNR–IBIM Clinical Epidemiology and Pathophysiology of Renal Diseases and Hypertension, Renal and Transplantation Unit, Ospedali Riuniti, 89125 Reggio Cal., Italy

3 Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, The Netherlands

Kidney International: Series on epidemiology

Agreement

• Situation in which one would like to compare different tests

• For example GFR estimate by 24 hour creatinine clearance and by inulin clearance

• Ideal method

– Determines whether a relationship is linear

– Detect a possible systematic difference

– Determine the amount of random difference

Pearson’s correlation coefficient

• Reflects degree of linear relation– -1 perfect negative relation – 0 no correlation– +1 perfect positive relation

• However, use has several limitations– Type of association

– Systematic difference

– Range of values

• Six different data collections

• Figures A, B, C & D show different types of associations

• Figures E and F show a systematic difference.

– In E all values are 5 points higher than A (dashed line)

– In F all values are multiplied by 2.5 and 10 was subtracted.

• All pearson’s correlation coefficients are 0.80

Shape of the curve and systematic

difference

Range of values

• Entire range of values– High correlation

coefficient

• Narrow range of values– Lower correlation

coefficient

Other methods

• Mean difference + paired t-test– Reveals systematic difference– But – mean difference can be zero if difference is not

constant

• Intraclass correlation coefficient (ICC)– Contain information on correlation and systematic difference

– But – sensitive to range of values

• Regression analysis– Useful for description amount of change

– But – no information on size and amount of difference

Bland – Altman plot

Difference between the paired measurements (A-B) is plotted against their mean value ([A+B]/2)

Bland - Altman

• Easy to assess – level of (systematic) difference

– the scatter of the values /random error

– shows relation between the values and measurement error

• Limits of agreement– Shows variation in results

– Useful for determining interchangeability

Repeated measurements

• In case of repeated measurements of two methods on the same subject

• Preferably use all data to compare the two methods.

• However => do not use the mean– Does result in an accurate estimate of the systematic difference

– but underestimation of standard deviation of the differences, and too narrow limits of agreement.

• Different accurate methods have been developed and described in the following papers– Bland JM, Altman DG: Measuring agreement in method comparison

studies. Stat Methods Med Res 8:135-160, 1999

– Myles PS, Cui J: Using the Bland-Altman method to measure agreement with repeated measures. Br J Anaesth 99:309-311, 2007

Calibration

• Plot of the difference against “truth”

• Easy to determine whether a technique should be changed

• Not useful to determine agreement

• As these plots show a correlation between the difference and the ‘truth’ even if there is no difference between the measurement error and the size of the values of the ‘truth’

In summary

• Hard to determine agreement

• Do not use the correlation coefficient– Sensitive for range of values

– Does not reveal a systematic difference

– No information on type of association

• Bland-Altman plot is highly preferred