Evaluating data quality in the Cancer Registry

Freddie Bray

Deputy Head, Section of Cancer Information

IARC

Dharmais Cancer Hospital · Jakarta · November 2010

Cancer Incidence in Five Continents: Vol 1 (1966) Introduction

Reliable cancer registries:

• Those able to amass information (diagnostic and personal) on virtually all cases of cancer among patients genuinely resident within a defined catchment area during a prescribed period of time;

• able to supplement this with death certificate data for patients not seen in hospital

• having an adequate system for eliminating duplicate entries for the same person

• and good population data - by sex and by 5-year age groups and, if relevant, by race/language

Data quality and its evaluation

• Evaluation of data quality in the cancer registry: Principles and methods.

• Part I: Comparability, validity and timeliness (Bray & Parkin)• Part II: Completeness (Parkin & Bray)• Eur J Cancer (2009) 45: 747-77, 756-64• Update of 1994 IARC Technical Report• Application to Cancer Registry of Norway:• Larsen et al (2009) Eur J Cancer 45:1218-31

• Standards and guidelines for cancer registration in Europe: The ENCR recommendations, vol 1. Lyon: IARC (2003).

Data quality and its evaluation

Conclusion:

“This review indicated that the routines in place at the Cancer Registry of Norway yield comparable data that can be considered reasonably accurate, close-to-completion and timely, and serves as a justification for our policy of reporting annual incidence one year after the close of registration.”

Data quality and its evaluation

Four “classical” dimensions of quality:

• Comparability

• Validity

• Completeness

• Timeliness

1. Comparability2. Completeness3. Validity4. Timeliness

Special Issue: Data Quality at the Cancer Registry of Norway

http://www.kreftregisteret.no

Data quality and its evaluation

Four “classical” dimensions of quality:

• Comparability

• Validity

• Completeness

• Timeliness

Data quality and its evaluation

Comparability

• Ensuring comparable standards of reporting and classification across registries and within registries over time;

• Reporting of routines, standards and practices in place and, especially, dates in changes of practice;

• Where standards within a registry differ from “accepted” practice, requirement to provide means of conversion from one to other.

Data quality and its evaluation

Comparability

• Classification and coding systems

• Definition of incidence date

• Handling of multiple primaries

• Incidental diagnosis (basis)

• Screening and testing

• Imaging

• Autopsy diagnosis (basis)

• Handling of death certificate information

Data quality and its evaluation

Bray and Parkin (2009) EJC 45:747

Data quality and its evaluation

Larsen et al (2009) EJC 45:1218

Note: Rates are age-adjusted to the 1970 U.S. standard. Rates from 1973-1987 are

based on data from the 9 standard registries. Data from San Jose and Los

Angeles are included in the rate calculation for 1988-1995.

Data quality and its evaluation

Four “classical” dimensions of quality:

• Comparability

• Validity

• Completeness

• Timeliness

Data quality and its evaluation

Validity

• Accuracy of reporting

• Do cases reported to have a specific characteristic truly have that characteristic

• Depends on

• Accuracy of source information

• Registry “skill” in abstracting, coding and reporting

Data quality and its evaluation

Validity – assessment procedures:

• Diagnostic criteria methods

• Histologic/microscopic verification (% HV/MV)

• Death certificate only (% DCO)

• Missing information (e.g. % PSU)

• Internal consistency checks (QC)

• Re-abstracting and recoding (QA)

Data quality and its evaluation

Microscopic verification (% MV)

• Varies by cancer site (and age);

• Depends on pathology/cytology service

• 100% not always best;

• Are statistical tests to compare % MV of a registry against standard, other registries or itself at different time.

Data quality and its evaluation

Larsen et al (2009) EJC 45:1218

Data quality and its evaluation

Death certificate only (% DCO)• Varies by cancer site (and age);

• Depends on clinical service;

• Associated with reduction in validity (especially site and diagnosis date) and increase in missing information;

• Other validity issues around “Death certificate notified (DCN)” or “Death certificate initiatied (DCI)”.

Data quality and its evaluation

Larsen et al (2009) EJC 45:1218

Data quality and its evaluation

Bray and Parkin (2009) EJC 45:747

Data quality and its evaluation

Missing information (e.g. % PSU)• Varies by cancer site and age;

• Varies by data item (e.g. stage);

• Depends on both registry and clinical record practice;

• Care required in codes used to define “primary site uncertain” (not just “Unknown primary site ICD-10 C80);

• Low % MV associated with high “PSU”.

Data quality and its evaluation

Larsen et al (2009) EJC 45:1218

Data quality and its evaluation

Internal consistency checks (QC)• Invalid (or unlikely) codes or combinations of codes

or sequences of dates;

• Can be operationalised within software (including during data entry);

• IARC has developed such tools (IARC-CHECK) within IARCcrgTools which can be downloaded from IACR website: www.iacr.com.fr

• Checks applied should be reported along with results.

Data quality and its evaluation

Re-abstracting and recoding (QA)

• Expensive and time consuming;

• Can be operationalised on sample basis;

• Can make use of other ad-hoc studies;

• Requires approaches to correct identified problems prospectively and retrospectively.

Data quality and its evaluation

Four “classical” dimensions of quality:

• Comparability

• Validity

• Completeness

• Timeliness

Data quality and its evaluation

Completeness:

• The extent to which all of the incident cancers occurring in a target population are included in the registry database;

• Key defining criterion for population basis to registration;

• No perfect assessment tool

Data quality and its evaluation

Completeness assessment:

• Methods based on comparisons and inspection;

• Methods based on independent assessment.

• Ad-hoc planned or incidental studies

• Use of multiple (independent) sources of notification especially death certificates

Data quality and its evaluation

Completeness assessment:

• Methods based on comparisons and inspection;

• Compare rates over time and/or with similar populations;

• Inspect age-incidence curves;

• Stability of childhood cancer rates.

Data quality and its evaluation

Completeness assessment:• Methods based on independent assessment

• Ad-hoc planned or incidental studies(comments as for validity)

• M/I ratios

• Capture-recapture methods

• The DC and M/I methodAjiki et al (1998) Nippon KEZ 45:1011

• The Flow method (also measures timeliness)Bullard et al (2000) B.J.Cancer 82:111

Read Parkin & Bray

(2009) for details

Data quality and its evaluation

Completeness assessment:• M/I ratios;• Number of incident cases during defined time period;• Number of deaths during the same time period;• Assumption that mortality data from a source

independent of cancer registration;• Should analyse by cancer site and by age group;• Absolute values depend on survival rates and quality

of both registration and death certification;• Not robust to (usually rare) short-term changes in

incidence or survival.

Data quality and its evaluation

Larsen et al (2009) EJC 45:1218

Data quality and its evaluation

Four “classical” dimensions of quality:

• Comparability

• Validity

• Completeness

• Timeliness

Data quality and its evaluation

Timeliness:

• Speed with which registry can collect, process and make available data at a given standard of completeness and quality;

• Often pressure to increase timeliness at expense of other quality indicators;

• Some registries (e.g. SEER) publish at a given time point and make estimates of under reporting;

• 12-24 months after year end represents current “standard”.

Data quality indicators CI5 vol. 9Breast cancer (f)

Registry No. MV% DCO% M/I%

Brazil

Sao Paulo

22598 82.2 4.6 22.8

SEER (14) 237378 98.5 0.6 21.3

Norway 12521 98.4 0.3 29.4

UK

Scotland

17137 96.4 0.3 32.9

Japan

Osaka

11103 90.1 2.5 30.5

Data quality indicators CI5 vol. 9Lung cancer (m)

Registry No. MV% DCO% M/I%

Brazil

Sao Paulo

6525 66.9 13.8 72.8

SEER (14) 123409 89.8 1.8 80.7

Norway 6516 87.4 1.0 88.6

UK

Scotland

12969 74.9 0.9 88.3

Japan

Osaka

16759 73.1 19.3 83.7

Data quality and its evaluation

• Cancer registration is a worldwide activity and leads the way in global surveillance for non communicable diseases;

• The benefit of population based registration to cancer control programs and to epidemiological research can be realised only to the extent that data are of a comparable, high quality standard;

• Reporting on data quality in a registry is as important as reporting analyses of the data.