Texas Department of State Health Services Texas Cancer Prevention and Control Program

CDC-RFA-DP17-1701-01CONT18 Texas Cancer Registry

Program 3: National Program of Cancer Registries

Texas Cancer Registry

Program Evaluation Plan

January 31, 2018

Texas Department of State Health Services

Division for Disease Control and Prevention Services

Texas Cancer Registry

P.O. Box 149347, Mail Code 1928

Austin, Texas 78714-9347

Table of Contents Plan Overview ...................................................................................... 1

Intended Use and Evaluation Users ......................................................... 1

Program Description ............................................................................. 2

Overview .......................................................................................... 2

Activities........................................................................................... 2

Priority Populations ............................................................................ 2

Beneficiary Impact from Programmatic Activities ................................... 2

Logic Model ....................................................................................... 2

Evaluation Focus .................................................................................. 4

Plan for Collecting Data ......................................................................... 4

Methods ........................................................................................... 4

Relevant Indicators and Data Sources .................................................. 4

Data Collection Responsibilities ............................................................ 5

Analysis and Interpretation .................................................................... 5

Dissemination and Use of Findings .......................................................... 6

Communication Strategies .................................................................. 6

Formats and Audience ........................................................................ 7

Dissemination Responsibilities ............................................................. 7

Evaluation Timeline ............................................................................... 9

Conclusion ......................................................................................... 10

Appendix ........................................................................................... 11

Texas Cancer Registry Evaluation Plan Page 1 of 13

Plan Overview

As required for DP17-1701 National Program of Cancer Registries

(NPCR) Grantees, the Texas Cancer Registry (TCR) is providing an evaluation and performance measurement plan for grant Years 1–5. The

general approach follows the CDC Framework for Program Evaluation in Public Health as the primary guide for evaluation questions, overall planning,

and implementation.1 Using this validated, well-established CDC framework fosters TCR performance improvement by 1) engaging key stakeholders in

evaluation design, monitoring, and decision making, 2) describing and benchmarking TCR strengths and weaknesses, 3) using multiple varied

credible data sources, data gathering, analyses, and evaluation methods, 4)

justifying conclusions, 5) communicating results, and 6) establishing future goals with an explicit emphasis on program improvement and accountability.

Evaluation questions are focused on the inputs (resources), core registry and specialized project activities, corresponding outputs, outcomes,

and desired impacts described in the NPCR Logic Model.2 More specifically, questions will evaluate the TCR on the NPCR 2017–2022 Program Standards

and Strategies: program collaboration, external partnerships, cancer data and surveillance, community level interventions and patient support, health

systems change, and program monitoring and evaluation (data completeness, timeliness, and quality).

Intended Use and Evaluation Users

The overall purpose of the TCR Program Evaluation is to systematically

and transparently evaluate TCR program inputs, grantee activities, outputs, outcomes, and impacts, as described in the NPCR Logic Model. The

evaluation will be used by stakeholders to improve TCR efficiency, effectiveness, and accountability in carrying out its core and specialized

activities. The subsequent improvement in these activities will in turn result in improved short, intermediate, and long-term outcomes, as well as positive

program impact.

Consistent with guidance provided by the CDC Framework for Program

Evaluation in Public Health, evaluation users with access to results for decision-making include persons and organizations having an investment in

what will be learned from the evaluation, and what will be done with that knowledge. To minimize the risk of stakeholders ignoring, criticizing, or

resisting the evaluation and results, stakeholders will be engaged in various aspects of the evaluation. The three principle evaluation stakeholder groups

include 1) those involved in program operations (e.g., NPCR, TCR and other DSHS staff), 2) those served and effected by program (e.g., researchers,

various types of cancer reporters), and primary evaluation users (e.g., TCR

managers, DSHS leadership, TCR Advisory Committee).

Texas Cancer Registry Evaluation Plan Page 2 of 13

Program Description

Overview—Supporting a comprehensive and coordinated approach to

inform policy, systems, and environmental changes to decrease the cancer

burden remains a leading priority for public health in Texas, and requires

intensive, high quality cancer data and surveillance from a broad variety of

sources. Currently, the TCR represents one of the largest and most

comprehensive high quality population-based cancer data resources and

surveillance systems in the Nation. The TCR collects cancer data for over

28.3 million Texas residents, including significant special populations, and

contributes a large amount of non-resident cancer cases to other state

cancer registries and the national cancer surveillance system.3

Activities— The Texas Cancer Registry (TCR) is a combination active

and passive surveillance system responsible for the collection, maintenance,

and dissemination of high quality population-based cancer data. The TCR

engages in all grantee activities listed in the NPCR Logic Model and Program

Standards. The TCR consists of 54 staff, a central office and two regional

offices, located in Austin, Arlington, and Houston, respectively. Over

260,000 reports of cancer were received in 2017 from over 550 hospitals,

cancer treatment centers, ambulatory surgical centers, and pathology

laboratories located across the state.

Priority Populations— Texas provides a broad array of population

characteristics important to national cancer surveillance and thought to

impact effective cancer prevention, diagnosis, and treatment. It is diverse,

with no ethnic majority. Texas has the second largest Hispanic, third largest

Black, and fourth largest Asian populations. According to the U.S.

Department of Agriculture Economic Research Service, Texas also has the

largest rural population in the U.S. (3,048,574).

Beneficiary Impact from Programmatic Activities—TCR data are

used in all seven CDC data use categories for assisting clinicians,

researchers, policy makers, cancer patients, and others in making better

informed decisions, recommendations, and new discoveries to improve

cancer prevention, diagnosis, treatment, monitoring of clinical conditions,

delivery of care, survival, and quality of life at the patient, health care

system, and population levels. On average, over 80 peer-reviewed journal

articles are published annually using TCR data, and the TCR currently

supports 63 DSHS-IRB approved studies funded for $88,049,520. In 2017,

15,253 queries were completed through the Cancer-Rates Info query tool,

and 388 additional data requests were completed using TCR data.

Logic Model—The TCR will use the NPCR Logic Model as primary

guides for evaluation questions and the general approach to evaluation.

Texas Cancer Registry Evaluation Plan Page 3 of 13

Texas Cancer Registry Evaluation Plan Page 4 of 13

Evaluation Focus

The evaluation purpose, stakeholder interests, the feasibility of answering questions, and in turn, being able to use that new knowledge in

an actionable way for program improvement were all taken into consideration in determining the evaluation focus. As stated previously in

the Plan Overview, all evaluation questions are tied to and based on the inputs (resources), core registry and specialized project activities,

corresponding outputs, outcomes, and desired impacts described in the NPCR Logic Model. Specifically, questions will evaluate the TCR on the NPCR

2017–2022 Program Standards and Strategies. These strategies include program collaboration, external partnerships, cancer data and surveillance

(legislative authority; data collection, content and format; data quality and education; data submission; data use and monitoring; electronic data

exchange), community level interventions and patient support, health

systems change (linkages), and program monitoring and evaluation (data completeness, timeliness, and quality).

Plan for Collecting Data

Methods—Both quantitative and qualitative data collection methods

will be used in conducting the TCR program evaluation, and these methods will align and vary according to the program strategy and grantee activity.

Web Plus, Prep Plus, CRS Plus, and eMaRC Plus, SAS, Tableau, Access, and Excel will serve as the primary software applications used for collecting

quantitative performance measure data used in the evaluation. Qualitative data will be collected through notes and captured using Survey Monkey,

Word, and Excel.

Relevant Indicators and Data Sources—The following relevant,

credible indicators and data sources will be used for the TCR evaluation: • TCR compliance, timeliness, completeness, and data quality reports

(for ongoing evaluation of timeliness, completeness, and data quality) • TCR passive and active case finding and data quality audit reports

• TCR business process maps • TCR data use and dissemination reports

• National quality assurance studies (SEER EOD, TNM Reliability Study) • NPCR data quality audits

• NPCR Hospital, Pathology Laboratory, and Physician Reporting Progress Report (to assess status and improvements in electronic case

capture) • NPCR Program Evaluation Instruments

• NPCR and North American Association of Central Cancer Registries (NAACCR) annual data submission data quality indicator and timeliness

reports (e.g., the NPCR-CSS Data Evaluation Reports, to assess NPCR

standards for data submission)

Texas Cancer Registry Evaluation Plan Page 5 of 13

• NPCR work plan interim and annual progress reports (includes measures for effective collaboration with NCCCP, NBCCEDP, and other

chronic disease programs, such as Tobacco Control) • Special studies presented at international/national meetings (e.g.,

NAACCR Annual Conference, National Cancer Registrar’s Association Annual Conference)

• Articles in the scientific literature (e.g., Journal of Registry Management)

The TCR will also ensure that all essential business functions are included

(see the Appendix, Texas Cancer Registry Work Flow Diagram, that documents these functions).

Data Collection Responsibilities—Data collection responsibilities will

be shared across TCR management. The TCR Epidemiology Group serves as

the lead for developing and maintaining the majority of current compliance, timeliness, completeness, data quality, and data dissemination/use reports.

However, the Operations, e-Reporting, and Quality Assurance Managers, team leads, and staff provide input when changes or additional information

in these reports is needed. Quality Assurance and Registry Operations Managers are primarily responsible for participating in and evaluating the

results for national quality assurance studies (e.g., SEER EOD, TNM Reliability Studies), TCR, and NPCR data quality audits. The Branch Manager

is the lead for the NPCR Program Evaluation Instruments, as well as grant interim and annual progress reports.

Analysis and Interpretation

A critical component for this evaluation plan is ensuring that

conclusions are justified, not only using accurate credible data sources and evidence, but also accurate analysis and interpretation. If appropriate

process, analysis, and interpretation are not used, the evaluation could in fact be harmful, due to inaccurate results and faulty conclusions. The

process for drawing appropriate, data-based conclusion includes engaging the right stakeholders at the right time, using standard high quality credible

data sources and methods of analysis. Analyses will focus on attempting to identify important patterns of evidence (e.g., certain types of reporters with

a high percentage of certain missing information), and isolating important findings (e.g., looking at each criterion used to determine NPCR’s 12 and 24-

month data standards). Analysis and interpretation responsibilities will be shared across TCR

managers. Primary responsibility for analysis and compilation will reside with

the TCR Branch and Epidemiology Managers. Primary interpretation responsibilities will vary according to the strategies and activities being

evaluated, and the stakeholder expertise. For example, Registry Operations,

Texas Cancer Registry Evaluation Plan Page 6 of 13

e-Reporting, and Quality Assurance managers who are Certified Tumor Registrars are the most qualified and appropriate subject matter experts to

review, analyze, and interpret national data quality study results and TCR/NPCR audit findings.

Key external program partners will also participate in the TCR program evaluation analysis and interpretation processes. The Texas Health and

Human Services Commission and Department of State Health Services have quarterly Coordination Meetings that include the Texas Comprehensive

Cancer Control Program (TCCCP) and Breast and Cervical Cancer Early Detection Services Program (BCCS). These meetings provide a regularly

recurring opportunity for the TCR to share analyses and request input on interpretation on evaluation questions and activities important to the

collaborations with their programs. The Advisory Committee to the Texas Cancer Registry (ACTCR), as

well as the Cancer Alliance of Texas (CAT) include a broad group of primarily

external stakeholders with expertise that can assist primarily with multiple aspects of the evaluation, including interpretation. The TCR convenes the

Advisory Committee at least twice annually and communicates with members monthly on items of interest and TCR updates. Stakeholders

represented on ACTCR Members include entities external to the DSHS such as the American Cancer Society, Texas Medical Association, Texas Hospital

Association, and Texas Tumor Registrar’s Association, and academic research institutions. They also include entities within the DSHS such as the

Health Promotion and Chronic Disease Prevention Section which administers the Comprehensive Cancer Control Program and the Texas Health and

Human Services Commission’s Breast and Cervical Cancer Services program. The full membership listing can be found at

https://www.dshs.texas.gov/tcr/actcr-members.aspx.

Dissemination and Use of Findings

To best ensure the use of TCR Program Evaluation findings and share

lessons learned, evaluation results must be shared using effective

communication strategies with the appropriate audience and using a variety

of formats. Responsibilities for dissemination and use of findings must be

clearly defined, and audience feedback considered.

Communication Strategies—Disseminating information on the

program evaluation plans, progress, preliminary, and final results to

stakeholders will use tailored communication strategies that will most

appropriately meet their particular needs. All TCR communication must be

transparent and accurate to foster trust in the evaluation process,

conclusions, and recommendations for improvement.

Texas Cancer Registry Evaluation Plan Page 7 of 13

Formats and Audience—The TCR will rely primarily on written

formats for evaluation plan dissemination and will tailor the formats and

content for the different stakeholder audience. The TCR will engage in

dissemination activities, including but not limited to:

• Emails to TCR staff, DSHS leadership, TCCCP and BCCS collaborators,

cancer reporters, data requestors, the Advisory Committee to the

Texas Cancer Registry, and the Cancer Alliance of Texas

• Updates to TCR staff and DSHS leadership at various regularly

occurring staff meetings (recurrence ranges from bi-weekly to bi-

monthly)

• TCR newsletter articles primarily targeted for cancer reporters

• Web page updates primarily targeted for TCR staff, cancer reporters,

and data requestors

• Internal Share Point updates primarily targeted for internal DSHS and

TCR staff/leadership

• Discussion and presentations at the quarterly Collaboration meetings

with the Texas TCCCP and BCCS programs

• Discussion and presentations at Advisory Committee to the Texas

Cancer Registry and Cancer Alliance of Texas meetings

• Progress reports to NPCR staff

Dissemination Responsibilities—As with data analysis and

interpretation, dissemination responsibilities will be shared across TCR

managers. Primary responsibility will reside with the TCR Branch and

Epidemiology Managers. Responsibilities will be shared according to the

intended audience, format, and internal TCR manager/staff/stakeholder

expertise and responsibilities within the TCR. For example, web updates are

mainly the responsibility of the Epidemiology group, but national data quality

study-related activities included in the evaluation plan require CTR expertise

from the Operations, e-Reporting, and/or Quality Assurance areas.

Audience Feedback and Action Steps—In order to maximize the

effectiveness of this program evaluation, mechanisms must be in place for monitoring and documenting audience feedback and following-up with any

resulting action steps. Feedback is the communication that occurs among all stakeholders engaged in the evaluation process. CDC program evaluation

guidance stresses that giving and receiving feedback creates an atmosphere of trust among stakeholders and keeps an evaluation on track by letting

those involved stay informed regarding how the evaluation is proceeding.4 TCR management will ensure that all primary program evaluation

participants, users, and other stakeholders have the right and ability to comment on decisions that impact actions steps and might affect the

likelihood of obtaining useful information. Stakeholder feedback is an

Texas Cancer Registry Evaluation Plan Page 8 of 13

integral part of evaluation, particularly for ensuring use. Obtaining feedback will be encouraged by keeping stakeholders regularly updated using the

variety of formats and mechanisms described in the Formats and Audience section, holding periodic discussions during each step of the

evaluation process, and routinely sharing interim findings, provisional interpretations, and draft reports. Feedback documentation and action steps

for the TCR program evaluation will be documented in specific discussion/meeting minutes, emails, and notes, saved in a specific TCR

Program Evaluation shared folder accessible to TCR management who have the primary responsibility in carrying out the evaluation.

Texas Cancer Registry Evaluation Plan Page 9 of 13

Evaluation Timeline

The TCR evaluation and performance measurement plan for grant

Years 1–5 will follow the recommended timeline provided by the NPCR.

Texas Cancer Registry Evaluation Plan Page 10 of 13

Conclusion

In conclusion, the TCR Program Evaluation seeks to systematically and

transparently evaluate TCR program inputs, grantee activities, outputs,

outcomes, and impacts, as described in the NPCR Logic Model. The

evaluation will be used by stakeholders to improve TCR efficiency,

effectiveness, and accountability in carrying out its core and specialized

activities. The subsequent improvement in these activities will in turn result

in improved short, intermediate, and long-term outcomes, as well as positive

program impact.

Cited References

1 Centers for Disease Control and Prevention. (1999, September).

Framework for program evaluation in public health. Morbidity and Mortality

Weekly Report, 48(RR-11). Available from:

ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf

2 NPCR Evaluation Plan Guide For DP17-1701 NPCR Grantees Provided by

the CDC Division of Cancer Prevention and Control Cancer Surveillance

Branch

3 Quick Facts United States. U.S. Census Bureau. Internet:

https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xht

ml?pid=PEP_2017_PEPANNRES&src=pt . Accessed 1/29/2018.

4 Developing an effective evaluation plan. Centers for Disease Control and

Prevention, National Center for Chronic Disease Prevention and Health

Promotion, Office on Smoking and Health; Division of Nutrition, Physical

Activity, and Obesity, 2011. Available from:

http://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf/

Texas Cancer Registry Evaluation Plan Page 11 of 13

Appendix