Automatic Formalization of Clinical Practice Guidelines

Matthew S. Gerber and Donald E. BrownDepartment of Systems and Information Engineering

University of Virginia

James H. HarrisonDepartment of Public Health Sciences

University of Virginia

2

• Many treatment options – what to do?

Clinical Practice GuidelinesB

enef

its /

cos

ts

Evidence quality

Randomized clinical trial: beneficial

Expert opinion: might be beneficial

Meta-analysis: usually beneficial

Strength

Recommended

Should consider

Might consider

3

Clinical Practice Guidelines

• Development– Expert synthesis of current evidence– Example from heart failure:

4

Clinical Practice Guidelines

• Expected outcomes– Evidence-based clinical decision aid– Reduction in cost and treatment/outcome variation– Improvement in patient health

• Challenges– A guideline for any occasion

– Guidelines change periodically – Lengthy (HFSA CPG is 259 pages)

Total (NGC) 2,269

Cardiovascular diseases 486

Heart failure 152

5

Clinical Decision Support Systems

• Goal: deliver CPG knowledge at point of care• Alleviate burden on clinician• Problem: CPGs contain minimally structured text

Formalization is required

6

Traditional CPG Formalization

Knowledge engineersMedical experts

Knowledge representation

Knowledge management software (e.g., Protégé)

CDSS

CPGCPG

Automatic formalization

…

Retrospective analyses

The Big Picture

7

Structured knowledge

Medical decision support

?

EndocrineInfections

…Cardiovascular

NLP

8

Data Collection

• Yale Guideline Recommendation Corpus– Hussain et al. (2009)– 1,275 recommendations– Representative sample of domains and rec. types

“Oral antiviral drugs are indicated within 5 days of the start of the episode and while new lesions are still forming.”

– Simplifications• Delimited recommendations• No inter-recommendation dependencies

• Random sub-sample of YGRC (n=200)

9

Recommendation Representation

• SNOMED-CT– Medical concept ontology– Broad coverage

Keywords ? Asbru, etc.

Fidelity: Low High

Automation: Trivial Impossible

10

Recommendation Representation

(Sundvalls et al., 2012)

11

Recommendation RepresentationPrimary Secondary Example recommendation

ACTION EVALUATION [computed tomography CT] should be used

PROCEDURE [red blood cell transfusion] is appropriate

…

EVIDENCE STRONG [it has been shown to reduce the occurrence of NTDs]

WEAK/NONE [there is insufficient evidence]

MODALITY OBLIGATORY computed tomography CT [should] be used

NEVER oral risedronate [should not] be used

OPTIONAL physician [may] choose

AGENT [physician] may choose

MORBIDITY prevent [preeclampsia]

POPULATION [obese women with gestational diabetes mellitus]

PURPOSE is used [to prevent osteoporotic fractures]

TEMPORAL [initial] treatment

SNOMED-CT CONCEPT: 129265001

12

Recommendation Annotation

• Task: manually identify representational elements within recommendations

• Example

Diuretics are recommended for patients with heart failure.

[DRUG Diuretics] are recommended for [POPULATION patients with [MORBIDITY heart failure]].

13

Methods

• Natural language processing• Supervised classification• Per-recommendation pipeline

1. Syntactic parsing

2. Parse node classification

3. Post-processing

14

Methods: (1) Syntactic Parsing

• Constituency parser (Charniak and Johnson, 2005)

15

Methods: (2) Parse Node Classification

• Unit of classification: node• Multi-class logistic regression• Example: 1 positive, 17 negative• Actual

– 12K nodes– 10 classes (primary)

16

Methods: (2) Parse Node Classification

• Linguistic features– Word stems under node– Syntactic configuration of node– …

17

Methods: (2) Parse Node Classification

• Learning– Forward feature selection– Per-class costs (LibLinear)

18

Methods: (3) Post-processing

• Remove duplicates• Other possible issues

– Conflicts– Embedding

19

Evaluation Results

Element # Precision (%) Recall (%) F1 (%)

ACTION 301 74.5 26.2 38.7

MODALITY 158 71.9 73.0 72.4

POPULATION 140 83.7 56.6 67.5

TEMPORAL 53 28.6 1.1 2.2

TRIGGER 45 81.7 93.3 87.1

PURPOSE 43 58.3 16.3 25.5

EVIDENCE 38 83.3 13.2 22.7

AGENT 37 76.0 47.6 58.5

MORBIDITY 19 50.0 10.5 17.4

Overall 834 75.3 41.7 53.7

• 10-fold cross-validation

20

Discussion

• High variance across classes• Alternative strategies

– Identify more informative features– Change the model formulation– Annotate more data

21

Conclusions

• CPGs are an important knowledge source• Difficult to use within CDSS• Prior CPG formalization

– Manual– Automatic for specific domains / recommendations

• Our contributions– SNOMED-CT representation– Manually annotated recommendation sample– Statistical NLP model / evaluation

22

Future Work

• Refined representation

• Model formulation

• Feature engineering

• Controlled natural language

23

Questions?

• References1. Charniak, E. & Johnson, M. Coarse-to-fine n-best parsing and MaxEnt

discriminative reranking. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, 2005, 173-180.

2. Hussain, T.; Michel, G. & Shiffman, R. N. The Yale Guideline Recommendation Corpus: A representative sample of the knowledge content of guidelines. I. J. Medical Informatics, 2009, 78, 354-363.

3. Fan, R.-E.; Chang, K.-W.; Hsieh, C.-J.; Wang, X.-R. & Lin, C.-J. LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research, 2008, 9, 1871-1874.

4. Sundvall, E.; Nystrom, M.; Petersson, H. & Ahlfeldt, H. Interactive visualization and navigation of complex terminology systems, exemplified by SNOMED CT. Studies in health technology and informatics, IOS Press; 1999, 2006, 124, 851.