Annotation and Image Markup:

Take AIM at Images!

David S. Channin M.D.Associate Professor of Radiology

Chief, Imaging InformaticsNorthwestern University

Feinberg School of MedicineDepartment of Radiology

Daniel Rubin M.D.Clinical Assistant Professor of Radiology

Research ScientistStanford University

Department of Medical Informatics

Principal Investigators:

Pat Mongkolwat PhD, Vladimir Kleper, Kaustubh Supekar

What is an Image Annotation?

Annotations are explanatory or descriptive information, generated by humans or machines, directly related to the content of a referenced

image or images

Image annotations let us capture information about the meaning of pixel information in images such that similar meaning in other images can be

found and used.

What is an Image Markup

An image markup is the graphical symbols associated with an image and optionally with one or more annotations of that same image.

An Image

An Image and an Image Markup

An Image, an Image Markup and an Annotation

The pixel at the tip of the arrow [coordinates (x,y)] inthis image[DICOM: 1.2.814.234543.23243]represents the Ascending Thoracic Aorta[SNOMED:A3310657]

What’s the problem?

• No agreed upon syntax for annotation and markup.

• No agreed upon semantics to describe annotations.

• No standard format (DICOM, XML, HL7, etc.) for

annotations and markup.

Why is this important?

What is the solution? The caBIG AIM Project

• An ontology of image annotations• An ontology of image markups

• An ontology defines concepts in a domain and the relationships between those concepts

• Use of controlled terminologies• EVS, RadLex, SNOMED, LOINC, UCUM

• A set of translatable, standards-based representations

The Deliverables

• An ontology of both annotation and markup• A UML model of AIM• Software to instantiate AIM XML• Software to generate DICOM S/R AIM Object (from AIM

XML)• Software to generate HL7 CDA (xml) (forthcoming)• An XIP Builder SceneGraph and XIP modules to

validate and transcode AIM annotations (ANIVATR)• An AIM library to create and render AIM annotations

and markups• (Integrated into the eXtensible Imaging Platform)

The Deliverables

• 1Q 2007• Project Management Plan• Communication Plan• Draft IOSA Document• Draft Reconciliation Document• Final IOSA Document• Final Reconciliation Document• Quarterly Progress Report

• 2Q 2007• Draft Mechanism for Free Text• Draft Mechanism for Arbitrary Calculation• Final Mechanism for Free Text• Final Mechanism for Arbitrary Calculation• Sample InstancesSample Instances• Quarterly Status Report

• 3Q 2007• Software DemonstrationSoftware Demonstration

• Import, validate, create, transcode, etc.• Quarterly Status Report

• 4Q 2007• RSNA 2007 Demonstration• Quarterly Status Report• Lessons Learned

Logical Diagram AIM

Logical Diagram AIM enlarged

What is an AIM Annotation?

• One creator (machine or human) at one instant in time• Annotating one series of images from one patient• 9 Types of Image Annotations, 5 Types of Annotation of Annotations• An annotation is assigned a unique identifier and can be assigned a name• An annotation has one or more anatomic entity that may be related to each other• An annotation has one or more imaging observations

• An imaging observation has more or more characteristics• An annotation has one or more geometric shapes

• Geometric shapes are: point, multipoint, circle, ellipse, polyline• An annotation has calculation(s)

• Calculations have result(s)• Results have data, dimensions, ordinates

• Calculations can be defined or arbitrary• Text can be intended for presentation (Text Annotation)

• Or not (intended just for reference) (Comments)

API

• Give me the XML schema for an XYZ AIM

• (Driven by Protégé)

• Constrain vocabulary choices

• Set the …

• Get the …

• Save As..

• DICOM S/R

• XML/CDA

What does this look like in practice?

Select anatomic entity

Select imaging observation

Select calculation type

Select output type

Summary

• Image annotations and markups are critical to “tagging” content in medical images• Such that images containing similar content can be

identified• The AIM project will deliver an information model and

encoding standards for the structure and content of image annotations

• AIM annotations will be critical components of future image based research.