Properties of Sections ERT 348 Controlled Environmental Design 1 Biosystem Engineering.
Tutorial on Medical Image Retrieval - IRMA€¦ · •IRMA code mono-hierarchical, multi-axial...
Transcript of Tutorial on Medical Image Retrieval - IRMA€¦ · •IRMA code mono-hierarchical, multi-axial...
Tutorial on Medical Image Retrieval- IRMA -
Medical Informatics Europe 2005
28.08.2005
2©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
indexingindexing
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
Information Processing (IRMA)
3©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
indexingindexing
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
Information Processing
4©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Arbitrarily taken from routineArbitrarily taken from routine
•• authenticauthentic
•• Manually classifiedManually classified
•• reliablereliable
Images: IRMA Reference Database
5©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Standardized nomenclatureStandardized nomenclature
•• Systemized Nomenclature in Medicine (SNOMED)Systemized Nomenclature in Medicine (SNOMED)
•• Medical Subject Headings (Medical Subject Headings (MeSHMeSH))
•• Unified Medical Language System (UMLS)Unified Medical Language System (UMLS)
•• Digital Imaging Communications in Medicine (DICOM)Digital Imaging Communications in Medicine (DICOM)
•• Example DICOMExample DICOM
•• Tag (0018,0015) Tag (0018,0015) Body Part ExaminedBody Part Examined, 26 valid entries, 26 valid entries
•• ExtremityExtremity, , HandHand, , ArmArm, ..., ...
•• ProblemsProblems
•• coarsecoarse, , ambiguousambiguous,, non non--hierachicalhierachical
Images: Consistent Categorization
6©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• IRMA codeIRMA code
mono-hierarchical, multi-axialmono-hierarchical, multi-axial
•• Technique, Direction, Anatomy, Technique, Direction, Anatomy, BiosystemBiosystem
(TTTT-DDD-AAA-BBB)(TTTT-DDD-AAA-BBB)
•• Example: 1121-127-720-500Example: 1121-127-720-500
•• rradiographyadiography, plain, analog, overview, plain, analog, overview
•• ccoronaloronal, AP, supine, AP, supine
•• aabdomenbdomen, middle, middle
•• uuropoetic ropoetic systemsystem
Images: Consistent Categorization
7©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
indexingindexing
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
Information Processing
8©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
indexingindexing
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categoriescategories
categorizationcategorization
Information Processing
9©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
Categorization of Medical Images
•• Pixel data Pixel data !! Semantical Semantical meaningmeaning
•• X-X-ray imageray image
of skullof skull
•• laterallateral view view
•• good qualitygood quality
•• no obviousno obvious
pathologypathology
•• ......
•• ApplicationsApplications
•• ddigital igital radiologyradiology & PACS & PACS
•• ccomputeromputer-aided diagnosis (CAD)-aided diagnosis (CAD)
•• ccontentontent-based image retrieval (CBIR)-based image retrieval (CBIR)
10©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Pixel data Pixel data !! Few numerical values Few numerical values
3.2783.278
1.4261.426
0.4500.450
......
3.263.26
•• FeaturesFeatures
•• colourcolour
•• texturetexture
•• shapeshape
Categorization: Global Features
11©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• SemanticalSemantical meaning meaning "" Few numerical values Few numerical values
3.2783.278
1.4261.426
0.4500.450
......
3.263.26
•• X-ray imageX-ray imageof skullof skull
•• lateral viewlateral view
•• good qualitygood quality
•• no obviousno obviouspathologypathology
•• ......
•• Common approachesCommon approaches
•• low number of categorieslow number of categories
•• IRMAIRMA approach approach
•• monomono--hierarchical coding schemehierarchical coding scheme
Semantical Gap
12©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Classified images at study dateClassified images at study date
•• 6,335 images, 401 different codes6,335 images, 401 different codes
•• MinimumMinimum class size class size
•• 1010 images images
•• Hierarchical code Hierarchical code ((TTxxTTxx--DxxDxx--AAxAAx--BxxBxx))
•• 6,231 images6,231 images, , 81 categories81 categories
•• Radiographs only Radiographs only (11xx-(11xx-DxxDxx--AAxAAx--BxxBxx))
•• 5,776 5,776 imagesimages, , 57 categories57 categories
•• Newer version of the database contains 10,000Newer version of the database contains 10,000
radiographs and was used for the ImageCLEFradiographs and was used for the ImageCLEF
2005 Evaluation2005 Evaluation
Categorization: Evaluation
13©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Texture: Tamura FeaturesTexture: Tamura Features
•• local features (multi-spectral)local features (multi-spectral)
•• histogram (384 bins)histogram (384 bins)
coarsenesscoarseness contrastcontrast directionalitydirectionality
•• DistanceDistance
•• Jensen-Shannon divergenceJensen-Shannon divergence
Texture
14©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Image FeaturesImage Features
•• rescaled imagesrescaled images
squaredsquared 32 x 3232 x 32 16 x 1616 x 16 8 x 88 x 8 4 x 44 x 4
•• DistancesDistances
•• ccrossross correlation functioncorrelation function
•• iimage distortion modelmage distortion model
•• tangent distancetangent distance
Images
15©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Pixel mappingPixel mapping
•• scans local neighborhood for best correspondencescans local neighborhood for best correspondence
•• allows local deformationsallows local deformations
•• ImprovementsImprovements
•• uuse image gradientsse image gradients
•• eevaluate local contextvaluate local context
Image Distortion Model
16©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Leaving-one-outLeaving-one-out
•• 6,231 images (all imaging techniques)6,231 images (all imaging techniques)
•• 81 categories81 categories
97.0 %90.1 %80.7 %82.3 %IDM
97.7 %86.6 %76.4 %76.1 %CCF
96.6 %80.4 %66.3 %66.4 %Tamura
within 10within 55-NN1-NNRepresentation
Results (Single Classifiers)
17©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Leaving-one-outLeaving-one-out
•• 5,776 images (radiographs only)5,776 images (radiographs only)
•• 57 categories57 categories
97.1 %90.0 %80.6 %81.8 %IDM
97.9 %86.4 %76.1 %75.4 %CCF
96.4 %79.3 %64.6 %64.5 %Tamura
within 10within 55-NN1-NNRepresentation
Results (Single Classifiers)
18©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• IDM + TamuraIDM + Tamura
•• Improvement (radiographs)Improvement (radiographs)
•• Correctness:Correctness: 81.8% 81.8% !! 85.0%85.0%
•• Within 5:Within 5: 90.0% 90.0% !! 92.8% 92.8%
95.2 %92.8 %85.0 %85.0 %Radiographs
(57 categories)
95.3 %93.0 %85.4 %85.5 %All images
(81 categories)
within 10within 55-NN1-NNSetup
Results (Parallel Combination)
19©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Absolute distance not evaluatedAbsolute distance not evaluated
•• Inhomogeneous category sizeInhomogeneous category size
•• High intra-category variabilityHigh intra-category variability
•• grouped codegrouped code
•• Low inter-category variabilityLow inter-category variability
•• misclassified body regionmisclassified body region
•• misclassified directionmisclassified direction
•• Collimation fields and shuttersCollimation fields and shutters
Reasons for Misclassifications
20©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• e.g.: radiographs only, min. 5 imagese.g.: radiographs only, min. 5 images
00
200200
400400
600600
800800
10001000
12001200
14001400
11 55 99 1313 1717 2121 2525 2929 3333 3737 4141 4545 4949 5353 5757
images images inin category category
category numbercategory number
recognition recognition raterate99.45 %99.45 %
99.84 %99.84 %
14.29 %14.29 %
0.00 %0.00 %
Inhomogeneous Category Size
21©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• e.g.: IRMA-code 1121-120-800-700e.g.: IRMA-code 1121-120-800-700
•• plain radiographyplain radiography
•• coronal, coronal, posterioanterior posterioanterior directiondirection
•• abdomenabdomen
•• musculoskeletal systemmusculoskeletal system
Intra-Category Variability
22©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Axial, Axial, craniocaudalcraniocaudal ((xxxxxxxx-310-xxx-xxx)-310-xxx-xxx)
•• OtherOther, , obliqueoblique ( (xxxxxxxx-410--410-xxxxxx--xxxxxx))
Inter-Category Similarity (Direction)
23©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Elbow (Elbow (xxxxxxxx-xxx-44x-xxx)-xxx-44x-xxx)
•• Knee (Knee (xxxxxxxx--xxxxxx--9494x-x-xxxxxx))
Inter-Category Similarity (Anatomy)
24©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Query imageQuery image
•• SystemSystem response response
Collimation Fields and Shutter
25©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• 85% correctness85% correctness
•• ~6,000 radiographs, ~80 categories~6,000 radiographs, ~80 categories
•• scaled representations, parallel combined classifierscaled representations, parallel combined classifier
•• 98 % correctness98 % correctness
•• within 10 most similar referenceswithin 10 most similar references
•• GlobalGlobal features applicable for many categories features applicable for many categories
if sufficient references are availableif sufficient references are available
Conclusion
26©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Thorax images in frontal view (1,278 images)Thorax images in frontal view (1,278 images)
•• Nearest neighborsNearest neighbors
Example: 99.5% Recognition
27©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
indexingindexing
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
Information Processing
28©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
indexingindexing
Information Processing
29©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• PreprocessingPreprocessing
•• categorization based on global featurescategorization based on global features
•• registration onto category prototypesregistration onto category prototypes
•• local feature extractionlocal feature extraction
(explosion of data volume)(explosion of data volume)
•• AbstractionAbstraction
•• significant reduction of informationsignificant reduction of information
Indexing
30©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Local FeaturesLocal Features
•• Each pixel Each pixel !! One feature vector One feature vector
•• Example: Example: Blobworld Blobworld featuresfeatures
polaritypolarity anisotropyanisotropy contrastcontrastgray valuegray value
Structured Analysis of Image Content
31©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Information abstractionInformation abstraction
•• image image pixels pixels regions regions blobs blobs
Structured Analysis of Image Content
32©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Hierarchical modelingHierarchical modeling
•• blobs blobs !! graph graph
(edge preserving) region growing(edge preserving) region growing
Structured Analysis of Image Content
33©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• ExampleExample
1
2
3
4
5
6
78
9
11
10
12
1
0
2
3
4
6
59
7
10
8
11
•• image image !! pixel pixel !! regions regions !! blobs blobs !! graphgraph
Structured Analysis of Image Content
34©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layerretrievalretrieval
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
indexingindexing
Information Processing
35©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layer
imagesimages
feature extractionfeature extraction
queryquery
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
categorizationcategorization
categoriescategories
indexingindexing
identificationidentification
retrievalretrieval
query resultsquery results
System Design
36©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• IRMA Database BrowserIRMA Database Browser
•• visual overview of all imagesvisual overview of all images
•• IRMA QueryIRMA Query
•• extended query refinementextended query refinement
•• IRMA Code DefinitionIRMA Code Definition
•• define IRMA code itemsdefine IRMA code items
•• IRMA Code EditorIRMA Code Editor
•• manual reference coding of imagesmanual reference coding of images
•• IRMA Code BrowserIRMA Code Browser
•• access images by IRMA codeaccess images by IRMA code
Exemplary Applications
37©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• German / English interface to code databaseGerman / English interface to code database
•• 738 entities in four axes738 entities in four axes
•• history logginghistory logging
Code Definition
38©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Reference categorization of imagesReference categorization of images
•• history logging of changeshistory logging of changes
currently over 15,000 history entriescurrently over 15,000 history entries
Code Editor
39©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• Browsing the reference database by IRMA codeBrowsing the reference database by IRMA code
•• e.g., e.g., anteroposterior anteroposterior chest radiographschest radiographs
(1123-111-500-000)(1123-111-500-000)
Code Browser
40©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• IRMAIRMA
•• general concept for medical image retrievalgeneral concept for medical image retrieval
•• any modality or body region (categorization)any modality or body region (categorization)
•• any level of details (hierarchical representation)any level of details (hierarchical representation)
•• successful evaluation of categorizationsuccessful evaluation of categorization
•• good results in the good results in the ImageCLEF ImageCLEF evaluationevaluation
Summary
41©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
•• IRMA is a joint ventureIRMA is a joint venture
•• Department of Medical InformaticsDepartment of Medical Informatics
•• Department of Diagnostic RadiologyDepartment of Diagnostic Radiology
•• Chair of Computer Science VIChair of Computer Science VI
•• FundsFunds
•• RWTH Aachen UniversityRWTH Aachen University
grant START 81/98grant START 81/98
•• German Research Foundation (DFG)German Research Foundation (DFG)
grants Le 1108/4-1, 1108/4-2, 1108/6grants Le 1108/4-1, 1108/4-2, 1108/6
!!http://http://irmairma-project.org-project.org
Acknowledgments
42©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layer
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
indexingindexing
retrievalretrieval
Information Processing
43©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
Retrieval = Similarity of graphsRetrieval = Similarity of graphs
•• QualitativelyQualitatively:: Visualization Visualization
•• QuantitativelyQuantitatively:: Graphmatching Graphmatching
Information Processing
44©2004 Hôpitaux Universitaires de Genève / RWTH Aachen
raw data layerraw data layer
registered data layerregistered data layer
feature layerfeature layer
scheme layerscheme layer
object layerobject layer
knowledge layerknowledge layer
imagesimages
feature extractionfeature extraction
query resultsquery results
queryquery
feature vectorsfeature vectors
blob-treesblob-trees
registrationregistration
RST-parametersRST-parameters
feature selectionfeature selection
identificationidentification
categorizationcategorization
categoriescategories
indexingindexing
retrievalretrieval
Information Processing