An Intelligent Visual Task System For Lateral SkullX-ray Images

D.N. Davis and C.J. TaylorWolfson Image Analysis Unit

Department of Medical BiophysicsUniversity of Manchester

Oxford RoadMANCHESTER M13 9PT

This paper describes research into structured,knowledge-based image interpretation. An integratedframework has been developed, within which most tasksassociated with the automatic interpretation andanalysis of lateral skull X-ray images (cephalometry)can be performed. A model-based image analysissystem makes use of a blackboard architecture andmultiple knowledge sources. Its performance comparesfavourably to previously published attempts to automatecephalometric analysis.

Cephalometric measurements, from lateral skullradiographs, can be of help to orthodontists in decidingupon the nature of any necessary orthodontictreatment, or in defining standards for classifyingcraniofacial growth. Manual and interactive methods ofperforming the cephalometric analysis areacknowledged as error prone [1,3]. An automatedanalysis, using computer vision techniques, that couldprovide systematic and accurate results would be ofbenefit. Such a system is being developed usingknowledge-based methods.

There are a number of different tasks that need to betackled in developing such a system. Reliable methodsneed to be implemented to segment individualanatomical features and landmarks. These must be ableto extract low contrast features in possibly noisy imageareas. We have previously reported one such systemwith promising results [2], though modifications arerequired to allow a greater use of feature appearancemodels. In order to control the segmentation system it isalso necessary to generate image areas within whichspecific features are expected to be located Anappropriate declarative knowledge-source, withassociated models, offers a satisfactory approach to theproblem. The full system must be able to demonstratevarious forms of behaviour. A dynamic means oforganising image interpretation tasks, allowing fordifferent analyses of the same image, must be present.Alternative solutions to a current task ought to beallowed. This will entail the use of some constraintapplication method that allows the most viable (iecorrect) interpretation to be produced. Blackboard

291

architectures, as developed for solving other complexproblems [4] requiring the use of a number of forms ofknowledge, offer a suitable framework for organisingthese tasks. Furthermore it would be sensible to developthe blackboard system within an integrated frameworkthat permits the use of model building facilities, andenables the segmentation system to be used in a standalone mode. Such facilities can be used to ensure thevalidity of newly developed models.

Background

A number of automated cephalometric systems havebeen developed [1,9,12]; the most successful being theParthasarathy et al [12] algorithmic implementation ofthe knowledge-based system of Levy-Mandell et al [9].This system improves on the performance of the originalby incorporating lessons that became apparent duringthe development of the knowledge-based system. TheLevy-Mandell system uses an elementaryknowledge-based approach that works only onrelatively good quality radiographs. It typifies amethodology which has often been applied to medicalimage interpretation. A high level module is used tomodel an application, so that particular declarativefeatures can be matched to the results of a rigidlydefined low-level segmentation module. Thesegmentation module typically convolves a low-pass, orsmoothed image, with a particular edge operator toproduce a list of edge segments. Domain knowledge,usually represented in production rule format or as astructured set of frames, is then applied to interpret thelist of line segments.

While such knowledge-based systems have achievedsome measure of success the results are heavilydependent upon the preprocessing and edge detectionprocesses. When the chosen operators fail, typicallybecause assumptions about the nature of the image areill founded, the whole system breaks down and acts in aquite unintelligent manner. It has been suggested[5,8,14] that alternative modes of segmentation,involving multiple low-level knowledge-based modulesmay provide the adaptability required for medical imagesegmentation. BMVC 1990 doi:10.5244/C.4.52

Nazif and Levine [11] demonstrated that heuristics,typically used in imperative segmentation systems, canbe expressed, to good effect, at the production rulelevel. Their system perhaps suffers from attempting toprovide a too general solution, leading to ambiguities inimage interpretation. The SIGMA system [9] makesuse of multiple knowledge sources; each with a clearlydesignated task. It can be seen as an advance on theNazif system, with modules for segmentation, localitydefinition and model access. However the nature ofbiomedical images and the biological variabilityassociated with even the most major of features requiremore sophisticated constraints and greater subtlety insystem control than industrial, machined part or evenaerial image systems [14].

Overview Of The Image Reasoning Tool

Our initial experiments with lateral skull X-ray imagesdemonstrated that no sequence of low-level imageprocessing could be guaranteed to provide cues tosupport the presence of a given feature or landmark.

(

B

LACKBOARD

MKSC ) M

ISS

ALS

CLS

FCS

PCS

OCS

CMS

CCS

MMS

MSS

TPS

TDS

Segmentation system

Location system 1

Location system 2

Constraint system 1

Constraint system 2

Constraint system 3

Ceph measurement

Measurement constraint

Model data gathering

Model system

Task panel management

Task definition system

ulti Knowledge Source Control

Figure 1 The Stylised Blackboard Architecture

The variations in image quality, both digital and on theX-ray film, and in the subtly varying morphology, shapeand visual definition of the biological features sought,

suggested a more adaptable approach to the problemwould be required.A model of the lateral view of the head could bedeveloped, using data gathering modules, that wouldencompass declarative object definitions and statisticsfor object feature appearance and expected areas ofinterest within an image. The principle function of sucha model would be to enable the image interpretationsystem to reason about application goals using evidencefrom the model, and any given images or other sourcesof information, in order that these goals be satisfied. It isenvisaged that at various stages in the chain ofreasoning, evidence from the chosen source imagewould be required, so that hypotheses regarding thelocation of selected features in the source image couldbe accepted or rejected. The architecture adopted mustoffer the adaptability and necessary structures withwhich to co-ordinate the variety of reasoning tasksrequired.

Figure 1 depicts the blackboard architecture which hasbeen adopted in this work. Each of the modularknowledge sources, whether declarative or imperative,is designed to undertake specific classes of task. Forinstance the intelligent segmentation system (ISS)attempts to find image feature objects given anapproximate location. Two location systems, both rulebased and using the same generic inference engine, areused to generate (ALS) and constrain (CLS)expectation windows for image features. The task panelmanagement system (TPS) accesses areas of theblackboard defining found, found and rejected,searched for but not found, and inactive objectstogether with a task specification structure, so that acurrently active object may be defined. The taskspecification structure, or task panel hierarchy, isproduced by the task definition system (TDS). Thisframe-based hierarchy is generated from a set ofunstructured frames defining object dependencies. TheTASK frame, or hierarchy root, becomes the object theblackboard system must ultimately satisfy. By findingobjects further down the hierarchy frames areinstantiated. The various levels reflect how frames aredefined on each other and are not dependent upon theclass of frame for any member at any task hierarchylevel.

Object Models

A variety of models are present and available for use inthe blackboard system. Some are purely declarative,whilst others rely on statistics generated from datagathering modules accessible from within the AITOOLsystem. A combination of these two classes of models isallowed.

The image feature appearance model mixes bothdeclarative and statistical knowledge. Hence we candeclaratively define the curvature of some modelprimitive as Convex, meaning the internal aspects of thiscurve are of a higher image intensity. We can also

292

generate population statistics from training exampleswhich circumscribe the nature of the curvature andquantify profile characteristics along the curve.Complex features can be built from more fundamentalobjects in the manner suggested by Tsotsos [12]. Thisallows models to be built from a base of perceptual (andmodel) primitives, in a constrained way alongrepresentational axes, to form fully specifiedcombinations.

For objects likely to require image segmentation animage location model is provided for use by theautomatic location system ALS. Rectangular imageregions, or windows, can be defined for all features ofinterest. Statistics can be generated not only for the size(width and height) of these windows, but also for theabsolute and relative positions of their centre points.

Further models describe statistical relationshipsbetween pairs of features, typically cephalometric pointsor feature location window centre points. Angular anddistance parameters can be generated from datagathered for the feature windows or from windows fittedto previously found features. This relative locationmodel can be used in either location generation by theALS or in constraint propagation (FCS and PCS). Asimilar statistical model exists for all aspects of trianglesdefined on points of window centres, or cephalometricpoints, and is used in constraint propagation (OCS andCCS).

The Reasoning Cycle

A default blackboard is constructed by calling theblackboard system and image references are written tothe blackboard when selected. The system keeps adisplayable track of generated windows, and all found,and rejected features. Task selection causes task framesto be loaded onto the blackboard. The taskspecification system will then create a blackboard taskpanel from the list of frames or references, within theframes, to objects defined within any of the declarativemodels. Any frames not used are removed, and anyframes containing slots referencing frames not availableare redefined without those slots. At this stage allknowledge sources, including the blackboard are fullyinitialised and image analysis can commence.

The blackboard control system (MKSC) commences ahypothesis and test reasoning cycle, calling upon verygeneral functions in a specified order to interact with theblackboard. Any one of these general functions cancause further, more specific reasoning cycles to beinitiated. Objects are selected and predictions, orhypotheses, are generated for the selected object.Candidate objects are found, then tested for validity. InThe blackboard can then build a reasoning profile ofwhat objects and systems have been used, what is left tobe done, what has been done but proved to beunsuccessful etc. In this way any attempt at finding anobject which has already been unsuccessfully searched

for, can use more exacting knowledge in successiveattempts.Objects, and current task panel levels, are specified bythe task management system (TMS). An objectbecomes active once selected as the blackboard currentobject. It remains active until a suitable candidate (s) isaccepted by all constraint systems or until objectingconstraint systems are overridden. Objects are chosenon the basis of a cost function related to statisticalvariance. Higher level objects, eg angles, are attributeda cost value related to the objects they are definedupon.

Intelligent Segmentation

An initial implementation of the ISS has been describedelsewhere [2]. The current ISS is much improved incomputational efficiency and in its functionality. Thesystem now makes no attempt to back chain, but usesmodel-directed objective-specific forward chaining,with an improved graph-based backtrackingmechanism, to find required objects. The rule baseplaces greater reliance on the interaction between imageparameters, as gathered by image processing toolswithin the ISS, and the new format feature appearancemodels to control, its actions.The production rules,within the segmentation knowledge-base, now takesaliency values and a greater wealth of rule syntax isprovided. The image processing and segmentationoperators are similarly improved, with greater relianceplaced on model parameters. The system allowssegment splitting and combination, duringsegmentation, in a model-based manner derived fromthe Nazif and Levine system.

Segmentation strategy borrows the Recognition andIdentification analogy used in human perception study[7]. Once the initial selection of binary cues have beenextracted from the source image, they are adapted andrejected until the set of objects are statisticallyrecognisable as members same class as the modelledfeature. Identification allows the system to select themost suitable candidates.

Object Expectation WindowsThe automated location system (ALS) is capable ofspecifying areas of an image likely to contain an object,utilising two of the models introduced above within adeclarative knowledge source. This it does in one of twomanners; the method used is dependent upon thenumber of features already found.If less than two image features are found, the systemaccesses data defining mean and variance for awindow's centre point, and convolves this with the meanand variance for window size. The generated window iswritten to the blackboard as the expected location area.However if the current feature is not a member of aspecified subset of features no location for the featurecan, as yet, be given. Typically in such a case, theblackboard task object would be reselected at a later

293

stage in the image analysis, allowing the second mode oflocation window generation to be used.

point 1

K\\\

point 2

0

\

1

y•IntersectionandHypothesisedWindow

Figure 2. Location Propagation

Where more than two features have already beenfound, the system uses further geometric constraints togenerate areas of intersection between pairs of foundfeatures and the required feature. Where a nullintersection is generated, the system backtracks toproduce the most viable non-null intersection. Theresult of this constraint propagation can be convolvedwith location window size statistics to produce anexpectation window of use to the rest of the blackboardsystem (Figure 2).

Cephalometric MeasurementsA relatively straightforward imperative system produceslines, ratios of distances and angles according to thespecification of the currently selected task panel frame.Angles can be defined upon pairs of lines, line and point(or point and line), and three or four points. Displaysallow the user to verify the correct measurement hastaken place.

Constraint SystemsThe three constraint systems can only be activated byfinding a candidate, or series of candidate, object(s).The FCS and PCS systems use similar mechanisms toverify that a new feature, or point, are statisticallycontingent with already known features. An insufficientfit causes an object to be rejected. Where more thanone candidate is acceptable, the most suitable candidatecan be selected using all available statistics. The thirdsystem (OCS) works in a similar manner for higherorder objects (eg lines, angles) but can have quitedifferent consequences. If the current object residesbeyond statistical limits, the system can be caused toreappraise the validity of all found objects.

Initial ResultsThe AITOOL and multi-knowledge sourcedblackboard system have been implemented in SunCommon LISP with image processing modules writtenin Pascal. Initial results for a system designed to locatefeatures have been obtained with X-ray images ofvarying quality. Figure 3 shows the results of a successful

analysis. The display shows 12 found features, theirexpectation windows and 10 fudicial points definedupon the features.

•MM

Zt7'

Figure 3. Features found with locations.

The modified ISS, used in interactive mode, nowoperates with close to 100 per cent reliability. Theautomatic location system (ALS) is available as part ofthe blackboard system or in a simplifiedmulti-knowledge (ISS, ALS, FCS, CMS, MSS)sourced image interpretation system (MKSS) whichfinds features in a predefined order, and has no realtask definition knowledge. Results using this system varywith image quality and the probability levels applied. Onaverage it performs with about 80% accuracy, but canrun at 100% on good quality images with high contrastfeatures. An initial implementation of the blackboardsystem is running but requires further development,particularly with regard to the handling of probabilities,constraint values and image interpretation alternatives.Some form of truth maintenance system, orprobabilistic belief system is necessary to allow thegeneration of the most viable image interpretation.

DiscussionPresently the blackboard system is running in anincomplete form, but it does drastically improve on theperformance of previous knowledge-based approachesto automating cephalometric analysis. The bestprocedural solution [12] runs at a slightly higher level ofaccuracy, but on a narrower range of test images. Thepresent results were gathered using test images thatincluded non-aligned images where double, orincomplete feature stimuli were present, and very poorquality images where a human observer would havedifficulty in finding the sought feature.It should be realised that successful imageinterpretations are achieved by allowing the system toconsider alternative, competitive interpretations. Thisoccurs not only at the gross level where alternative sets

294

of candidate feature combinations exist, but also wherealternative interpretations of segments are considered.For instance, it may be possible to generate a set of edgesegments in a particular area of the image when seekingthe forehead contour. These segments aresystematically broken down and combined, usingmodel-based parameters, to offer the greatest possiblerange of segments. The segmentation phases ofrecognition, and identification then allow the validmembers of this extended segment set to be selected. Ata higher level, within the blackboard system's constraintmechanisms, these viable alternative foreheads areconsidered not only in terms of how well they fit thesegmentation appearance model but also, on how thesealternatives fit with location models constraining thejuxtaposition of different features within an image.Similarly a candidate forehead may be preferentiallyselected upon evidence related to cephalometric pointsdefined upon the forehead.

It is possible in low contrast, or poorly aligned imagesfor segmentation to initially fail for particular contours.In previous attempts at automating the cephalometricanalysis [9,12] this would lead to an imageinterpretation failure. An initial failure to find a feature,in the present system, causes the selection cost value ofthat feature to be increased. This allows other featuresto be found in preference to the failed feature. Thesefound features will then be used in applying greaterconstraints to the finding of the failed feature when it isreselected as the currently required object. Where suchactions still fail, the blackboard system should be able toactivate highly sensitive one-dimensional edge evidencecollectors. A model-based operator based on the dipole[6] is currently being developed for this purpose.Evidence gathered in such a fashion, together with themodel's available to the system, should enable thesought feature to be inferred. It should then be possibleto perform full automatic cephalometric analyses oneven the poorest quality radiographs.

We have chosen cephalometry as our exemplar for theapplication of knowledge-based and blackboardsystems to medical image interpretation, but the types ofbehaviour displayed by this system should be applicableto other medical or difficult domains. The use of generalreasoning methods and the implementation of thesystem using generic processing should allow the systemto be successful in other complex image interpretationdomains.

Acknowledgements

This work is supported by a SERC research studentshipand a North West Regional Health Authority projectgrant. We thank Professor W.C. Shaw and Dr. F.MacKay, of the Department of Orthodontics, for theirassistance on this project.

References

[I] Cohen A.M., Ip G.C. & Linney A.D. 'A preliminarystudy of computer recognition and identification ofskeletal landmarks as a new method of cephalometricanalysis.' British Journal of Orthodontics, Vol. 11 No. 3(1984) ppl34-154.

[2] Davis D.N. & Taylor C.J. 'An intelligent segmen-tation system for lateral skull X-ray images.'AVC89, Proceedings of the Fifth Alvey Vision Con-ference pp 251-255, University of Reading, Sep-tember 1989.

[3] Davis D.N. & McKay F. 'Reliability Of Measure-ments Taken From Cephalometric Analyses.' Brit-ish Journal Of Orthodontics, (In preparation).

[4] Engelmore R. & Morgan T. (Eds.) BlackboardSystems. Addison-Wesley, 1988.

[5] Garbay C. 'KISS: A knowledge-based image segmenta-tion system.' Proceedings of the Ninth International Con-ference on Pattern Recognition, Beijing, China, (1988).

[6] Graham J. & Taylor C.J. 'Boundary cue operators formodel based image processing.' Proceedings AVC88(1988) pp 59-64.

[7] Green R.T. & Curtis M.C. 'Information theoryand figure perception: the metaphor that failed.'Acta Psychologica 25 :12-36, 1966.

[8] Hu Z.P., Pun T. & Pellegrini C. 'An expert systemfor guiding image segmentation.' ComputerisedMedical Imaging and Graphics, 14(1): 13-24,1990.

[9] Levy-Mandell A.D., Venetsanopoulos A.N. & TsotsosJ.K. 'Knowledge-based landmarking ofcephalograms.' Computers & Biomedical Research, Vol.19 (1986) pp 282-309.

[10] Matsuyama T. & Hwang V. 'SIGMA : A frame-work for image understanding.' Proceedings ofNinth International Joint Conference in ArtificialIntelligence, Milan, 1985, pp 908-915.

[II] Nazif A.M. & Levine M.D. 'Low level image segmen-tation: An expert system.' IEEE Transactions on Pat-tern Analysis and Machine Intelligence, PAMI-6(5)(1984) pp 555-577.

[12] Parthasarathy S., Nugent S.T., Gregson P.G. &Fay D.F. 'Automatic Landmarking ofCephalograms.' Computers and Biomedical Re-search Vol. 22, 1989, pp 248-269.

[13] Tsotsos J.K. 'Representational axes and temporal co-operative processes.' Technical Report RCBV-TR-84-2. The University of Toronto, (1984).

[14] Walker N. & Fox J. 'Knowledge-based interpre-tation of images: a biomedical perspective.' TheKnowledge Engineering Review, 2(4):249-264,1988.

295