Structured Literature Image Finder

Amr Ahmed1,6, Andrew Arnold1, Luıs Pedro Coelho2,3,4, Joshua Kangas2,3,4,Abdul-Saboor Sheikh3, Eric Xing1,5,6, William Cohen1, Robert F. Murphy1,2,3,4,5,7∗

Abstract

The Structured Literature Image Finder tacklestwo related problems posed by the vastness of thebiomedical literature: how to make it more accessi-ble to scientists in the field and how to take advan-tage of the primary data often locked inside papers.Towards this goal, the slif project developed an in-novative combination of text and image processingmethods.

Images from papers are classified according totheir type (fluorescence microscopy image, gel, . . . )and their caption is parsed for biologically relevantentities such as protein names. This enables tar-geted queries for primary data (a feature that auser study revealed to be highly valued by scien-tists). Finally, using a novel extension to latenttopic models, we model papers at multiple levelsand provide the ability to find figures similar to aquery and refine these findings with interactive rel-evance feedback.

Slif is most advanced in processing fluorescentmicroscopy images which are further categorised ac-cording to the depicted subcellular localization pat-tern.

The results of SLIF are made available to thecommunity through a user friendly web interface(http://slif.cbi.cmu.edu).

1 Introduction

Biomedical research worldwide results in a veryhigh volume of information in the form of publica-tions. Biologists are faced with the daunting task of

1Machine Learning Department, 5Department of Bi-ological Sciences, and 7Department of Biomedical Engi-neering, Carnegie Mellon University; 2Joint Carnegie Mel-lon University–University of Pittsburgh Ph.D. Program inComputational Biology; 3Center for Bioimage Informatics,Carnegie Mellon University; 4Lane Center for Computa-tional Biology, Carnegie Mellon University; 6Language Tech-nologies Institute, Carnegie Mellon University. ∗ to whomcorrespondence should be addressed.

Figure 1: Screenshot of the SLIF search engineshowing the results of a search.

querying and searching these publications to keepup with recent developments and to answer specificquestions.

In the biomedical literature, data is most of-ten presented in the form of images. A fluores-cent micrograph image (fmi) or a gel is sometimesthe key to a whole paper. Compared to figuresin other scientific disciplines, biomedical figures areoften a stand alone source of information that sum-marizes the findings of the research under consid-eration. A random sampling of such figures inthe publicly available PubMed Central databasereveals that in some, if not most of the cases, abiomedical figure can provide as much informationas a normal abstract. The information-rich, highly-evolving knowledge source of the biomedical liter-ature calls for automated systems that would helpbiologists find information quickly and satisfacto-rily. These systems should provide biologists witha structured way of browsing the otherwise unstruc-tured knowledge in a way that would inspire themto ask questions that they never thought of before,or reach a piece of information that they would havenever considered pertinent to start with.

Relevant to this goal, we developed the first sys-tem for automated information extraction from im-ages in biological journal articles (the “SubcellularLocation Image Finder,” or slif, first described in

1

2001 [8]). Since then, we have made major enhance-ments and additions to the SLIF system [2, 7, 6],and now report not only additional enhancementsbut the broadening of its reach beyond fluorescentmicroscopy images. Reflecting this, we have nowrechristened slif as the “Structured Literature Im-age Finder.”

SLIF reached the final stage in the ElsevierGrand Challenge (4 out of 70), a contest sponsoredby Elsevier to “improve the way scientific informa-tion is communicated and used.”

2 Overview

SLIF provides both a pipeline for extracting struc-tured information from papers (illustrated in Fig-ure 2) and a web-accessible searchable database ofthe processed information (depicted in Figure 1).

The pipeline begins by finding all figure-captionspairs. Each caption is then processed to identifybiological entities (e.g., names of proteins and celllines) and these are linked to external databases.Pointers from the caption to the image are identi-fied, and the caption is broken into “scopes” so thatterms can be linked to specific parts of the figure.

The image processing module begins by splittingeach figure into its constituent panels, and thenidentifying the type of image contained in eachpanel. The patterns in FMIs are described usinga set of biologically relevant image features [8], andthe subcellular location depicted in each image isrecognized.

The last step in the pipeline is to discover latenttopics that are present in the collection of papers.These topics serve as the basis for visualization andsemantic representation. Each topic consists of atriplet of distributions over words, image features,and proteins (possibly extended to include gene on-tology terms and subcellular locations). Each figurein turn is represented as a distribution over thesetopics, and this distribution reflects the themes ad-dressed in the figure. This representation servesas the basis for various tasks like image-based re-trieval, text-based retrieval and multimodal-basedretrieval. Moreover, these discovered topics providean overview of the information content of the col-lection, and structurally guide its exploration.

All results of processing are stored in a database,which is accessible via a web interface or SOAPqueries. The results of queries always include linksback to the panel, figure, caption and the full paper.

Users can query the database for various informa-tion appearing in captions or images, including spe-cific words, protein names, panel types, patterns infigures, or any combination of the above. Using thelatent topic representation, we built an innovativeinterface that allows browsing through figures bytheir inferred topics and jumping to related figuresfrom any currently viewed figure.

3 Caption Processing

In order to identify the protein depicted in an im-age, we look for protein names in the caption.The structure of captions can be complex (espe-cially for multipanel figures). We therefore imple-mented a system for processing captions with threegoals: identifying the “image pointers” (e.g., “(A)”or“(red)”) in the caption that refer to specific panellabels or panel colors in the figure [2], dividing thecaption into fragments that refer to an individualpanel, color, or the entire figure, and recognizingprotein and cell types.

Errors in optical character recognition can leadto low accuracy in matching image pointers to panellabels. Using regularities in the arrangement of thelabels (e.g., if the letters A through D are found asimage pointers and the panel labels are recognizedas A,B,G and D, then the G should be corrected toa C) corrects some of the errors [6]. Using a testset from PNAS, the precision of the final matchingprocess was found to be 83% and the recall to be74% [4].

Recognition of named entities (such as proteinand cell types) in free text is a difficult task thatmay be even more difficult in condensed text suchas captions. We have implemented two schemes forrecognizing protein names. The first (which is alsoused for cell type recognition) uses prefix and suffixfeatures along with immediate context to identifycandidate protein names. This approach has a lowprecision but an excellent recall (which is useful toenable database searches on abbreviations or syn-onyms that might not be present in structured pro-tein databases). The second approach [5] uses a dic-tionary of names extracted from protein databasesin combination with soft match learning methodsto obtain a recall and precision above 70%. Theoccurrences of the names found in the captions arestored as being associated either with a panel or afigure, depending on the scope in which the proteinname was found. The system also assigns subcellu-lar locations to proteins using lookup of GO terms

2

Figure 2: SLIF Pipeline. This figure shows the general pipeline through papers are processed.

in the Uniprot database, making it possible to findimages depicting particular subcellular patterns.

Finally, the task of simply segmenting a paperand extracting the caption, even without namedentity recognition or panel scoping, has proven veryuseful to our users, allowing easy search of free textwhich can be limited to the captions, and thereforethe figures, of a paper.

4 Image Processing

Since, in most cases, figures are composed of mul-tiple panels, the first step in our image process-ing pipeline is to divide the figures into panels.We employ a figure-splitting algorithm that recur-sively finds constant-intensity boundary regions inbetween panels, a method which we have previ-ously shown can effectively split figures with com-plex panel layouts [8].

Slif was originally designed to process only fmipanels, and subsequent systems created by othershave included classifiers to distinguish other figuretypes [10, 3]. We have now expanded the classifi-cation to other panel types: (1) fmi, (2) gel, (3)graph or illustration, (4) photograph, (5) X-ray, or(6) light microscopy. Using active learning [9], weselected ca. 700 panels to label.

Given its importance to the working scientists,we focused on the gel class. Currently, the systemproceeds through 3 classification levels: the firstlevel, classifies the image into fmi or non-fmi us-ing image based features (as previously reported);the second level, uses textual features to identifygels with high-precision (91%, and moderate recall:66%); finally, if neither classifier has fired, a generalpurpose support vector machine classifier, operat-ing on image-based features does the final classifi-cation (accuracy: 61%).

Perhaps the most important task that slif sup-

ports is the classification of fmi panels based on thedepicted subcellular localization. To provide train-ing data for pattern classifiers, we hand-labeled aset of images into four different subcellular locationclasses: (1) nuclear, (2) cytoplasmic, (3) punctate,and (4) other, again selected through active learn-ing.

We computed previously described features torepresent the image patterns. If the scale is in-ferred from the image, then we normalize this fea-ture value to square microns. Otherwise, we as-sume a default scale of 1µm/pixel. On the 3 mainclasses (Nuclear, Cytoplasmic, and Punctate), weobtained 75% accuracy (as before, reported accu-racies are estimated using 10 fold cross-validationand the classifier used was libsvm based). On thefour classes, we obtained 61% accuracy.

5 Topic Discovery

The goal of the topic discovery phase is to en-able the user to structurally browse the otherwiseunstructured collection. This problem is reminis-cent of the actively evolving field of multimediainformation management and retrieval. However,structurally-annotated biomedical figures pose a setof new challenges to mainstream multimedia infor-mation management systems due to the hierarchicalstructure of the domain (panels contained withinfigures within papers) and the presence of bothfree form and annotated text in the caption (pro-tein names should be handled differently from freetext).

Our model, which we call multimodal factored,scoped, correspondence topic model, addresses theaforementioned challenges [1]. The input to thetopic modeling system is the panel-segmented,structurally and multimodally annotated biomedi-cal figures. The goal of our approach is to discover a

3

set of latent themes in the collection. These themesare called topics and serve as the basis for visualiza-tion and semantic representation. Each biomedicalfigure, panel, and protein entity is then representedas a distribution over these latent topics. This rep-resentation serves as the basis for various tasks likeimage-based retrieval, text-based image retrieval,multimodal-based image retrieval and image anno-tation. We compared our model to various base-lines over the aforementioned tasks with favorableresults [1].

The latent topic representation facilitates the im-plementation of features such as finding similar ob-jects to an example that the user has found as inter-esting (this can be done at any level: panel, figure,or paper).

6 Discussion

We have presented SLIF, a system which analyzesimages in the biomedical literature. It processesboth text and image, combining them through la-tent topic discovery. This enables users to browsethrough a collection of papers by looking for re-lated topics or images that are similar to an imageof interest.

Although it is crucial that individual componentsachieve good results (and we have shown good re-sults in our sub-tasks), good component perfor-mance is not sufficient for a working system. Slifis a production system that has been shown to yieldusable results in real collections of papers.

The project is on-going and many avenues forimprovement are being exploited. Among those arebetter semantic understanding of FMI data, moreadvanced image processing of gels, exploitation ofthe full-text, as well as a continuing improvementof all the components in the pipeline.

References

[1] A. Ahmed, E. Xing, W. Cohen, and R. F. Murphy.Modeling structurally-annotated figures using fac-tored scoped correspondence topic models. In ACMconference of Knowledge Discovery and Data Min-ing (submitted), 2009.

[2] W. Cohen, R. Wang, and R. F. Murphy. Under-standing captions in biomedical publications. InKDD 2003: Proceedings of the ninth ACM SIGKDDinternational conference on Knowledge discoveryand data mining, pages 499–504, New York, USA,

ACM.

[3] J.-M. Geusebroek, M. Anh Hoang, J. van Gern-ert, and M. Worring. Genre-based search throughbiomedical images. volume 1, pages 271–274 vol.1,2002.

[4] Z. Kou, W. Cohen, and R. F. Murphy. Extractinginformation from text and images for location pro-teomics. In Mohammed Javeed Zaki, Jason Tsong-Li Wang, and Hannu Toivonen, editors, BIOKDD,pages 2–9, 2003.

[5] Z. Kou, W. Cohen, and R. F. Murphy. High-recallprotein entity recognition using a dictionary. InISMB (Supplement of Bioinformatics), pages 266–273, 2005.

[6] Z. Kou, W. Cohen, and R. F. Murphy. A stackedgraphical model for associating sub-images withsub-captions. In R. B. Altman, A. K. Dunker,L. Hunter, T. Murray, and T. E. Klein, editors, Pa-cific Symposium on Biocomputing, pages 257–268.World Scientific, 2007.

[7] R. F. Murphy, Z. Kou, J. Hua, M. Joffe, and W.Cohen. Extracting and structuring subcellular loca-tion information from on-line journal articles: Thesubcellular location image finder. In IASTED In-ternational Conference on Knowledge Sharing andCollaborative Engineering, pages 109–114, 2004.

[8] R. F. Murphy, M. Velliste, J. Yao, and G. Por-reca. Searching online journals for fluorescence mi-croscope images depicting protein subcellular loca-tion patterns. In BIBE ’01: Proceedings of the 2ndIEEE International Symposium on Bioinformaticsand Bioengineering, pages 119–128, Washington,DC, USA, 2001. IEEE Computer Society.

[9] N. Roy and A. Mccallum. Toward optimal activelearning through sampling estimation of error re-duction. In In Proc. 18th International Conf. onMachine Learning, pages 441–448. Morgan Kauf-mann, 2001.

[10] H. Shatkay, N. Chen, and D. Blostein. Integratingimage data into biomedical text categorization. InISMB (Supplement of Bioinformatics), pages 446–453, 2006.

4