Benchmarking Infrastructure for Mutation Text MiningArtjom Klein*, Alexandre Riazanov, Christopher J. O. Baker

Computer Science and Applied Statistics, University of New Brunswick, Saint John, Canada.

 Matthew Hindle

Synthetic and Systems Biology, Edinburgh University, 

Edinburgh, UK.

AIMM 2012 September 9th

Basel, Switzerland.

Mutation Text Mining

Mutation text mining facilitates a wide range of activities in multiple scenarios in bioinformatics and systems biology, including:• Modeling of cell signalling pathways• Protein structure annotation• Expansion of disease-mutation database annotations• Development of tools predicting the impacts of mutations

Useful text mining tasks:• Simple identification of mutation mentions • Linking (”grounding”) identified mutations to the

corresponding genes and proteins• Identifying mutation impacts and/or related phenotypes.

Sample Mutation Text Mining Task: Grounding

“ Haloalkane dehalogenase (DhlA) from Xanthobacter autotrophicus GJI0 hydrolyses terminally chlorinated and brominated n-alkanes to the corresponding alcohols.” 

“The A149T mutant showed only a slight reduction of dehalogenase activity (Vmax) and while D260N resulted in a larger increase of Km with 1,2-dibromoethane.”

Mutation / Protein / Gene / Organism / Direction / Protein Property / Chemical

Sequence

Protein ID

“Haloalkane dehalogenase (DhlA) from Xanthobacter autotrophicus GJI0 hydrolyses terminally chlorinated and brominated n-alkanes to the corresponding alcohols.”

 

“The A147F mutant showed only a slight reduction of the enzyme activity (Vmax) and while D157P resulted in a larger increase of Km with 1,2-dibromoethane.”

Mutation / Protein / Gene / Organism / Direction / Protein Property / Chemical

• Event: Mutation Impacts (Protein Property + Impact Direction)

• Impact Direction: Positive / Negative

• Linking / Grounding Impact mentions to Mutation mentions in text

• Linking / Grounding Mutation Mentions to Protein mentions in text

• Linking / Grounding Protein Property mentions to GO Molecular Function terms

• Normalizing Mutation mentions (e.g. to HGVS Nomenclature)

Sample Mutation Text Mining Tasks: Relations

Performance of Mutation Grounding Systems

Mut. Gr. System Text Type Coverage Size of test doc. corpus

Performance

Horn et al., 2004 full text G protein-coupled receptors and nuclear hormone receptors

914 + 1094 P: 0.87R: 0.64

Krallinger et al., 2009 full textabstracts

human kinase mutations

714 abstracts3486 full texts

P: 0.72R: ?

Winnenburg et al., 2009

abstracts ten species 508 abstracts P: ?R: ?

Laurila et al., 2010 full text no restriction, but evaluated on 4 corpora for 7 specific proteins.

76 P: 0.84R: 0.65

Baker, Kanagasabai, 2011

full text no restriction, but evaluated on corpora for specific proteins.

96 P: 0.819 R: 0.601

Klein et al., 2012 full text no restriction, tested on 91 differentUniProt identifiers

331 P: 0.82R: 0.77

Benchmarking Resources

The Developer needs:

• Annotated Corpora (training and dev.)

• Robust tools to evaluate the performance of– different systems – different runs of an evolving prototype system against a

gold standard corpus.

• Tools for migrating / uploading system outputs

• Semantic models for integrating data

• Appropriate Metrics

Typical Benchmarking Challenges

• Benchmarks (annotated corpora) often not open / published• Must be build from scratch• Annotation Set (Semantic Types, Ontology classes, …)

• Different formats, different annotation schemas• Choice of Representation Format (TXT, XML, TAB, RDF)

• Different metrics used for evaluation• What and How to evaluate it

• Complex submission procedures• Cumbersome / Slow

mutation-text-mining

We leverage the semantic web standards: OWL, RDF and SPARQL

Our Representation Format: RDF

Why not XML? - XML is a widely used standard format for corpora annotation and is supported by a large number of tools.

• The processing of complex annotations in XML – parsing, storing, querying, evaluation – is usually virtually impossible with off-the-shelf XML tools.

• Developers need to develop schema-specific parsers and processing scripts and change them each time when the schema is changed or extended.

RDF: Extensibility• Different mutation text mining tasks and all requirements can not

be foreseen• Same data may be used for different tasks

=> We need extensible representations.

OWL/RDF ontologies are highly extensible data schemas providing:• easy integration of new corpora with annotation schemas that

need not be identical, as long as they are compatible. • easy merging of data defined modulo one ontology with data

modulo another ontology. • additional alignments between the ontologies can be provided by

the annotation providers – corpus curators or text mining system developers.

RDF: Tool Availability

• OWL reasoners for data integrity checking• RDF and OWL APIs for multiple programming

languages facilitate easy programmatic generation and manipulation of annotations or RDF data representing text mining results.

• SPARQL query language can be directly used for calculating system performance metrics as well as for various searches in the gold standard corpora. 

• Multiple implementations of RDF databases (triplestores) are available that facilitate efficient storing and querying of large volumes of annotations.

Semantic Model: Mutation Impact Extraction Ontology (MIEO)

 

Riazanov A, Laurila JB and Baker C, Deploying mutation impact text-mining software with the SADI Semantic Web Services framework, BMC Bioinformatics 2011, 12(Suppl 4):S6 and Nona Naderi and René Witte, Automated extraction and semantic analysis of mutation impacts from the biomedical literature, BMC Genomics 2012, 13(Suppl 4):S10 

Benchmarking with SPARQL

• SPARQL is query language for RDF data.• Create a new SPARQL query or change an existing

one is usually easier than create or rewrite some scripts.

• We use named graphs (identified subsets of RDF statements) to separate results coming from different systems or different experiments and gold-standard data: results from different experiments, and even gold standard data from different corpora.

• Basically, metrics are calculated by comparing 2 graphs and finding overlaps between them.

Corpus Development: KinMutBase

• A subset of 201 documents annotated with singular amino acid substitutions grounded to proteins

• Curation – using MutationFinder (high recall) and comparing the results with the annotations in the database. Based on this comparison, we discarded about 70 documents that appear annotated with protein-level mutations not explicitly mentioned in the documents

• The final size of the corpus is 128 documents. In total, we have 271 mutations linked to 26 different UniProt identifiers.

• Primarily for Mutation Grounding Evaluations

Stenberg KA, Riikonen PT, Vihinen M., KinMutBase, a database of human disease-causing protein kinase mutations , Nucleic Acids Res. 1999 Jan 1;27(1):362-4.

Corpus Development: EnzyMiner•Full text documents (38) randomly selected from EnzyMiner* abstracts•Documents with proteins from 49 UniProt Ids and 24 different species. •Coverage: 488 statements (occurrences of impact information in text),

61 molecular functions and 29 combined mutations.

Annotated Information:• Studied protein-level mutations, in the form of singular amino acid substitutions.

For situations when effects of several simultaneous amino acid substitutions are studied, we annotate them as combined mutations.

• Proteins to which the mutations are related are identified with UniProt IDs. Host organisms / sets of specific protein sequences can be identified via UniProt IDs.

• Protein properties specified as Gene Ontology Molecular function classes.• Mutation impacts qualified as Positive, Negative or Neutral.• Text fragments from where information was extracted from. Typical fragments

contain mentions of protein properties, impact directionality words, such as “increased” or “worse”, mutation mentions, protein and organism names, etc.

• Documents identified with PubMed IDs.

* Yeniterzi S, Sezerman U., EnzyMiner: automatic identification of protein level mutations and their impact on target enzymes from PubMed abstracts.BMC Bioinformatics. 2009 Aug 27;10 Suppl 8:S2.

Corpora Statistics

Corpus Size UniProt IDs Mutations (per document)

EnzyMiner (Dev.) 38 49 176

KinMutBase(published set)

128 26 271

DHLA 13 4 52

PIK3CA 30 1 169

FGFR3 26 1 175

MEN1 7 1 22

Utilities

As a part of our infrastructure, we created a small set of simple utilities, which facilitate data access:

• The evaluator utility calculates standard performance metrics by executing some user-provided SPARQL queries, counting the results and making necessary calculations. The user can supply the queries in a simple configuration file.

• The Sesame loader and query client are simple command line applications that allow loading RDF graphs into a Sesame triplestore and executing queries from files.

Mutation Grounding Metrics

• Precision - the fraction of correctly grounded mutations (true positives) over all grounded mutations (true positives + false positives)

• Recall - the fraction of correctly grounded mutations over all mutations in the gold standard (true positives + false negatives). 

• A mutation is considered correctly grounded if it is mapped to a sequence corresponding to the UniProt ID specified by the corresponding gold standard corpus. 

Witte R, Baker CJO (2007) "Towards a systematic evaluation of protein mutation extraction " proposes over 15 different metrics to evaluate protein mutation extraction systems.

Evaluation Example

• There are 3 SPARQL queries to calculate: • The number of all correctly grounded mutations (query1), • All grounded mutations (query2), • All mutations in gold-standard (query 3).

• Queries defined in a configuration file of evaluator tool.• Evaluator tool executes 3 queries and combines results

in recall and precision formulas which are also defined in the configuration file as mathematical expressions.

E.g. • precision = query1/query2, • recall = query1/query3,

SPARQL Query 1

• Singular mutation mention recognition. (True positives)SELECT ?doc ?singl_mut1 FROM NAMED <http://unbsj.biordf.net/misc/text_mining_experiment255.rdf> WHERE {

GRAPH <http://unbsj.biordf.net/misc/text_mining_experiment255.rdf> {?doc sio: 'refers to' ?singl_mut1 . ?singl_mut1 a mieo:AminoAcidSubstitution . ?singl_mut1 mieo:mutationHasWildtypeResidue ?wt_residue . ?singl_mut1 mieo:mutationHasMutantResidue ?mut_residue . ?singl_mut1 mieo:mutationHasPosition ?pos1 . ?pos1 sio:’has_value’ ?pos_value .

} . GRAPH goldst:v0.0 {

?doc sio: 'refers to' ?singl_mut2 . ?singl_mut2 a mieo:AminoAcidSubstitution . ?singl_mut2 mieo:mutationHasWildtypeResidue ?wt_residue . ?singl_mut2 mieo:mutationHasMutantResidue ?mut_residue . ?singl_mut2 mieo:mutationHasPosition ?pos2 . ?pos2 sio:’has_value’ ?pos_value .

} }

We select single mutations from system output, which match single mutations from gold standard (overlap: wild type residue, position and mutant residue)

Testing the Infrastructure

For concept validation the infrastructure was used for testing and iterative performance evaluation during a project dedicated to the development of a robust mutation grounding system.EnzyMiner was used as development corpus.• originally created for mutation impact extraction • it only contains information about mutations whose

impact is studied• there may be other mutations associated with specific

proteins but not with impacts• we only compute our performance metrics on the

subsets of mutations mentioned in the annotationsAll other corpora were used as test corpora.

Evaluation ResultsOriginal Prototype

Original Prototype

New System New System

Precision Recall Precision Recall

EnzyMiner (Dev.)

0.31 0.12 0.75 0.72

KinMutBase (225 docs)

0.36 0.14 0.92 0.92

DHLA 0.83 0.73 0.96 0.94

PIK3CA 0.86 0.70 0.98 0.81

FGFR3 0.89 0.66 0.27 0.25

MEN1 0.54 0.32 0.54 0.52

Total w/o EnzMiner

0.64 0.35 0.82 0.77

On all corpora the new system outperforms the original prototype ….

mutation-text-mining

http://code.google.com/p/mutation-text-mining/

Future work

• Further stress-test infrastructure with text mining tasks other than mutation grounding and mutation impact extraction, and a third-party mutation text mining system.

• Extend the ontology based on the new requirements identified through community involvement and our own research.

• Extend the infrastructure to include protein properties other than molecular functions, such as enzyme kinetics, and DNA-level mutations.

• Modeling sentence level provenance to provide more precise pointers to text fragments supporting annotations.

• Want participants: Open Mutation Text Mining Competition

Acknowledgements

This research was funded in part by the New Brunswick Innovation Foundation, New Brunswick, Canada; the NSERC, Discovery Grant Program, Canada and the Quebec - New Brunswick University Co-operation in Advanced Education – Research Program, Government of New Brunswick, Canada.