EARL: Joint Entity and Relation Linking forQuestion Answering over Knowledge Graphs

Mohnish Dubey1,2, Debayan Banerjee1, Debanjan Chaudhuri1,2, and Jens Lehmann1,2

1 Smart Data Analytics Group (SDA), University of Bonn, Germany{dubey, chaudhur, jens.lehmann}@cs.uni-bonn.de

debayan@uni-bonn.de2 Fraunhofer IAIS, Bonn, Germany

jens.lehmann@iais.fraunhofer.de

Abstract. In order to answer natural language questions over knowledge graphs,most processing pipelines involve entity and relation linking. Traditionally, entitylinking and relation linking have been performed either as dependent sequentialtasks or independent parallel tasks. In this paper, we propose a framework calledEARL, which performs entity linking and relation linking as a joint single task.EARL uses a graph connection based solution to the problem. We model the link-ing task as an instance of the Generalised Travelling Salesman Problem (GTSP)and use GTSP approximate algorithm solutions. We later develop EARL whichuses a pair-wise graph-distance based solution to the problem.The system de-termines the best semantic connection between all keywords of the question byreferring to a knowledge graph. This is achieved by exploiting the “connectiondensity” between entity candidates and relation candidates. The “connection den-sity” based solution performs at par with the approximate GTSP solution. We haveempirically evaluated the framework on a dataset with 5000 questions. Our systemsurpasses state-of-the-art scores for entity linking task by reporting an accuracy of0.65 against 0.40 from the next best entity linker.

Keywords: Entity Linking, Relation Linking, GTSP, Question Answering

1 Introduction

Question answering over knowledge graphs (KGs) is an active research area concernedwith techniques that allow obtaining information from knowledge graphs based onnatural language input. Specifically, Semantic Question Answering (SQA) as definedin [9] is the task of users asking questions in natural language (NL) to which theyreceive a concise answer generated by a formal query over a KG. Semantic questionanswering systems can broadly be classified as either semantic parsing based systemssuch as AskNow [5] or end-to-end systems [18] (usually based on neural networks). Themain challenges faced in both types of SQA are (i) entity identification and linking (ii)relation identification and linking (iii) query intent identification and (iv) formal querygeneration.

Some QA systems have achieved good performance on simple questions [12],i.e. those questions which can be answered by linking to at most one relation and

arX

iv:1

801.

0382

5v2

[cs

.AI]

16

Jan

2018

at most one entity in the KG. Recently, the focus has shifted towards complex ques-tions [26], comprising of multiple entities and relations. Usually, all entities and relationsneed to be correctly linked to the knowledge graph in order to generate the correct formalquery and successfully answer the question of a user. Hence, it is of utmost importanceto perform the linking process with high accuracy and indeed that is a major bottle-neck for the widespread proliferation of current SQA systems. In most entity linkingsystems [13,22] disambiguation is performed by looking at other entities present in theinput text. However in the case of short text fragments, such as we typically have inuser questions, there is often no other entity to be found. To deal with such cases, webelieve that it is required to consider entity and relation candidates in input questions incombination to maximise the available evidence for the candidate selection process.

To achieve this, we propose EARL (Entity and Relation Linker), a system for jointlylinking entities and relations in a question to a knowledge graph. EARL treats entitylinking and relation linking as a single task and thus aims to reduce the error causedby the dependent steps. To realise this, EARL uses the knowledge graph to jointlydisambiguate entity and relations. EARL obtains the context for entity disambiguationby observing the relations surrounding the entity. Similarly, it obtains the context forrelation disambiguation by looking at the surrounding entities. We support multipleentities and relations occurring in complex questions by modelling the joint entity andrelation linking task as an instance of the Generalised Travelling Salesman Problem(GTSP). Since the problem is NP-hard, we investigate state-of-the-art approximateGTSP solvers. Since it turns out that the approximation results in a relevant reductionin accuracy, we devise approximation techniques specific for the QA setting. Thisallows us to obtain an accuracy closer to exact GTSP solvers while being significantlymore efficient. In our experimental setting involving 3000 questions, we improve thestate-of-the-art performance for entity linking from 40% to 65% accuracy.

To provide an intuition on why the joint linking process can improve accuracy, letus consider the question "Where was the founder of Tesla and SpaceX born?". In thiscase, the entity linker needs to perform the disambiguation for the keyword "Tesla"between, e.g., "Nikolas Tesla" the scientist and "Tesla Motors" the car company. Injoint process used by EARL, all other entities and relations (SpaceX, founder, born) areconsidered. We do this by analysising the subdivision graph of the knowledge graphfragment containing the candidates for relevant entities and relations (see Figure 1 showsan excerpt). When performing the joint analysis, EARL in this case detects that thereis no likely combination of candidates, which supports the disambiguation of "Tesla"as "Nikolas Teslas" whereas there is a plausible combination of candidates for the carcompany "Tesla Motors".

Overall, our contributions in this paper are as follows:

1. A formalisation of the joint entity and relation linking problem as an instance of theGeneralised Traveling Salesman problem.

2. A novel approach (EARL), which performs joint entity linking and relation linkingfor natural language questions that is close to an exact solution of the GTSP in termsof accuracy while being significantly more efficient.

3. A fully annotated version of LC-QuAD data-set, where entity and relations arelinked to the KG.

Fig. 1. An excerpt of the subdivision knowledge graph for the example question "Where was thefounder of Tesla and Space X born?". Note that both entities and relations are nodes in the graph.

4. A novel character embedding based LSTM (long short-term memory recurrentneural network) architecture for predicting entities and relations from phrases in anatural language query.

5. A large set of labels for DBpedia predicates and entities covering the syntactic andsemantic variations.

2 Related Work

The entity and relation linking challenge has attracted a wide variety of solutions overtime. Linking natural language phrases to DBpedia resources, Spotlight [13] breaksdown the process of entity spotting into four phases. It first identifies the entity usinga list of surface forms and then generates DBpedia resources as a list of candidates. Itthen disambiguates entity based on surrounding context to find the correct resource.While it relies primarily on natural language based features, AGDISTIS [22] followsthe inherent structure of the target knowledge base more closely to solve the problem.Being a graph based disambiguation system, AGDISTIS does disambiguation based onthe hop-distance between the candidates for the entity for a given text where there aremultiple entities present. On the other hand, S-MART [25] is often appropriated as anentity linking system over Freebase resources. It generates multiple regression tress andthen applies sophisticated structured prediction techniques to link entities to resources.

While relation linking is generally considered to be a more problem-specific task,few general purpose relation linking systems are in-use. Iterative bootstrapping strategiesfor extracting RDF resources from unstructured text have been explored in BOA [6]PATTY [14]. It consists of natural language patterns corresponding to relations presentin the knowledge graph. Word embedding models are also frequently used to overcomethe linguistic gap for relation linking.

Many QA systems use an out-of-the-box entity linking tool, often one of the afore-mentioned ones. However, these tools are not tailor made for questions, and are insteadtrained on large text corpora, typically devoid of questions. This can create several prob-lems as questions don’t span over more than a sentence, thereby rendering context baseddisambiguation relatively ineffective. Further, graph based systems rely on presence ofmultiple entities in the source text and disambiguates them based on each other. Thisbecomes difficult while dealing with questions, as often they do not consist of more thanone entity and relation.

Thus, to avoid above issues, a variety of approaches have been employed for entityand relation linking for question answering. Semantic parsing based systems such as As-

kNow [5], TBSL [21] first link the entities and generate a list of candidate relations basedon the identified resources. They use several string similarity and semantic similaritytechniques to finally select the correct entity and relation candidates for the question. Inthese systems, the process of relation linking depends on linking the entities. Generatingentity and relation candidates have also been explored by [26], which uses these candi-dates to create staged query graphs, and later re-ranks them based on textual similaritybetween the query and the target question, computed by a Siamese architecture basedneural network. There are some QA systems such as Xser [24], which performs relationlinking independent of entity linking. STAGG[26] takes top 10 entity given by the entitylinker then it tries to make chains query-subgraph, corresponding to the question. Thisapproach considers ranked list of entity candidate from the entity linker and choose thebest candidate based on the query subgraph formed. Generally, semantic parsing basedsystem treat entity linking and relation linking as separate task which could be observedin the generalised pipeline of QANARY [3] and OKBQA 3.

LinkingApproach

QA System Advantage Disadvantage

Sequential [5] [3], - Reduces candidate searchspace for Relation Linking

- Relation Linking informationcannot be exploited in EntityLinking process

-Allows schema verification - Errors in Entity Linking cannotbe overcome

Parallel [23] [24] [1][15]

- Lower runtime - EL process cannot use informa-tion from RL process and viceversa

- Re-ranking of Entities possiblebased on RL

- Does not allow schema verifi-cation

Joint [2] [26] - Potentially high accuracy - Complexity increase(with - Reduces error propagation - Larger search spacelimited - Better disambiguationcandidate - Allows schema verificationset) - Allows re-ranking

Table 1. State of the art for Entity and Relation linking in Question Answering

3 Problem Statement and Formalisation

3.1 Overview and Research Questions

Typically, in question answering, the tasks of entity and relation linking are performedeither sequentially or in parallel. In sequential systems, usually the entity linking task isperformed first followed by relation linking. Naturally, information in the relation linkingphase cannot be exploited during entity linking in this case. In parallel systems, entity and

3http://www.okbqa.org/

relation linking are performed independently. While this is efficient in terms of runtimeperformance, the entity linking process cannot benefit from further information obtainedduring relation linking and vice versa. We illustrate the advantages and disadvantages ofboth approaches as well as systems following the approaches in Table 1.

Our main contribution in this paper is the provision of a system, which takes candi-dates for entity and relation linking as input and performs a joint optimisation selectingthe best combination (with respect to a distance function) of entity and relation candi-dates.

Postulates We have two postulates, which we want to verify based on our approach:H1: Given a candidate list of entities and relations that form a question, the correct

candidates exhibit relatively dense and short-hop connections among themselves in theknowledge graph compared to wrong candidate sets.

H2: Jointly linking entity and relation leads to higher accuracy compared to perform-ing these tasks separately.

We will re-visit both in the evaluation section of the paper.

3.2 Preliminaries

We will first introduce basic notions from graph theory:

Definition 1 (Graph). A (simple, undirected) graph is an ordered pair G = (V,E)where V is a set whose elements are called vertices and E is a set of pairs of verticeswhich is called edges.

Definition 2 (Knowledge Graph). Within the scope of this paper, we define a knowl-edge graph as a labelled directed multi-graph. A labelled directed multi-graph is atuple KG = (V,E, L) where V is a set called vertices, L is a set of edge labels andE ⊆ V × L× V is a set of ordered triples.

It should be noted that our definition of knowledge graphs captures basic aspects ofRDF datasets as well as property graphs [7]. The knowledge graph verticies represententities and the edges represent relationships between those entities.

Definition 3 (Subdivision Graph). The subdivision graph [20] S(G) of a graph G isthe graph obtained from G by replacing each edge e = (u, v) of G by a new vertex we

and 2 new edges (u,we) and (v, we) .

3.3 Formulating Joint Entity and Relation Linking as Optimisation Problem

In general, entity linking is a two step process. The first step is to identify and spot thespan of the entity. The second step is to disambiguate or link the entity to knowledgegraph. For linking, the candidates are generated for the spotted span of the entity andthen the best candidate is chosen for the linking. These two steps are similarly followedin standard relation linking approaches. In our approach, we first spot the spans ofentities and relations. After that, the linking task is performed jointly for both entities and

Fig. 2. GTSP modelled for EARL

relations. We assume this to work well for question answering tasks due to the postulatesformulated above.

The entity and relation linking process can be formalised via spotting and candidategeneration functions as follows: Let S be the set of all strings. We assume that thereis a function spot : S → 2S which maps a string s (the input question) to a set Kof substrings of s. We call this set K the keywords occurring in our input. Moreover,we assume there is a function candKG : K → 2V ∪L which maps each keyword toa set of candidate node and edge labels for our knowledge graph G = (V,E,L).The goal of joint entity and relation linking is to find combinations of candidates,which are closely related. How closely nodes are related is modelled by a cost functioncostKG : (V ∪ L) × (V ∪ L) → [0, 1]. Lower values indicate closer relationships.According to our first postulate, we aim to encode graph distances in the cost functionto reward those combinations of entities and relations, which are located close to eachother in the input knowledge graph. To be able to consider distances between bothrelations and entities, we transform the knowledge graph into its subdivision graph (seeDefinition 3). This subdivision graph allows us to elegantly define the distance functionas illustrated in Figure 4.

Given the knowledge graph KG and the functions spot, cand and cost, we cancast the problem of joint entity and relation linking as an instance of the GeneralisedTraveling Salesman (GTSP) problem: We construct a graph G with V =

⋃k∈K cand(k).

Each node set cand(k) is called a cluster in this vertex set. The GTSP problem is to finda subset V ′ = (v1, . . . , vn) of V which contains exactly one node from each cluster andthe total cost

∑n−1i=1 cost(vi, vi+1) is minimal with respect to all such subsets. Please

note that in our formalisation of the GTSP, we do not require V ′ to be a cycle, i.e. v1and vn can be different. Moreover, we do not require clusters to be disjoint, i.e. differentkeywords can have overlapping candidate sets.

Figure 2 illustrates the problem formulation. Each candidate set for a keyword formsa cluster in the graph. The weight of each edge in this graph is given by the cost function,which includes the distance between the nodes in the subdivision graph of the inputknowledge graph as well as the confidence scores of the candidates. The GTSP requiresto visit one element per cluster and minimises the overall distance.

3.4 Approximate GTSP Solvers and Connection Density Techniques

In order to solve the joint entity and relation linking problem, the corresponding GTSPinstance needs to be solved. Unfortunately, the GTSP is NP-hard [11] and hence it isintractable. However, since GTSP can be reduced to standard TSP, several polynomialapproximation algorithms exist to solve GTSP. The state of the art approximate GTSPsolver is the Lin–Kernighan–Helsgaun Algorithm [8]. Here, a GTSP instance is trans-formed into standard asymmetric TSP instances using the Noon-Bean transformation. Itallows the heuristic TSP solver LKH to be used for solving the initial GTSP. AmongLKH’s characteristics, its use of 1-tree approximation for determining a candidate edgeset, the extension of the basic search step, and effective rules for directing and pruningthe search contribute to its efficiency.

While a GTSP based solution would be suitable to solve the joint entity and relationlinking problem, it has the drawback that it can only provide the best candidate for eachkeyword given the list of candidates. Most approximate GTSP solutions do not exploreall possible paths and nodes and hence a comprehensive scoring and re-ranking of nodesis not possible. Ideally, we would like to go beyond this and re-rank all candidates for agiven keyword. This would open up new opportunities from a QA perspective, i.e. a usercould be presented with a sorted list of multiple possible answers to select from.

Thus, we will in addition develop a solution which aims to be competitive with theaccuracy of GTSP solvers and is, in addition to this, able to produce a re-ranked list ofcandidates for each keyword. We will do this via a new set of features which we denoteas connection density described in detail in the next section. In the evaluation section weempirically show:

i) Connection density results in a similar accuracy to that of exact GTSP solvers.ii) We also show that connection density has a similar runtime performance to that of

the approximate GTSP solver LKH.iii) Connection Density can produce re-ranked lists. This allows to apply adaptive

learning techniques for further improving the accuracy of the system.

In the next section we discuss EARL’s architecture and how it uses connection densityto solve the joint linking problem.

4 Architecture of EARL

EARL uses a new set of features named Connection Density to solve the problem of jointlinking of entities and relations. Via those features, we aim to approximate the global totalpath costs of GTSP by local per-node features that we feed into a classifier to score andrank each node based on its likelihood of being the target entity or relation. Although wedo not use total paths costs, the local features capture inter-node connectivity informationand our hypothesis is that this results in similar accuracy to that of an exact GTSP solverin a question answering setting. The connection density not only looks for the distancebetween two nodes as done in TSP problem, but also checks the number of connectionsa particular node has with other candidate sets. EARL uses the following steps, whichare described in more detail in the remainder of the section:

Fig. 3. EARL architecture diagram

Step 1: Shallow Parsing: Given a question, extract all keyword phrases out of it. EARL usesSENNA[4] as the keyword extractor. The latter uses a Neural Network Architecturefor predicting a noun or verb phrase(s) from a sentence.

Step 2: E/R prediction: For a extracted keyword phrases it is predicted to be an entity orrelation candidate

Step 3: Candidate List Generation: retrieve candidate URIs for keywords from the searchindex by taking the top-k results.

Step 4: Connectivity Detection: taking candidate lists from the previous step and see howwell each candidate is connected to other candidates in other lists. This informationis distilled into two features, namely connection count and hop count. These twofeatures for each candidate with their text search ranks are passed to next phase.

Step 5: Re-ranking candidate lists: From the top-k candidate list for each entity/relation, wepredict the most probable candidate. More on this in Section 4.4.

4.1 E/R Prediction

Once keyword phrases are extracted from the questions, the next step in EARL is topredict whether a phrase extracted from the previous phase is an entity or a relation. Weuse a character embedding based long-short term memory network (LSTM) to do thesame. The network is trained using labels for resources, ontology and properties in theknowledge graph. For handling OOV words [16], and also to encode the knowledge graphstructure in the network, we take a multi-task learning approach with hard parametersharing. Our model is trained on a custom loss given by:

L = (1− α) ∗ LBCE + α ∗ LED (1)

Where, LBCE is the binary cross entropy loss for the learning objective of a phrasebeing an entity or a relation. And, LEd is the squared eucledian distance betweenthe predicted embedding and the correct embedding for that label. The value of α isempirically selected as 0.25.

We use pre-trained label embeddings from RDF2Vec [17] which are trained onknowledge graphs. RDF2Vec provides latent representation for entities and relations

in RDF graphs. It efficiently captures the semantic relatedness between entities andrelations.

We use a hidden layer size of 128 for the LSTM, followed by two dense layers ofsizes 512 and 256 respectively. A dropout value of 0.5 is used in the dense layers. Thenetwork is trained using Adam optimizer [10] with a learning rate of 0.0001 and a batchsize of 128.

4.2 Candidate List Generation

This module retrieves a candidate list for each keyword identified in the natural languagequestion by the shallow parser. To retrieve the top candidates for a keyword we createan Elasticsearch4 index of URI-label pairs. Since EARL requires an exhaustive list oflabels for an URI in the knowledge graph, we expanded the labels beyond the providedlabels. We used Wikidata labels using for entities which are in same-as relation in theknowledge base. For relations we require labels which were semantically equivalent(such as writer, author) for which we took synonyms from the Oxford Dictionary API 5.We disable the default TF-IDF based scoring so that no bias is held towards or againstpopular entities and relations.

4.3 Joint Connectivity Detection

With the candidate lists returned by the previous text search phase, we form a sub graphof the underlying knowledge graph. Each candidate is a node in the graph. It must benoted here that while relations are conventionally considered as edges in a knowledgegraph, we consider both entity and relation as nodes for the purpose of our paper.

We develop connection density consisting of two features, namely connection countand hop count. Once we have received candidate lists from the previous text searchphase, we take each candidate from each list and check whether they are knowledge-graph-connected to any other candidate in all other lists. A candidate may directly beconnected to another candidate (1 hop) or it may be connected to another candidate viaone or more intermediate nodes in the subdivision graph (2 or more hops).

To explain the concept of hops further we present the following example:

In Fig 4(d) the following sum of hops exist:U1 : 3, V1 : 1, W3 : 2, V4 : 3, X4 : 3

We normalize by dividing these by the number of lists, which in Fig 4(d) is 4. This isour feature hop count.

In Fig 4(d) the following sum of connections exist:U1 : 2, V1 : 1, W3 : 1, V4 : 1, X4 : 1

We normalize by dividing these by the number of lists, which in Fig 4(d) is 4. This isour feature connection count.

4https://www.elastic.co/products/elasticsearch5https://developer.oxforddictionaries.com/

Fig. 4. (a) A set of triples represented in a knowldge graph (b) Converted to a subdivision graph(c) Each item of the tuple is a node in the subdivision graph (d) Connections and correspondinghop lengths among candidate nodes for a sample natural question. U,V,W,X are chunks from thenatural language question, and below them are corresponding candidate URIs returned by textsearch. U1 and V1 are at a hop distance of 1 in the subdivision graph.

A low hop count and a large connection count for a given node means it is a strongcandidate, since it is well connected to neighbouring clusters.

We now present a pseudocode version of the algorithm to calculate these features:

function :ConnectionDensity( )input :Clusters // an array of arraysoutput :connectionStore, hopStore

1 connectionStore ={ } // number of connections for each node2 hopStore = { } // hop lengths are these connections3 foreach (clusterA, clusterB) ∈ Clusters do4 foreach (nodeA) ∈ ClusterA do5 foreach (nodeB) ∈ ClusterB do6 if nodeA and nodeB are connected in KG then ;;7 connectionStore[nodeA] += 18 connectionStore[nodeB] += 19 hopStore[nodeA] += getHopsBetween(nodeA,nodeB)

10 hopStore[nodeB] += getHopsBetween(nodeA,nodeB)

11 foreach (node, score) ∈ connectionStore : do12 connectionStore[node] = connectionStore[node]/len(Clusters)

13 foreach (node, score) ∈ hopStore do14 hopStore[node] = hopStore[node]/len(Clusters)

15 return (connectionStore, hopStore)Algorithm 1: Connection Density.

This algorithm (Algorithm 1 Connection Density) has a time complexity given byO(N2L2) where N is the number of nodes per cluster, and L is the number of clusters.

4.4 Candidate Re-ranking

In this step, we use a machine learning algorithm C to get the most probable label for theidentified keyword phrases in the query. A feature space X is created from the connection

Fig. 5. MRR. with different algorithms on cross-validation sets

density of the labels and the rank of the retrieved top-k candidate lists. The lists arere-ranked based on the probability assigned by C. The performance is reported on 5-foldcross-validation. For each fold, we train C on X to classify which is the correct label. Weuse this trained model to assign a score ∈ [0, 1] for the candidate in the test set beinga correct label. The fetched candidate lists are re-ranked based on the former assignedscore.

We experiment with three different classifiers for this re-ranking task, namely, treeboosting(xgboost), support vector machine with linear kernel and logistic regression.Figure 5 reports the performance of the classifiers for MRR. on top-10 fetched candi-date lists. Clearly, xgboost is the best performing algorithm. Hence, for all our furtherexperiments we use xgboost for re-ranking.

4.5 Adaptive E/R prediction module

A bottleneck for EARL is if there is an error from the E/R prediction module, it will besequentially propagated. To trammel this, we implement an adaptive approach. Intuitively,if the assigned probability for all the candidates in the top-k list is too low, then none ofthem is a correct candidate. To act upon it, if the maximum probability from all the top-kcandidate is less than a very small threshold ther value of 0.05, we alter the predictionfrom the E/R prediction phase and redo the consecutive steps, hence creating a feedbackloop between the re-ranking phase and the E/R prediction phase. The adaptive approachis empirically evaluated in Table 4.

5 Evaluation

Data Set: LC-QuAD [19] is is the largest complex questions data set available forquestion answering over RDF. We have annotated this data set manually to create agold label data set for entity and relation linking. We published a set of 3000 annotatedquestions from LC-QuAD. Each question in this data contains an annotated set ofknowledge graph (in this case DBpedia) URIs and the corresponding SPARQL queriesrequired to fetch the answer for the respective question. The dataset is available at thefollowing persistent link: https://figshare.com/projects/EARL/28218

5.1 Comparison of GTSP, LKH and Connection Density

Aim: We wanted to evaluate our hypothesis that the connection density techniquesresult in an accuracy similar to an exact GTSP solver. We also evaluated the LKHapproximation solution of GTSP for doing the joint linking task. Moreover, we comparethe time complexity of the three different appraoches.Results: Connection density results in a similar accuracy as that of an exact GTSP witha much better time complexity. Connection density has a slightly worse time complexityto that of approximate GTSP solver LKH (assuming the best case of equal cluster sizesfor LKH) and provides better significantly better accuracy.

Approach Accuracy (K=30) Accuracy (K=10) Time Complexity

Brute Force GTSP 0.61 0.62 O(n22n)LKH - GTSP 0.59 0.58 O(nL2)Connection Density 0.61 0.62 O(N2L2)

Table 2. Empirical comparison of Connection Density and GTSP: n = number of nodes in graph;L = number of clusters in graph; N = number of nodes per cluster.

5.2 Experiments

Experiment 1: Disambiguation performance evaluationAim: The aim of this experiment is to evaluate our second hypothesis (H2) for joint

linking of entity and relations. Moreover, we use them to optimise the value of k to beused for our next set of experiments.Metrics: We use Mean Reciprocal Rank (MRR) as a statistical measure for evaluation.Given a question, and a set of phrases marked as entity and relations from the question,we evaluate the performance of the disambiguation module, which choose the correctKG URIs from the k candidates for each phrase.Result: The results in Table 3 show an improvement in the disambiguation task over theinitial list provided by naive text search for candidates. Also, there’s no considerableimprovement in performance after a k value of 30. EARL out performs AGDISTIS inthe disambiguation task, also EARL provides disambiguation for the relations at thesame time.

System WithoutReranking

Re-ranking withPath Scores

Re-ranking with Path Scoresand candidate Rank

EARL (k=10) 0.543 0.689 0.708EARL (k=30) 0.544 0.666 0.735EARL (k=50) 0.543 0.617 0.737EARL (k=100) 0.540 0.534 0.733AGDISTIS[22] - - 0.645

Table 3. Evaluating EARL’s disambiguation performance

Experiment 2: Evaluating entity linkingAim: To evaluate the performance of EARL with other state-of-the-art systems on the

entity linking task:Metrics: We are reporting the performance on accuracy. Accuracy is defined by the ratioof the correctly identified entities over the total number of entities present.Result: As reported in Table 4, EARL performs better entity linking than the othersystems, namely, DBpediaSpotlight and FOX + AGDISTIS. We conducted this test onthe LC-QuAD dataset. The value of k is set to 30 while re-ranking and fetching the mostprobable entity. We also evaluate our algorithm on the QALD-7 dataset 6.

Dataset System Accuracy

LC-Quad EARL with no adaptive learning 0.61EARL with adaptive learning 0.65DBpediaSpotlight 0.40FOX + AGDISTIS 0.36

QALD-7 EARL with no adaptive learning 0.55EARL with adaptive learning 0.57DBpediaSpotlight 0.42FOX + AGDISTIS 0.30

Table 4. Evaluating EARL’s Entity Linking performance

Experiment 3: Evaluating relation disambiguationAim: Given a set of correct entities for a question, the task is to the predict the relations

in the question using EARL.Metrics: We use the same accuracy metric as in the previous experiment: correctlylinked relations divided by present relations.Result: For relation disambiguation, given all the entities in a question are linked, thetask is of link the relation keywords correctly to the knowledge graph. EARL achievesan accuracy of 0.85 in this task.

6 Discussion

Our analysis shows that we have provided a tractable (polynomial with respect to thenumber of clusters and the elements per cluster) approximation of the GTSP instance,which solves the joint entity and relation linking problem as explained in Section 3. Inour experiments, we achieve similar accuracy as the exact GTSP solution with runtimes,which allow to actually use the system in QA engines. Currently, the system was testedon DBpedia, but is not restricted to a particular knowledge graph.

There are some limitations: The current approach does not tackle questions withhidden relations. Such as "How many shows does HBO have?". Here the seman-tic understanding of the corresponding SPARQL query is to count all TV shows

6https://project-hobbit.eu/challenges/qald2017/

"dbo:TelevisionShow" which are owned by i.e."dbo:company" the HBO "dbr:HBO".Here "dbo:company" is the hidden relation which we do not attempt to link. However, itcould be argued that this problem goes beyond the scope of relation linking and could bebetter handled by the query generation phase of a semantic QA system.

Aother limitation is that EARL cannot be used as inference tool for entities asrequired by some benchmark questions. For example Taikonaut is an astronaut withChinese nationality. The system can only link taikonaut to dbr:Astronaut, but additionalinformation can not be captured. Please note that EARL can tackle the problem of the"lexical gap" to a great extent as it has synonyms with the grammar inflection forms.

Our approach has a O(N2L2) time complexity where N is the number of candidatesper list, and L is the number of lists. This time complexity allows us to process a questionin a few hundred milliseconds on a standard desktop computer on average. The fullyannotated LC-QuAD dataset, result logs, experimental setup and source code of oursystem are publicly available at https://github.com/AskNowQA/EARL.

7 Conclusions and Future Work

In this paper we propose EARL, a framework for joint entity and relation linking. Wecast the joint linking problem into an instance of the Generalised Travelling Salesmanproblem. We then optimised the solution to the specific application scenario – entityand relation linking for question answering over knowledge graphs. Our experimentsresulted in scores, which are significantly above the results of current state-of-the-artapproaches for entity linking and high for relation linking. For future work, we willimprove the candidate generation phase to ensure that a higher percentage of correctcandidates are retrieved.

References

1. R. Beaumont, B. Grau, and A.-L. Ligozat. Semgraphqa@ qald5: Limsi participation atqald5@ clef.

2. J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, volume 2, page 6, 2013.

3. A. Both, D. Diefenbach, K. Singh, S. Shekarpour, D. Cherix, and C. Lange. Qanary–amethodology for vocabulary-driven open question answering systems. In InternationalSemantic Web Conference, pages 625–641. Springer, 2016.

4. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natu-ral language processing (almost) from scratch. Journal of Machine Learning Research,12(Aug):2493–2537, 2011.

5. M. Dubey, S. Dasgupta, A. Sharma, K. Höffner, and J. Lehmann. Asknow: A framework fornatural language query formalization in sparql. In International Semantic Web Conference,pages 300–316. Springer, 2016.

6. D. Gerber and A.-C. N. Ngomo. Bootstrapping the linked data web. In 1st Workshop on WebScale Knowledge Extraction@ ISWC, volume 2011, 2011.

7. A. Gubichev and M. Then. Graph pattern matching: Do we have to reinvent the wheel? InProceedings of Workshop on GRAph Data management Experiences and Systems, pages 1–7.ACM, 2014.

8. K. Helsgaun. Solving the equality generalized traveling salesman problem using the lin–kernighan–helsgaun algorithm. Mathematical Programming Computation, 7(3):269–287,2015.

9. K. Höffner, S. Walter, E. Marx, R. Usbeck, J. Lehmann, and A.-C. Ngonga Ngomo. Surveyon challenges of question answering in the semantic web. Semantic Web, (Preprint).

10. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.

11. G. Laporte, H. Mercure, and Y. Nobert. Generalized travelling salesman problem through nsets of nodes: the asymmetrical case. Discrete Applied Mathematics, 18(2):185 – 197, 1987.

12. D. Lukovnikov, A. Fischer, J. Lehmann, and S. Auer. Neural network-based question an-swering over knowledge graphs on word and character level. In Proceedings of the 26thInternational Conference on World Wide Web, pages 1211–1220, 2017.

13. P. N. Mendes, M. Jakob, A. García-Silva, and C. Bizer. Dbpedia spotlight: shedding lighton the web of documents. In Proceedings of the 7th international conference on semanticsystems, pages 1–8. ACM, 2011.

14. N. Nakashole, G. Weikum, and F. Suchanek. Patty: A taxonomy of relational patterns withsemantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in NaturalLanguage Processing and Computational Natural Language Learning, pages 1135–1145.Association for Computational Linguistics, 2012.

15. S. Park, S. Kwon, B. Kim, and G. G. Lee. Isoft at qald-5: Hybrid question answering systemover linked data and text data. In CLEF (Working Notes), 2015.

16. Y. Pinter, R. Guthrie, and J. Eisenstein. Mimicking word embeddings using subword rnns. InEMNLP, pages 102–112, 2017.

17. P. Ristoski and H. Paulheim. Rdf2vec: Rdf graph embeddings for data mining. In InternationalSemantic Web Conference, pages 498–514. Springer, 2016.

18. I. V. Serban, A. García-Durán, C. Gulcehre, S. Ahn, S. Chandar, A. Courville, and Y. Bengio.Generating factoid questions with recurrent neural networks: The 30m factoid question-answercorpus. arXiv preprint arXiv:1603.06807, 2016.

19. P. Trivedi, G. Maheshwari, M. Dubey, and J. Lehmann. Lc-quad: A corpus for complexquestion answering over knowledge graphs. In International Semantic Web Conference, pages210–218. Springer, 2017.

20. R. J. Trudeau. Introduction to graph theory (corrected, enlarged republication. ed.), 1993.21. C. Unger, L. Bühmann, J. Lehmann, A.-C. Ngonga Ngomo, D. Gerber, and P. Cimiano.

Template-based question answering over rdf data. In Proceedings of the 21st internationalconference on World Wide Web, pages 639–648. ACM, 2012.

22. R. Usbeck, A.-C. N. Ngomo, M. Röder, D. Gerber, S. A. Coelho, S. Auer, and A. Both.Agdistis-graph-based disambiguation of named entities using linked data. In InternationalSemantic Web Conference, pages 457–471. Springer, 2014.

23. A. P. B. Veyseh. Cross-lingual question answering using common semantic space. InTextGraphs@ NAACL-HLT, pages 15–19, 2016.

24. K. Xu, S. Zhang, Y. Feng, and D. Zhao. Answering natural language questions via phrasalsemantic parsing. In Natural Language Processing and Chinese Computing, pages 333–344.Springer, 2014.

25. Y. Yang and M.-W. Chang. S-mart: Novel tree-based structured learning algorithms appliedto tweet entity linking. In ACL 2015, 2015.

26. W.-t. Yih, M.-W. Chang, X. He, and J. Gao. Semantic parsing via staged query graphgeneration: Question answering with knowledge base. In Proceedings of the 53rd ACL Conf.,volume 1, pages 1321–1331, 2015.