Can a Suit of Armor Conduct Electricity?A New Dataset for Open Book Question Answering

Todor Mihaylov‡ and Peter Clark† and Tushar Khot† and Ashish Sabharwal†

† Allen Institute for Artificial Intelligence, Seattle, WA, U.S.A.‡ Research Training Group AIPHES & Heidelberg University, Heidelberg, Germany

{peterc,tushark,ashishs}@allenai.org, mihaylov@cl.uni-heidelberg.de

Abstract

We present a new kind of question answeringdataset, OpenBookQA, modeled after openbook exams for assessing human understand-ing of a subject. The open book that comeswith our questions is a set of 1326 elementarylevel science facts. Roughly 6000 questionsprobe an understanding of these facts and theirapplication to novel situations. This requirescombining an open book fact (e.g., metals con-duct electricity) with broad common knowl-edge (e.g., a suit of armor is made of metal) ob-tained from other sources. While existing QAdatasets over documents or knowledge bases,being generally self-contained, focus on lin-guistic understanding, OpenBookQA probes adeeper understanding of both the topic—in thecontext of common knowledge—and the lan-guage it is expressed in. Human performanceon OpenBookQA is close to 92%, but manystate-of-the-art pre-trained QA methods per-form surprisingly poorly, worse than severalsimple neural baselines we develop. Our or-acle experiments designed to circumvent theknowledge retrieval bottleneck demonstratethe value of both the open book and additionalfacts. We leave it as a challenge to solve theretrieval problem in this multi-hop setting andto close the large gap to human performance.

1 Introduction

Open book exams are a common mechanismfor assessing human understanding of a subject,where test takers are allowed free access to a rel-evant book, study guide, or class notes when an-swering questions. In this context, the goal is notto evaluate memorization but a deeper understand-ing of the material and its application to new situa-tions (Jenkins, 1995; Landsberger, 1996). The ap-plication, in turn, often requires combining a factin the book (e.g., metals conduct electricity) withadditional common knowledge the test taker is ex-

Question:Which of these would let the most heat travel through?A) a new pair of jeans.B) a steel spoon in a cafeteria.C) a cotton candy at a store.D) a calvin klein cotton hat.

Science Fact:Metal is a thermal conductor.

Common Knowledge:Steel is made of metal.Heat travels through a thermal conductor.

Figure 1: An example for a question with a given setof choices and supporting facts.

pected to have acquired by this stage (e.g., a suitof armor is made of metal).

Motivated by this setting, we present a new kindof question answering dataset, OpenBookQA,1

that consists of two parts: Q, a set of 5957multiple-choice questions, andF , a set of 1326 di-verse facts about elementary level science. F hasthree key characteristics of an ‘open book’: (a) itforms the basis for generating Q; (b) it has beendeemed central to scientific explanations (Jansenet al., 2018); and (c) by itself, F is generally in-sufficient to answer questions in Q. Faced with aquestion q ∈ Q, a student or system S is expectedretrieve a relevant fact f ∈ F , and appeal to theirown common knowledge, KS , when applying f toanswer q.

Figure 1 provides an example. Here, metals arethermal conductors is a core scientific fact avail-able in F . One way to apply this fact to decidewhether a steel spoon would let the most heattravel through is to appeal to common knowledgethat steel is metallic and heat travels through ther-mal conductors. In general, the expected commonknowledge is relatively simple (taxonomic facts,

1The dataset and the code for the models are available athttp://data.allenai.org/OpenBookQA.

arX

iv:1

809.

0278

9v1

[cs

.CL

] 8

Sep

201

8

definitions, object properties, etc.); the difficultylies in identifying it and meaningfully combiningit with a core fact from F to answer the question.

OpenBookQA questions are challenging as theyrequire multi-hop reasoning with partial contextprovided by F . Specifically, unlike existingdatasets for reading comprehension (RC), answer-ing questions on the back of a textbook (TQA),2

as well as question answering over structuredknowledge-bases (KBQA), the open book F thatcomes with OpenBookQA is not self-contained.A successful system must therefore go beyondthe typical challenges such as paraphrase match-ing and coreference resolution, without benefitingfrom the canonicalized and complete informationin KBQA.

Generating interesting open book questions isa difficult task. We used a multi-stage processstarting with F , using crowd-sourcing to generate(noisy) questions based on F that probe novel sit-uations, using an automatic filter to ensure hard-ness for retrieval and association based systems,using a crowd filter to ensure answerability by alay person, and further using an expert filter to en-sure higher quality in Dev and Test sets.

We evaluate a number of existing QA systemsfor science (without retraining) on OpenBookQA,finding that they perform surprisingly close to therandom guessing baseline of 25%. Human perfor-mance, on the other hand, is close to 92%.3

Motivated by recent findings of gameability ofNLP datasets (Gururangan et al., 2018), we alsodevelop and evaluate simple, attention-based, neu-ral baselines including a plausible answer detector(which ignores the question text completely) andan odd-one-out solver. These highlight inevitablehuman bias in any crowdsourced dataset, increas-ing performance on OpenBookQA to 48%.

Building upon a recent neural model for incor-porating external knowledge in the story cloze set-ting (Mihaylov and Frank, 2018), we propose aknowledge-aware neural baseline that can utilizeboth the open book F and common knowledge re-trieved from sources such as ConceptNet (Speeret al., 2017). While retrieving the most usefulpieces of knowledge remains an open challenge,our ‘oracle’ experiments with the fact f used whilegenerating a question q and an interpretation (by

2Only ∼5% of the TQA questions of Kembhavi et al.(2017) require additional common knowledge.

3To avoid ambiguity in the term ‘human performance’,Section 3.2 describes the specific randomized model we use.

the question author) of the additional knowledgek needed for q, provides valuable insight into thenature of this dataset: Facts from the open book Fare valuable (5% improvement) but not sufficient.Using both f and k increases the accuracy to 76%,but is still far from human level performance, sug-gesting the need for non-trivial reasoning to com-bine these facts.

To encourage further research on this new task,for each Train and Dev question q, OpenBookQAalso includes f as intermediate supervision signal,which may be viewed as a partial explanation forq. We leave closing the large gap to human perfor-mance as a challenge for the NLP community.

2 Related Work

By construction, answering OpenBookQA ques-tions requires (i) some base science facts froma provided ‘open book’, (ii) broader understand-ing about the world (common or commonsenseknowledge), and (iii) an ability to combine thesefacts (reasoning). This setup differs from severalexisting QA tasks, as summarized below.

Reading Comprehension (RC) datasets havebeen proposed as benchmarks to evaluate the abil-ity of systems to understand a document by an-swering factoid-style questions over this docu-ment. These datasets have taken various forms:multiple-choice (Richardson et al., 2013), cloze-style (Hermann et al., 2015; Onishi et al., 2016;Hill et al., 2016), and span prediction (Rajpurkaret al., 2016; Trischler et al., 2017; Joshi et al.,2017) However, analysis (Chen et al., 2016; Sug-awara et al., 2017) of these datasets has shown thatmany of the questions can be solved with contexttoken matching (Chen et al., 2017a; Weissenbornet al., 2017) or relatively simple paraphrasing.

To focus on the more challenging problemof reasoning across sentences, new datasetshave been proposed for multi-step RC. QAnga-roo (Welbl et al., 2018) have used a knowledge-base to identify entity pairs (s, o) with a knownrelation, r, which is also supported by a multi-hop path in a set of documents. They use struc-tured tuple queries (s, r, ?) and use all the docu-ments along the path as the input passage. Nar-rativeQA (Kocisky et al., 2017) is an RC datasetthat has been shown to require an iterative reason-ing about the narrative of a story. Similar to Open-BookQA, the questions were generated to ensurethat the answer is not a direct match or paraphrase

that can be retrieved with an IR approach. Mostrecently, Khashabi et al. (2018) proposed Mul-tiRC, a multiple-choice RC dataset that is de-signed to require multi-sentence reasoning and canhave multiple correct answers. Again, like mostRC datasets, it is self-contained.

Tasks with external knowledge. While manyof the RC datasets could benefit from common-sense or background knowledge, they are designedto be self-contained, i.e., solvable by the documentcontext alone. Datasets such as the Story ClozeTest (Mostafazadeh et al., 2016), MCScript,4 andProPara (Mishra et al., 2018) do require addi-tional domain knowledge about everyday events,scripts, and processes, respectively. However,these datasets need domain-specific modeling ofevents, whereas OpenBookQA appeals to broadcommon knowledge cutting across a variety oftypes and topics.

Stasaski and Hearst (2017) explore the creationof multi-hop questions and propose generatingstronger distractors for the multiple-choice setting.Their work, however, starts with structured knowl-edge, specifically a Biology ontology.

Lastly, many Science Question Answeringdatasets (e.g. Clark et al., 2016, 2018) have beenreleased that need broad external knowledge toanswer the questions. However, these questionsare not associated with a core set of facts, i.e., an“open book” used to define these questions. Asa result, the questions vary widely in style andcomplexity (Clark et al., 2018). In contrast, Open-BookQA focuses on a more well-defined subset ofscience QA, appealing to one core fact from theopen book and one (or few) relatively simple com-monly known supporting facts.

3 OpenBookQA Dataset

The OpenBookQA dataset consists of about 6,0004-way multiple-choice questions, each associatedwith one core fact from a “book” F of 1326 suchfacts, and an auxiliary set K of about 6000 ad-ditional facts. The questions were created via amulti-stage crowdsourcing and partial expert fil-tering process, discussed in Section 3.1.

The small “book” F consists of recurring sci-ence themes and principles, each of which can be(and here is) instantiated into multiple questions.

4SemEval-2018 Task 11: Machine Comprehension usingCommonsense Knowledge https://competitions.codalab.org/competitions/17184

For F , we use a subset of the WorldTree corpuswhich Jansen et al. (2018) have analyzed for suf-ficiency for elementary level science. The subsetwe use is taken from the 2287 WorldTree facts thatwere marked as “central” by the original authorsin at least one explanation. We further filter themdown to 1326 that appear general enough to be ap-plicable to multiple situations.

OpenBookQA additionally requires broad com-mon knowledge, which is expected to come fromlarge corpora, such as ConceptNet, Wikipedia, ora corpus with 14M science-related sentences usedby some existing baselines. The crowdsourcingprocess below also asks workers to mark a secondfact, k, needed for each question q, in addition tof . These second facts, unfortunately, were oftenincomplete, over-complete, or only distantly re-lated to q. We thus include in OpenBookQA theset K of such second facts only as auxiliary datafor optional use. We emphasize that K should notbe viewed as ‘gold’ additional facts, or as a substi-tute for broad common knowledge.

3.1 Crowdsourcing Process

The overall question generation and filteringpipeline is summarized in Figure 2. Given the“book” F of core facts, the process proceeds asfollows, starting with an empty question set Qsand an empty ‘second facts’ set K:

1. A crowd-worker5 w is shown a random sci-ence fact f from the set F .

2. w is asked to think of a second common fact,k, that may be combined with f to derive a new,valid assertion s.

3. w then converts s into a question-answer pairand extends this into a 4-way multiple choicequestion by adding 3 incorrect answer choices,qmc = (q, {c1, c2, c3, c4}), where one of the ci’sis the unique correct answer.

4. The system verifies qmc passes basic checkssuch as uniformity of answer choices.6

5. w then feeds the multiple-choice question qmc

to an information retrieval solver (Clark et al.,

5 We used Amazon Mechnical Turk, with workers fromNorth America and with a ‘masters’ level qualification.

6Specifically, it looks for: 1) exactly 4 answer choices; 2)no negation words to trivially fool baselines (no, none, not,isn’t, doesn’t, aren’t, don’t, won’t, except, can’t, shouldn’t,wouldn’t, couldn’t, mustn’t); 3) uniform answer choicelength: all with at most 3 or at least 4 words.

Fact, fFact, f

Core Facts

Write question, q

Verify hardness Fact, f

No

YesHard

Questions

Verify answerability

OpenBookQA

Hard AnswerableQuestions

De-biaschoices

Hard for IR Hard for ACME Uniform ChoiceLength

Figure 2: OpenBookQA question generation pipeline

2016) and a word association based solver (Tur-ney, 2017), and verifies that (a) neither of them an-swers qmc correctly and (b) the top 3 IR retrievedsentences are insufficient to answer qmc; if not, thequestion is edited and re-tried.

6. Question qmc is then shown to 5 new crowd-workers, who are asked to answer it.

7. If at least 4 out of 5 workers answer qmc cor-rectly, it is deemed answerable and the processcontinues. If not, qmc is discarded.

8. The answer choices of qmc are randomly shuf-fled to avoid unintended bias.7

9. qmc is associated with f as the core sciencefact and added to the question setQ. k is added tothe set K of additional (noisy) facts.

The Dev and Test splits were further filtered byan in-house expert to ensure higher quality.

3.2 Human PerformanceTo assess human accuracy on this dataset, we con-sider the following model: Each question q ∈ Qhas some (unknown) human accuracy pq, definedas the probability that a random human subject,chosen uniformly from a large pool H, wouldanswer q correctly. Thus, we can think of thisas defining a Bernoulli random variable, Xq ∼B(pq), whose mean is (unknown) pq. The aver-age human accuracy on Q under this model is:

H(Q) = 1

|Q|∑q∈Q

pq

where {pq | q ∈ Q} are unknown.With H as the set of crowd-workers (cf. Foot-

note 5), step 6 of the above question generation7Choice ‘A’ was the correct answer in 69% of the ques-

tions at the end of Step 4.

process is equivalent to obtaining 5 independentsamples, Xq,i, i ∈ I, |I| = 5, from B(pq). Wemust, however, be careful when using this data toestimate pq, as the same 5 samples were used todecide whether q makes it into the question set Qor not. For instance, if we had kept only thosequestions that all 5 workers answered correctly, itwould clearly be inaccurate to claim that the hu-man accuracy on Q is 100%. Nevertheless, it ispossible to re-use the judgments from Step 6 toapproximate H(Q) with high confidence, withoutposing the questions to new workers.

Intuitively, if all questions in Q were difficultto answer (i.e., all pq were small), it would be un-likely that all |Q| questions would pass the test inStep 6. We can use the contrapositive of this obser-vation to conclude that pq, on average, must havebeen high for q ∈ Q.

Formally, aggregating across all questions givesthe following empirical estimate of H(Q):

H(Q) = 1

|Q|∑q∈Q

1

|I|∑i∈I

Xq,i

=1

|Q||I|∑

q∈Q,i∈|I|

Xq,i

For analysis, we assume all samplesXq,i are in-dependent, i.e., every answer is obtained indepen-dently.8 An application of Hoeffding’s Inequal-ity (Hoeffding, 1963) shows that H(Q) convergestoH(Q) very rapidly as n = |Q||I| grows; specif-ically, H(Q) ≤ H(Q)+ t with probability at least1−exp(−2nt2); similarly for H(Q) ≥ H(Q)−t.In our Dev and Test sets, where |Q| = 500 and|I| = 5, this translates into H(Q) being at least

8Realistically, there is some dependence across questionsas a single worker may answer multiple questions. We leavea formal analysis of this setting as future work.

OpenBookQA Statistics# of questions 5957# of choices per question 4Avg. question sentences 1.08 (6)Avg. question tokens 11.46 (76)Avg. choice tokens 2.89 (23)Avg. science fact tokens 9.38 (28)Vocabulary size (q+c) 11855Vocabulary size (q+c+f) 12839Answer is the longest choice 1108 (18.6%)Answer is the shortest choice 216 (3.6%)

Table 1: Statistics for full OpenBookQA dataset. Par-enthetical numbers next to each average are the max.

H(Q) − 3% with probability over 98.8% and atleast H(Q) − 2.5% with prob 95.6%; we reportthe former as our conservative estimate on humanperformance.

3.3 Question Set AnalysisOpenBookQA consists of 5957 questions, with4957/500/500 in the Train/Dev/Test splits.9 Ta-ble 1 summarizes some statistics about the fulldataset. Each question has exactly four answerchoices and one associated fact used in the cre-ation process. We report the average length ofquestions, candidate choices, and associated facts,as well as how often is the longest/shortest choicethe correct one.

We analyzed 100 questions in the Train set tocapture the kind of common knowledge and rea-soning needed. For each, we wrote down the addi-tional common knowledge needed to answer thisquestion in addition to the original science fact. In21% of the cases, the crowdsourced question ac-tually tests for a fact that doesn’t necessarily needthe original science fact. For example, the ques-tion: “On a rainy day the clouds are (A) low (B)white (C) small (D) gray” was written based onthe science fact “clouds produce rain” but doesn’tneed this fact to answer it. We ignore such ques-tions in our analysis. For the remaining ques-tions, we categorized the additional facts into fivehigh-level categories (and collapsed the remain-ing facts into a catch-all OTHERS category) basedon previous approaches on similar science ques-tions (Clark et al., 2018; Jansen et al., 2016):

1. ISA: Basic taxonomic facts such as isa(tree,9Overall, 8140 questions were collected, of which 2183

were discarded in crowdsourcing Step 7.

Fact Type % Questions % FactsPROPERTY 29.11% 25.81%ISA 20.25% 17.20%BASIC 17.72% 19.35%DEFINITION 17.72% 15.05%CAUSAL 11.39% 9.68%OTHERS 13.92% 12.90%

Table 2: Percentage of questions and facts for the fivemost common type of additional facts. Note that %Questions does not add up to 100% since we count thepercentage of questions where at least one such fact isneeded.

living thing), isa(granite, rock).2. PROPERTY: Properties of objects such as

madeof(belt buckle, metal), has(mammals,four legs), contains(lemon juice, citric acid).

3. DEFINITION: Definitions of objects that maybe based on their appearance (tape is a plasticwith markings), working mechanism (tele-scope is a device that uses mirrors to viewobjects), etc.

4. CAUSAL: Causal facts such as causes(addinglemon juice to milk, milk to break down).

5. BASIC: General scientific fact that did not fitabove, e.g. squirrels eat nuts for food.

Table 2 presents the proportions of these factsin our analyzed question set. For each type offact, we calculate the percentage of questions thatneed at least one such fact (shown as % Ques-tions). We also calculate the overall percentageof each fact type across all the common knowl-edge facts (shown as % Facts). Most of our ques-tions need simple facts such as isa knowledge andproperties of objects, further confirming the needfor simple reasoning with common knowledge.Apart from these five major categories of facts,the catch-all OTHERS category contains common-sense facts (e.g., it is dark at night), world knowl-edge (e.g., Japan is often hit by earthquakes) andlexical rewrites10 (e.g., ad infinitum means overand over).

Most of our questions need simple facts thatshould be easily retrievable from any knowledge-base/textual corpora. On an average, each ques-tion needed 1.16 additional facts ignoring any lin-guistic variations. Despite the simplicity of theknowledge needed for these questions, as we show

10Of course, every question had lexical variations. Wemarked it when this was the only change to the core fact.

empirically, most baseline approaches achieve arelatively low score on this dataset (even whenthe core fact is provided). We claim that this isdue to the fact that the reasoning needed to answerthese questions is non-trivial. Table 3 shows fewquestions with the associated facts and high-levelreasoning needed to answer these questions. As-suming a model can extract the described relations(e.g. defn, contains), the QA system still needs tobe able to chain these facts together, identify theresulting relation and verify its expression for eachchoice. In the extreme case (as shown in the lastexample), even though only one additional fact isneeded to answer the question, it needs a systemto apply the core “general” science fact to a “spe-cific” situation.

4 Baseline Models

We evaluate the performance of several baselinessystems on the Dev and Test subsets of Open-BookQA. For each question, a solver receives 1point towards this score if it chooses the correctanswer, and 1/k if it reports a k-way tie thatincludes the correct answer. The “Guess All”baseline, which always outputs a 4-way tie, thusachieves a score of 25%, same as the expected per-formance of a uniform random baseline.

4.1 No Training, External Knowledge Only

Since OpenBookQA is a set of elementary levelscience questions, one natural baseline categoryis existing systems that have proven to be effec-tive on elementary- and middle-school level sci-ence exams. These pre-trained systems, however,rely only on their background knowledge and donot take the set F of core facts into account. Fur-ther, their knowledge sources and retrieval mecha-nism are close to those used by the IR solver that,by design, is guaranteed to fail on OpenBookQA.These two aspects place a natural limit on the ef-fectiveness of these solvers on OpenBookQA, de-spite their excellent fit for the domain of multiple-choice science questions. We consider four suchsolvers.

PMI (Clark et al., 2016) uses pointwise mutualinformation (PMI) to score each answer choice us-ing statistics based on a corpus of 280 GB of plaintext. It extracts unigrams, bigrams, trigrams, andskip-bigrams from the question q and each answerchoice ci. Each answer choice is scored based onthe average PMI across all pairs of question and

answer n-grams.TableILP (Khashabi et al., 2016) is an Integer

Linear Programming (ILP) based reasoning sys-tem designed for science questions. It operatesover semi-structured relational tables of knowl-edge. It scores each answer choice based on theoptimal (as defined by the ILP objective) “sup-port graph” connecting the question to that an-swer through table rows. The small set of theseknowledge tables, however, often results in miss-ing knowledge, making TableILP not answer 24%of the OpenBookQA questions at all.

TupleInference (Khot et al., 2017), alsoan ILP-based QA system, uses Open IE tu-ples (Banko et al., 2007) as its semi-structured rep-resentation. It builds these subject-verb-object tu-ples on-the-fly by retrieving text for each questionfrom a large corpus. It then defines an ILP pro-gram to combine evidence from multiple tuples.

DGEM (Khot et al., 2018) is a neural entail-ment model that also uses Open IE to produce asemi-structured representation. We use the adap-tation of this model to multiple-choice questionanswering proposed by Clark et al. (2018), whichworks as follows: (1) convert q and each ci intoa hypothesis, hi, and each retrieved fact into apremise pj ; and (2) return the answer choice withthe highest entailment score, argmaxi e(pj , hi).

4.2 No Training; F and Extr. Knowledge

We also consider providing the set F of corefacts to two existing solvers: the IR solver ofClark et al. (2016) (to assess how far simple word-overlap can get), and the TupleInference solver.

4.3 Trained Models, No Knowledge

We consider several neural baseline models thatare trained using Train set of OpenBookQA. Forease of explanation, we first define the notationused in our models. For a given question qmc =(q, {c1, c2, c3, c4}), we define the set of token se-quences , S = {q, c1, c2, c3, c4}. For each tokensequence s ∈ S, wsj is the jth and esj = Emb(wsj )is the embedding for this token. We use ns toindicate the number of tokens in s and d for thedimensionality of the embeddings.11 We modelmultiple-choice QA as multi-class classification:Given qmc, predict one of four class labels L =

11For all experiments we use d = 300 GloVe (Penningtonet al., 2014) embeddings pre-trained on 840B tokens fromCommon Crawl (https://nlp.stanford.edu/projects/glove/).

Question Science Fact Common Knowledge(Type)

ReasoningChallenge

What is the most likely to be an effect of acid rainon an aquatic environment? (A) increase in plantgrowth (B) increase in fish population (C)decrease in plant life (D) cleaner and clearer water

acid rain has anegative impact onwater quality

decrease in waterquality leads to adecrease in aquatic life(CAUSAL)

causes(x, y) ∧causes(y, z)⇒causes(x, z)

The moon’s surface (A) is smooth on the entiresurface (B) contains an internal core of cheese (C)is filled with lakes (D) contains large cavitiescause by explosions

the moon’s surfacecontains manycraters

Craters are largecavities caused byexplosions(DEFINITION)

contains(x, y) ∧defn(y, z)⇒contains(x, z)

As a car approaches you in the night (A) theheadlights remain at a constant (B) the headlightsturn off (C) the headlights become more intense(D) the headlights recede into the dark

as a source of lightbecomes closer,that source willappear brighter

Headlights of a car aresource of light(PROPERTY)

[lhs⇒ rhs]⇒[ground(lhs)⇒

ground(rhs)]

Table 3: Example training questions (with their correct choices marked) along with the facts and reasoningneeded. In the last example, the science fact states that lhs=“source of light becomes closer” implies rhs=“sourcewill appear brighter”. Grounding this rule based on the common-knowledge fact, produces a new rule: “As head-lights of the car come closer, headlights will appear brighter”

{1, 2, 3, 4}, where the true label is the correct an-swer index.

Embeddings + Similarities as Features. Wefirst experiment with a simple logistic regressionmodel (Mihaylov and Nakov, 2016; Mihaylov andFrank, 2016, 2017) that uses centroid vectors remb

s

of the word embeddings of tokens in s, and thencomputes the cosine similarities between the ques-tion and each answer choice, rcos

q,ci :

rembs =

1

ns

ns∑j=1

esj ∈ Rd

rcosq,ci = cos(remb

q , rembci ) ∈ R1

For each training instance, we build a feature rep-resentations ~f by concatenating these vectors andtrain an L2 logistic regression classifier:

~f = [rembq ; remb

c1..4 ; rcosq,c1..4 ] ∈ R5d+4

BiLSTM Max-Out Baselines. As a simpleneural baseline, we adapt BiLSTM max-outmodel (Conneau et al., 2017) to our QA task. Thatis, we first encode the question tokens and choicetokens ws1...ns

, independently with a bi-directionalcontext encoder (LSTM) to obtain a context (ctx)representation hctx

s1...ns= BiLSTM(es1...ns

) ∈Rns×2h Next, we perform an element-wise aggre-gation operation max on the encoded representa-tions hctx

s1..nsto construct a single vector:

rctxs = max(hctx

s1..ns) ∈ R2h. (1)

Given the contextual representations for eachtoken sequence, we experiment with three config-urations for using these representations for QA:

(a) Plausible Answer Detector. This baselinegoes to the extreme of completely ignoring q andtrying to learn how plausible it is for ci to be thecorrect answer to some question in this domain.This captures the fact that certain choices like ‘amagical place’ or ‘flying cats’ are highly unlikelyto be the correct answer to a science question with-out negation (which is the case for OpenBookQA).

We implement a plausible answer detector us-ing a choice-only model for predicting the an-swer by obtaining a score αci as: αci =W Tc r

ctxci ∈ R1, where W T

c ∈ R2h is aweights vector optimized during training, i ={1..4} is the index of the choice. To ob-tain the answer choice from the set of choicescores αc1..4 using argmax(softmax(αc1..4)),

where softmax(αci) =exp(αci )∑4

j=1 exp(αcj )as usual.

(b) Odd-One-Out Solver. It considers all 4 an-swer options jointly and selects the one that is leastsimilar to the others. This captures bias in humanauthored questions arising from the fact that cre-ating good quality incorrect answers is difficult.Workers generally start with the correct answer,and then come up with three incorrect ones. Thelatter often tend to be homogeneous or share othercommon properties (e.g., non-scientific terms) un-characteristic of the correct answer.

We implement this using a choice-to-choices at-tention model. For each choice ci, we calculate theattention to the other choices as αci,cj . We thensum these attention values to compute the atten-tion for ci to the rest of the choices, αci2cr(est) , andreturn the choice with the lowest sum. The atten-

tion is computed as αci,cj = Att(rctxci , r

ctxcj ) where

Att(u, v) =W T ([u; v;u · v; |u− v|]) ∈ R1

is a linear attention function and W ∈ R8h isa weight vector. We then compute αci2cr(est) =∑4

j=1 αci,cj (j 6= i) and select the answer with theindex ac2cr = argmin(softmax(αc1..42cr)).

(c) Question Match. This solver tries to predictwhich choice best matches the question (Nakovet al., 2016), without relying on external knowl-edge. To achieve that, we compute an attentionscore αq,ci between q and each of the choices qi asαq,ci = Att(rctx

q , rctxci ), and select the one with the

highest score. We also experiment with a modelwhere rctx

q and rctxci are obtained using token-wise

interaction proposed in ESIM (Chen et al., 2017b).

4.4 Trained Model with External KnowledgeLastly, we implement a two stage model for in-corporating external common knowledge, K. Thefirst module performs information retrieval on Kto select a fixed size subset of potentially relevantfacts KQ,C for each instance in the dataset (seeAppendix A). The second module is a neural net-work that takes (Q, C, KQ,C) as input to predictthe answer aq,c to a question Q from the set ofchoices C.

Knowledge-Enhanced Reader. As a baseknowledge-aware model, we use a variant ofthe model of Mihaylov and Frank (2018), im-plemented by extending our BiLSTM max-outquestion-match baseline (c). For each instancethe model reads the question q and answersc1..4 independently and attends to the set ofretrieved external knowledge facts KQ,C . Weencode each fact kj from KQ,C = k1..Nk

(Nk

is the number of facts) with same BiLSTMas used for q and c1..4 and construct a singlevector rctx

kj∈ R2h using Eq. 1. Having such

representations for each kj results in knowledgememory matrix Mk = rctx

k1..Nk∈ RNk×2h. Note

that Mk is dynamic memory, specific for eachinstance in the batch and is encoded in eachstep during training. This memory is used tocalculate a knowledge-aware representation,rkns =

∑((MT

k rctxs ).Mk) ∈ R2h. Each context

(ctx) representation rctxs (s ∈ S) is combined with

rkns to obtain a knowledge-enhanced representa-

tion rctx+kns = (rctx

s + rkns )/2. We then model

the knowledge-enhanced attention αknq,ci between

Solver Dev Test

Human solver 89.3* 91.7*Guess All (“random”) 25.0 25.0

NO TRAINING, KB ONLY (§4.1)TupleInference 15.9 17.9PMI (Waterloo corpus) 19.7 21.2TableILP 20.0 23.4DGEM 27.4 24.4

NO TRAINING, KB + F (§4.2)IR with F 25.5 24.8TupleInference with F 23.6 26.6DGEM with F 28.2 24.6

TRAINED MODELS, NO F OR KB (§4.3)Embedd+Sim 44.6 41.8ESIM 53.9±0.4 48.9±1.1Plausible Answer Detector 54.4±0.7 49.6±0.7Odd-one-out Solver 56.9±0.5 50.2±1.6Question Match 54.6±1.2 50.2±0.9

ORACLE MODELS, F AND/OR KB (§4.4)f 63.0±2.3 55.8±2.3f + WordNet 57.6±1.4 56.3±1.3f + ConceptNet 57.0±1.6 53.7±1.5f + k 80.2±1.1 76.9±0.7

Table 4: Scores obtained by various solvers on Open-BookQA, reported as a percentage± the standard devi-ation across 5 runs with different random seeds. Otherbaselines are described in the corresponding referencedsection. For oracle evaluation, we use the gold sciencefact f associated with each question, and optionally theadditional fact k provided by the question author. Bolddenotes the best Test score in each category.

q and ci as a linear combination of the ctx, kn andctx + kn representations as

αq,ci =W T [Att(rctxs , rctx

ci ); Att(rkns , r

knci );

Att(rctx+kns , rctx

ci ); Att(rctxs , rctx+kn

ci );

Att(rctxs , rkn

ci ); Att(rkns , r

ctxci );

Att(rctx+kns , rkn

ci ); Att(rkns , r

ctx+knci );

Att(rctx+kns , rctx+kn

ci )],

where W ∈ R9 is a weight vector initialized withthe ones vector and optimized during training. Wethen select the answer ci with the highest score.

5 Baseline Performance

The results for various baseline models are sum-marized in Table 4, grouped by method category.We make a few observations:

First, the task is largely solvable by a lay-person, as evidenced by the 92% score of crowd-workers. This is measured as described in Sec-

tion 3.2. We use annotations from Step 6 of thequestion generation process and report H(Q)−3%as a conservative lower estimate. As an additionalassessment, we also obtained 5 new annotationsfor 100 randomly chosen questions from each ofTrain, Dev, and Test sets. The performance re-mained similar at 88.6%, 90.2%, and 91.6%, resp.

The second group shows that pre-trainedstate-of-the-art solvers for multiple-choice sciencequestions perform poorly. One explanation is theircorrelation with the the IR method used for ques-tion filtering, as mentioned in Section 4.1.

The third group of results suggests that addingF to pre-trained models has a mixed effect, im-proving TupleInference by 8.7% but not changingDGEM.12 Unlike DGEM, TupleInference relieson brittle word-overlap similarity measures verysimilar to the ones used by IR. Since IR (KB) gets0% by design, TupleInference (KB) also has poorperformance and adding F helps it find better sup-port despite the brittle measures.

The fourth group demonstrates that carefullydesigned trainable neural models—even if sim-plistic and knowledge-free—can be surprisinglypowerful. For example, the “plausible answerdetector” can predict the correct answer with49.6% accuracy without even looking at the ques-tion. The “odd-one-out” solver, by consideringother answer choices, raises this to 50.2%. The“question match” solver, which simply comparesthe BiLSTM max-out encoding of the questionwith that of various answer choices, also achieves50.2%.13 Similar findings have been reported forseveral recent datasets (Gururangan et al., 2018),making it imperative to perform such tests early.

Interestingly, all of these neural knowledge-freebaselines simultaneously succeed on 34.4% of theDev questions, and simultaneously fail on 23.6%.For Question Match and ESIM we also experi-ment with ELMo (Peters et al., 2018) which im-proved their score on Test with 0.4% and 1.8%.

The final group demonstrates the need for ex-ternal knowledge and deeper reasoning. When the“oracle” science fact f used by the question authoris provided to the knowledge-enhanced reader,

12By design, IR with its default corpus gets 0% on Open-BookQA. Hence we don’t consider the effect of adding F ,which appears artificially magnified.

13This model also achieves the current best score, 33.87%,on the ARC Reasoning Challenge (Clark et al., 2018).When adapted for the textual entailment task by comparingBiLSTM max-out encodings of premise and hypothesis, itachieves 85% on the SciTail dataset (Khot et al., 2018).

it improves over the knowledge-less models byabout 5%. However, there is still a large gap,showing that the core fact is insufficient to answerthe question. When we also include facts retrievedfrom WordNet (Miller et al., 1990), the score im-proves by about 0.5%. Unlike the WordNet gain,adding ConceptNet (Speer et al., 2017) introducesa distraction and reduces the score. This suggeststhat ConceptNet is either not a good source ofknowledge for our task, or only a subset of itsrelations should be considered. Overall, externalknowledge helps, although retrieving the right bitsof knowledge remains difficult. In the last row ofTable 4, we use the oracle core fact along withquestion author’s interpretation of the additionalfact k. This increases the scores substantially, toabout 76%. This big jump shows that improvedknowledge retrieval should help on this task. Atthe same time, we are still not close to the hu-man performance level of 92% due to various rea-sons: (a) the additional fact needed can be sub-jective, as hinted at by our earlier analysis; (b)the authored facts K tend to be noisy (incomplete,over-complete, or only distantly related), also asmentioned earlier; and (b) even given the true goldfacts, performing reliable “reasoning” to link themproperly remains a challenge.

Sample predictions and analysis of questionsfrom Dev are provided in Appendix D.

6 Conclusion

We present a new dataset, OpenBookQA, of about6000 questions for open book question answering.The task focuses on the challenge of combining acorpus of provided science facts (open book) withexternal broad common knowledge. We show thatthis dataset requires simple common knowledgebeyond the provided core facts, as well as multi-hop reasoning combining the two. While simpleneural methods are able to achieve an accuracy ofabout 50%, this is still far from the human perfor-mance of 92% on this task. We leave closing thisgap for future research, and illustrate, via oracle-style experiments, the potential of better retrievaland reasoning on this task.

AcknowledgmentsThe authors would like to thank Lane Aasen forhelping develop the infrastructure for the crowd-sourcing task, and Madeleine van Zuylen for pro-viding expert annotation for the Dev and Testquestions.

ReferencesM. Banko, M. J. Cafarella, S. Soderland, M. Broad-

head, and O. Etzioni. 2007. Open information ex-traction from the web. In IJCAI.

D. Chen, J. Bolton, and C. D. Manning. 2016. Athorough examination of the cnn/daily mail readingcomprehension task. In ACL, pages 2358–2367.

D. Chen, A. Fisch, J. Weston, and A. Bordes. 2017a.Reading wikipedia to answer open-domain ques-tions. In ACL.

Q. Chen, X. Zhu, Z.-H. Ling, S. Wei, H. Jiang, andD. Inkpen. 2017b. Enhanced lstm for natural lan-guage inference. In ACL, pages 1657–1668.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabhar-wal, C. Schoenick, and O. Tafjord. 2018. Think youhave solved question answering? Try ARC, the AI2reasoning challenge. CoRR, abs/1803.05457.

P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord,P. D. Turney, and D. Khashabi. 2016. Combiningretrieval, statistics, and inference to answer elemen-tary science questions. In AAAI, pages 2580–2586.

A. Conneau, D. Kiela, H. Schwenk, L. Barrault, andA. Bordes. 2017. Supervised learning of universalsentence representations from natural language in-ference data. In EMNLP, pages 670–680.

M. Gardner, J. Grus, M. Neumann, O. Tafjord,P. Dasigi, N. F. Liu, M. Peters, M. Schmitz, andL. S. Zettlemoyer. 2017. AllenNLP: A deep seman-tic natural language processing platform. CoRR,abs/1803.07640.

S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz,S. R. Bowman, and N. A. Smith. 2018. Annota-tion artifacts in natural language inference data. InNAACL.

K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espe-holt, W. Kay, M. Suleyman, and P. Blunsom. 2015.Teaching machines to read and comprehend. InNIPS, pages 1693–1701.

F. Hill, A. Bordes, S. Chopra, and J. Weston. 2016. Thegoldilocks principle: Reading children’s books withexplicit memory representations. In ICLR.

W. Hoeffding. 1963. Probability inequalities for sumsof bounded random variables. Journal of the Amer-ican Statistical Association, 58(301):13–30.

P. Jansen, N. Balasubramanian, M. Surdeanu, andP. Clark. 2016. What’s in an explanation? charac-terizing knowledge and inference requirements forelementary science exams. In COLING.

P. A. Jansen, E. Wainwright, S. Marmorstein, and C. T.Morrison. 2018. WorldTree: A corpus of explana-tion graphs for elementary science questions sup-porting multi-hop inference. In LREC.

T. Jenkins. 1995. Open book assessment in comput-ing degree programmes 1. Technical Report 95.28,University of Leeds.

M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017.TriviaQA: A large scale distantly supervised chal-lenge dataset for reading comprehension. In ACL,pages 1601–1611.

A. Kembhavi, M. J. Seo, D. Schwenk, J. Choi,A. Farhadi, and H. Hajishirzi. 2017. Are yousmarter than a sixth grader? textbook question an-swering for multimodal machine comprehension. InCVPR, pages 5376–5384.

D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay,and D. Roth. 2018. Looking beyond the surface: Achallenge set for reading comprehension over multi-ple sentences. In NAACL.

D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Et-zioni, and D. Roth. 2016. Question answering viainteger programming over semi-structured knowl-edge. In IJCAI.

T. Khot, A. Sabharwal, and P. Clark. 2017. Answer-ing complex questions using open information ex-traction. In ACL.

T. Khot, A. Sabharwal, and P. Clark. 2018. SciTail:A textual entailment dataset from science questionanswering. In AAAI.

D. P. Kingma and J. L. Ba. 2015. Adam: a Method forStochastic Optimization. International Conferenceon Learning Representations 2015, pages 1–15.

T. Kocisky, J. Schwarz, P. Blunsom, C. Dyer, K. M.Hermann, G. Melis, and E. Grefenstette. 2017.The NarrativeQA reading comprehension challenge.CoRR, abs/1712.07040.

J. Landsberger. 1996. Study guides and strategies.Http://www.studygs.net/tsttak7.htm.

T. Mihaylov and A. Frank. 2016. Discourse relationsense classification using cross-argument semanticsimilarity based on word embeddings. In CoNLL-16 shared task, pages 100–107.

T. Mihaylov and A. Frank. 2017. Story Cloze EndingSelection Baselines and Data Examination. In LSD-Sem Shared Task.

T. Mihaylov and A. Frank. 2018. KnowledgeableReader: Enhancing Cloze-Style Reading Compre-hension with External Commonsense Knowledge.In ACL, pages 821–832.

T. Mihaylov and P. Nakov. 2016. SemanticZ atSemEval-2016 Task 3: Ranking relevant answers incommunity question answering using semantic sim-ilarity based on fine-tuned word embeddings. In Se-mEval ’16.

G. A. Miller. 1995. Wordnet: a lexical database forenglish. Communications of the ACM, 38(11):39–41.

G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, andK. J. Miller. 1990. Introduction to WordNet: An on-line lexical database. International Journal of Lexi-cography, 3(4):235–244.

B. D. Mishra, L. Huang, N. Tandon, W. tau Yih, andP. Clark. 2018. Tracking state changes in procedu-ral text: A challenge dataset and models for processparagraph comprehension. In NAACL.

N. Mostafazadeh, N. Chambers, X. He, D. Parikh,D. Batra, L. Vanderwende, P. Kohli, and J. Allen.2016. A Corpus and Evaluation Framework forDeeper Understanding of Commonsense Stories. InNAACL.

P. Nakov, L. Marquez, A. Moschitti, W. Magdy,H. Mubarak, a. A. Freihat, J. Glass, and B. Ran-deree. 2016. Semeval-2016 task 3: Communityquestion answering. In SemEval ’16, pages 525–545.

T. Onishi, H. Wang, M. Bansal, K. Gimpel, andD. McAllester. 2016. Who did what: A large-scaleperson-centered cloze dataset. In EMNLP, pages2230–2235, Austin, Texas.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang,Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, andA. Lerer. 2017. Automatic differentiation in py-torch. In NIPS-W.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,B. Thirion, O. Grisel, M. Blondel, P. Pretten-hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas-sos, D. Cournapeau, M. Brucher, M. Perrot, andE. Duchesnay. 2011. Scikit-learn: Machine learningin Python. Journal of Machine Learning Research,12:2825–2830.

J. Pennington, R. Socher, and C. Manning. 2014.GloVe: Global vectors for word representation. InEMNLP, pages 1532–1543.

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner,C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deepcontextualized word representations. In NAACL.

P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016.SQuAD: 100,000+ questions for machine compre-hension of text. In EMNLP, pages 2383–2392.

M. Richardson, C. J. Burges, and E. Renshaw. 2013.MCTest: A challenge dataset for the open-domainmachine comprehension of text. In EMNLP, pages193–203.

P. Singh, T. Lin, E. Mueller, G. Lim, T. Perkins, andW. Zhu. 2002. Open mind common sense: Knowl-edge acquisition from the general public. In Lec-ture Notes in Computer Science, volume 2519, pages1223–1237.

R. Speer, J. Chin, and C. Havasi. 2017. ConceptNet5.5: An open multilingual graph of general knowl-edge. In AAAI.

K. Stasaski and M. A. Hearst. 2017. Multiplechoice question generation utilizing an ontology. InBEA@EMNLP, 12th Workshop on Innovative Use ofNLP for Building Educational Applications.

S. Sugawara, H. Yokono, and A. Aizawa. 2017. Pre-requisite skills for reading comprehension: Multi-perspective analysis of mctest datasets and systems.In AAAI, pages 3089–3096.

A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni,P. Bachman, and K. Suleman. 2017. NewsQA: Amachine comprehension dataset. In Proceedings ofthe 2nd Workshop on Representation Learning forNLP, pages 191–200.

P. D. Turney. 2017. Leveraging term banks for answer-ing complex questions: A case for sparse vectors.CoRR, abs/1704.03543.

D. Weissenborn, G. Wiese, and L. Seiffe. 2017. Mak-ing neural qa as simple as possible but not simpler.In CoNLL, pages 271–280.

J. Welbl, P. Stenetorp, and S. Riedel. 2018. Construct-ing datasets for multi-hop reading comprehensionacross documents. TACL.

Y. Zhang, H. Dai, K. Toraman, and L. Song. 2018.KGˆ2: Learning to Reason Science Exam Questionswith Contextual Knowledge Graph Embeddings. InarXiv.

A Knowledge Retrieval Module

This module is the first part of a two stage modelfor incorporating knowledge from an externalsource K. For each instance (q, C) in the dataset,where q is a question and C = {c1, . . . , c4} aset of answer choices, it performs information re-trieval (IR) on K to select a fixed size subset Kq,C

of potentially relevant facts. The second module isa neural network that takes (q, C,Kq,C) as input,and predicts the answer aq,C .

For the IR module, we use TfIdfVectorizer14 tobuild vector representations ~qtfidf , ~citfidf and ~ktfidf

for the question q, choice ci ∈ C, and fact k ∈ Kbased on the tokens in the training set. We thencalculate similarity scores tq,k and tq,ci,k betweenq and ci, resp., and each of the external facts ink ∈ K:

tq,k = 1− sim(~qtfidf ,~ktfidf)

tq,ci,k = 1− sim(~citfidf ,~ktfidf) · tq,k,

where sim is implemented as cosine distance.Based on these similarity scores, we obtain a setKq,C of facts for each (q, C,K) as Kq ∪

⋃iKq,ci ,

where Kq and Kq,ci are the top Nk facts each withhighest similarity tq,k and tq,ci,k, respectively. Nk

is a hyper-parameter chosen from {5, 10, 20} so asto yield the best Dev set performance.

For experimentation with knowledge, we con-sider the ‘open book’ set of facts F in conjunc-tion with two sources of common knowledge: theOpen Mind Common Sense (Singh et al., 2002)part of ConceptNet (Speer et al., 2017), and itsWordNet (Miller, 1995) subset.

B Implementation and Training

Our neural models are implemented with Al-lenNLP15 (Gardner et al., 2017) and PyTorch16

(Paszke et al., 2017). We use cross-entropy lossand the Adam optimizer (Kingma and Ba, 2015)with initial learning rate 0.001. For the neuralmodels without external knowledge, we typicallytrain the model with a maximum of 30 epochs andstop training early if the Dev set accuracy does notimprove for 10 consecutive epochs. We also halvethe learning rate if there is no Dev set improve-ment for 5 epochs. For the neural models with ex-ternal knowledge, we typically train for 60 epochs

14Term frequency, Inverse document frequency based vec-torizer from scikit-learn (Pedregosa et al., 2011).

15https://allennlp.org16https://pytorch.org

with a patience of 20 epochs. For most of our neu-ral models, we use h = 128 as the LSTM hiddenlayer size. The embedding dropout rate is chosenfrom {0.1, 0.2, 0.5}, again based on the best Devset performance.

For each model configuration, we perform 5 ex-periments with different random seeds. For eachrun, we take the model with the best performanceon Dev and evaluate on Test. We report the aver-age accuracy for the best Dev score and the aver-age of the corresponding Test score± the standarddeviation across the 5 random seeds.

The code for the models and the configurationfiles required for reproducing the results are avail-able at http://data.allenai.org/OpenBookQA.

C Additional Experiments

C.1 Question Answering: ARC

We also perform experiments with the QuestionMatch system on the Challenge (hard) set of theAI2 Reasoning Challenge or ARC (Clark et al.,2018). We train several models with differentLSTM hidden sizes (128, 256, 384 (best), 512),and dropout of the embedding layer (0.0 (best),0.2, 0.5) on the questions from the ChallengeTrain set and take the model that has the highestaccuracy on the Dev set. The resulting systemscores 33.87% on the Challenge Test set, whichis 2.17% higher than the previous best score byZhang et al. (2018). The code and model con-figuration are available at https://github.com/allenai/ARC-Solvers.

C.2 Textual Entailment: SciTail

We perform textual entailment experiments on theScience enTailment dataset SciTail (Khot et al.,2018). We change the Question Match model to aclassic BiLSTM Max-Out (Conneau et al., 2017)for textual entailment, by replacing the questionq and a choice ci with the premise p and thehypothesis h, resp., and perform binary classifi-cation on the entailment labels (Entail, Neural).We run experiments with BiLSTM encoders withLSTM hidden size of 384 and share the encoderparameters between the premise and the hypoth-esis. Without additional hyper-parameter tuning,this yields entailment accuracy scores of 87.9%and 85.4% on the Dev and Test sets, respectively.

D Success and Failure Examples

We give some examples of questions that were an-swered correctly/incorrectly by various groups ofmodels. We include here the first three questionsin each case.

D.1 Neural Baseline SuccessesWe begin with three examples of questions thatall neural models without external knowledge(namely Question Match, Plausible Answer, One-Odd-Out, and ESIM from the fourth group in Ta-ble 5) predicted correctly.

A body may find its temperature to be lowered af-ter (A) water is heated up (B) fluid spreads frompores (C) the air becomes arid (D) the sky staysbrightOil is a non-renewable resource which tells usthat when (A) it can be remade (B) it can be foundin other places (C) there is an endless supply (D)the final barrel is gone, there supply is finishedMagma contains (A) particles of iron (B) Loadsof leaves (C) Soda (D) Silly Putty

Table 5: Sample questions predicted correctly(172/500) by all trained neural models without exter-nal knowledge.

In these examples, we observe that the correctanswer usually contains a word that is semanti-cally closer (than words in other answer choices)to an important word from the question: pores tobody; non-renewable (negative sentiment) to gone,finished (also negative sentiment); iron to magma(liquid rock).

D.2 Neural Baseline Failures, Oracle SuccessTable 6 shows example questions (with the Oraclefacts) from the Dev set that were predicted cor-rectly by the f + k Oracle model (405/500) butincorrectly by all of the 4 neural models withoutknowledge (69/405). In contrast to Table 5, a sim-ple semantic similarity is insufficient. The ques-tions require chaining of multiple facts in order toarrive at the correct answer.

D.3 Neural Baseline and Oracle Failures42/500 questions in the Dev set were predicted in-correctly by all models without external knowl-edge, as well as by the Oracle f + k model. InTable 7 we show 3 such questions. In all cases,the Oracle f + k model made an incorrect predic-tion with confidence higher than 0.9.

Frilled sharks and angler fish live far beneaththe surface of the ocean, which is why they areknown as (A) Deep sea animals (B) fish (C)Long Sea Fish (D) Far Sea Animals. Oraclefacts: (f ) deep sea animals live deep in theocean. (k) Examples of deep sea animals are an-gler fish and frilled sharks.Gas can fill any container it is given, and liquid(A) is standard weight and size (B) is the oppositeof variable (C) only needs a few (D) uses whatit needs. Oracle facts: (f ) Matter in the liq-uid phase has definite volume. (k) liquid cannotspread endlessly.When birds migrate south for the winter, they doit because (A) they are genetically called to (B)their children ask for them to (C) it is importantto their happiness (D) they decide to each year.Oracle facts: (f ) migration is an instinctive be-havior. (k) instinctive is genetic.

Table 6: Sample questions predicted correctly by thef+k Oracle model (405/500) but were predicted incor-rectly by all of the 4 neural models without knowledge(total of 69 out of 405).

As noted earlier, there are several broad reasonswhy even this so-called oracle model fails on cer-tain questions in OpenBookQA. In some cases, thecore fact f associated with a question q isn’t actu-ally helpful in answering q. In many other cases,the corresponding second fact k is noisy, incom-plete, or only distantly related to q. Finally, evenif f and k are sufficient to answer q, it is quite pos-sible for this simple model to be unable to performthe reasoning that’s necessary to combine thesetwo pieces of textual information in order to arriveat the correct answer.

In the shown examples, the first question fallsoutside the domain of Science where most of thecore facts come from. The scientific fact “(f ) Anexample of collecting data is measuring” is trans-formed into a question related to the law and judi-cial domain of collecting data for a (court) case.This is an indication that the model trained onthe Train set does not perform well on distant do-mains, even if the core facts are provided.

In the second question, we have an option allof these. Indeed, the selected answer seems themost relevant (a generalized version of the othertwo), but the model did not know that if we havean option all of these and all answers are plausible,

An example of data collection is: (A - 0.9977)Deleting case files on the computer, (B - 0.0000)Touching evidence without gloves, (C - 0.0004)speaking with a witness, (D - 0.0019) Throw-ing documents in the trash. Oracle facts: (f ) Anexample of collecting data is measuring. (k) In-terviews are used to collect data.If a farmland up the hill gets rainfall, what couldhappen to lower lands? (A - 0.0005) all of these,(B - 0.0245) they could get fertilizer washed tothem, (C - 0.9542) they could experience un-favorable chemical change in their lands, (D -0.0208) they could have their lands poisoned.Oracle facts: (f ) runoff contains fertilizer fromcropland. (k) fertilizers for certain crops couldpoison other crops or soil types.Layers of the earth include all but: (A - 0.0429)mantle, (B - 0.0059) center, (C - 0.0334) crust,(D - 0.9177) inner core. Oracle facts: (f ) thecrust is a layer of the Earth. (k) the last layer isthe outer core.

Table 7: Sample questions predicted incorrectly by allmodels models w/o knowledge, as well as the f + kOracle model, even though the Oracle model has con-fidence higher than 0.90.

it should decide if all answers are correct and notpick the “most likely” individual answer.

The third question again requires the model toselect a special type of aggregate answer (“all butxyz”), but the related Oracle facts are pointing toa specific answer.