Optimization Method for Weighting Explicit and Latent Concepts in Clinical Decision Support Queries

1/27

Introduction Method Experiments Conclusions

Optimization Method for Weighting Explicitand Latent Concepts in Clinical Decision

Support Queries

Saeid Balaneshin-kordan Alexander Kotov

Wayne State University

September 16, 2016

Balaneshin, Kotov Wayne State University

Optimization Method for Weighting Explicit and Latent Concepts in Clinical Decision Support Queries

2/27


Introduction

Method

Experiments

Conclusions



3/27


Introduction

Method

Experiments

Conclusions



4/27


Queries and Explicit and Latent Concepts (Example)

§ Query: 33-year-old male presents with severe abdominalpain one week after a bike accident, in which he sustainedabdominal trauma. He is hypotensive and tachycardic,and imaging reveals a ruptured spleen andintraperitoneal hemorrhage

§ Explicit concepts: ”bike accident”, “abdominal trauma”,“tachycardia”, “splenic rupture”, “intraperitonealhemorrhage”

§ Latent concepts: “splenic trauma”, “Injury of spleen”,“Traffic accidents”



5/27


Sources of Latent Concepts for Query Expansion

§ Top retrieved documents§ External domain-specific knowledge repositories (e.g.,

UMLS)§ External general-purpose resources (e.g., Wikipedia)



6/27


Retrieval model

§ Retrieval score of document D with respect to query Q:

sc(Q,D) =ÿ

TPTQ

ÿ

cPCT

λT(c)fT(c,D)

§ Weight of concept c with type T:

λT(c) =N

ÿ

n=1

wnφφn ,

§ Objective: find the weights wnφ that maximize infNDCG



6/27


Retrieval model§ Retrieval score of document D with respect to query Q:

sc(Q,D) =ÿ

TPTQ

ÿ

cPCT

λT(c)fT(c,D)

§ Matching function of concept c in document D (two-stagesmoothing method [Zhai and Lafferty, SIGIR’02]):

fT(c,D) = log((1´ λ)n(c,D) + µ

n(c,Col)|Col|

|D|+ µ+ λ

n(c,Col)|Col|

)


λT(c) =N

ÿ

n=1

wnφφn ,




6/27


Retrieval model

§ Retrieval score of document D with respect to query Q:

sc(Q,D) =ÿ

TPTQ

ÿ

cPCT

λT(c)fT(c,D)


λT(c) =N

ÿ

n=1

wnφφn ,




7/27


infNDCG retrieval metric by varying the weight ofone of the features

0.325

0.330

0.335

0.340

0.0 0.1 0.2 0.3 0.4

wGI

infN

DC

G



8/27


Proposed Method

§ Represent verbose domain-specific queries usingweighted unigram, bigram and multi-term concepts in aquery itself, top retrieved documents and knowledgebases.

§ Leverage Graduated Non-Convexity Optimization(GNC) method to jointly determine the importanceweights for the query and expansion concepts dependingon their type and source.



9/27


Introduction

Method

Experiments

Conclusions



10/27


0.325

0.330

0.335

0.340

0.0 0.1 0.2 0.3 0.4

wGI

infN

DC

G

∆ 0.025

0.33

0.34

0.0 0.1 0.2 0.3 0.4w GI

infN

DC

G

§ Sample infNDCG for the followingvalues of wφ:

ws,φ = [wφ,´M, . . . ,wφ,0, . . . ,wφ,M]

§ Using the fixed sampling interval(∆wφ):

wφ,m = wφ,0 +m∆wφ, m P [´M, . . . ,M]

§ Polynomial of degree K is used forsmoothing the objective function:

E(wφ,m) =K

ÿ

k=0

akmk, m P [´M, . . . ,M]



11/27


0.325

0.330

0.335

0.340

0.0 0.1 0.2 0.3 0.4

wGI

infN

DC

G

∆ 0.025

0.33

0.34

0.0 0.1 0.2 0.3 0.4w GI

infN

DC

G

§ Cost Function (Mean Square Error(MSE)):

εφ =1

2M + 1

Mÿ

m=´M

(E(wφ,m)´E(wφ,m))2

§ Optimal a:

a = (JTJ)´1JTws,φ

§ J is a Jacobian of the vector:

[J]m,k = (m´M)k, m P [0, 2M], k P [0,K] .



12/27


The first three iterations...

∆ 0.025

0.33

0.34

0.0 0.1 0.2 0.3 0.4w GI

infN

DC

G

(a) 1st Iteration

∆

0.338

0.340

0.342

0.344

0.00 0.01 0.02 0.03 0.04w GI

infN

DC

G

(b) 2nd Iteration

∆w

0.3426

0.3429

0.3432

0.3435

0.023 0.024 0.025 0.026 0.027w GI

infN

DC

G

(c) 3rd Iteration



13/27


Multi-variate optimization

§ Using Powell’s method, the multi-variate objectivefunction is decomposing into a series of univariateobjective functions to weight the n-th feature at the j-thiteration:

En,j(wnφ) = E([w1

φ, . . . , wn´1φ ,wn

φ, wn+1φ , . . . , wN

φ])



14/27


Algorithm

1: Identify explicit and latent concepts2: Randomly initialize the feature weights vector (wφ)3: for j = 1 : jmax do4: Randomly shuffle wφ

5: for n = 1 : N do6: for each sampling policy do7: Sample En,j(wn

φ)

8: Obtain En,j(wnφ,m)

9: Obtain the optimum point wnφ

10: Update n-th element of wφ by wnφ

11: end for12: end for13: if Convergence then14: Break15: end if16: end for



15/27


Introduction

Method

Experiments

Conclusions



16/27


Experimental Setup

§ All retrieval methods were implemented using Indri 5.9 IR toolkit§ We used INQUERY stoplist and Porter stemmer§ Very frequent terms (that occur in more than 15% of documents) and very rare

terms (that occur in less than 5 documents) were ignored for estimation oftranslation models, LDA and NMF.

§ We used the implementation of NMF from scikit-learn 17.1 (Double SingularValue Decomposition for non-random initialization of factors and CoordinateDescent solver for optimization)

§ Gibbs sampler for posterior inference of LDA parameters was run for 200iterations, and the convergence threshold for NMF was set to 10´5

§ Word embeddings were obtained by using word2vec (version 0.1c) tool 3 (byusing Skip-gram architecture for its training)



17/27


Explicit and Latent Query Concept Types

Concept Type Concept Source(s) Concept Representation Concept ExtractionQU Query unigrams all query unigramsQOB Query ordered bigrams all query bigramsQUB Query unordered bigrams all query bigramsQUU Query, UMLS unigrams MetaMapQUOB Query, UMLS ordered bigrams MetaMapQUUB Query, UMLS unordered bigrams MetaMapQDU Query, Wikipedia unigrams health-relatedness measureQDOB Query, Wikipedia ordered bigrams health-relatedness measureQDUB Query, Wikipedia unordered bigrams health-relatedness measureTU Top-docs unigrams direct identificationTOB Top-docs ordered bigrams direct identificationTUB Top-docs unordered bigrams direct identificationTUU Top-docs, UMLS unigrams MetaMapTUOB Top-docs, UMLS ordered bigrams MetaMapTUUB Top-docs, UMLS unordered bigrams MetaMapTDU Top-docs, Wikipedia unigrams health-relatedness measureTDOB Top-docs, Wikipedia ordered bigrams health-relatedness measureTDUB Top-docs, Wikipedia unordered bigrams health-relatedness measureUU UMLS unigrams UMLS relationshipsUOB UMLS ordered bigrams UMLS relationshipsUUB UMLS unordered bigrams UMLS relationships



18/27


Features (when concept sources are Query and UMLS)§ TF-IDF of concept c in the collection,§ Number of top retrieved documents containing concept c,§ Sum of retrieval scores of top-ranked documents

containing concept c,§ Average/Maximum collection co-occurrence of concept c

with other concepts in the query,§ Average/Maximum co-occurrence of concept c with other

query concepts in top retrieved documents,§ Does concept c have a UMLS semantic type that is effective

for medical query expansion?§ Node degree of concept c in the UMLS concept graph,§ Average distance between concept c in the UMLS concept

graph and other query, top document and related UMLSconcepts identified for a query.



19/27


Considered Semantic Types of UMLS concepts

Obtained using backward feature elimination process from[Limsopatham et al., OAIR’13]:

Semantic Group Semantic TypeChemicals & Drugs Clinical DrugDisorders Disease or SyndromeDisorders Injury or PoisoningDisorders Sign or SymptomProcedures Therapeutic or Preventive Procedure



20/27


Number of Top-docs and number of their Concepts

●●

● ●

●

● ● ● ●● ● ● ●

●●

●

●●

0.26

0.28

0.30

0.32

5 10 15

Number of Top−docs

infN

DC

G

● ●●

●● ●

● ●● ●

●● ● ● ● ● ● ●

0.28

0.29

0.30

0.31

0.32

0.33

5 10 15

Number of Concepts

infN

DC

G



21/27


Performance ComparisonQuery set TREC 2014 CDS track TREC 2015 CDS trackMethod infNDCG infAP P@5 infNDCG infAP P@5Two-Stage 0.1945 0.0493 0.3533 0.2110 0.0449 0.4200Wiki-Orig 0.2069 0.0550 0.3533 0.2193 0.0457 0.4133UMLS-Orig 0.2074 0.0569 0.3867 0.2206 0.0478 0.4400UMLS-TD˚ 0.2724 0.0810 0.4133 0.2503 0.0614 0.4667PQE 0.2796 0.0873 0.4733 0.2792 0.0762 0.4400Wiki-TD˚ 0.2883 0.0944 0.4600 0.2519 0.0633 0.4600TREC best 0.2631 0.0757 0.4067 0.2928 0.0777 0.4467INTGR-LS 0.3114 0.0993 0.4867 0.2987 0.0792 0.4800INTGR 0.3401 0.1229 0.5267 0.3135 0.0873 0.5200

§ INTGR significantly out- performs all of the best performing baselines in termsof all retrieval metrics.

§ Using graduated non-convexity as a uni-variate optimization method results in:§ 5-9% improvement of retrieval accuracy in terms of infNDCG,§ 10-23% improvement in terms of infAP and§ 8% improvement in terms of P@5 on different query sets.



22/27


Effectiveness of different type of knowledge bases

Table : TREC 2014 CDS track topics

Method infNDCG infAP P@5INTGR using no knowledge bases 0.2673 0.0875 0.4601

INTGR using only Wikipedia 0.2975 0.0936 0.4533(11.30%) (6.97%) (-1.47%)

INTGR using only UMLS 0.3309 0.1170 0.5200(23.79%) (33.71%) (13.02%)

INTGR using UMLS and Wikipedia 0.3401 0.1229 0.5267(27.23%) (40.46%) (14.47%)

§ the methods based on semantic concepts, UMLS is a better choice thanWikipedia with respect to all metrics, if only one knowledge repository is used

§ combining both knowledge bases results in better retrieval accuracy than usingany one of them individually



22/27


Effectiveness of different type of knowledge bases

Table : TREC 2015 CDS track topics

Method infNDCG infAP P@5INTGR using no knowledge bases 0.2771 0.0758 0.4633

INTGR using only Wikipedia 0.2954 0.0779 0.4667(6.60%) (2.77%) (0.09%)

INTGR using only UMLS 0.3012 0.0786 0.5033(8.67%) (3.93%) (7.93%)

INTGR using UMLS and Wikipedia 0.3135 0.0873 0.5200(13.14%) (15.17%) (11.52%)

§ the methods based on semantic concepts, UMLS is a better choice thanWikipedia with respect to all metrics, if only one knowledge repository is used

§ combining both knowledge bases results in better retrieval accuracy than usingany one of them individually



23/27


Comparison of INTGR with the baselines in terms ofP@k for k ď 10:

0.3

0.4

0.5

0.6

1 2 3 4 5 6 7 8 9 10

k

P@

k

(d) TREC 2014 CDS track topics

0.40

0.45

0.50

0.55

1 2 3 4 5 6 7 8 9 10

k

P@

k

(e) TREC 2015 CDS track topics

§ INTGR has slightly lower accuracy improvement over its best-performingbaseline and Two-Stage on the testing query set than on the training query set

§ Improvement that INTGR achieves over Two-Stage is much higher than theimprovement of the best performing base- line over Two-Stage.



24/27


Topic-level differences of the infNDCG values

16

27

15

10

19

24

29

9 26

4 2 25

12

18

23

21

20

8 6 11

22 3

13

17

28

30 1 5

71

4

mean = 0.0518σ = 0.12

−0.2

0.0

0.2

0.4

Query ID

infN

DC

G−

diffe

rence

(f) TREC 2014 CDS track topics

§ infNDCG of INTGR is greater than that of Wiki-TD‹ on 67% of the queries

§ infNDCG of INTGR is greater than that of PQE on 73% of the queries

§ Highest Improvement: 28-year-old female with neck pain and left arm numbness 3 weeks after

working with stray animals. Physical exam initially remarkable for slight left arm tremor and spasticity.

Three days later she presented with significant arm spasticity, diaphoresis, agitation,

difficulty swallowing, and hydrophobia.§ Lowest Degradation: 85-year-old man who was in a car accident 3 weeks ago, now with 3 days of

progressively decreasing level of consciousness and impaired ability to perform activities of daily living.

§ Highest Improvement:A 46-year-old woman with sweaty hands, exophthalmia, and weight loss

despite increased eating.§ Lowest Degradation:An 89-year-old man with progressive change in personality, poor memory, and

myoclonic jerks.



24/27


Topic-level differences of the infNDCG values

61

52

22

83

04 1

62

37 1

23 9 27

18

26

17

10

29

24

19

25

2 5 12

01

11

32

11

4 8

mean = 0.0345

σ = 0.0734

−0.2

0.0

0.2

0.4

Query ID

infN

DC

G−

diffe

rence

(g) TREC 2015 CDS track topics

§ infNDCG of INTGR is greater than that of Wiki-TD‹ on 67% of the queries

§ infNDCG of INTGR is greater than that of PQE on 73% of the queries§ Highest Improvement:A 46-year-old woman with sweaty hands, exophthalmia, and weight loss

despite increased eating.

§ Lowest Degradation:An 89-year-old man with progressive change in personality, poor memory, and

myoclonic jerks.



25/27


Introduction

Method

Experiments

Conclusions



26/27


Conclusions

§ We proposed a method to represent CDS queries usingexplicit concepts from the original query and the latentconcepts from the top retrieved documents and knowledgebases

§ We proposed the features to individually weigh eachquery concept depending on its type and source

§ We proposed to use graduated optimization method todirectly optimize the parameters of the concept basedretrieval model with respect to the target retrieval metric

§ Proposed method significantly outperformsstate-of-the-art baselines as well as best systems in CDStrack of TREC 2014 and 2015



Many Thanks to the ACM SIGIR Student Travel Grant!

27/27

Optimization Method for Weighting Explicit and Latent Concepts in Clinical Decision Support Queries

Science

Transcript of Optimization Method for Weighting Explicit and Latent Concepts in Clinical Decision Support Queries