H. Lundbeck A/S 21 Apr 2023 1
Assessing the effectiveness of your current search and retrieval function
Anna G. Eslau, Information Specialist, H. Lundbeck A/SMarianne Lykke Nielsen, Associate Professor, Royal School of Library and
Information Science
Case story evaluating human metadata indexing versus automatic query expansion using a
corporate thesaurus
H. Lundbeck A/S 21 Apr 2023 2
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 3
Motivation
• A lot of money has been invested – but does our current search and retrieval function perform as expected?
• An advanced and time consuming indexing task has been laid upon our end users – but is our current indexing strategy effective?
• Do we have - as high quality - alternatives to manual indexing?
H. Lundbeck A/S 21 Apr 2023 4
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 5
Case study - Research partners
• H. Lundbeck A/S– Pharmaceutical company– 5000 employees, in > 40 countries– Information systems with electronic documents– Corporate thesaurus– Users and search requests
• Royal School of Librarianship– Thesaurus research expertise– Domain knowledge from former research project
• Ensight A/S– Verity K2 search engine and Intelligent Classifier– Technical expertise
H. Lundbeck A/S 21 Apr 2023 6
Purpose of case study
To evaluate1. Information retrieval based on
controlled, human indexing (controlled metadata)
2. Information retrieval based on full-text indexing, with thesaurus-based automatic query expansion
H. Lundbeck A/S 21 Apr 2023 7
Case study – Retrieval system and indexing policy
• Electronic document management system (EDMS) and bibliographic information system containing research documentation
• Indexing policy– Written indexing policy– Mandatory training of indexers – Corporate Thesaurus– Human, controlled indexing– Topical checklist/Facetted indexing
• Searching by controlled metadata and full-text• Domain specific thesaurus containing 5,500
concepts and 16,000 terms
EDMS 1/2 - Indexing
EDMS 2/2 – Searching
H. Lundbeck A/S 21 Apr 2023 10
Lundbeck Thesaurus 1/3
H. Lundbeck A/S 21 Apr 2023 11
Lundbeck Thesaurus 2/3
H. Lundbeck A/S 21 Apr 2023 12
Lundbeck Thesaurus 3/3
H. Lundbeck A/S 21 Apr 2023 13
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 14
Test design - Retrieval performance of different search strategies
• Three different search strategies were evaluated:1. Searches based on natural language
(words from original request) in full text 2. Searches based on natural language in full
text expanded with words from thesaurus (query expansion with synonyms and narrower terms)
3. Searches based on (manually assigned) controlled keywords in selected metadata fields
H. Lundbeck A/S 21 Apr 2023 15
Test design - Query expansion
• Search for information about intravenous administration of a drug AND Alzheimer’s disease:
’Intravenous OR IV OR Intravenously OR…’AND’Alzheimer’s disease OR Alzheimer’s disorders
OR Alzheimer type dementia OR…..’
H. Lundbeck A/S 21 Apr 2023 16
Lundbeck Thesaurus
H. Lundbeck A/S 21 Apr 2023 17
Test design - Test persons and retrieval system
• Persons– Query expansion tests were carried out by the
thesaurus manager and did not involve end-users– Evaluation of search results were carried out by end
users – 4 subject experts (Medical advisers) who had formerly answered the search requests
• System– Verity K2 search system was used as test retrieval
system for the query expansion test work– Original document management systems were used
as retrieval system for the metadata searches
H. Lundbeck A/S 21 Apr 2023 18
Test design - Test thesaurus
• The Lundbeck Thesaurus was the test thesaurus. The thesaurus formed basis for query formulations: - Synonyms and narrower terms were picked
from the thesaurus for the test searches based on expansion of natural language in full text searches
- Preferred keywords were picked from the thesaurus for the test searches based on controlled keywords in selected metadata fields.
H. Lundbeck A/S 21 Apr 2023 19
Test design - Test collection
• 25,384 document objects from two different sources– 24,369 document objects from a
bibliographical (BRS) information system (internal research reports and published research articles)
– 1015 documents from the full-text EDMS system (internal research reports)
H. Lundbeck A/S 21 Apr 2023 20
Test design - Search requests
• 10 search requests were selected from a set of searches which in real life had been carried out in the corporate information systems
Work task 7: You are a medical reviewer. A physician has contacted you. He would like to have data on the use of Citalopram and Reboxetine together to treat resistant depression. He wants any reporting of possible interactions.
Indicative request: Find reports, papers or case stories that investigate the possible interaction of Citalopram and Reboxetine
on resistant depression
H. Lundbeck A/S 21 Apr 2023 21
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 22
Findings – Performance
Recall (% relevant docs retrieved out of total no. of relevant docs)
Searchstrategy
SJ1 SJ2 SJ3 SJ4 SJ5 SJ6 SJ7 SJ8 SJ9 SJ10
Full-text 42 52 88 38 79 54 39 3 12 7
Full-text with
QE (syn)
64 68 100 76 89 100 39 100 100 68
Full-text with
QE (syn, nt)
100 90 100 87 89 100 39 100 100 73
Metadata 0 0 0 33 29 61 100 1 0 45
Precision (% relevant docs out of all retrieved docs) went down from 33% to 24% with query expansion
SJ = Search Job, QE = Query Expansion
Findings – Human indexing problemsIndexing problems Frequency
(%)N = 156
Explanations
1. Conceptual analysis
A1 Omission of topic 69 • Indexers fail to remember facets and topics that are not explicitly mentioned in indexing policy or checklist• Indexing policy recommend to check specific document sections such as title, table of content, etc. why indexers, especially in long documents, tend to omit topics from other document sections
÷
A2 Misinterpretation and wrong perspective of topic
14 • Indexers misunderstand topic due to lack of topical and domain knowledge ÷
A3 Omission of implicit topic
2 • Difficult for indexers to determine degree of topical interpretation and domain-orientation ÷
2. Translation
B1 Topic indexed at BT level
7
B2 Topic indexed with incorrect keyword
8 • Indexers misunderstand meaning and use of keywords ÷
H. Lundbeck A/S 21 Apr 2023 24
Findings – Other metadata
• Topical retrieval and situational relevance ranking - the importance of contextual parameters– Document type– Publication year– Source– Language– Author
H. Lundbeck A/S 21 Apr 2023 25
Findings – Thesaurus
• Thesaurus– Relevant synonyms (acronyms with multiple
meanings should be omitted)– Logical hierarchies– High topical relevance
H. Lundbeck A/S 21 Apr 2023 26
Findings – Documents and search requests
• Document collection– OCR scanned documents may contain errors =>
false positive hits– Large (>100 pages) full text documents lower
precision (irrelevant hits)
• Search requests– If people are searching using very general terms, QE
will be extremely complicated/extensive, the more levels of QE we choose to add
– Different types of facets result in• Different relevance assessment according to
document types• Different recall in metadata search
H. Lundbeck A/S 21 Apr 2023 27
Findings – Search software
• Search software settings are important– Stemming– Case sensitivity– Character sensitivity (())– Number of search terms allowed– Zoning
H. Lundbeck A/S 21 Apr 2023 28
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 29
Conclusion – Thesaurus and QE
• A domain specific thesaurus are well suited for QE
• QE improves recall but decreases precision
• QE with synonyms only are in most cases sufficient
H. Lundbeck A/S 21 Apr 2023 30
Conclusion - Search result display
• Users want to see all hits (recall is important)
• Manual sorting of search results by (other than topical) metadata is requested by the users
• Ranking based on e.g. zoning is not always useful
H. Lundbeck A/S 21 Apr 2023 31
Conclusion – Indexing policy
• Difficult to obtain complete, accurate and exhaustive human indexing
• Findings suggest that searching for specific topics should be based on full-text indexing, supported by thesaurus based query expansion
• Human indexing should focus on few, important, well-defined topics, e.g. used to develop taxonomies for broad browsing
• Non-Topical context metadata are important in assessment of document relevance– Document type– Publication year– Source– Language– Author
H. Lundbeck A/S 21 Apr 2023 32
Conclusion – Implications for Lundbeck
• Lundbeck Thesaurus has been integrated with bibliographic information system to perform automated QE
• EDMS upgrade planned where QE should be possible
• OCR scanning of existing documents are considered
• Metadata on document types in EDMS are evaluated and under revision (simplified)
• New models on how to add metadata are considered (dictionaries)
• New indexing tools for the users are developed (indexing keys)
H. Lundbeck A/S 21 Apr 2023 33
Agenda
• Motivation• Case study
– Research partners– Purpose– Test design– Findings– Conclusions
• Summing up
H. Lundbeck A/S 21 Apr 2023 34
Summing up
• If your current search and retrieval function does NOT perform as expected, your organisation may loose important information
• You may have an indexing strategy (which is good…) but evaluation may reveal that the resource investments could be used even better
• Evaluation is important, it may save your organisation money over time
Top Related