Annual reports and feedback from UMLS licensees
Kin Wah Fung MD, MSc, MAKin Wah Fung MD, MSc, MA
The UMLS TeamThe UMLS Team
National Library of MedicineNational Library of Medicine
Workshop on the Future of the UMLS Semantic Network
2
BackgroundBackground
One of the requirements of the UMLS license is to submit One of the requirements of the UMLS license is to submit an annual report of UMLS usage within 30 days of the end an annual report of UMLS usage within 30 days of the end of each yearof each year
Up to last year, a free-form email (with suggested section Up to last year, a free-form email (with suggested section headings) was the main way to collect the annual reports. headings) was the main way to collect the annual reports. Disadvantages:Disadvantages: low response rate (usually less than 100 received each year)low response rate (usually less than 100 received each year) difficult to analyzedifficult to analyze
From 2005, online form to collect annual reports. From 2005, online form to collect annual reports. Advantages:Advantages: more structured questionsmore structured questions easier to analyzeeasier to analyze targetted email reminders possible => higher response ratetargetted email reminders possible => higher response rate
3
Format of online annual reportFormat of online annual report
Logon page – identity authenticated with UMLS license Logon page – identity authenticated with UMLS license numbernumber
Main report page:Main report page: user info: affiliation, activity type, computing environmentuser info: affiliation, activity type, computing environment usage of UMLSusage of UMLS feedback on user support and documentationfeedback on user support and documentation
Optional sections for feedback on:Optional sections for feedback on: MetathesaurusMetathesaurus Semantic NetworkSemantic Network Lexical toolsLexical tools UMLS Knowledge Source ServerUMLS Knowledge Source Server
4
Results (up to March 31, 2005)Results (up to March 31, 2005)
Total responsesTotal responses 1,0051,005
Valid responsesValid responses 988988
Have used the UMLSHave used the UMLS 495495
Have used the Semantic Have used the Semantic NetworkNetwork
210210
Provided feedback on the Provided feedback on the Semantic NetworkSemantic Network
7676
5
Results (up to March 31, 2005)Results (up to March 31, 2005)
Total responsesTotal responses 1,0051,005
Valid responsesValid responses 988988
Have used the UMLSHave used the UMLS 495495
Have used the Semantic Have used the Semantic NetworkNetwork
210210
Provided feedback on the Provided feedback on the Semantic NetworkSemantic Network
7676
6
AffiliationAffiliation
AcademicAcademic 54 (71%)54 (71%)
CommercialCommercial 12 (16%)12 (16%)
Self-employedSelf-employed 4 (5%)4 (5%)
Government – non NLMGovernment – non NLM 3 (4%)3 (4%)
Government – NLMGovernment – NLM 2 (3%)2 (3%)
Not-for-profitNot-for-profit 1 (1%)1 (1%)
7
Main activity of organization/companyMain activity of organization/company
ResearchResearch 27 (36%)27 (36%)
EducationEducation 21 (28%)21 (28%)
Software developmentSoftware development 11 (14%)11 (14%)
Health care providerHealth care provider 10 (13%)10 (13%)
Information serviceInformation service 2 (3%)2 (3%)
Health insuranceHealth insurance 1 (1%)1 (1%)
OtherOther 3 (4%)3 (4%)
MissingMissing 1 (1%)1 (1%)
8
How long have you been using the UMLS?How long have you been using the UMLS?
Range 2 months – 15 yearsRange 2 months – 15 years Average: 4.3 yearsAverage: 4.3 years Median 2.3 yearsMedian 2.3 years Significantly longer than the rest of respondents - Significantly longer than the rest of respondents -
average 2.6 years, median 1 year average 2.6 years, median 1 year (Independent-samples T test, 2- tailed p = 0.003)(Independent-samples T test, 2- tailed p = 0.003)
9
How are you using the UMLS?How are you using the UMLS?
Terminology researchTerminology research 4949
Map between terminologiesMap between terminologies 3939
Indexing and information retrievalIndexing and information retrieval 3636
Support natural language processingSupport natural language processing 3434
Discover concepts in biomedical textsDiscover concepts in biomedical texts 3232
Acquire biomedical knowledgeAcquire biomedical knowledge 3232
Design, construct and maintain your terminologyDesign, construct and maintain your terminology 2929
Support terminology serviceSupport terminology service 1515
Access specific terminologiesAccess specific terminologies 1515
Publish and distribute your terminologyPublish and distribute your terminology 66
OtherOther 22
10
What kind of information do you use the What kind of information do you use the UMLS to process?UMLS to process?
Clinical informationClinical information Non-clinical informationNon-clinical information
Findings & diagnosisFindings & diagnosis 4646 Genomics & proteomicsGenomics & proteomics 2323
ProceduresProcedures 3131 Bibliographic informationBibliographic information 2121
Laboratory reportsLaboratory reports 2727 Consumer health informationConsumer health information 1010
DrugsDrugs 2626 BillingBilling 33
Clinical guidelinesClinical guidelines 1414 OtherOther 33
Clinical outcomeClinical outcome 1010
Nursing informationNursing information 88
OtherOther 88
TotalTotal 170170 6060
11
Usage of the Semantic NetworkUsage of the Semantic Network
How have you used the Semantic Network?How have you used the Semantic Network? in a research setting – 69in a research setting – 69 in a production environment – 7in a production environment – 7
Ways in which Semantic Network is used:Ways in which Semantic Network is used: in natural language processingin natural language processing
word sense disambiguationword sense disambiguation tagging of natural language expressionstagging of natural language expressions natural language generationnatural language generation
ontological uses ontological uses finding concept distancefinding concept distance reference for building biomedical ontologiesreference for building biomedical ontologies
other usesother uses suggesting relationships between UMLS conceptssuggesting relationships between UMLS concepts restricting search of terms in the Metathesaurusrestricting search of terms in the Metathesaurus research on structure of Semantic Networkresearch on structure of Semantic Network
12
Components of Semantic Network usedComponents of Semantic Network used
Do you use the Semantic Network on its own or in Do you use the Semantic Network on its own or in conjunction with the Metathesaurus concepts?conjunction with the Metathesaurus concepts? on its own - 15on its own - 15 in conjunction - 58in conjunction - 58
Semantic typesSemantic types 64 (84%)64 (84%)
Semantic groupsSemantic groups 48 (63%)48 (63%)
Semantic relationships (e.g. location_of, Semantic relationships (e.g. location_of, causes)causes)
43 (57%)43 (57%)
Semantic relations (e.g. Tissue|Semantic relations (e.g. Tissue|location_of|Neoplastic Process)location_of|Neoplastic Process)
39 (51%)39 (51%)
13
Have you extended or modified the Semantic Have you extended or modified the Semantic Network?Network?
Semantic Semantic TypeType
Semantic Semantic RelationshipRelationship
Semantic Semantic RelationRelation
Area of applicationArea of application
11 ++ ++ creating terminologycreating terminology
22 ++ supporting terminology serversupporting terminology server
33 ++ ++ building knowledge basesbuilding knowledge bases
44 ++ ++ natural language generationnatural language generation
55 ++ ++ ++ creating terminologycreating terminology
66 -- natural language processing of clinical natural language processing of clinical documentsdocuments
77 -- natural language processing of clinical and natural language processing of clinical and biomedical documentsbiomedical documents
88 -- +/-+/- concept distance estimationconcept distance estimation
99 +/-+/- +/-+/- +/-+/- clinical information systemsclinical information systems
14
What improvements would you suggest?What improvements would you suggest?
more semantic types (4) more semantic types (4) better description of semantic types and how they are assigned (2) better description of semantic types and how they are assigned (2) multiple hierarchiesmultiple hierarchies more semantic relationsmore semantic relations more uniform granularitymore uniform granularity extension of relations to concept levelextension of relations to concept level evolve towards a formal ontologyevolve towards a formal ontology
16
One possible way to prioritize changes?One possible way to prioritize changes?
Higher demandHigher demand Lower demandLower demand
17
One possible way to prioritize changes?One possible way to prioritize changes?
Higher demandHigher demand Lower demandLower demand
Less work Less work involved involved
More work More work involvedinvolved
18
One possible way to prioritize changes?One possible way to prioritize changes?
Higher demandHigher demand Lower demandLower demand
Less work Less work involved involved
- better description of semantic - better description of semantic types and how they are types and how they are assigned (2)assigned (2)
- multiple hierarchies- multiple hierarchies
- more semantic relations- more semantic relations
More work More work involvedinvolved
- more semantic types (4)- more semantic types (4) - more uniform granularity- more uniform granularity
- extension of relations to concept - extension of relations to concept levellevel
- evolve towards a formal - evolve towards a formal ontologyontology
19
One possible way to prioritize changes?One possible way to prioritize changes?
Higher demandHigher demand Lower demandLower demand
Less work Less work involved involved
- better description of semantic - better description of semantic types and how they are types and how they are assigned (2)assigned (2)
- multiple hierarchies- multiple hierarchies
- more semantic relations- more semantic relations
More work More work involvedinvolved
- more semantic types (4)- more semantic types (4) - more uniform granularity- more uniform granularity
- extension of relations to concept - extension of relations to concept levellevel
- evolve towards a formal - evolve towards a formal ontologyontology
Top Related