An(Introduc=on(to(( Clinical(Natural(( … Ques&ons!addressed!in!this!½!day!tutorial!!! (((•...

Leonard D’Avolio Dina Demner-‐Fushman Wendy W. Chapman

An Introduc=on to Clinical Natural

Language Processing

Ques&ons addressed in this ½ day tutorial

•  What is natural language processing (NLP)?

•  Why does it maDer?

•  How is it being used? •  What are the basic approaches to it?

•  What considera=ons are there in using it?

•  How should you evaluate it? •  Where is the field today?

•  Where is it headed?

•  How can I learn more? 2

Format

•  Focus on clinical NLP •  Some discussion of literature & phenotyping

•  70% basic, 30% intermediate

•  A lot of material covered at a high level

•  PLEASE interrupt with ques=ons •  Planned 15 minute break

•  Don’t forget your survey •  Part 2 in Jefferson East

Outline

1.  What is NLP and how is it used in medicine? (Dina)

2.  Goals and challenges of clinical NLP (Wendy)

3.  The methods of NLP (Leonard)

4.  Annota=on & evalua=on (Dina) 5.  Implementa=on considera=ons (Wendy)

6.  Current state, future progress, available resources (Leonard)

Dina Demner-‐ Fushman

What is NLP and how is it being used

in medicine?

Why natural language processing?

•  Increasing amounts of biomedical literature

–  Extrac=ng facts, rela=ons, events into knowledge repositories (text mining)

–  Model organism database cura=on

–  Ques=on answering (TREC Genomics track)

–  Literature based discovery

•  Increasing demands for use of EMR data –  Phenotyping for genomic-‐

related analysis –  Linking evidence for

Evidence-‐based medicine –  Biosurveillance –  Quality measures

•  Majority of EMR data is free text

•  Classify

•  Extract

•  Summarize

What is natural language processing? Electronic Medical Records

MEDLINE Ar=cles / Abstracts

Natural Language Processing

Structured Data (Machine

interpretable)

Examples of Uses of Clinical NLP

•  Classify

•  Extract

•  Summarize

BioNLP Examples •  Classify

•  Extract

•  Summarize

Classify a chief complaint into a syndrome category

“SOB/cough” = Respiratory

Triage of ar=cles likely to have experimental evidence

Find evidence to assign top-‐level GO terms

Examples of Uses of Clinical NLP

# of lymph nodes removed during colorectal cancer surgery

Extract bio-‐molecular events

phosphoryla=on of TRAF2 -‐> (Type:Phosphoryla=on,

Theme:TRAF2)

•  Classify

•  Extract

•  Summarize

•  Classify

•  Extract

•  Summarize

BioNLP Examples

Examples of Uses of Clinical NLP •  Classify

•  Extract

•  Summarize

From a H&P note, list chronic condi=on

Summarize family history of prostate cancer

BioNLP Examples •  Classify

•  Extract

•  Summarize

Summarize full text documents

Gene Reference into func=on (GeneRif)

Biomedical Usage

Wendy Chapman

Goals and Challenges of Clinical NLP

Detect Nosocomial Infec=ons

An=bio=c Assistant* (LDS Hospital)

* Evans RS, et al. N Eng J Med 1998

temperature

white blood cell count

infiltrate compa=ble with pneumonia

1) Alert physician: pa=ent might need An=infec=ve therapy

2) Suggest type and dose of an=bio=c -‐ allergies -‐  insurance -‐  age -‐  renal func=on

infiltrate compa=ble with pneumonia chest x-‐ray report

Phenotyping Iden=fy symptoms that co-‐occur with lung cancer

ED Report

NLP System

Feature 1: Feature 2:

… Feature n:

Classifier

Predic=ve of Lung Cancer

Two Simple NLP Tasks

1.  Find all relevant phrases in ED Report

2.  Map individual phrases to standard features

Your Task

•  Highlight every instance of features in sample report

•  Mark most specific instance –  E.g., “chest pain” preferred over “pain”

•  Do not mark =me, nega=on, or uncertainty

Find Relevant Features in ED Report

Produc=ve cough

Dyspnea

Sinusi=s

Pneumonia

Wheezing

Tachypnea

Cervical adenopathy

Possible values: acute, historical, absent 16

How did you do?

Why is NLP Difficult?

Named en=ty recogni=on

Linguis=c varia=on Polysemy

Finding valida=on Implica=on

Contextual aDribute assignment

Nega=on Uncertainty Temporality

Discourse processing

Report structure Coreference

Linguis=c Varia=on Different Words with the Same Meaning

Deriva=on medias=nal = medias=num

Inflec=on opacity = opaci=es; cough = coughed

Synonymy Addison’s Disease: Addison melanoderma, adrenal insufficiency, adrenocor=cal insufficiency, asthenia pigemntosa, bronzed disease, melasma addisonii, …

Chest wall tenderness: chest wall did demonstrate some slight tenderness when the pa=ent had pressure applied to the right side of the thoracic cage 19

Polysemy One Word With Mul=ple Meanings

General polysemy Pa=ent was prescribed codeine upon discharge The discharge was yellow and purulent

Acronyms and Abbrevia=ons APC: ac=vated protein c, adenomatosis polyposis coli, adenomatous polyposis coli, an=gen presen=ng cell, aerobic plate count, advanced pancrea=c cancer, age period cohort, alfalfa protein concentrated, allophycocyanin, anaphase promo=ng complex, anoxic precondi=oning, anterior piriform cortex, an=body producing cells, atrial premature complex, …

Nega=on Approximately half of all clinical concepts in dictated

reports are negated*

Explicit nega=on “The medias=num is not widened”

Medias=nal widening: absent

Implied absence without nega=on “Lungs are clear upon ausculta=on”

Rales/crackles: absent Rhonchi: absent Wheezing: absent

*Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. Evalua=on of nega=on phrases in narra=ve clinical reports. Proc AMIA Sym. 2001:105-‐9.

Uncertainty

Unsure

treated for a presump=ve sinusi=s

Reasoning

It was felt that the pa=ent probably had a cerebrovascular accident involving the lev side of the brain. Other differen=als entertained were perhaps seizure and the pa=ent being post-‐ictal when he was found, although this considera=on is less likely

Reason for exam

R/O out pneumonia. 22

Temporality Clinical reports tell a story

Past medical history History of CHF presen=ng with shortness of lev-‐sided chest pain.

Hypothe=cal or non-‐specific men=ons He should return for fever or increased shortness of breath.

Temporal course of disease Pa=ent presents with chest pain … Aver administra=on of nitroglycerin, the chest pain resolved.

Finding Valida=on Men=on of a finding in the text does not guarantee the pa=ent has the finding

She received her influenza vaccine His temperature was taken in the ED

Some findings require values Fever

Temperature 38.5C Oxygen desatura=on

Oxygen satura=on low Oxygen satura=on 85% on room air

Implica=on

Audience for pa=ent reports is physicians Lay people less accurate at determining if a chest x-‐ray report shows evidence of Pneumonia Pneumonia not men=oned in 2/3 of posi=ve reports

Sentence level inference “There were hazy opaci=es in the lower lobes” à

Localized infiltrate Report level inference

Localized infiltrates à Probable pneumonia

Report Structure

Anatomic Loca=on some=mes in sec=on header NECK: no adenopathy.

Some sec=ons carry more weight IMPRESSION: atelectasis

Some reports contain pasted text difficult to process

Cardiovascular: [ ] Angina [ ] MI [x ] HTN [ ] CHF [ ] PVD [ ] DVT [ ] Arrhythmias [ ] Previous PTCA [ ] Previous Cardiac Surgery [ ] Nega=ve -‐ Denies CV problems

Coreference

Chest x-‐ray again shows a well-‐circumscribed nodule located in the lev upper lobe. The tumor has increased in size since the last exam with a diameter of approximately 2 cm. How big is the nodule? Has the nodule increased in size? Where is the tumor?

References "Mutalik PG, Deshpande A, Nadkarni PM. Use of general-purpose negation detection to augment concept indexing of medical documents: a quantitative study using the UMLS. J Am Med Inform Assoc. 2001 Nov-Dec;8(6):598-609."

"Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001 Oct;34(5):301-10."

"Uzuner O, Zhang X, Sibanda T. Machine learning and rule-based approaches to assertion classification. J Am Med Inform Assoc. 2009 Jan-Feb;16(1):109-15."

"Sneiderman CA, Rindflesch TC, Aronson AR. Finding the findings: identification of findings in medical literature using restricted natural language processing. Proc AMIA Annu Fall Symp. 1996:239-43"

"Fiszman M, Chapman WW, Aronsky D, Evans RS, Haug PJ. Automatic detection of acute bacterial pneumonia from chest X-ray reports. J Am Med Inform Assoc. 2000 Nov-Dec;7(6):593-604."

"Zhou L, Melton GB, Parsons S, Hripcsak G. A temporal constraint structure for extracting temporal information from clinical narrative. J Biomed Inform. 2005 Sep 15."

"Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform. 2009 Oct;42(5):839-51."

Leonard D’Avolio

The Methods of Clinical NLP

how this stuff works

Developing / using NLP is a process The NLP Process

Find the right documents

The NLP Process

Create the “gold standard”

The NLP Process

Train the system The NLP Process

Evaluate the system

The NLP Process

Methods of NLP

•  A number of approaches have evolved

•  Simple rules-‐based

•  Symbolic, gramma=cal NLP

•  Machine learning

•  NLP can be considered a series of transforms

Think “PIPELINE”

Research Scenario Posi=ve margins aver RRP = 2 x 4 =mes risk of cancer recurrence

Goal: EXTRACT MARGIN STATUS

Simple Rules-‐Based Approach

Heuris=cs, Probabili=es, Combina=on of the two

Simple Rules-‐Based Approach

Simple

Regular expressions included in many programming languages

Great for semi-‐structured (consistently formaDed) targets

PaDerns must consider all possible configura=ons.

Symbolic or Gramma=cal NLP Approach

Many of the same components…

…plus…

POS tagging & phrase chunking are ac=ve areas of research

Some=mes called “concept mapping”

Robust – reduces complexity by mapping to standard terms

Great for mapping large numbers of concepts

Complex – more steps, more opportuni=es to introduce error

Which controlled vocabulary?

Can be slow 48

Classifica=on Model

Several open source ML packages available

(decision trees, SVMs, neural nets)

Machine Learning Approach

Machine Learning: Which ‘features’ to learn from?

Targeted approach = high accuracy

Capable of learning from examples

Great for extrac=ng few predetermined targets

Requires manual training

New target = new training effort

Machine Learning Approach

Also used increasingly in POS tagging & mapping to ontologies

Machine Learning Approach Not limited to a step in the pipeline

What if RegExs don’t cut it?

Swap them out for Gramma=cal NLP approach

The Hybrid Approach

References

Natural language processing: Manning & Schutze. Founda=ons of Natural Language Processing. MIT Press. 1999

Regular Expressions: Java Tutorial, hDp://java.sun.com/docs/books/tutorial/essen=al/regex/

Machine learning: WiDen & Frank. Data Mining, Prac=cal Machine Learning Tools and Techniques with Java Implementa=ons. Academic Press. 2001

Dina Demner-‐Fushman

Annota=on and Evalua=on

Manual Annota=on

•  Purposes •  Levels •  Guidelines •  Methods (manual, assisted) •  Tools •  Format (embedded/standoff) •  Collec=on size (number needed, representa=ve sample)

•  Annotators (linguists, domain experts, crowdsourcing)

•  Annotator agreement •  Preserva=on/dissemina=on/repor=ng 57

Annota=on purposes

•  System development –  Rule genera=on (manual or automa=c) –  Sta=s=cal modeling – supervised machine learning (training + valida=on/op=miza=on)

•  Evalua=on –  Tes=ng on a held-‐out set or cross-‐valida=on

•  Clinical data quality assurance •  Reusable collec=on (corpus)

Annota=on levels •  Meta –  informa=on about the corpus

•  Document –  type, relevancy to topic, quality, structure

•  Pragma=c –  purpose of a sentence interpreted in context using world knowledge, involves inference

•  Discourse –  contextual features, links between instances of concepts, or concepts across sentences

•  Seman=cs –  formal representa=on of meaning using concepts, frames

•  Syntax –  part of speech, phrases, rela=ons between phrases

•  Lexical 59

Meta: XYZ hospital, respiratory problems, … Document: Pa=ent #13, Discharge summary # 1, …

Annota=on levels example

Annota=on guidelines

•  Define task and annota=on purpose •  Be clear •  Be concise •  Avoid bias •  Itera=vely refine using representa=ve sample

•  Come to consensus

•  Finalize before annota=ng reference standard

Annota=on methods Trade-‐off of manual vs assisted annota=on

Accuracy / consistency

Speed-‐up

Training

Annota=on tools

•  Read and write formaDed text (markup language)

•  Allow to define/ link annota=on schema

•  Provide for span selec=on & markup (color-‐coding)

•  Minimize annota=on steps and naviga=on

•  Link ontologies •  Compute inter-‐annotator agreement

•  Provide for reconcilia=on of annotator disagreement

•  Provide web-‐service/API 63

References Linguis=c annota=on: Wynne M (editor). 2005. Developing

Linguis4c Corpora: a Guide to Good Prac4ce. Oxford: Oxbow Books. Available from hDp://ahds.ac.uk/linguis=c-‐corpora/

Issues: Hovy E, Lavid J. Corpus annota=on tutorial. hDp://www.lrec-‐conf.org/lrec2008/IMG/pdf/Corpus_annota=on.Tutorial-‐outline.pdf

Clinical text AMIA NLP-‐SIG annota=on project Available from hDp://understandit.net/r02.01.11/index.php?=tle=Annota=onProjectAnnota=onSchema

Annotator agreement: Hripcsak G, Rothschild AS. Agreement, the f-‐measure, and reliability in informa=on retrieval. J Am Med Inform Assoc. 2005 May-‐Jun;12(3):296-‐8. hDp://www.ncbi.nlm.nih.gov/pmc/ar=cles/PMC1090460/

Overview Judges Metrics

Evalua=on methods Large-‐scale evalua=ons

Dina Demner-‐Fushman

Evalua=ng NLP

Evalua=on roots Human/biomedical studies

Subjects Outcomes/Sta=s=cs

Sovware/NLP evalua=on Quality of the algorithm Quality of implementa=on Quality of results

Human-‐computer interac=on Usability tes=ng

Heuris=c User-‐centered Scenario-‐based

What is evaluated?

Sovware System components Black/Glass-‐box (results/algorithm and implementa=on)

Task-‐specific (intrinsic/extrinsic) Manual/automa=c

Applica=on Interface (HCI)

Qualita=ve/quan=ta=ve Access (API, service)

Impact/Outcome Healthcare process Pa=ent’s experience

Judges: who is evalua=ng?

•  Experts vs. convenience popula=on vs. end-‐users •  How many?

•  Consensus (reliability, agreement) vs. pyramid

•  Capturing judgments in reusable test collec=ons

Evalua&on Metrics

Reference Standard

NLP output

posi&ve nega&ve

posi&ve a (TP) b (FP)

nega&ve c (FN) d (TN)

Recall (Sensi=vity) = a / (a + c) Precision (PPV) = a / (a + b) Fall-‐out (1-‐Specificity) = b / (b + d) = 1 -‐ d/(b+d)

Evalua&on Metrics

F-‐measure: harmonic mean of precision and recall

What if enumera=ng all true posi=ve/nega=ve examples is not possible or prac=cal?

Mean average precision, binary preference 70

Reference Standard

NLP output

posi&ve nega&ve

posi&ve a (TP) b (FP)

nega&ve c (FN) d (TN)

Sovware evalua=on

•  Establish strong baseline –  For extrac=on of pa=ent-‐oriented outcomes from MEDLINE abstracts selec=ng

3 last sentences achieves 75% accuracy

•  Select evalua=on metrics appropriate for the task –  U=lity measure for text categoriza=on (Genomics track) –  “This measure contains coefficients for the u=lity of retrieving a relevant and

retrieving a nonrelevant document normalized by the best possible score” hDp://ir.ohsu.edu/genomics/2005protocol.html

End-‐user evalua=on

Use Log files observa=on Surveys

Impact Cost / =me Outcomes

Community-‐wide evalua=ons

Format Post-‐hoc ( TREC ) Gold standard provided Pros/cons cost vs. coverage

Clinical I2b2 cmc

References Friedman CP, WyaD JC. Evalua=on Methods in Biomedical Informa=cs. 2nd

ed., New York: Springer, 2006.

Sparck-‐Jones K, Galliers JR. Evalua=ng Natural Language Processing Systems. Springer, 1996.

van Rijsbergen C.J. Informa=on Retrieval, 2nd ed. London: BuDerworths, 1979. hDp://www.dcs.gla.ac.uk/Keith/pdf/Chapter7.pdf

Hripcsak G, Wilcox A. Reference standards, judges, and comparison subjects: roles for experts in evalua=ng system performance. J Am Med Inform Assoc. 2002 Jan-‐Feb;9(1):1-‐15. Available online from hDp://www.ncbi.nlm.nih.gov/pmc/ar=cles/PMC349383

Passonneau RJ, Nenkova A. Evalua=ng Content Selec=on in Human-‐ or Machine-‐Generated Summaries: The Pyramid Scoring Method hDp://www1.cs.columbia.edu/~library/TR-‐repository/reports/reports-‐2003/cucs-‐025-‐03.pdf

Wendy Chapman

Implementa=on Considera=ons

Implemen&ng NLP

• Ge�ng an NLP system up and running •  Case study

Preprocessing Post-‐processing NLP System

The devil is in the details

Remove extraneous characters control characters foreign characters (é)

Remove extra line feeds, etc.

pul-‐_monary

Preserve/enhance sec=on labels “IMPRESSION:_”

Reformat to improve readability

De-‐iden=fy

Obtain source feeds

Assess completeness

De-‐duplicate

Clean, “sec=onize,” format

De-‐iden=fy

Load database

Hand-‐off to NLP system

Quality assurance

Slide courtesy David Carrell 78

Obtain source feeds

Assess completeness

De-‐duplicate

Clean, “sec=onize,” format

De-‐iden=fy

Load database

Sample

Hand-‐off to NLP system

Quality assurance

Human Subjects/IRB

Source system manager

Network/database administrator

Programmer

Investigator

Informatics/NLP expert

Clinician (“domain expert”)

Chart abstractor

Slide courtesy David Carrell A lot of tasks and a lot of people 79

80 Slide courtesy David Carrell

Which CUIs map to Produc4ve Cough?

Which combina=on of radiological findings & aDributes = evidence of acute bacterial pneumonia?

Does the pa=ent have a recurrent breast cancer?

Map NLP output to your vocabulary and

your task

Instance

Report

Pa=ent

Case Study Case-‐control observa=onal GWAS study

Hypothesis Biomarkers in pa=ents with prostate cancer can be used to predict =me to survival, informing course of care

Targeted Phenotype

•  Prostate cancer •  Co-‐morbidi=es •  Basic demographics •  Disease characteris=cs

•  TNM Staging •  Gleason score •  PSA

•  Treatments administered •  Surgery •  Chemo •  Watchful wai=ng

The NLP Process

Defining the target

Data challenges

Prostate cancer •  ICD-‐9 codes don’t cut it

•  VA Boston: 18% of path reports 60 days before / aver 1st ICD-‐9 were prostate cancer related

•  No standardized =tles on path reports •  Biopsy? •  Post-‐op?

Into NLP just to find the right documents Phenotyping with NLP is really several projects in one

The NLP Process

Extrac=ng key variables

Annota=on challenges

TNM Staging Gleason score PSA

•  Gleason score different on post-‐op than biopsy •  Pathological vs. es=mate

•  Pa=ent level vs. document level •  PSA at 4 visits •  Conflic=ng Gleason scores

•  Start with a wish list and whiDle down •  Cost vs. benefit will become clear

•  Define categorical variables •  Versus highligh=ng strings

•  Create clear instruc=ons •  Training •  Pilot, pilot, pilot

•  Plan for several itera=ons

The NLP Process

Crea=ng your training / test sets

Designing your “gold standard”

•  Several variables = several measures of accuracy •  What if tumor staging F – measure is .97 but co-‐

morbidi=es is .6? •  Effects must be accounted for in study design

The NLP Process

Extrac=ng key variables

NLP algorithm development

•  Are you reinven=ng the wheel?* •  Is it important that it scale

•  Other projects? •  Beyond your ins=tu=on?

*PraD, AW. 1969“Automated processing of medical English” Interna=onal Conference on Computa=onal Linguis=cs, Sweden

Current state, future progress, available

resources

Leonard D’Avolio

State of the Science

NLP is not “off the shelf” • Opportunity to reduce effort

Several approaches can yield similar performance

• i2b2 challenge First increase in open source components

• Weka, MMTx, Stanford parser

• Lots of ‘glue code’ Now increase in open source frameworks

• GATE, UIMA

End-‐to-‐end informa=on retrieval using open source frameworks

•  ARC 94

Progression of Field—More Resources

“Closed” Concept Mapping Systems •  MedLEE •  Knowledge Map •  MVCS

Open Components •  Stanford Parser •  IBM Parser •  OpenNLP •  Weka (ML) •  MALLET (ML) •  UMLS •  NegEx (nega=on)

Open Frameworks •  UIMA •  GATE

Open Concept Mapping Systems •  MetaMap •  HITEx (GATE) •  Topaz (GATE) •  cTAKES (UIMA) •  MedKAT (UIMA)

Open Corpora •  Cincinna= •  PiDsburgh NLP Repository •  i2b2 •  MIMIC 1 & 2

Open IR Systems •  ARC (UIMA + MALLET)

Tools Registries •  RDS •  ORBIT •  Eagle-‐I Hosted Environments •  iDASH •  VINCI

Future of NLP

Informa=on quality – context is key •  Error propagates in pipelines •  Informa=on not captured for our secondary uses •  Scrap idealized test sets

Greater code reuse •  Less glue code •  Will allow focus on improving specific components

Increase in open source data sets & shared task challenges Drive adop=on of NLP

•  More data driving greater demand / new uses •  Reduce current dependency on system developers

Current Process

D’Avolio et. al. “Evalua=on of a generalizable approach to informa=on retrieval using the Automated Retrieval Console.” 2011. 17(4)

What it should be

D’Avolio et. al. “Evalua=on of a generalizable approach to informa=on retrieval using the Automated Retrieval Console.” 2011. 17(4)

Best approach to NLP?

Worst approach to NLP?

Resources WEKA: hDp://www.cs.waikato.ac.nz/ml/weka/

MALLET: hDp://mallet.cs.umass.edu/

MetaMap: hDp://mmtx.nlm.nih.gov/

UMLS: hDp://www.nlm.nih.gov/research/umls/

OpenNLP: hDp://opennlp.sourceforge.net/

HITEx (hosted by i2b2): hDps://www.i2b2.org/resrcs/hive.html

cTAKES: hDps://cabig-‐kc.nci.nih.gov/Vocab/KC/index.php/OHNLP_Documenta=on_and_Downloads

UIMA: hDp://incubator.apache.org/uima/

GATE: hDp://gate.ac.uk/

ARC: hDp://research.maveric.org/mig/arc.html

Resources (cont) Topaz: hDp://www.dbmi.piD.edu/blulab/resources.asp#Topaz

NegEx: hDp://code.google.com/p/negex/

ConText: hDp://www.dbmi.piD.edu/chapman/ConText.html

Cincinna= Pediatric Corpus: hDp://www.computa=onalmedicine.org/project/cpc.php

PiDsburgh NLP Repository: hDp://www.dbmi.piD.edu/blulab/nlprepository.html

MIT MIMIC Repository (structured and unstructured):

hDp://mimic.mit.edu/mimic-‐ii-‐database.html

ORBIT Project: hDp://orbit.nlm.nih.gov/

iDASH: hDp://iDash.ucsd.edu

Contact informa&on

Leonard D’Avolio, PhD Associate Center Director for Biomedical

Informa=cs MAVERIC, VA Boston Healthcare System Leonard.davolio@va.gov

Wendy Chapman, PhD Division of Biomedical Informa=cs University of CA, San Diego wwchapman@UCSD.edu

Dina Demner-‐Fushman, MD, PhD Na=onal Library of Medicine ddemner@mail.nih.gov

An(Introduc=on(to(( Clinical(Natural(( … Ques&ons!addressed!in!this!½!day!tutorial!!! (((•...

Documents

Transcript of An(Introduc=on(to(( Clinical(Natural(( … Ques&ons!addressed!in!this!½!day!tutorial!!! (((•...

Introduc)on Results* Conclusions

Enterprise Application Integration Whatis It

CSE351 Lec1b Whatis - GitHub Pages

Introduc neumologia 1

Introduc)on - CTgoodjobs

Whatis%a%Hallucinogen?% · Some%HallucinogenicHistory% • Use’of’natural’hallucinogens’ predates’historical’record’ • Work’on’psychopharmacology’of’

ProvisionalDiagnoses,ProblemDescriptors… ARefresher* · Introduc

Whatis Social Media and Why Should I Care?

Gpj whatis xd_v4

00: Introduc

WHATIS - AMS

Whatis ADR

Introduc)on*to* - University at Buffalobina/cse487/fall2011/Oozie.pdf · Whatis* Oozie? • Oozie*is*aworkﬂow*scheduler*for*Hadoop* • Originally,*designed*atYahoo!*for*their*complex*search*

Whatis YA Lit 2007

Whatis happeningsyriafactsheet 09_30_11

00 introduc-to piping

Introduc)on - Ecommerce Europe

GraphdatabaseIntroducon - Georgia State Universitytinman.cs.gsu.edu/~raj/8711/sp15/GraphDatabases.pdf · 2018-05-17 · Agenda • Whatis%NoSQL?% • Whatis%aGraph,%Anyway?% • Whatis%aGraph%Database?%

Whatis a national plus school IV

Ch 2 whatis marketingppt

Introduc)onto - University at Buffalobina/cse487/fall2011/Oozie.pdf · Whatis* Oozie? • OozieisaworkﬂowschedulerforHadoop • Originally,designedatYahoo!fortheircomplexsearch*