DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial...
Transcript of DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial...
![Page 1: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/1.jpg)
DEEP LEARNING INNON-COGNITIVE DOMAINS
Truyen TranPRaDA, Deakin
2/04/2019 1
![Page 2: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/2.jpg)
DEEP LEARNING IN COGNITIVE DOMAINS
2/04/2019 2
![Page 3: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/3.jpg)
DEEP LEARNING IN NON-COGNITIVE DOMAINS
Where humans need extensive training to do well
Domains with great diversity but small in size
Domains with great uncertainty, low-quality/missing data
Domains that demand transparency & interpretability.
… healthcare, security, foods, water, manufacturing
2/04/2019 3
![Page 4: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/4.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 4
![Page 5: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/5.jpg)
CENTRE FOR PATTERN RECOGNITION AND DATA ANALYTICS
2/04/2019 5
![Page 6: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/6.jpg)
PRADA: MAKING DATA SPEAK
DomainsHealth Pervasive computing Social mediaManufacturingCybersecurityNow: in collaboration with UoW –software engineering and process mining
MethodsDeep learning Bayesian nonparametrics (topic models included) Sparse methods (e.g., compress sensing) Probabilistic graphical modelsDistributed computingOptimization
2/04/2019 6
![Page 7: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/7.jpg)
Accelerated Learningfor children withAutism
4 Startups CollaboratorsPRADA: THE MAKING
2/04/2019 7
![Page 8: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/8.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 8
![Page 9: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/9.jpg)
INTRODUCTION TO DEEP LEARNING
Google Brain
Google DeepMind
Facebook (FAIR)
Baidu
Microsoft
Twitter Cortex
IBM
2/04/2019 9
![Page 10: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/10.jpg)
DEEP LEARNING: MACHINE THAT LEARNS EVERYTHING
End-to-end machine learning – no human involved.Models are compositional, e.g., object is composed of parts.
→ Models can be complex, but building block is simple and universal!
→ Learning is more efficient in multiple steps
Things can be learn: Feature | Selectivity | Invariance | Dynamics | Memory encoding and forgetting | Attention | Planning
2/04/2019 10
![Page 11: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/11.jpg)
THE BUILDING BLOCKS: FEATURE DETECTOR
∫ ∫
signals
feedbacks
2/04/2019 11
![Page 12: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/12.jpg)
WHY FEATURE LEARNING?
In typical machine learning projects, 80-90% effort is on feature engineeringA right feature representation doesn’t need much work. Simple linear methods often work well.
Vision: Gabor filter banks, SIFT, HOG, BLP, BOW, etc.Text: BOW, n-gram, POS, topics, stemming, tf-idf, etc.SW: token, LOC, API calls, #loops, developer reputation, team complexity, report readability, discussion length, etc. Try yourself on Kaggle.com!
2/04/2019 12
![Page 13: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/13.jpg)
(Bengio, DLSS 2015)2/04/2019 13
![Page 14: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/14.jpg)
2/04/2019 14
![Page 15: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/15.jpg)
THREE MAIN ARCHITECTURES
Deep (DNN): Vector to vectorMost existing ML/statistics fall into this category
Recurrent (RNN): Sequence to sequence Temporal, sequential. E.g., sentence, actions, DNA, EMR Program evaluation/execution. E.g., sort, traveling salesman problem
Convolutional (CNN): Image to vector/sequence/image In time: Speech, DNA, sentences In space: Image, video, relations
2/04/2019 15
![Page 16: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/16.jpg)
DNN FOR VEC2VEC MAPPING
2/04/2019 16
![Page 17: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/17.jpg)
RNN FOR SEQ2SEQ MAPPING
Source: http://karpathy.github.io/assets/rnn/diags.jpeg
deep neural nets with parameter tying
2/04/2019 17
![Page 18: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/18.jpg)
CNN FOR TRANSLATION INVARIANCE
translation
(Quoc Le, 2015)
(Russ Salakhutdinov, 2015)2/04/2019 18
![Page 19: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/19.jpg)
WHEN DOES DEEP LEARNING WORK?
Lots of data (e.g., millions of images)
Strong, clean training signals (e.g., when human can provide correct labels – cognitive domains)
Data structures are well-defined (e.g., image, speech, NLP, video)
Data is compositional (luckily, most data is like this)
The more primitive (raw) the data, the more benefit of using deep learning.
2/04/2019 19
![Page 20: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/20.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X (non-cognitive) is: Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 20
![Page 21: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/21.jpg)
X = PROSPECTIVE HEALTHCARE
Promises yet to be delivered.
Bottleneck: data – providers not willing to share. Many issues – privacy, ethics, governance.
Requirements Transparency & interpretability Correctly model characteristics of healthcare data (e.g., Irregular timing, Interventions, Regular motifs)
A wide range of modalities: health processes, EMR, questionaires, imaging, biomarkers, NLP (clinical notes, pubmed, social media), wearable devices, genomics.
2/04/2019 21
![Page 22: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/22.jpg)
Risk: LOW
Score = 1
2/04/2019 22
![Page 23: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/23.jpg)
Score = 2
Risk: Moderate
2/04/2019 23
![Page 24: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/24.jpg)
Risk: High
Score = 2
2/04/2019 24
![Page 25: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/25.jpg)
Risk: high
Score = 1
2/04/2019 25
![Page 26: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/26.jpg)
2/04/2019 26
![Page 27: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/27.jpg)
DEEP ARCHITECTURES FOR HEALTHCARE
Our primary goal: predicting future risk!
DNND – vector input & vector output, no sequences
DEEPR (CNN) – repeated motifs, short sequences
DEEPCARE (RNN) – long-term dependencies, long sequences
INTEGRATED DATA VIEW OF MULTIPLE HOSPITAL SYSTEMS
MULTI DATA INPUT METHODS
FLEXIBLE TO CREATE, EASY TO USE
SECURE AND ACCESSIBLE, ANYWHERE
Patient data
2/04/2019 27
![Page 28: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/28.jpg)
DNND
Input vector --redundancy
Output vector – multiple prediction horizons
Tricks: Dropout
assessment
15 days30 days 60 days 120 days 180 days
hidden layers
pooling
historyfuture
360 days
fragmentation
[0-3]m[3-6]m[6-12]mdata segments
[12-24]m[24-48]m
2/04/2019 28
![Page 29: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/29.jpg)
SUICIDE RISK PREDICTION: MACHINE VERSUS CLINICIAN
2/04/2019 29
![Page 30: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/30.jpg)
DEEPR: CNN FOR REPEATED MOTIFS AND SHORT SEQUENCES
output
max-pooling
convolution --motif detection
embedding
sequencing
medical record
visits/admissions
time gaps/transferphrase/admission
prediction
1
2
3
4
5
time gaprecord vector
word vector
?
prediction point
2/04/2019 30
![Page 31: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/31.jpg)
DISEASE EMBEDDING & MOTIFS DETECTION
E11 I48 I50Type 2 diabetes mellitusAtrial fibrillation and flutterHeart failure
E11 I50 N17Type 2 diabetes mellitusHeart failureAcute kidney failure
2/04/2019 31
![Page 32: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/32.jpg)
EMBEDDING OF PATIENTS: LINEARIZING DECISION BOUNDARY
2/04/2019 32
![Page 33: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/33.jpg)
DEEPCARE: LONG-TERM MEMORY
Illness states are a dynamic memory process → moderated by time and intervention
Discrete admission, diagnosis and procedure → vector embedding
Time and previous intervention → “forgetting” of illness
Current intervention → controlling the risk states
2/04/2019 33
![Page 34: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/34.jpg)
DEEPCARE: DYNAMICS
memory
*input gate
forget gate
prev. memory
output gate
*
*
input
aggregation over time → prediction
previous intervention
history states
current data
time gap
current intervention
current state
New in DeepCare2/04/2019 34
![Page 35: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/35.jpg)
DEEPCARE: STRUCTURE
Time gap
LSTM
Admission(disease)
(intervention)
Vector embedding
Multiscale poolingNeural network
Future risks
Long short-term memory
Latent states
FutureHistory
LSTM LSTM LSTM
2/04/2019 35
![Page 36: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/36.jpg)
DEEPCARE: TWO MODES OF FORGETTING AS A FUNCTION OF TIME
→ decreasing illness
→ Increasing illness
2/04/2019 36
![Page 37: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/37.jpg)
DEEPCARE: PREDICTION RESULTS
Intervention recommendation (precision@3) Unplanned readmission prediction (F-score)
12 months 3 months 12 months 3 months
2/04/2019 37
![Page 38: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/38.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X (non-cognitive) is: Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 38
![Page 39: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/39.jpg)
X = SOFTWARE ANALYTICS
Goal: To model code, text, team, user, execution, project & enabled business process answer any queries by developers, managers, users and business
End-to-end Compositional
For now: LSTM for report representation DeepSoft vision paper Stacked/deep inference (later)
2/04/2019 39
![Page 40: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/40.jpg)
LONG SHORT-TERM MEMORY FOR TEXT REPRESENTATION
LSTM
Authenticate
[0.1 0.3 -0.2]
quota
[-1 -2.1 0.5]
requests
[1.5 0.5 -1.2]
quota requests <EOS>
[1 -0.5 -3] [-1.3 0 2] [-0.5 -0.5 -1]
[-0.27 -0.33 -0.67]
LSTMLSTMSequence embedding
Word embedding
Output states
2/04/2019 40
![Page 41: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/41.jpg)
DEEPSOFT: COMPOSITIONAL DEEP NET FOR SW PROJECT
2/04/2019 41
![Page 42: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/42.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X (non-cognitive) is: Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 42
![Page 43: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/43.jpg)
X = RANKING
Raking web documents in search engines Movie recommendation Advertisement placement Tag recommendation Expert finding in a community network Friend ranking in a social network ???
*(these graphics were downloaded using image.google.com)2/04/2019 43
![Page 44: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/44.jpg)
LEARNING-TO-RANK
Learn to rank responses to a query
A ML approach to Information Retrieval Instead of hand-engineering similarity measures, learn it
Two key elementsChoice model rank loss (how right/wrong is a ranked list?) Scoring function mapping features into score (how good is the choice?)
Web documents in search engines query: keywords
Movie recommendation query: an user
Advertisement placement query: a Web page
Tag recommendation query: a web object
Friend ranking in a social network query: an user
2/04/2019 44
![Page 45: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/45.jpg)
CHOICE BY ELIMINATION
gate
input
rank score
gate
A
B
C
A
C C
Recurrent highway networks
The networks represent the scoring function
All networks are linked through the rank loss – neural choice by elimination
It is a structured output problem (permutation)
2/04/2019 45
![Page 46: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/46.jpg)
YAHOO! L2R CHALLENGE (2010)
19,944 queries
473,134 documents
519 unique features
Performance measured in:
Expected Reciprocal Rank (ERR)
Normalised Discounted Cumulative Gain (NDCG)
2/04/2019 46
![Page 47: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/47.jpg)
RESULTS
As of 2011 – Forward selection + quadratic rank function
As of 2016 – Backward elimination + deep nets
Rank 41 out of 1500
Rank?2/04/2019 47
![Page 48: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/48.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 48
![Page 49: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/49.jpg)
X = ANOMALY DETECTION
London, July 7, 2005
2/04/2019 49
![Page 50: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/50.jpg)
Real world - what are the operators monitoring?2/04/2019 50
![Page 51: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/51.jpg)
Strategy: learn normality, anything does not fit in is abnormal2/04/2019 51
![Page 52: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/52.jpg)
MAD: MULTILEVEL ANOMALY DETECTION
2/04/2019 52
![Page 53: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/53.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 53
![Page 54: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/54.jpg)
X =MULTI-RELATIONAL DATABASES
The world is multi-relational (e.g., friend, class-mate, collaborator, flat-mate).
Stacked inference (with Hoa & Morakot)
Deep inference
Shallow Stacked inference Deep inference2/04/2019 54
![Page 55: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/55.jpg)
STACKED INFERENCE RESULT
Latest update: Deep Inference is now better!
2/04/2019 55
![Page 56: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/56.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 56
![Page 57: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/57.jpg)
REPRESENTATION OF MATRIX AND TENSORScolumn-specific model
row-specific model
2/04/2019 57
![Page 58: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/58.jpg)
REPRESENTATION OF MIXED-TYPES
2/04/2019 58
![Page 59: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/59.jpg)
1
2
3
Gaussian RBM
THURSTONIAN BOLTZMANN MACHINE
2/04/2019 59
![Page 60: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/60.jpg)
AGENDA
Introduction to PRaDA
Introduction to deep learning
Deep learning for [X], where X = Healthcare Software engineering Choice and ranking Anomaly detection Multi-relational databases Representation
The open room
2/04/2019 60
![Page 61: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/61.jpg)
THE BEST STRATEGY TO PLAY THIS DEEP LEARNING GAME?
“[…] the dynamics of the game will evolve. In the long run, the right way of playing football is to position yourself intelligently and to wait for the ball to come to you. You’ll need to run up and down a bit, either to respond to how the play is evolving or to get out of the way of the scrum when it looks like it might flatten you.” (Neil Lawrence)
http://inverseprobability.com/2015/07/12/Thoughts-on-ICML-2015/2/04/2019 61
![Page 62: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/62.jpg)
“A NEW IDEA IS JUST RE-PACKAGING OF OLD IDEAS”
2/04/2019 62
![Page 63: DEEP LEARNING IN Truyen Tran NON -COGNITIVE DOMAINS … · Type 2 diabetes mellitus. Atrial fibrillation and flutter. Heart failure. E11 I50 N17. Type 2 diabetes mellitus. Heart failure.](https://reader030.fdocuments.us/reader030/viewer/2022041208/5d65d73388c993047c8b8d1b/html5/thumbnails/63.jpg)
DEEP (LEARNING) QUESTIONS
Is this just yet-another-toolbox or a way of thinking?
Is this a right approach to AI?
2/04/2019 63