Applying Deep Learning in Information Retrieval System

http://twchrislin.com/

My projecthttp://twchrislin.com/project/

Tools: python numpy pandas scikit-learn xgboost theano lasagne

id LifeTime dira FlightDistance FlightDistanceError … signal

18453471 0.00157832377590239050.999999344348907614.0333347320556640.6814009547233582 … 0

5364094 0.00098804314620792870.99970477819442765.536156654357910.3023407459259033 … 0

11130990 0.00087667378829792140.99998414516448966.1173019409179690.27646270394325256… 0

15173787 0.0008541746647097170.99990290403366085.2280673980712890.22073912620544436… 0

1102544 0.00112933525815606140.999995291233062939.069534301757811.8981972932815552 … 0

https://youtu.be/_SjgCoR7e1s

http://www.emotibot.com/

:

python numpy pandas

scikit-learn xgboost theano lasagne

Machine Learning Deep Learning

Outline

1. Information Retrieval 2. Neural Networks and Convolutional Neural Networks 3. Feature Vector Representation 4. Sentence Retrieval 5. Image Retrieval 6. Conclusion

Information Retrieval

Input Query

Encoding Database Matching

Ranking

Neural Networks

http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf

Single Neuron, Logistic Regression

Sigmoid funciton

28 pixels

28 pixels

784 pixels

Neural Networks

http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf

Multiclass 10

Neural Networks

http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20(v4).pdf

MLP Multiclass Z1xL1=X1x785 W785xL1

+f(Z1xL1)

Sigmoid funciton

Z1xL2=X1xL1 WL1xL2

+f(Z1xL2)

Convolutional Neural Networks

http://www.slideshare.net/tw_dsconf/ss-62245351

translation invariance

subsampling

Conv + Pooling Conv + Pooling MLP

Feature Vector Representation

Bengio, 2014, Representation Learning: A Review and New Perspectives

Feature Vector

Representation Learning

http://deeplearning4j.org/convolutionalnets

NN feature

extractor

Feature Vector Representation

InputNeural Networks

including MLP, CNN, RNN

Feature Vector

[ 0.25, 0.33…, 0.78 ]

Encoding

Matching

Feature Vector 1

Feature Vector 2

Similarity Function Score

• cosine similarity • v1TM v2

Outline

1. Information Retrieval 2. Neural Networks and Convolutional Neural Networks 3. Feature Vector Representation 4. Sentence Retrieval 5. Image Retrieval 6. Conclusion

<Pad> <Pad> <Pad> <Pad>

Sentence Retrieval

http://emnlp2014.org/papers/pdf/EMNLP2014181.pdf

Yoon Kim, 2014, Convolutional Neural Networks for Sentence Classification

Neural Net Architecture

NN feature

extractor

1 of K (one-hot) +

word embedding

Conv + MaxPooling

Sentence Retrieval

Results: • Training acc: 99% up • Validation acc: 99% up

Experiment: • 5 classes sentences • Training Set 80% • Validation Set 20%

Feature vector PCA tSNE

Sentence Retrieval

Recurrent Neural Network

Kyunghyun Cho et. al., 2014, Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

https://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part-2/

Skip-Thought Vectors

Skip-Thought Vectors http://arxiv.org/pdf/1506.06726v1.pdf

Encode a sentence into a thought vector hi by predicting its neighbor sentence.

Ryan Kiros et. al., 2015, Skip-Thought Vectors

… I just got back home. I could see the cat on the steps. This was strange. …

Channel A

http://www.cherry123.com/data/book/14299/book14299_2.html

Skip-Thought Vectors

? ?

?

?

?

?

?

! ? !

?

, ,

Semantic Similarity

Skip-Thought Vectors

query sen skt result skt score

… … 0.934

0.918

0.901

0.898

0.889

0.861

0.836

0.831

0.812

0.777

0.765

Reference Relationship

Skip-Thought Vectors

query sen skt result skt score

0.898

0.916

0.836

… … 0.755

0.757

0.763

LimitationsSkip-Thought Vectors

• Requiring huge size of corpus • … I just got back home. I could see the cat on the steps. This was strange. … • … I got back to office. I could see the cat on the steps. This was cool. …

11,051 novels with 17,515,150 sentences

• Scenario dependency

query sen skt result skt score

0.697

… … 0.697

0.697

Something InterestingSkip-Thought Vectors

query sen skt result skt score

0.889

0.697

skip-thought knows: = I’m ?

skip-thought knows: = ?

Image Retrieval

“A cat” Image Input

Expected Outcome:

Feature Vector Calculation

Matching Database

Image Retrieval

Input Image

pre-processing

(2)

(1)

Hashing Vector Layer

FC + Softmax Layer

.

.

.

.

.

.

CNN network

1st: various targets

Step1 Training

copy

CNN network

Step2Training

Kevin Lin et. al, 2015, Deep Learning of Binary Hash Codes for Fast Image Retrieval

.

.

.

.

.

.

2nd: task specific targets

encoding application Hashing

Vector Layer

copy

CNN network

copy

Feature Vector

AlexNet VGG

Image Retrieval Kevin Lin et. al, 2015, Deep Learning of Binary Hash Codes for Fast Image Retrieval

Demohttp://tweddielin.pythonanywhere.com/

https://github.com/tweddielin/flask-imsearchGithub

Conclusion

1. Deep Learning => Trainable Feature Extractor 2. Type of Neural Nets: MLP, CNN, RNN 3.

• http://www.slideshare.net/tw_dsconf/ss-62245351

• http://speech.ee.ntu.edu.tw/~tlkagk/courses_MLSD15_2.html

Backup

Convolutional Neural Networks

http://cs231n.github.io/convolutional-networks/More Information

1 1 2 4

5 6 7 8

3 2 1 -2

1 2 3 4

ReLU

max(0,x)

Convolved Feature

MaxPooling