Post on 14-Jan-2017
Inria, SequeL
1
Inria, Sequel – Meetup Juin 2016
Deep Learning
Collaborative Filtering
and
Inria, SequeL
2
Collaborative filtering
Bob
Inria, SequeL
3
Bob
Collaborative filtering
Tim
Inria, SequeL
4
Bob
Tim
Collaborative filtering
Inria, SequeL
5
Collaborative filtering
Bob
Tim
Inria, SequeL
6
Collaborative filtering
Bob
Tim
Inria, SequeL
7
Collaborative filtering
Bob
Tim
Inria, SequeL
8
Goal is to predict the rating Tim would have given to Jurassic Park
Collaborative filtering
Tim
Inria, SequeL
9
Collaborative filtering
Inria, SequeL
10
Goal : Predict the missing rating
Collaborative filtering
Bob ? 3 5 ?Ana ? 4 ? 5Elain 5 ? 4 ?Sulman 2 ? 1 4
Inria, SequeL
11
Goal : Predict the missing rating
Bob 4 3 5 1Ana 2 4 2 5Elain 5 4 4 3Sulman 2 3 1 4
Collaborative filtering
Inria, SequeL
12
Goal : Predict the missing rating
Bob 4 3 5 1Ana 2 4 2 5Elain 5 4 4 3Sulman 2 3 1 4
To be recommended
Collaborative filtering
Inria, SequeL
13
Bob ? 3 5 ?Ana ? 4 ? 5Elain 5 ? 4 ?Sulman 2 ? 1 4
Score: Average|Predicted rating – real rating|
Collaborative filtering
Inria, SequeL
14
Bob ? 3 5 ?Ana ? 4 ? 5Elain 5 ? 4 ?Sulman 2 ? 1 4
Score = Average |Predicted rating – real rating|2
Collaborative filtering
Inria, SequeL
15
Neural Networks
Inria, SequeL
16
Input
Plane ?
Dog ?
Cat ?
Boat ?
Truck ?
Neural Networks
Inria, SequeL
17
Plane ?
Dog ?
Cat ?
Boat ?
Forw
ard
Back
war
d
The model is updated to fit the target
The error is backpropagated into the model
Truck ?
Neural Networks
Inria, SequeL
18
Plane ?
Dog ?
Cat ?
Boat ?
Truck ?
Neural Networks
Inria, SequeL
19
5 3 1 4
RUsers
Items v
u
2
Neural Networks
Inria, SequeL
20
5 ? ? 4 sparse
RUsers
Items v
u
2
5 3 1 4 2
Neural Networks
Inria, SequeL
21
sparse
dense
RUsers
Items v
u
5 ? ? 4 2
5 3 1 4 2
Neural Networks
Inria, SequeL
22
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
�̂�user �̂�item
U-CFN V-CFN
Neural Networks
Inria, SequeL
23
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
inputs
Dimension reduction
Input reconstruction
5 3 4 2 1
4.9 2.8 4 .1 2.0 1.1
The model
Inria, SequeL
24
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
3 4 1
2.8 4 .1 1.1
Training stepTesting step
The model
Inria, SequeL
25
1 ? 2 ? 1
1.2 4.9 1.1
1 0 2 0 1
1.2 2.1 4.9 2.0 1.1
Forw
ard
The model
Inria, SequeL
26
1 ? 2 ? 1
1.2 4.9 1.1
1 0 2 0 1
0.2 0 -0.1 0 0.1
Back
war
d
The model
Inria, SequeL
27
β*0.2 0 α*1.1 0 β*1.1
1 0 2 0 1
1 0 0 0 1
Forward
1.2 2.3 3.1 4.2 1.1
Back
war
d
1 ? 2 ? 1True input
0.2 0 1.1 0 0.1error 0.2 ? 1.1 ? 0.1
α = Prediction = supervizedβ = Reconstruction = unsupervized
The model
1 ? 2 ? 1
Inria, SequeL
28
β*0.2 0 α*1.1 0 β*1.1
1 0 2 0 1
1 0 0 0 1
Forward
1.2 2.3 3.1 4.2 1.1
Back
war
d
1 ? 2 ? 1True input
0.2 0 1.1 0 0.1error 0.2 ? 1.1 ? 0.1
α = Prediction = supervizedβ = Reconstruction = unsupervized
The model
1 ? 2 ? 1
Inria, SequeL
29
The model
300.000 to 50.000 entries
Real life
Inria, SequeL
30
MovieLens :
- 10 million ratings : 72.000 users / 10.000 movies
- 20 million ratings : 138.000 users / 28.000 movies
Results
Inria, SequeL
31
Quadratic error
MovieLens-10M MovieLens-20MBPMF 0.8213 0.8123ALS-WR 0.7830 0.7746 LLORMA 0.7949 0.7843 U-CFN 0.7767 0.7663 V-CFN 0.7767 0.7663
RMSE for train/test 90/10
BPMF : rank=10ALS-WE : rank = 200LLORMA : rank = 20, anchor point = 30
A. Mnih and R. Salakhutdinov, “Probabilistic matrix factorization,” in Advances in neural information processing systems, 2007, pp. 1257– 1264.B. J. Lee, S. Kim, G. Lebanon, and Y. Singerm, “Local low-rank matrix approximation,” in Proc. of ICML’13, 2013, pp. 82–90.C. Y.Zhou,D.Wilkinson,R.Schreiber,andR.Pan,“Large-scaleparallel collaborative filtering for the netflix prize,” in Algorithmic Aspects in Information and Management. Springer, 2008, pp. 337–348.
Results
Inria, SequeL
32
Singular value decomposition (SVD)
Other algorithms:●Alternating Least Square Weighted Lambda-Regularization (ALS-WR)●Probabilistic Matrix Factorization (PMF, BPMF, NLPMF)●Local Low Rank Matrix Approximation (LLORMA)
RUsers
Items v
u
Link with matrix factorization (optional)
Inria, SequeL
33
dense
sparse
RUsers
Items v
u
5 ? ? 4 2
5 3 1 4 2
z=W 3𝐚𝐜𝐭𝐢𝐯𝐚𝐭𝐢𝐨𝐧2
ri=V ui
activ
ation
Link with matrix factorization (optional)
Inria, SequeL
34
u i
V
Link with matrix factorization (optional)
5 ? ? 4 2
5 3 1 4 2
activ
ation
ri=V uidense
sparse
RUsers
Items v
u
CFN computes a Non-Linear Matrix Factorization
Inria, SequeL
35Discussion
Why nobody tried it?
Image/sounds are dense
Few works about sparsity
No tools!
Inria, SequeL
36
DiscussionTorch framework
Reproduce results
Inria, SequeL
37
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
Actor : Dicaprio
Genre: action
Year: 2010
Adding external information
Inria, SequeL
38
4.9 2.8 4 .1 2.0 1.1
Content-based Filtering
Extensions
Inria, SequeL
39
Thank you for listening!
Inria, SequeL
40
Input reconstruction
5 ? 4 ? 1
4.9 2.8 4 .1 2.0 1.1
Encoder
Decoder
Autoencoders:
Matrix factorizationz=h (W ui ,𝐝𝐞𝐧𝐬𝐞 )
u i ,𝐬𝐩𝐚𝐫𝐬𝐞
u i ,𝐝𝐞𝐧𝐬𝐞
Link with Matrix Factorization