Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou...
Transcript of Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou...
![Page 1: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/1.jpg)
Bayesian Nonparametric Matrix Factorization for Recorded Music
Reading Group Presenter:
Shujie Hou
Cognitive Radio Institute
Friday, October 15, 2010
Authors: Matthew D. Hoffman, David M. Blei, Perry R. cook
Princeton University, Department of Computer Science, 35 olden St., Princeton, NJ, 08540 USA
![Page 2: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/2.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and contribution of this paper
■ Gap-NMF Model(Gamma Process Nonnegative Matrix Factorization )
■ Variational Inference■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 3: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/3.jpg)
Terminology(1)
■ Nonparametric Statistics:□ The term non-parametric is not meant to imply that such models
completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance.
■ Nonnegative Matrix Factorization:□ Non-negative matrix factorization (NMF) is a group of algorithms
in multivariate analysis and linear algebra where a matrix, is factorized into (usually) two matrices with all elements are greater than or equal to 0 WHX
The above two definitions are cited from Wikipedia
![Page 4: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/4.jpg)
Terminology(2)
■ Variational Inference:□ Variational inference approximates the posterior distribution with
a simpler distribution, whose parameters are optimized to be close to the true posterior.
■ Mean-field Variational Inference:□ In mean-field variational inference, each variable is given an
independent distribution, usually of the same family as its prior.
![Page 5: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/5.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 6: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/6.jpg)
Problem Statement and Contribution
■ Research Topic:□ Breaking audio spectrograms into separate sources of sound
using latent variable decompositions. E.g., matrix factorization.
■ A potential problem :□ The number of latent variables must be specified in advance
which is not always possible.
■ Contribution of this paper□ The paper develops Gamma Process Nonnegative Matrix
Factorization (GaP-NMF), a Bayesian nonparametric approach to decompose spectrograms.
![Page 7: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/7.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 8: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/8.jpg)
Dataset on GaP-NMF Model
■ What are given is a M by N matrix
in which is the power of audio signal at time window n and frequency bin m.
If the number of latent variable is specified in advance:■ Assuming the audio signal is composed of K static sound
sources. The problem is to decompose , in which is M by K matrix, is K by N matrix. In which cell is the average amount of energy source k exhibits at frequency m. cell is the gain of source k at time n.
■ The problem is solved by
X
mnX
WHX
H
W
knH
mkW
![Page 9: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/9.jpg)
GaP-NMF Model
If the number of latent variable is not specified in advance:
■ GaP-NMF assumes that the data is drawn according to the following generative process:
Based on the formula that(Abdallah&Plumbley (2004))
![Page 10: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/10.jpg)
GaP-NMF Model
If the number of latent variable is not specified in advance:
■ GaP-NMF assumes that the data is drawn according to the following generative process:
The overall gain of the corresponding source l
Based on the formula that(Abdallah&Plumbley (2004))
Used to control the number of latent variables
![Page 11: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/11.jpg)
GaP-NMF Model
■ The number of nonzero is the number of the latent variables K.
■ If L increased towards infinity, the nonzero L which expressed by K is finite and obeys:
Kingman ,1993
![Page 12: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/12.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 13: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/13.jpg)
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior.
■ Under this paper’s condition:
Posterior Distribution
What measured
![Page 14: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/14.jpg)
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior.
■ Under this paper’s condition:
Variational distribution assumption with free parameters
Variational Distribution Posterior Distribution
Approximates
What measured
![Page 15: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/15.jpg)
Definition of Variational Inference
■ Variational inference approximates the posterior distribution with a simpler distribution, whose parameters are optimized to be close to the true posterior.
■ Under this paper’s condition:
Variational distribution assumption with free parameters
Variational Distribution Adjust Parameters Posterior Distribution
Approximates
What measured
![Page 16: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/16.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 17: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/17.jpg)
Variational Objective Function
■ Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family:
![Page 18: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/18.jpg)
Variational Objective Function
■ Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family:
It is Gamma family
![Page 19: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/19.jpg)
Variational Objective Function
■ Assume each variable obeys the following Generalized Inverse-Gaussian (GIG) family:
Denotes a modified Bessel function of the second kind
It is Gamma family
![Page 20: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/20.jpg)
Deduction(1)
■ The difference between the left and right sides is the Kullback-Leibler divergence between the true posterior and the variational distribution q.
■ Kullback-Leibler divergence : for probability distributions P and Q of a discrete random variable their K–L divergence is defined to be
From Jordan et al., 1999
![Page 21: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/21.jpg)
Deduction(2)
![Page 22: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/22.jpg)
Deduction(2)
Using Jensen’s inequality
![Page 23: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/23.jpg)
Objective function
■ L=
■ The objective function becomes
Bounded by
+
![Page 24: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/24.jpg)
■ Maximize the objective function defined above with the corresponding parameters.
■ The distribution is obtained:
■ Because these three distributions are independent, we gain
approximates
![Page 25: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/25.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 26: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/26.jpg)
Coordinate Ascent Algorithm(1)
■ The derivative of the objective function with respect to variational parameters equals to zero to obtain:
■ Similarly:
![Page 27: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/27.jpg)
Coordinate Ascent Algorithm(2)
■ Using Lagrange multipliers, then the bound parameters become
■ Then updating bound parameters and variational parameters according to equations 14,15,16,17 and18 to ultimately reaching a local minimum.
![Page 28: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/28.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches■ Evaluation
![Page 29: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/29.jpg)
Other Approaches
■ Finite Bayesian Model ( also called GIG-NMF).■ Finite Non-Bayesian Model.■ EU-Nonnegative Matrix Factorization.■ KL-Nonnegative Matrix Factorization.
![Page 30: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/30.jpg)
Outline
■ Introduction■ Terminology■ Problem statement and Contribution of this Paper
■ Gap-NMF Model■ Variational Inference
■ Definition■ Variational Objective Function■ Coordinate Ascent Optimization
■ Other Approaches
■ Evaluation
![Page 31: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/31.jpg)
Evaluation on Synthetic data(1)
■ The data is generated according to the following model:
![Page 32: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/32.jpg)
Evaluation on Synthetic data(2)
![Page 33: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/33.jpg)
Evaluation on Recorded Music
![Page 34: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/34.jpg)
Conclusion
■ Gap-NMF model is capable of determining the number of latent source automatically.
■ The key step of the paper is to use variational distribution to approximate posterior distribution.
■ Gap-NMF can work well on analyzing and processing recorder music, it can be applicable to other types of audio.
![Page 35: Bayesian Nonparametric Matrix Factorization for Recorded Music Reading Group Presenter: Shujie Hou Cognitive Radio Institute Friday, October 15, 2010 Authors:](https://reader031.fdocuments.us/reader031/viewer/2022032723/56649d1f5503460f949f35a9/html5/thumbnails/35.jpg)
■Thank you!