A Generalization of PCA to the Exponential Family

Collins, Dasgupta and Schapire

Presented by Guy Lebanon

Two Viewpoints of PCA

• Algebraic Given data find a linear

transformation such that the sum of squared distances is minimized (over all linear transformation )

• Statistical Given data assume that each point

is a random variable. Find the maximum likelihood estimator under the constraint that are in a K dimensional subspace and are linearly related to the data.

Nin RxxxD },,,{ 1

KN RRT :

KN RR

i ii Txx2

Nin RxxxD },,,{ 1 ix

),( IN i i }ˆ{ i

• The Gaussian assumption may be inappropriate – especially if the data is binary valued or non-negative for example.

• Suggestion: replace the Gaussian distribution by any exponential distribution.

Given data such that each point comes from an exponential family distribution , find the MLE for

under the assumption that it lies in a low dimensional subspace.

Nin RxxxD },,,{ 1

ix)()(, iiii cxtxe

i

• The new algorithm finds a linear transformation in the parameter space but a nonlinear subspace in the original coordinates .

• The loss functions may be cast in terms of Bregman distances.

• The loss function is not convex in the general case.

• The authors use the alternating minimization algorithm (Csiszar and Tsunadi) to compute the transformation.

i

x