Independent Component Analysis

Revision as of 00:47, 18 June 2011 (view source)

Revision as of 08:22, 19 June 2011 (view source)

Line 49:

In practice, the learning rate <math>\alpha</math> is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting <math>W \leftarrow (WW^T)^{-\frac{1}{2}} W</math>, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening).

-

~~== Reconstruction ICA ==~~

-

~~In reconstruction ICA, we drop the constraint that we want an orthonormal basis, replacing it with a reconstruction error term. Hence, the new reconstruction ICA objective is:~~

-

~~:<math>~~

-

~~\begin{array}{rcl}~~

-

~~{\rm minimize} & \lVert Wx \rVert_1 + \lVert W^TWx \rVert_2^2 \\~~

-

~~\end{array}~~

-

~~</math>~~

-

where <math>W^T(Wx)</math> is the reconstruction of the input from the features (i.e. <math>W^T</math> is supposed to play the role of <math>W^{-1}</math>). If you recall the [[Sparse Coding: Autoencoder Interpretation | sparse coding]] objective,

-

~~:<math>~~

-

~~J(A, s) = \lVert As - x \rVert_2^2 + \lambda \lVert s \rVert_1~~

-

~~</math>~~

-

you will notice that these two objectives are identical, except that whereas in sparse coding, we are trying to learn a matrix <math>A</math> that maps the feature space to the input space, in reconstruction ICA, we are trying to learn a matrix <math>W</math> that maps the input space to the feature space instead.

-

~~As such, it is obvious that reconstruction ICA does not learn independent or orthonormal components.~~

== Topographic ICA ==

Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term.

Independent Component Analysis

From Ufldl

Revision as of 08:22, 19 June 2011

Views

Personal tools

ufldl resources

wiki

Search

Toolbox

@@ Line 49: / Line 49: @@
 In practice, the learning rate <math>\alpha</math> is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting <math>W \leftarrow (WW^T)^{-\frac{1}{2}} W</math>, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening).
-== Reconstruction ICA ==
-In reconstruction ICA, we drop the constraint that we want an orthonormal basis, replacing it with a reconstruction error term. Hence, the new reconstruction ICA objective is:
-:<math>
-\begin{array}{rcl}
-     {\rm minimize} & \lVert Wx \rVert_1 + \lVert W^TWx \rVert_2^2 \\
-\end{array}
-</math>
-where <math>W^T(Wx)</math> is the reconstruction of the input from the features (i.e. <math>W^T</math> is supposed to play the role of <math>W^{-1}</math>). If you recall the [[Sparse Coding: Autoencoder Interpretation | sparse coding]] objective,
-:<math>
-J(A, s) = \lVert As - x \rVert_2^2 + \lambda \lVert s \rVert_1
-</math>
-you will notice that these two objectives are identical, except that whereas in sparse coding, we are trying to learn a matrix <math>A</math> that maps the feature space to the input space, in reconstruction ICA, we are trying to learn a matrix <math>W</math> that maps the input space to the feature space instead.
-As such, it is obvious that reconstruction ICA does not learn independent or orthonormal components.
 == Topographic ICA ==
 Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term.