Independent Component Analysis

From Ufldl

Jump to: navigation, search
 
Line 20: Line 20:
As is usually the case in deep learning, this problem has no simple analytic solution, and to make matters worse, the orthonormality constraint makes it slightly more difficult to optimize for the objective using gradient descent - every iteration of gradient descent must be followed by a step that maps the new basis back to the space of orthonormal bases (hence enforcing the constraint).  
As is usually the case in deep learning, this problem has no simple analytic solution, and to make matters worse, the orthonormality constraint makes it slightly more difficult to optimize for the objective using gradient descent - every iteration of gradient descent must be followed by a step that maps the new basis back to the space of orthonormal bases (hence enforcing the constraint).  
-
In practice, optimizing for the objective function while enforcing the orthonormality constraint (as described in [[Independent Component Analysis#Orthonormal ICA | Orthonormal ICA]] section below) is feasible but slow. Hence, the use of orthonormal ICA is limited to situations where it is important to obtain an orthonormal basis ([[TODO]]: what situations) . In other situations, the orthonormality constraint is replaced by a reconstruction cost instead to give [[Independent Component Analysis#Reconstruction ICA | Reconstruction ICA]], which is very similar to [[Sparse Coding: Autoencoder Interpretation | sparse coding]], but no longer strictly finds independent components.
+
In practice, optimizing for the objective function while enforcing the orthonormality constraint (as described in [[Independent Component Analysis#Orthonormal ICA | Orthonormal ICA]] section below) is feasible but slow. Hence, the use of orthonormal ICA is limited to situations where it is important to obtain an orthonormal basis ([[TODO]]: what situations) .
-
=== Orthonormal ICA ===
+
== Orthonormal ICA ==
The orthonormal ICA objective is:
The orthonormal ICA objective is:
Line 50: Line 50:
In practice, the learning rate <math>\alpha</math> is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting <math>W \leftarrow (WW^T)^{-\frac{1}{2}} W</math>, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening).
In practice, the learning rate <math>\alpha</math> is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting <math>W \leftarrow (WW^T)^{-\frac{1}{2}} W</math>, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening).
-
=== Reconstruction ICA ===
+
== Topographic ICA ==
-
In reconstruction ICA, we drop the constraint that we want an orthonormal basis, replacing it with a reconstruction error term. Hence, the new reconstruction ICA objective is:
+
Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term.
-
:<math>
+
-
\begin{array}{rcl}
+
-
    {\rm minimize} & \lVert Wx \rVert_1 + \lVert W^TWx \rVert_2^2 \\
+
-
\end{array}
+
-
</math>
+
-
where <math>W^T(Wx)</math> is the reconstruction of the input from the features (i.e. <math>W^T</math> is supposed to play the role of <math>W^{-1}</math>). If you recall the [[Sparse Coding: Autoencoder Interpretation | sparse coding]] objective,
 
-
:<math>
 
-
J(A, s) = \lVert As - x \rVert_2^2 + \lambda \lVert s \rVert_1
 
-
</math>
 
-
you will notice that these two objectives are identical, except that whereas in sparse coding, we are trying to learn a matrix <math>A</math> that maps the feature space to the input space, in reconstruction ICA, we are trying to learn a matrix <math>W</math> that maps the input space to the feature space instead.
 
-
As such, it is obvious that reconstruction ICA does not learn independent or orthonormal components.
+
{{Languages|独立成分分析|中文}}
-
 
+
-
=== Topographic ICA ===
+
-
 
+
-
Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term.
+

Latest revision as of 04:35, 8 April 2013

Retrieved from "http://ufldl.stanford.edu/wiki/index.php/Independent_Component_Analysis"
Personal tools