Sparse Coding

From Ufldl

Jump to: navigation, search
Line 30: Line 30:
So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model.  
So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model.  
-
Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent causal features <math>\mathbf{\phi}_i</math> with some additive noise <math>\nu</math>:
+
Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent source features <math>\mathbf{\phi}_i</math> with some additive noise <math>\nu</math>:
:<math>\begin{align}
:<math>\begin{align}
\mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x})
\mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x})
Line 37: Line 37:
Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that  
Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that  
:<math>\begin{align}
:<math>\begin{align}
-
P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i} }{2\sigma^2})
+
P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2})
 +
\end{align}</math>
 +
In order to determine the distribution <math>P(\mathbf{x}\mid\mathbf{\phi})</math>, we also need to specify the prior distribution <math>P(\mathbf{a})</math>. Assuming the independence of our source features, we can factorize our prior probability as
 +
:<math>\begin{align}
 +
P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2})
\end{align}</math>
\end{align}</math>
-
 
-
 
-
In order to define the distribution <math>P(\mathbf{x}\mid\mathbf{\phi})</math>, we must first specify a prior distribution over the amplitudes <math>a_i</math>
 

Revision as of 11:29, 21 March 2011

Personal tools