Sparse Coding
From Ufldl
Line 30: | Line 30: | ||
So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. | So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. | ||
- | Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent | + | Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent source features <math>\mathbf{\phi}_i</math> with some additive noise <math>\nu</math>: |
:<math>\begin{align} | :<math>\begin{align} | ||
\mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) | \mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) | ||
Line 37: | Line 37: | ||
Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that | Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that | ||
:<math>\begin{align} | :<math>\begin{align} | ||
- | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i} }{2\sigma^2}) | + | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) |
+ | \end{align}</math> | ||
+ | In order to determine the distribution <math>P(\mathbf{x}\mid\mathbf{\phi})</math>, we also need to specify the prior distribution <math>P(\mathbf{a})</math>. Assuming the independence of our source features, we can factorize our prior probability as | ||
+ | :<math>\begin{align} | ||
+ | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) | ||
\end{align}</math> | \end{align}</math> | ||
- | |||
- | |||
- |