# Sparse Coding

### From Ufldl

Line 30: | Line 30: | ||

So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. | So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. | ||

- | Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent | + | Consider the problem of modelling natural images as the linear superposition of <math>k</math> independent source features <math>\mathbf{\phi}_i</math> with some additive noise <math>\nu</math>: |

:<math>\begin{align} | :<math>\begin{align} | ||

\mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) | \mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) | ||

Line 37: | Line 37: | ||

Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that | Assuming <math>\nu</math> is Gaussian white noise with variance <math>\sigma^2</math>, we have that | ||

:<math>\begin{align} | :<math>\begin{align} | ||

- | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i} }{2\sigma^2}) | + | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) |

+ | \end{align}</math> | ||

+ | In order to determine the distribution <math>P(\mathbf{x}\mid\mathbf{\phi})</math>, we also need to specify the prior distribution <math>P(\mathbf{a})</math>. Assuming the independence of our source features, we can factorize our prior probability as | ||

+ | :<math>\begin{align} | ||

+ | P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) | ||

\end{align}</math> | \end{align}</math> | ||

- | |||

- | |||

- |