# Sparse Coding

 Revision as of 11:20, 21 March 2011 (view source)Zhenghao (Talk | contribs)← Older edit Revision as of 11:29, 21 March 2011 (view source)Zhenghao (Talk | contribs) Newer edit → Line 30: Line 30: So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. So far, we have considered sparse coding in the context of finding a sparse, over-complete set of basis vectors to span our input space. Alternatively, we may also approach sparse coding from a probabilistic perspective as a generative model. - Consider the problem of modelling natural images as the linear superposition of $k$ independent causal features $\mathbf{\phi}_i$ with some additive noise $\nu$: + Consider the problem of modelling natural images as the linear superposition of $k$ independent source features $\mathbf{\phi}_i$ with some additive noise $\nu$: :\begin{align} :[itex]\begin{align} \mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) \mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} + \nu(\mathbf{x}) Line 37: Line 37: Assuming [itex]\nu is Gaussian white noise with variance $\sigma^2$, we have that Assuming $\nu$ is Gaussian white noise with variance $\sigma^2$, we have that :\begin{align} :[itex]\begin{align} - P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i} }{2\sigma^2}) + P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) + \end{align} + In order to determine the distribution $P(\mathbf{x}\mid\mathbf{\phi})$, we also need to specify the prior distribution $P(\mathbf{a})$. Assuming the independence of our source features, we can factorize our prior probability as + :\begin{align} + P(\mathbf{x} \mid \mathbf{a}, \mathbf{\phi}) = \frac{1}{Z} \exp(- \frac{(\mathbf{x}-\sum^{k}_{i=1} a_i \mathbf{\phi}_{i})^2}{2\sigma^2}) \end{align} \end{align}[/itex] - - - In order to define the distribution $P(\mathbf{x}\mid\mathbf{\phi})$, we must first specify a prior distribution over the amplitudes $a_i$