# Linear Decoders

## Sparse Autoencoder Recap

In the sparse autoencoder implementation, we had 3 layers of neurons: the input layer, a hidden layer and an output layer. Recall that each neuron (in the output layer) computes the following:

\begin{align} z^{(3)} &= W^{(2)} a^{(2)} + b^{(2)} \\ a^{(3)} &= f(z^{(3)}) \end{align}

where a(3) is the reconstruction of the input (layer a(1)).

Notice that due to the choice of the sigmoid function for f(z(3)) we need to constrain the inputs to be in the range [0,1].

While some datasets like MNIST fit well into this criteria, this hard constraint can sometimes be awkward to satisfy. For example, if one uses PCA whitening, the input is no longer constrained to [0,1] and it is not clear what kind of scaling is appropriate to fit the data into the constrained range.

## Linear Decoder

One easy fix for the fore-mentioned problem is to use a linear-decoder, that is, we set a(3) = z(3).

For a linear decoder, the activation function of the output unit is effectively the identity function. Formally, to reconstruct the input from the features using a linear decoder, we simply set $\hat{x} = a^{(3)} = z^{(3)} = W^{(2)}a + b^{(2)}$ instead, without applying the sigmoid function. Now the reconstructed output $\hat{x}$ is a linear function of the activations of the hidden units, which means that by varying W, each output unit $\hat{x}$ can be made to produce any activation without the previous constraints. This allows us to train the sparse autoencoder on any input that takes on real values without any additional pre-processing. (Note that the hidden units are still sigmoid units, that is, a = σ(W(1) * x + b(1)), where x is the input, and W(1) and b(1) are the weight and bias terms for the hidden units)

Of course, now that we have changed the activation function of the output units, the gradients of the output units must be changed accordingly. Recall that for each output unit, we set the error as follows:

\begin{align} \delta_i = \frac{\partial}{\partial z_i} \;\; \frac{1}{2} \left\|y - \hat{x}\right\|^2 = - (y_i - \hat{x}_i) \cdot f'(z_i) \end{align}

(where y = x is the desired output, $\hat{x}$ is the reconstructed output of our autoencoder, z is the input to the the output units, and f(x) is our activation function)

Since the activation function for the output units for a linear decoder is just the identity function, the above reduces to:

\begin{align} \delta_i = - (y_i - \hat{x}_i) \end{align}