So far, we have described the application of neural networks to supervised learning, in which we have labeled training examples. Now suppose we have only a set of unlabeled training examples **autoencoder** neural network is an unsupervised learning algorithm that applies backpropagation, setting the target values to be equal to the inputs. I.e., it uses

Here is an autoencoder:

The autoencoder tries to learn a function

Our argument above relied on the number of hidden units

Informally, we will think of a neuron as being “active” (or as “firing”) if its output value is close to 1, or as being “inactive” if its output value is close to 0. We would like to constrain the neurons to be inactive most of the time. This discussion assumes a sigmoid activation function. If you are using a tanh activation function, then we think of a neuron as being inactive when it outputs values close to -1.

Recall that

Further, let

be the average activation of hidden unit

where

To achieve this, we will add an extra penalty term to our optimization objective that penalizes

Here,

where

This penalty function has the property that

We see that the KL-divergence reaches its minimum of 0 at

, and blows up (it actually

approaches

Our overall cost function is now

where

To incorporate the KL-divergence term into your derivative calculation, there is a simple-to-implement trick involving only a small change to your code. Specifically, where previously for the second layer (

now instead compute

One subtlety is that you’ll need to know

The full derivation showing that the algorithm above results in gradient descent is beyond the scope of these notes. But if you implement the autoencoder using backpropagation modified this way, you will be performing gradient descent exactly on the objective

### Visualizing a Trained Autoencoder

Having trained a (sparse) autoencoder, we would now like to visualize the function learned by the algorithm, to try to understand what it has learned. Consider the case of training an autoencoder on

We will visualize the function computed by hidden unit

By displaying the image formed by these pixel intensity values, we can begin to understand what feature hidden unit

If we have an autoencoder with 100 hidden units (say), then we our visualization will have 100 such images—one per hidden unit. By examining these 100 images, we can try to understand what the ensemble of hidden units is learning.

When we do this for a sparse autoencoder (trained with 100 hidden units on 10x10 pixel inputs^{1} we get the following result:

Each square in the figure above shows the (norm bounded) input image

These features are, not surprisingly, useful for such tasks as object recognition and other vision tasks. When applied to other input domains (such as audio), this algorithm also learns useful representations/features for those domains too.

^{1}”The learned features were obtained by training on ”‘whitened”’ natural images. Whitening is a preprocessing step which removes redundancy in the input, by causing adjacent pixels to become less correlated.