Visualizing a Trained Autoencoder

From Ufldl

Revision as of 23:33, 22 April 2011 by Jngiam (Talk | contribs)
Jump to: navigation, search

Having trained a (sparse) autoencoder, we would now like to visualize the function learned by the algorithm, to try to understand what it has learned. Consider the case of training an autoencoder on \textstyle 10 \times 10 images, so that \textstyle n = 100. Each hidden unit \textstyle i computes a function of the input:

\begin{align}
a^{(2)}_i = f\left(\sum_{j=1}^{100} W^{(1)}_{ij} x_j  + b^{(1)}_i \right).
\end{align}

%This is the activation function \textstyle g(\cdot) applied to an affine function of the input. We will visualize the function computed by hidden unit \textstyle i---which depends on the parameters \textstyle W^{(1)}_{ij} (ignoring the bias term for now) using a 2D image. In particular, we think of \textstyle a^{(1)}_i as some non-linear feature of the input \textstyle x. We ask: What input image \textstyle x would cause \textstyle a^{(1)}_i to be maximally activated? %(Less formally, what is the feature that hidden unit \textstyle i is looking for?) For this question to have a non-trivial answer, we must impose some constraints on \textstyle x. If we suppose that the input is norm constrained by \textstyle ||x||^2 = \sum_{i=1}^{100} x_i^2 \leq 1, then one can show (try doing this yourself) that the input which maximally activates hidden unit \textstyle i is given by setting pixel \textstyle x_j (for all 100 pixels, \textstyle j=1,\ldots, 100) to

\begin{align}
x_j = \frac{W^{(1)}_{ij}}{\sqrt{\sum_{j=1}^{100} (W^{(1)}_{ij})^2}}.
\end{align}

By displaying the image formed by these pixel intensity values, we can begin to understand what feature hidden unit \textstyle i is looking for.

If we have an autoencoder with 100 hidden units (say), then we our visualization will have 100 such images---one per hidden unit. By examining these 100 images, we can try to understand what the ensemble of hidden units is learning.

When we do this for a sparse autoencoder (trained with 100 hidden units on 10x10 pixel inputs1 we get the following result:

ExampleSparseAutoencoderWeights.png

Each square in the figure above shows the (norm bounded) input image \textstyle x that maximally actives one of 100 hidden units. We see that the different hidden units have learned to detect edges at different positions and orientations in the image.

These features are, not surprisingly, useful for such tasks as object recognition and other vision tasks. When applied to other input domains (such as audio), this algorithm also learns useful representations/features for those domains too.


1 The learned features were obtained by training on whitened natural images. Whitening is a preprocessing step which removes redundancy in the input, by causing adjacent pixels to become less correlated.

Personal tools