Exercise:Self-Taught Learning

From Ufldl

(Difference between revisions)
Jump to: navigation, search
m (Step One: Generate the input and test data sets: Rephrased data set counts)
 
(30 intermediate revisions not shown)
Line 3: Line 3:
In this exercise, we will use the self-taught learning paradigm with the sparse autoencoder and softmax classifier to build a classifier for handwritten digits.
In this exercise, we will use the self-taught learning paradigm with the sparse autoencoder and softmax classifier to build a classifier for handwritten digits.
-
In this context, the self-taught learning paradigm can be used as follows. First, the sparse autoencoder is trained on a unlabeled training dataset of images of handwritten digits. This produces feature detectors that correspond to the activation pattern for each hidden unit. The features learned are expected to be pen strokes to resemble segments of digits.  
+
You will be building upon your code from the earlier exercises. First, you will train your sparse autoencoder on an "unlabeled" training dataset of handwritten digits. This produces feature that are penstroke-like. We then extract these learned features from a labeled dataset of handwritten digits. These features will then be used as inputs to the softmax classifier that you wrote in the previous exercise.  
-
The sparse autoencoder is then used to trainsform the raw labeled training dataset into representation of features, which are then used to train train the softmax classifier. For each example in the the labeled training dataset, <math>\textstyle x^{(k)}</math>, forward propogation is used to is used to obtain the activation of the hidden units <math>\textstyle a^{(k)}</math>. The data point <math>\textstyle a^{(k)}</math> and its corresponding label is then used to train the softmax classifier. Finally, the test dataset is also transformed to hidden unit activations and passed to the softmax classifier to obtain predictions.
+
Concretely, for each example in the the labeled training dataset <math>\textstyle x_l</math>, we forward propagate the example to obtain the activation of the hidden units <math>\textstyle a^{(2)}</math>. We now represent this example using <math>\textstyle a^{(2)}</math> (the "replacement" representation), and use this to as the new feature representation with which to train the softmax classifier.  
-
To demonstrate the effectiveness of this method, we will train the sparse autoencoder on an unlabeled data set comprised out of all the digits 0 to 9, and then test them on the digits 5 to 9. The purpose of this is to demonstrate that self-taught learning can be surprisingly effective in improving results even if some data set items do not fall within the classes of our classification task.
+
Finally, we also extract the same features from the test data to obtain predictions.
-
===Step One: Generate the input and test data sets===
+
In this exercise, our goal is to distinguish between the digits from 0 to 4.  We will use the digits 5 to 9 as our
 +
"unlabeled" dataset which which to learn the features; we will then use a labeled dataset with the digits 0 to 4 with
 +
which to train the softmax classifier.
-
Download and decompress <tt>stl_exercise.zip</tt>, which contains starter code for this exercise. Additionally, you will need to download the datasets from the MNIST Handwritten Digit Database for this project.
+
In the starter code, we have provided a file '''<tt>stlExercise.m</tt>''' that will help walk you through the steps in this exercise.
-
To separate the the cases, you will need to fill in <tt>filterData.m</tt>, which should return a subset of the data set containing examples with labels within a certain bound. After implementing this function, run this step. To test your function, the script will output the number of examples in the supervised training and the STL training data set, which should be 60000 and 30596 respectively.
+
=== Dependencies ===
-
===Step Two: Train the sparse autoencoder===
+
The following additional files are required for this exercise:
 +
* [http://yann.lecun.com/exdb/mnist/ MNIST Dataset]
 +
* [[Using the MNIST Dataset | Support functions for loading MNIST in Matlab ]]
 +
* [http://ufldl.stanford.edu/wiki/resources/stl_exercise.zip Starter Code (stl_exercise.zip)]
-
Next we will train the input data set on the sparse autoencoder, using the same <tt>sparseAutoencoderCost.m</tt> function from the previous assignments. (Use the frameworks from previous assignments to ensure that your code is working and vectorized, as no testing facilities are provided in this assignment.) After doing so, running the training step should take about half an hour (on a reasonably fast computer). When it is completed, a visualization of pen strokes should be displayed.
+
You will also need your code from the following exercises:
 +
* [[Exercise:Sparse Autoencoder]]
 +
* [[Exercise:Vectorization]]
 +
* [[Exercise:Softmax Regression]]
-
Hint: This step takes a very long time, so you might want to avoid running it on subsequent trials! To do so, uncomment the lines containing the function <tt>save(...);</tt>, or run the save function from the command line. In future trials, you may skip the training set and load the save weights from disk.
+
''If you have not completed the exercises listed above, we strongly suggest you complete them first.''
 +
 
 +
===Step 1: Generate the input and test data sets===
 +
 
 +
Download and decompress <tt>[http://ufldl.stanford.edu/wiki/resources/stl_exercise.zip stl_exercise.zip]</tt>, which contains starter code for this exercise. Additionally, you will need to download the datasets from the MNIST Handwritten Digit Database for this project.
 +
 
 +
===Step 2: Train the sparse autoencoder===
 +
 
 +
Next, use the unlabeled data (the digits from 5 to 9) to train a sparse autoencoder, using the same <tt>sparseAutoencoderCost.m</tt> function as you had written in  the previous exercise.  (From the earlier exercise, you should have a working and vectorized implementation of the sparse autoencoder.) For us, the training step took less than 25 minutes on a fast desktop.  When training is complete, you should get a visualization of pen strokes like the image shown below:
[[File:selfTaughtFeatures.png]]
[[File:selfTaughtFeatures.png]]
-
===Step Three: Training the logistic regression model===
+
Informally, the features learned by the sparse autoencoder should correspond to penstrokes.
-
After the sparse autoencoder is trained, we can use it to detect pen strokes in images. To do so, you will need to modify the sparse autoencoder function to output its hidden unit activation.
+
===Step 3: Extracting features===
-
Change the method signature of sparseAutoencoderCost from
+
After the sparse autoencoder is trained, you will use it to extract features from the handwritten digit images.  
-
<tt>function [cost, grad] = sparseAutoencoderCost(...)</tt> to <tt>function [cost, grad, activation] = sparseAutoencoderCost(...)</tt> where activation should be a matrix with each column corresponding to activation of the hidden layer i.e. the vector <math>a^{(2)}</math> corresponding to activation of layer <math>L_{2}</math>. The remainder of the function should remain unchanged.
+
-
After doing so, running this step should
+
Complete <tt>feedForwardAutoencoder.m</tt> to produce a matrix whose columns correspond to activations of the hidden layer for each example, i.e., the vector <math>a^{(2)}</math> corresponding to activation of layer 2.  (Recall that we treat the inputs as layer 1).
-
===Step Four: Training and testing the logistic regression model===
+
After completing this step, calling <tt>feedForwardAutoencoder.m</tt> should convert the raw image data to hidden unit activations <math>a^{(2)}</math>.
-
After completing these steps, running the entire script in <tt>trainSelfTaught.m</tt> will use your sparse autoencoder to train the logistic model, then measure how well this system performs on the test set. Statistics about the model will be displayed afterwards. If you've done all the steps correctly, you should get an accuracy of about X percent.
+
===Step 4: Training and testing the logistic regression model===
 +
 
 +
Use your code from the softmax exercise (<tt>softmaxTrain.m</tt>) to train a softmax classifier using the training set features (<tt>trainFeatures</tt>) and labels (<tt>trainLabels</tt>).
 +
 
 +
===Step 5: Classifying on the test set===
 +
 
 +
Finally, complete the code to make predictions on the test set (<tt>testFeatures</tt>) and see how your learned features perform! If you've done all the steps correctly, you should get an accuracy of about '''98%''' percent.
 +
 
 +
As a comparison, when ''raw pixels'' are used (instead of the learned features), we obtained a test accuracy of only around 96% (for the same train and test sets).
[[Category:Exercises]]
[[Category:Exercises]]
 +
 +
 +
{{STL}}

Latest revision as of 11:02, 26 May 2011

Contents

Overview

In this exercise, we will use the self-taught learning paradigm with the sparse autoencoder and softmax classifier to build a classifier for handwritten digits.

You will be building upon your code from the earlier exercises. First, you will train your sparse autoencoder on an "unlabeled" training dataset of handwritten digits. This produces feature that are penstroke-like. We then extract these learned features from a labeled dataset of handwritten digits. These features will then be used as inputs to the softmax classifier that you wrote in the previous exercise.

Concretely, for each example in the the labeled training dataset \textstyle x_l, we forward propagate the example to obtain the activation of the hidden units \textstyle a^{(2)}. We now represent this example using \textstyle a^{(2)} (the "replacement" representation), and use this to as the new feature representation with which to train the softmax classifier.

Finally, we also extract the same features from the test data to obtain predictions.

In this exercise, our goal is to distinguish between the digits from 0 to 4. We will use the digits 5 to 9 as our "unlabeled" dataset which which to learn the features; we will then use a labeled dataset with the digits 0 to 4 with which to train the softmax classifier.

In the starter code, we have provided a file stlExercise.m that will help walk you through the steps in this exercise.

Dependencies

The following additional files are required for this exercise:

You will also need your code from the following exercises:

If you have not completed the exercises listed above, we strongly suggest you complete them first.

Step 1: Generate the input and test data sets

Download and decompress stl_exercise.zip, which contains starter code for this exercise. Additionally, you will need to download the datasets from the MNIST Handwritten Digit Database for this project.

Step 2: Train the sparse autoencoder

Next, use the unlabeled data (the digits from 5 to 9) to train a sparse autoencoder, using the same sparseAutoencoderCost.m function as you had written in the previous exercise. (From the earlier exercise, you should have a working and vectorized implementation of the sparse autoencoder.) For us, the training step took less than 25 minutes on a fast desktop. When training is complete, you should get a visualization of pen strokes like the image shown below:

SelfTaughtFeatures.png

Informally, the features learned by the sparse autoencoder should correspond to penstrokes.

Step 3: Extracting features

After the sparse autoencoder is trained, you will use it to extract features from the handwritten digit images.

Complete feedForwardAutoencoder.m to produce a matrix whose columns correspond to activations of the hidden layer for each example, i.e., the vector a(2) corresponding to activation of layer 2. (Recall that we treat the inputs as layer 1).

After completing this step, calling feedForwardAutoencoder.m should convert the raw image data to hidden unit activations a(2).

Step 4: Training and testing the logistic regression model

Use your code from the softmax exercise (softmaxTrain.m) to train a softmax classifier using the training set features (trainFeatures) and labels (trainLabels).

Step 5: Classifying on the test set

Finally, complete the code to make predictions on the test set (testFeatures) and see how your learned features perform! If you've done all the steps correctly, you should get an accuracy of about 98% percent.

As a comparison, when raw pixels are used (instead of the learned features), we obtained a test accuracy of only around 96% (for the same train and test sets).


Self-Taught Learning | Exercise:Self-Taught Learning

Personal tools