Exercise:Sparse Coding

From Ufldl

Jump to: navigation, search
(Step 3: Iterative optimization)
(Sparse Coding)
 
Line 1: Line 1:
== Sparse Coding ==
== Sparse Coding ==
-
In this exercise, you will implement [[sparse coding]] and topographic sparse coding on black-and-white natural images.  
+
In this exercise, you will implement [[Sparse Coding: Autoencoder Interpretation | sparse coding]] and [[Sparse Coding: Autoencoder Interpretation | topographic sparse coding]] on black-and-white natural images.  
In the file <tt>[http://ufldl.stanford.edu/wiki/resources/sparse_coding_exercise.zip sparse_coding_exercise.zip]</tt> we have provided some starter code. You should write your code at the places indicated "YOUR CODE HERE" in the files.
In the file <tt>[http://ufldl.stanford.edu/wiki/resources/sparse_coding_exercise.zip sparse_coding_exercise.zip]</tt> we have provided some starter code. You should write your code at the places indicated "YOUR CODE HERE" in the files.
Line 36: Line 36:
Once you have implemented these functions, you should check the gradients numerically.  
Once you have implemented these functions, you should check the gradients numerically.  
-
'''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(s + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be totally incorrect.
+
'''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(x + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be wildly inaccurate.
=== Step 3: Iterative optimization ===
=== Step 3: Iterative optimization ===
-
In this step, you will iteratively optimize for the weights and features to learn a basis for the data. However, you need to first fill in the analytic solution to the the optimization problem with respect to the weight matrix, given the feature matrix. Once that is done, you should check that your solution is correct using the given checking code, which checks that the gradient at the point determined by your analytic solution is close to 0. Once your solution has been verified, comment out the checking code, and run the iterative optimization code. 200 iterations should take less than 45 minutes to run, and by 100 iterations you should be able to see bases that look like edges, similar to those you learned in [[Exercise:Sparse Autoencoder | the sparse autoencoder exercise]].  
+
In this step, you will iteratively optimize for the weights and features to learn a basis for the data, as described in the section on [[Sparse Coding: Autoencoder Interpretation | sparse coding]]. Mini-batching and initialization of the features <math>s</math> has already been done for you. However, you need to still need to fill in the analytic solution to the the optimization problem with respect to the weight matrix, given the feature matrix.  
 +
 
 +
Once that is done, you should check that your solution is correct using the given checking code, which checks that the gradient at the point determined by your analytic solution is close to 0. Once your solution has been verified, comment out the checking code, and run the iterative optimization code. 200 iterations should take less than 45 minutes to run, and by 100 iterations you should be able to see bases that look like edges, similar to those you learned in [[Exercise:Sparse Autoencoder | the sparse autoencoder exercise]].  
For the non-topographic case, these features will not be "ordered", and will look something like the following:
For the non-topographic case, these features will not be "ordered", and will look something like the following:

Latest revision as of 06:59, 29 May 2011

Personal tools