Exercise:Sparse Coding

From Ufldl

Jump to: navigation, search
(Step 3: Iterative optimization)
(Step 2: Implement and check sparse coding cost functions)
Line 36: Line 36:
Once you have implemented these functions, you should check the gradients numerically.  
Once you have implemented these functions, you should check the gradients numerically.  
-
'''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(s + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be totally incorrect.
+
'''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(x + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be wildly inaccurate.
=== Step 3: Iterative optimization ===
=== Step 3: Iterative optimization ===

Revision as of 07:22, 28 May 2011

Personal tools