Gradient checking and advanced optimization
From Ufldl
m |
|||
Line 1: | Line 1: | ||
Backpropagation is a notoriously difficult algorithm to debug and get right, | Backpropagation is a notoriously difficult algorithm to debug and get right, | ||
- | especially since many subtly buggy implementations of it | + | especially since many subtly buggy implementations of it—for example, one |
that has an off-by-one error in the indices and that thus only trains some of | that has an off-by-one error in the indices and that thus only trains some of | ||
- | the layers of weights, or an implementation that omits the bias term | + | the layers of weights, or an implementation that omits the bias term—will |
manage to learn something that can look surprisingly reasonable | manage to learn something that can look surprisingly reasonable | ||
(while performing less well than a correct implementation). Thus, even with a | (while performing less well than a correct implementation). Thus, even with a |