Backpropagation Algorithm
From Ufldl
m (Fixed quotes) |
|||
Line 115: | Line 115: | ||
the algorithm using matrix-vectorial notation. | the algorithm using matrix-vectorial notation. | ||
We will use "<math>\textstyle \bullet</math>" to denote the element-wise product | We will use "<math>\textstyle \bullet</math>" to denote the element-wise product | ||
- | operator (denoted | + | operator (denoted "<tt>.*</tt> |
+ | in Matlab or Octave, and also called the Hadamard product), | ||
so | so | ||
that if <math>\textstyle a = b \bullet c</math>, then <math>\textstyle a_i = b_ic_i</math>. | that if <math>\textstyle a = b \bullet c</math>, then <math>\textstyle a_i = b_ic_i</math>. | ||
Line 153: | Line 154: | ||
below, <math>\textstyle \Delta W^{(l)}</math> is a matrix (of the same dimension as <math>\textstyle W^{(l)}</math>), and | below, <math>\textstyle \Delta W^{(l)}</math> is a matrix (of the same dimension as <math>\textstyle W^{(l)}</math>), and | ||
<math>\textstyle \Delta b^{(l)}</math> is a vector (of the same dimension as <math>\textstyle b^{(l)}</math>). Note that in this notation, | <math>\textstyle \Delta b^{(l)}</math> is a vector (of the same dimension as <math>\textstyle b^{(l)}</math>). Note that in this notation, | ||
- | + | "<math>\textstyle \Delta W^{(l)}</math>" is a matrix, and in particular it isn't "<math>\textstyle \Delta</math> times <math>\textstyle W^{(l)}</math>." | |
We implement one iteration of batch gradient descent as follows: | We implement one iteration of batch gradient descent as follows: | ||