Neural Networks
From Ufldl
Line 101: | Line 101: | ||
to apply to vectors in an element-wise fashion (i.e., | to apply to vectors in an element-wise fashion (i.e., | ||
<math>f([z_1, z_2, z_3]) = [f(z_1), f(z_2), f(z_3)]</math>), then we can write | <math>f([z_1, z_2, z_3]) = [f(z_1), f(z_2), f(z_3)]</math>), then we can write | ||
- | + | the equations above more | |
compactly as: | compactly as: | ||
:<math>\begin{align} | :<math>\begin{align} | ||
Line 109: | Line 109: | ||
h_{W,b}(x) &= a^{(3)} = f(z^{(3)}) | h_{W,b}(x) &= a^{(3)} = f(z^{(3)}) | ||
\end{align}</math> | \end{align}</math> | ||
- | More generally, recalling that we also use <math>a^{(1)} = x</math> to also denote the values from the input layer, | + | We call this step '''forward propagation.''' More generally, recalling that we also use <math>a^{(1)} = x</math> to also denote the values from the input layer, |
then given layer <math>l</math>'s activations <math>a^{(l)}</math>, we can compute layer <math>l+1</math>'s activations <math>a^{(l+1)}</math> as: | then given layer <math>l</math>'s activations <math>a^{(l)}</math>, we can compute layer <math>l+1</math>'s activations <math>a^{(l+1)}</math> as: | ||
:<math>\begin{align} | :<math>\begin{align} | ||
Line 126: | Line 126: | ||
layer <math>\textstyle l</math> is densely connected to layer <math>\textstyle l+1</math>. In this setting, to compute the | layer <math>\textstyle l</math> is densely connected to layer <math>\textstyle l+1</math>. In this setting, to compute the | ||
output of the network, we can successively compute all the activations in layer | output of the network, we can successively compute all the activations in layer | ||
- | <math>\textstyle L_2</math>, then layer <math>\textstyle L_3</math>, and so on, up to layer <math>\textstyle L_{n_l}</math>, using | + | <math>\textstyle L_2</math>, then layer <math>\textstyle L_3</math>, and so on, up to layer <math>\textstyle L_{n_l}</math>, using the equations above that describe the forward propagation step. This is one |
example of a '''feedforward''' neural network, since the connectivity graph | example of a '''feedforward''' neural network, since the connectivity graph | ||
does not have any directed loops or cycles. | does not have any directed loops or cycles. |