|
|
| Line 44: |
Line 44: |
| | == 示例 == | | == 示例 == |
| | 为了阐述如何使用反向传导思想计算关于输入的导数，我们要在示例1,示例2中用[[Sparse Coding: Autoencoder Interpretation | sparse coding]]一节中的两个函数。在示例3中，我们使用[[Independent Component Analysis | independent component analysis]]一节中的一个函数来说明使用此思想计算关于权重的偏导的方法，以及在这种特殊情况下，如何处理相同或重复的权重的情况 | | 为了阐述如何使用反向传导思想计算关于输入的导数，我们要在示例1,示例2中用[[Sparse Coding: Autoencoder Interpretation | sparse coding]]一节中的两个函数。在示例3中，我们使用[[Independent Component Analysis | independent component analysis]]一节中的一个函数来说明使用此思想计算关于权重的偏导的方法，以及在这种特殊情况下，如何处理相同或重复的权重的情况 |
| - |
| |
| - | [原文]
| |
| | | | |
| | === 示例1:稀疏编码中权重矩阵的目标函数 === | | === 示例1:稀疏编码中权重矩阵的目标函数 === |
| - | 回顾[[Sparse Coding: Autoencoder Interpretation | sparse coding]]一节中，给定特征矩阵<math>s</math>，权重矩阵<math>A</math>的目标函数为:
| + | 回顾在[[Sparse Coding: Autoencoder Interpretation | sparse coding]]一节中，给定特征矩阵<math>s</math>，权重矩阵<math>A</math>的目标函数为: |
| | :<math>F(A; s) = \lVert As - x \rVert_2^2 + \gamma \lVert A \rVert_2^2</math> | | :<math>F(A; s) = \lVert As - x \rVert_2^2 + \gamma \lVert A \rVert_2^2</math> |
| | | | |
| Line 63: |
Line 61: |
| | [[File:Backpropagation Method Example 1.png | 400px]] | | [[File:Backpropagation Method Example 1.png | 400px]] |
| | | | |
| - | 该网络的权重和激励函数如下所示：
| + | 该网络的权重和激励函数如下表所示： |
| | <table align="center"> | | <table align="center"> |
| | <tr><th width="50px">层</th><th width="200px">权重</th><th width="200px">激励函数 <math>f</math></th></tr> | | <tr><th width="50px">层</th><th width="200px">权重</th><th width="200px">激励函数 <math>f</math></th></tr> |
| Line 84: |
Line 82: |
| | 为了使<math>J(z^{(3)}) = F(x)</math>，我们可令<math>J(z^{(3)}) = \sum_k J(z^{(3)}_k)</math>。 | | 为了使<math>J(z^{(3)}) = F(x)</math>，我们可令<math>J(z^{(3)}) = \sum_k J(z^{(3)}_k)</math>。 |
| | | | |
| - | 一旦我们将<math>F</math>看成神经网络，梯度<math>\nabla_X F</math>就很容易求了——用反向传导得到： | + | 一旦我们将<math>F</math>看成神经网络，梯度<math>\nabla_X F</math>就很容易求了——使用反向传导得到： |
| | <table align="center"> | | <table align="center"> |
| | <tr><th width="50px">层</th><th width="200px">激励函数的导数<math>f'</math></th><th width="200px">Delta</th><th>该层输入<math>z</math></th></tr> | | <tr><th width="50px">层</th><th width="200px">激励函数的导数<math>f'</math></th><th width="200px">Delta</th><th>该层输入<math>z</math></th></tr> |
| Line 116: |
Line 114: |
| | [原文] | | [原文] |
| | | | |
| - | === Example 2: Smoothed topographic L1 sparsity penalty in sparse coding [例2：稀疏编码中的平滑地形L1稀疏罚函数] === | + | === 示例2：稀疏编码中的平滑地形L1稀疏罚函数 === |
| - | | + | 回顾[[Sparse Coding: Autoencoder Interpretation | sparse coding]]一节中对<math>s</math>的平滑地形L1稀疏罚函数： |
| - | Recall the smoothed topographic L1 sparsity penalty on <math>s</math> in [[Sparse Coding: Autoencoder Interpretation | sparse coding]]:
| + | |
| | :<math>\sum{ \sqrt{Vss^T + \epsilon} }</math> | | :<math>\sum{ \sqrt{Vss^T + \epsilon} }</math> |
| - | where <math>V</math> is the grouping matrix, <math>s</math> is the feature matrix and <math>\epsilon</math> is a constant.
| + | 其中<math>V</math>是分组矩阵，<math>s</math>是特征矩阵，<math>\epsilon</math> 是一个常数. |
| | | | |
| - | [初译]
| + | 我们希望求得<math>\nabla_s \sum{ \sqrt{Vss^T + \epsilon} }</math> 。像上面那样，我们把这一项看做一个神经网络的实例： |
| - | | + | |
| - | 例2：稀疏编码中的平滑地形L1稀疏罚函数
| + | |
| - | | + | |
| - | 回想稀疏编码一节中的平滑地形L1稀疏罚函数<math>s</math>：
| + | |
| - | :<math>\sum{ \sqrt{Vss^T + \epsilon} }</math>
| + | |
| - | ,其中<math>V</math>是分组矩阵，<math>s</math>是特征矩阵，<math>\epsilon</math> 是一个常数.
| + | |
| - | | + | |
| - | [一审]
| + | |
| - | | + | |
| - | | + | |
| - | 例2：稀疏编码中的平滑地形L1稀疏罚函数
| + | |
| - | | + | |
| - | 回顾稀疏编码一节中对<math>s</math>的平滑地形L1稀疏罚函数：
| + | |
| - | :<math>\sum{ \sqrt{Vss^T + \epsilon} }</math>
| + | |
| - | ,其中<math>V</math>是分组矩阵，<math>s</math>是特征矩阵，<math>\epsilon</math> 是一个常数.
| + | |
| - | | + | |
| - | [原文]
| + | |
| - | | + | |
| - | We would like to find <math>\nabla_s \sum{ \sqrt{Vss^T + \epsilon} }</math>. As above, let's see this term as an instantiation of a neural network:
| + | |
| | | | |
| | [[File:Backpropagation Method Example 2.png | 600px]] | | [[File:Backpropagation Method Example 2.png | 600px]] |
| - |
| |
| - | [初译]
| |
| - |
| |
| - | 我们希望求<math>\nabla_s \sum{ \sqrt{Vss^T + \epsilon} }</math> 。像上面那样，我们把这一项看做一个具体的神经网络：
| |
| - |
| |
| - | [一审]
| |
| - |
| |
| - | 我们希望求<math>\nabla_s \sum{ \sqrt{Vss^T + \epsilon} }</math> 。像上面那样，我们把这一项看做一个神经网络的实例：
| |
| - |
| |
| - | [原文]
| |
| - |
| |
| - | The weights and activation functions of this network are as follows:
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">Layer</th><th width="200px">Weight</th><th width="200px">Activation function <math>f</math></th></tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>I</math></td>
| |
| - | <td><math>f(z_i) = z_i^2</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>V</math></td>
| |
| - | <td><math>f(z_i) = z_i</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>I</math></td>
| |
| - | <td><math>f(z_i) = z_i + \epsilon</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td>N/A</td>
| |
| - | <td><math>f(z_i) = z_i^{\frac{1}{2}}</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - | To have <math>J(z^{(4)}) = F(x)</math>, we can set <math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>.
| |
| - |
| |
| - | [初译]
| |
| | | | |
| | 该网络的权重和激励函数如下表所示： | | 该网络的权重和激励函数如下表所示： |
| Line 208: |
Line 148: |
| | </table> | | </table> |
| | 为使 <math>J(z^{(4)}) = F(x)</math>，我们可令 <math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。 | | 为使 <math>J(z^{(4)}) = F(x)</math>，我们可令 <math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。 |
| - |
| |
| - | [一审]
| |
| - |
| |
| - | 该网络的权重和激励函数如下表所示：
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">层</th><th width="200px">权重</th><th width="200px">激励函数 <math>f</math></th></tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>I</math></td>
| |
| - | <td><math>f(z_i) = z_i^2</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>V</math></td>
| |
| - | <td><math>f(z_i) = z_i</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>I</math></td>
| |
| - | <td><math>f(z_i) = z_i + \epsilon</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td>N/A</td>
| |
| - | <td><math>f(z_i) = z_i^{\frac{1}{2}}</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - | 为使 <math>J(z^{(4)}) = F(x)</math>，我们可令 <math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。
| |
| - |
| |
| - | [原文]
| |
| - |
| |
| - | Once we see <math>F</math> as a neural network, the gradient <math>\nabla_X F</math> becomes easy to compute - applying backpropagation yields:
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">Layer</th><th width="200px">Derivative of activation function <math>f'</math>
| |
| - | </th><th width="200px">Delta</th><th>Input <math>z</math> to this layer</th></tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td><math>f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}</math></td>
| |
| - | <td><math>f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}</math></td>
| |
| - | <td><math>(Vss^T + \epsilon)</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( I^T \delta^{(4)} \right) \bullet 1</math></td>
| |
| - | <td><math>Vss^T</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( V^T \delta^{(3)} \right) \bullet 1</math></td>
| |
| - | <td><math>ss^T</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| |
| - | <td><math>\left( I^T \delta^{(2)} \right) \bullet 2s</math></td>
| |
| - | <td><math>s</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - |
| |
| - | Hence,
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_X F & = I^T V^T I^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\
| |
| - | & = V^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\
| |
| - | & = V^T (Vss^T + \epsilon)^{-\frac{1}{2}} \bullet s
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | [初译]
| |
| - |
| |
| - | 一旦我们把<math>F</math>看做一个神经网络，梯度<math>\nabla_X F</math> 变得好求了——使用反向传导产生
| |
| - |
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">层</th><th width="200px">激励函数的导数 <math>f'</math>
| |
| - | </th><th width="200px">Delta</th><th>该层输入<math>z</math></th></tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td><math>f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}</math></td>
| |
| - | <td><math>f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}</math></td>
| |
| - | <td><math>(Vss^T + \epsilon)</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( I^T \delta^{(4)} \right) \bullet 1</math></td>
| |
| - | <td><math>Vss^T</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( V^T \delta^{(3)} \right) \bullet 1</math></td>
| |
| - | <td><math>ss^T</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| |
| - | <td><math>\left( I^T \delta^{(2)} \right) \bullet 2s</math></td>
| |
| - | <td><math>s</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - | 因此
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_X F & = I^T V^T I^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\
| |
| - | & = V^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\
| |
| - | & = V^T (Vss^T + \epsilon)^{-\frac{1}{2}} \bullet s
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | [一审]
| |
| | | | |
| | 一旦我们把<math>F</math>看做一个神经网络，梯度<math>\nabla_X F</math> 变得很容易计算——使用反向传导得到： | | 一旦我们把<math>F</math>看做一个神经网络，梯度<math>\nabla_X F</math> 变得很容易计算——使用反向传导得到： |
| Line 361: |
Line 189: |
| | [原文] | | [原文] |
| | | | |
| - | === Example 3: ICA reconstruction cost [例3：ICA重建成本] === | + | === 示例3：ICA重建代价 === |
| | + | 回顾[[Independent Component Analysis | independent component analysis (ICA)]]一节重建代价一项：<math>\lVert W^TWx - x \rVert_2^2</math> ，其中<math>W</math>是权重矩阵，<math>x</math>是输入。 |
| | | | |
| - | Recall the [[Independent Component Analysis | independent component analysis (ICA)]] reconstruction cost term:
| + | 我们希望计算 <math>\nabla_W \lVert W^TWx - x \rVert_2^2</math>——对于'''权重矩阵'''的导数，而不是像前两例中对于'''输入'''的导数。不过我们仍然用类似的方法处理，把该项看做一个神经网络的实例： |
| - | <math>\lVert W^TWx - x \rVert_2^2</math>
| + | |
| - | where <math>W</math> is the weight matrix and <math>x</math> is the input.
| + | |
| - | | + | |
| - | [初译]
| + | |
| - | | + | |
| - | 例3：ICA重建成本
| + | |
| - | | + | |
| - | 回故独立成分分析（ICA）一节重建成本项：<math>\lVert W^TWx - x \rVert_2^2</math> ，其中<math>W</math>是权矩阵，<math>x</math> 是输入。
| + | |
| - | | + | |
| - | [一审]
| + | |
| - | | + | |
| - | 例3：ICA重建成本
| + | |
| - | 回顾独立成分分析（ICA）一节重建成本一项：<math>\lVert W^TWx - x \rVert_2^2</math> ，其中<math>W</math>是权矩阵，<math>x</math>是输入。
| + | |
| - | | + | |
| - | [原文]
| + | |
| - | | + | |
| - | We would like to find <math>\nabla_W \lVert W^TWx - x \rVert_2^2</math> - the derivative of the term with respect to the '''weight matrix''', rather than the '''input''' as in the earlier two examples. We will still proceed similarly though, seeing this term as an instantiation of a neural network:
| + | |
| | | | |
| | [[File:Backpropagation Method Example 3.png | 400px]] | | [[File:Backpropagation Method Example 3.png | 400px]] |
| - |
| |
| - | [初译]
| |
| - |
| |
| - | 我们希望计算 <math>\nabla_W \lVert W^TWx - x \rVert_2^2</math> ——该项关于权矩阵的导数，而不是像前两例中关于输入的导数。不过我们仍然使用相似的过程，把该项看做一个具体的神经网络:
| |
| - |
| |
| - | [一审]
| |
| - |
| |
| - | 我们希望计算 <math>\nabla_W \lVert W^TWx - x \rVert_2^2</math> ——该项关于权重矩阵的导数，而不是像前两例中对于输入的导数。不过我们仍然用类似的方法处理，把该项看做一个神经网络的实例：
| |
| - |
| |
| - | [原文]
| |
| - |
| |
| - | The weights and activation functions of this network are as follows:
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">Layer</th><th width="200px">Weight</th><th width="200px">Activation function <math>f</math></th></tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>W</math></td>
| |
| - | <td><math>f(z_i) = z_i</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>W^T</math></td>
| |
| - | <td><math>f(z_i) = z_i</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>I</math></td>
| |
| - | <td><math>f(z_i) = z_i - x_i</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td>N/A</td>
| |
| - | <td><math>f(z_i) = z_i^2</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - | To have <math>J(z^{(4)}) = F(x)</math>, we can set <math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>.
| |
| - |
| |
| - | [初译]
| |
| | | | |
| | 该网络的权重和激励函数如下表所示： | | 该网络的权重和激励函数如下表所示： |
| Line 448: |
Line 222: |
| | 为使<math>J(z^{(4)}) = F(x)</math>，我们可令<math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。 | | 为使<math>J(z^{(4)}) = F(x)</math>，我们可令<math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。 |
| | | | |
| - | [一审]
| + | 既然我们可将<math>F</math>看做神经网络，我们就能计算出梯度 <math>\nabla_W F</math> 。然而，我们现在面临的难题是<math>W</math> 在网络中出现了两次。幸运的是，可以证明如果<math>W</math> 在网络中出现多次，那么对于<math>W</math> 的梯度是对网络中每个<math>W</math> 实例的梯度的简单相加（你需要自己给出对这一事实的严格证明来说服自己）。知道这一点后，我们将首先计算delta： |
| - | 该网络的权重和激励函数如下表所示：
| + | |
| - | <table align="center">
| + | |
| - | <tr><th width="50px">层</th><th width="200px">权重</th><th width="200px">激励函数 <math>f</math></th></tr>
| + | |
| - | <tr>
| + | |
| - | <td>1</td>
| + | |
| - | <td><math>W</math></td>
| + | |
| - | <td><math>f(z_i) = z_i</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>2</td>
| + | |
| - | <td><math>W^T</math></td>
| + | |
| - | <td><math>f(z_i) = z_i</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>3</td>
| + | |
| - | <td><math>I</math></td>
| + | |
| - | <td><math>f(z_i) = z_i - x_i</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>4</td>
| + | |
| - | <td>N/A</td>
| + | |
| - | <td><math>f(z_i) = z_i^2</math></td>
| + | |
| - | </tr>
| + | |
| - | </table>
| + | |
| - | 为使<math>J(z^{(4)}) = F(x)</math>，我们可令<math>J(z^{(4)}) = \sum_k J(z^{(4)}_k)</math>。
| + | |
| - | | + | |
| - | [原文]
| + | |
| - | | + | |
| - | Now that we can see <math>F</math> as a neural network, we can try to compute the gradient <math>\nabla_W F</math>. However, we now face the difficulty that <math>W</math> appears twice in the network. Fortunately, it turns out that if <math>W</math> appears multiple times in the network, the gradient with respect to <math>W</math> is simply the sum of gradients for each instance of <math>W</math> in the network (you may wish to work out a formal proof of this fact to convince yourself). With this in mind, we will proceed to work out the deltas first:
| + | |
| - | | + | |
| - | <table align="center">
| + | |
| - | <tr><th width="50px">Layer</th><th width="200px">Derivative of activation function <math>f'</math>
| + | |
| - | </th><th width="200px">Delta</th><th>Input <math>z</math> to this layer</th></tr>
| + | |
| - | <tr>
| + | |
| - | <td>4</td>
| + | |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| + | |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| + | |
| - | <td><math>(W^TWx - x)</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>3</td>
| + | |
| - | <td><math>f'(z_i) = 1</math></td>
| + | |
| - | <td><math>\left( I^T \delta^{(4)} \right) \bullet 1</math></td>
| + | |
| - | <td><math>W^TWx</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>2</td>
| + | |
| - | <td><math>f'(z_i) = 1</math></td>
| + | |
| - | <td><math>\left( (W^T)^T \delta^{(3)} \right) \bullet 1</math></td>
| + | |
| - | <td><math>Wx</math></td>
| + | |
| - | </tr>
| + | |
| - | <tr>
| + | |
| - | <td>1</td>
| + | |
| - | <td><math>f'(z_i) = 1</math></td>
| + | |
| - | <td><math>\left( W^T \delta^{(2)} \right) \bullet 1</math></td>
| + | |
| - | <td><math>x</math></td>
| + | |
| - | </tr>
| + | |
| - | </table>
| + | |
| - | | + | |
| - | [初译]
| + | |
| - | | + | |
| - | 既然我们可将<math>F</math>看做神经网络，我们就能计算出梯度 <math>\nabla_W F</math> 。然而，我们还要面对<math>W</math>在网络中出现两次的困难。幸运的是，可以证明如果<math>W</math>在网络中出现两次，那么关于<math>W</math>的梯度是网络中每个含<math>W</math>项的简单相加（你可能希望给出这一事实的严格证明来说服自己）。有了这个事实，我们接下来先计算delta。 | + | |
| | <table align="center"> | | <table align="center"> |
| | <tr><th width="50px">层</th><th width="200px">激励函数的导数 <math>f'</math> | | <tr><th width="50px">层</th><th width="200px">激励函数的导数 <math>f'</math> |
| - | </th><th width="200px">Delta</th><th>该层输入 <math>z</math></th></tr> | + | </th><th width="200px">Delta</th><th>该层输入<math>z</math></th></tr> |
| | <tr> | | <tr> |
| | <td>4</td> | | <td>4</td> |
| Line 539: |
Line 251: |
| | </tr> | | </tr> |
| | </table> | | </table> |
| - |
| |
| - | [一审]
| |
| - |
| |
| - | 既然我们可将<math>F</math>看做神经网络，我们就能计算出梯度 <math>\nabla_W F</math> 。然而，我们现在面临<math>W</math> 在网络中出现两次的困难。幸运的是，可以证明如果<math>W</math> 在网络中多次，那么对于<math>W</math> 的梯度是对网络中每个<math>W</math> 实例的梯度的简单相加（你需要自己给出对这一事实的严格证明来说服自己）。知道这一点后，我们接下来先计算delta。
| |
| - | <table align="center">
| |
| - | <tr><th width="50px">层</th><th width="200px">激励函数的导数 <math>f'</math>
| |
| - | </th><th width="200px">Delta</th><th>该层输入 <math>z</math></th></tr>
| |
| - | <tr>
| |
| - | <td>4</td>
| |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| |
| - | <td><math>f'(z_i) = 2z_i</math></td>
| |
| - | <td><math>(W^TWx - x)</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>3</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( I^T \delta^{(4)} \right) \bullet 1</math></td>
| |
| - | <td><math>W^TWx</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>2</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( (W^T)^T \delta^{(3)} \right) \bullet 1</math></td>
| |
| - | <td><math>Wx</math></td>
| |
| - | </tr>
| |
| - | <tr>
| |
| - | <td>1</td>
| |
| - | <td><math>f'(z_i) = 1</math></td>
| |
| - | <td><math>\left( W^T \delta^{(2)} \right) \bullet 1</math></td>
| |
| - | <td><math>x</math></td>
| |
| - | </tr>
| |
| - | </table>
| |
| - |
| |
| - | [原文]
| |
| - |
| |
| - | To find the gradients with respect to <math>W</math>, first we find the gradients with respect to each instance of <math>W</math> in the network.
| |
| - |
| |
| - | With respect to <math>W^T</math>:
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_{W^T} F & = \delta^{(3)} a^{(2)T} \\
| |
| - | & = 2(W^TWx - x) (Wx)^T
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | With respect to <math>W</math>:
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_{W} F & = \delta^{(2)} a^{(1)T} \\
| |
| - | & = (W^T)(2(W^TWx -x)) x^T
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | [初译]
| |
| - |
| |
| - | 为计算关于<math>W</math>的梯度，首先计算网络中每个含<math>W</math>项的梯度。
| |
| - |
| |
| - | 关于 <math>W^T</math>:
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_{W^T} F & = \delta^{(3)} a^{(2)T} \\
| |
| - | & = 2(W^TWx - x) (Wx)^T
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | 关于 <math>W</math>:
| |
| - | :<math>
| |
| - | \begin{align}
| |
| - | \nabla_{W} F & = \delta^{(2)} a^{(1)T} \\
| |
| - | & = (W^T)(2(W^TWx -x)) x^T
| |
| - | \end{align}
| |
| - | </math>
| |
| - |
| |
| - | [一审]
| |
| | | | |
| | 为计算对于<math>W</math>的梯度，首先计算对网络中每个<math>W</math>实例的梯度。 | | 为计算对于<math>W</math>的梯度，首先计算对网络中每个<math>W</math>实例的梯度。 |
| Line 632: |
Line 270: |
| | </math> | | </math> |
| | | | |
| - | [原文]
| + | 最后进行求和，得到对于<math>W</math>的最终梯度，注意我们需要对<math>W^T</math>梯度进行转置，来得到关于<math>W</math>的梯度（原谅我在这里稍稍滥用了符号）： |
| - | | + | |
| - | Taking sums, noting that we need to transpose the gradient with respect to <math>W^T</math> to get the gradient with respect to <math>W</math>, yields the final gradient with respect to <math>W</math> (pardon the slight abuse of notation here):
| + | |
| - | | + | |
| - | :<math>
| + | |
| - | \begin{align}
| + | |
| - | \nabla_{W} F & = \nabla_{W} F + (\nabla_{W^T} F)^T \\
| + | |
| - | & = (W^T)(2(W^TWx -x)) x^T + 2(Wx)(W^TWx - x)^T
| + | |
| - | \end{align}
| + | |
| - | </math> | + | |
| - | | + | |
| - | [初译]
| + | |
| - | | + | |
| - | 求和得到关于<math>W^T</math>的最终梯度，注意我们需要求关于<math>W^T</math>梯度的转制来得到关于<math>W^T</math>的梯度（原谅我在这里稍稍乱用了符号）：
| + | |
| - | :<math>
| + | |
| - | \begin{align}
| + | |
| - | \nabla_{W} F & = \nabla_{W} F + (\nabla_{W^T} F)^T \\
| + | |
| - | & = (W^T)(2(W^TWx -x)) x^T + 2(Wx)(W^TWx - x)^T
| + | |
| - | \end{align}
| + | |
| - | </math>
| + | |
| - | | + | |
| - | [一审]
| + | |
| - | | + | |
| - | 求和得到对于<math>W^T</math>的最终梯度，注意我们需要求对于<math>W^T</math>梯度的转置来得到关于<math>W^T</math>的梯度（原谅我在这里稍稍滥用了符号）：
| + | |
| | :<math> | | :<math> |
| | \begin{align} | | \begin{align} |
