# Deriving gradients using the backpropagation idea

## Introduction

In the section on the backpropagation algorithm, you were briefly introduced to backpropagation as a means of deriving gradients for learning in the sparse autoencoder. It turns out that together with matrix calculus, this provides a powerful method and intuition for deriving gradients for more complex matrix functions (functions from matrices to the reals, or symbolically, from $\mathbb{R}^{r \times c} \rightarrow \mathbb{R}$).

First, recall the backpropagation idea, which we present in a modified form appropriate for our purposes below:

1. For each output unit i in layer nl (the final layer), set
$\delta^{(n_l)}_i = \frac{\partial}{\partial z^{(n_l)}_i} \;\; J(z^{(n_l)})$
where J(z) is our "objective function" (explained below).
2. For $l = n_l-1, n_l-2, n_l-3, \ldots, 2$
For each node i in layer l, set
$\delta^{(l)}_i = \left( \sum_{j=1}^{s_{l+1}} W^{(l)}_{ji} \delta^{(l+1)}_j \right) \bullet \frac{\partial}{\partial z^{(l)}_i} f^{(l)} (z^{(l)}_i)$
3. Compute the desired partial derivatives,
\begin{align} \nabla_{W^{(l)}} J(W,b;x,y) &= \delta^{(l+1)} (a^{(l)})^T, \\ \end{align}

Quick notation recap:

• l is the number of layers in the neural network
• nl is the number of neurons in the lth layer
• $W^{(l)}_{ji}$ is the weight from the ith unit in the lth layer to the jth unit in the (l + 1)th layer
• $z^{(l)}_i$ is the input to the ith unit in the lth layer
• $a^{(l)}_i$ is the activation of the ith unit in the lth layer
• $A \bullet B$ is the Hadamard or element-wise product, which for $r \times c$ matrices A and B yields the $r \times c$ matrix $C = A \bullet B$ such that $C_{r, c} = A_{r, c} \cdot B_{r, c}$
• f(l) is the activation function for units in the lth layer

Let's say we have a function F that takes a matrix X and yields a real number. We would like to use the backpropagation idea to compute the gradient with respect to X of F, that is $\nabla_X F$. The general idea is to see the function F as a multi-layer neural network, and to derive the gradients using the backpropagation idea.

To do this, we will set our "objective function" to be the function J(z) that when applied to the outputs of the neurons in the last layer yields the value F(x). For the intermediate layers, we will also choose our activation functions f(l) to this end.

## Examples

We will use two functions from the section on sparse coding to illustrate the method of computing gradients of functions on matrices using the backpropagation idea.

### Example 1: Objective for weight matrix in sparse coding

Recall the objective function for the weight matrix A, given the feature matrix s:

$F(A; s) = \lVert As - x \rVert_2^2 + \gamma \lVert A \rVert_2^2$

We would like to find the gradient of F with respect to A, or in symbols, $\nabla_A F(A)$. Since the objective function is a sum of two terms in A, the gradient is the sum of gradients of each of the individual terms. The gradient of the second term is trivial, so we will consider the gradient of the first term instead.

The first term, $\lVert As - x \rVert_2^2$, can be seen as an instantiation of neural network taking s as an input, and proceeding in four steps, as described and illustrated in the paragraph and diagram below:

1. Apply A as the weights from the first layer to the second layer.
2. Subtract x from the activation of the second layer, which uses the identity activation function.
3. Pass this unchanged to the third layer, via identity weights. Use the square function as the activation function for the third layer.
4. Sum all the activations of the third layer.

The weights and activation functions of this network are as follows:

LayerWeightActivation function f
1 A f(zi) = zi (identity)
2 I (identity) f(zi) = zixi
3 N/A $f(z_i) = z_i^2$

To have J(z(3)) = F(x), we can set $J(z^{(3)}) = \sum_k J(z^{(3)}_k)$.

Once we see F as a neural network, the gradient $\nabla_X F$ becomes easy to compute - applying backpropagation yields:

LayerDerivative of activation function f'DeltaInput z to this layer
3 f'(zi) = 2zi f'(zi) = 2zi Asx
2 f'(zi) = 1 $\left( I^T \delta^{(3)} \right) \bullet 1$ As
1 f'(zi) = 1 $\left( A^T \delta^{(2)} \right) \bullet 1$ s

Hence,

\begin{align} \nabla_X F & = A^T I^T 2(As - x) \\ & = A^T 2(As - x) \end{align}

### Example 2: Smoothed topographic L1 sparsity penalty in sparse coding

Recall the smoothed topographic L1 sparsity penalty on s in sparse coding:

$\sum{ \sqrt{Vss^T + \epsilon} }$

where V is the grouping matrix, s is the feature matrix and ε is a constant.

We would like to find $\nabla_s \sum{ \sqrt{Vss^T + \epsilon} }$. As above, let's see this term as an instantiation of a neural network:

The weights and activation functions of this network are as follows:

LayerWeightActivation function f
1 I $f(z_i) = z_i^2$
2 V f(zi) = zi
3 I f(zi) = zi + ε
4 N/A $f(z_i) = z_i^{\frac{1}{2}}$

To have J(z(4)) = F(x), we can set $J(z^{(4)}) = \sum_k J(z^{(4)}_k)$.

Once we see F as a neural network, the gradient $\nabla_X F$ becomes easy to compute - applying backpropagation yields:

LayerDerivative of activation function f' DeltaInput z to this layer
4 $f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}$ $f'(z_i) = \frac{1}{2} z_i^{-\frac{1}{2}}$ (VssT + ε)
3 f'(zi) = 1 $\left( I^T \delta^{(3)} \right) \bullet 1$ VssT
2 f'(zi) = 1 $\left( V^T \delta^{(3)} \right) \bullet 1$ ssT
1 f'(zi) = 2zi $\left( I^T \delta^{(2)} \right) \bullet 2s$ s

Hence,

\begin{align} \nabla_X F & = I^T V^T I^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\ & = V^T \frac{1}{2}(Vss^T + \epsilon)^{-\frac{1}{2}} \bullet 2s \\ & = V^T (Vss^T + \epsilon)^{-\frac{1}{2}} \bullet s \end{align}