Feedforward Artificial Neural Network pt2

Feedforward Artificial Neural Network pt2

In the previous post, we briefly looked at the general structure of a Feedforward ANN and why such a construction should be useful. Now that we have a general handle of the idea behind a Feedforward ANN, we need to train the network - find a method to calculate the appropriate weights for our hidden layer neurons. In the context of our classification case, we want to train the network by providing it a dataset with labels so that it can predict the labels of new data once trained - a classic supervised learning scenario.

Backpropagation

Backpropagation is the algorithm we'll use to train (i.e minimize the error function of) our ANN. It is essentially repeated applications of the chain rule - chaining together all possible paths in the network to establish how the error changes as a function of the inputs and outputs of each neuron. This may not seem like a very profound approach, nor may it be clear that this use of the chain rule should deserve it's own name, however the power from this method comes from the implementation. The set up of backpropagation allows the derivatives we use in the chain rule to be calculated using intermediate results in our network. This becomes especially important when the size of our network grows - it is efficient in that it saves the recalculation of derivatives.

Some notation

Suppose we have two neurons labelled by $i$ and $j$ respectively, as depicted below:

where $\sigma_j$ is a label for the neuron with the following meaning:

• We define the activation function for neuron $i$ as $a_j(w_{i \rightarrow j} \cdot a_i)$, where $w_{i \rightarrow j}$ is the weight applied to the input $a_i$ of neuron $j$. 

Gradient Descent

In order to minimize the Loss function, one might think to simply take the derivatives and throw them into a prebuilt optimisation routine from your favourite library. This approach is fine when the network is small, but when it scales to millions of neurons it becomes computationally intractable. We will in fact use a standard optimisation routine, but it is the clever implementation which makes it so attractive. The optimisation method we'll be using is called the batch gradient descent method, this is in contrast to the stochastic gradient descent method. It is a fairly standard iterative method to minmize a loss function $L\left(w_{i \rightarrow j} \right)$ by updating the target variables by

$$ w_{i\rightarrow j} \rightarrow w_{i\rightarrow j} - \eta \frac{\partial L}{\partial w_{i\rightarrow j}} $$

at each iteration. $\eta$ is the so called learning rate which determines how big/small the steps are at each iteration. Intuitively small steps will take a longer time for the algorithm to converge, while larger steps cover more ground, they may overshoot the true minimum - the ideal value is dependent on the loss function and is somewhere in between.

The Loss Function

Once we have the architecture of our ANN set up, we need to measure its performance. In order to do so, we borrow from information theory, a loss function which is commonly found in ANN classification tasks: the cross entropy loss
$$ L = -\frac{1}{N} \sum_{n \in N} \sum_{ i \in C} y_{n,i} \ln \hat{y}_{n,i} $$
where we sum over N, the number of training examples and C, the number of classes in our classification problem. The hat $\hat{y}$ indicates an output classification from our network, where $y$ is the actual classification of the training sample. This is the Loss function we will use once we begin to classify the examples from the previous blog post. Note that if we were building the ANN with a continuous output (i.e. regression) we could use the usual Residual Sum of Squares

$$L_{reg} = \sum_{i=1}^{N} \frac{1}{2} \left( \hat{y_{i}} - y_{i} \right)^{2} $$

where the factor of a half is chosen for convenience (it prevents stray factors of $\frac{1}{2}$ floating around when we differentiate).
Thus the task of training our neural network then becomes an optimisation problem; minimize the classification error function $L$ with respect to the weights $w_{i \rightarrow j}$ of each neuron in the hidden layer -  see below for an explanation of the notation. Note this is ignoring any potential problems of overfitting.

Illuminating example - a sample ANN

I'm still working my way around TikZ which will allow me to make nice diagrams in LaTeX, but in the interest of time I've borrowed the below image from here. The network pictured has a single input $x_i$, 2 hidden layers and a single output. We will not assume the functional form of the loss function $L$ or the activation functions $\sigma$. 

We want to understand how the output $\hat{y}_i$ changes with respect to the input weight $w_{in \rightarrow i}$, we can use the chain rule to get this relationship. Noting that there are two paths from the input layer to the output, we need to consider each path separately. 

We define the following in the network, where $i$, $j$, $k$, $o$ index the neurons 

$$z_i = w_{in \rightarrow i} x_i,  \quad a_i = \sigma(z_i)$$

$$z_j = w_{i \rightarrow j} a_i, \quad a_j = \sigma(z_j)$$

$$z_k = w_{i \rightarrow k} a_i, \quad a_j = \sigma(z_k)$$

$$z_o = w_{k \rightarrow o} a_k + w_{j \rightarrow o} a_j, \quad \hat{y} \equiv a_j = output(z_o)$$

where $output(x)$ is the output function, which for now we will keep generic and $\sigma(x)$ is the activation function, which is the same for each neuron in the network.

We decompose $\frac{\partial L}{\partial w_{in \rightarrow j}}$: starting at the right side of the graph and moving through $o \rightarrow k \rightarrow i$ as below:

$$\frac{\partial L}{\partial z_o} \times \frac{\partial z_o}{\partial z_k} \times \frac{\partial z_k}{\partial z_i} \times \frac{\partial z_i}{\partial w_{in \rightarrow j}}$$

Let's look at an individual component $$\frac{\partial z_o}{ \partial z_k} = \frac{\partial}{\partial z_k} \left( w_{k \rightarrow o} a_k + w_{j \rightarrow o} a_j \right)$$ noting the second term does not depend on $z_k$

$$\frac{\partial}{\partial z_k}  w_{k \rightarrow o} \sigma(z_k)  = w_{k \rightarrow o} \sigma ' (z_k)$$With a similar analysis we see that the product of derivatives can be expressed as

$$\frac{\partial L}{\partial z_o} \times w_{k \rightarrow o} \sigma ' (z_k) \times w_{i \rightarrow k} \sigma ' (z_i) \times x_i$$

and now moving through $o \rightarrow j \rightarrow i$

$$\frac{\partial L}{\partial z_o} \times \frac{\partial z_o}{\partial z_j} \times \frac{\partial z_j}{\partial z_i} \times \frac{\partial z_i}{\partial w_{in \rightarrow j}}$$

$$=\frac{\partial L}{\partial z_o} \times w_{j \rightarrow o} \sigma ' (z_j) \times w_{i \rightarrow k} \sigma ' (z_i) \times x_i$$

Putting this all together we have $$\frac{\partial L}{\partial w_{in \rightarrow i}} = \frac{\partial L}{\partial z_o} \left[ w_{k \rightarrow o} \sigma ' (z_k) \times w_{i \rightarrow k} \sigma '(z_i) + w_{j \rightarrow o} \sigma ' (z_j) \times w_{i \rightarrow j} \sigma '(z_i) \right] x_i$$

where we have yet to specify the exact form of the loss function $L$ and activation function $\sigma(x)$ in this example. Once these are specified, we can just plug them in to the above formula. Hopefully this simplified example illuminates how the derivatives of the loss function wrt to each neuron is calculated. In terms of backpropagation, we can think of starting out the output and working backwards through each neuron in the path leading to it and understanding how the error changes at each point. In the next section I will generalise these results so they are easily extendable to a larger network.

Error propagation

We will now generalise the above computation, in order to get the derivative of the loss function with respect to any $z_j$ in any of the layers of the network. We define the error signal $$\delta_j = \frac{\partial L}{\partial z_j}$$ Now, using the chain rule as in the example above we have $$\delta_j = \frac{\partial L}{\partial \hat{y}} \times \frac{\partial \hat{y}}{\partial z_j}$$ Once we specify our loss function $L$ we'll have a nice expression for the first term in the expression above, for now let's keep it generic. Now using the definitions of the $z_i$ and $a_i$ above: $$\frac{\partial \hat{y}}{\partial z_j} = \frac{\partial \hat{y}}{\partial a_j} \times \frac{\partial a_j}{\partial z_j} = \sigma ' (z_j) \sum_{k \in path(j)} \frac{\partial \hat{y}}{\partial z_k} \frac{\partial z_k}{\partial a_j}$$ where the sum is over the $k$ paths which are connected to the output of neuron $j$. We have $z_k = a_j w_{j \rightarrow k}$ so our expression becomes $$\frac{ \partial \hat{y}}{\partial z_j} = \sigma '(z_j) \sum_{k \in path(j)} \frac{\partial \hat{y}}{\partial z_k} w_{j \rightarrow k}$$ $$\implies \delta_j = \sigma '(z_j) \sum_{k \in path(j)} \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial z_k} w_{j \rightarrow k}$$ Note the first two terms in the summation are actually $\delta_k$ in disguise! Thus we have a recursive definition for our error propagation: $$\delta_j =  \sigma ' (z_j) \sum_{k \in path(j)} \delta_k w_{j \rightarrow k}$$
Thus if we have $\delta_\text{output}$, we can work our way backwards through the neural network, calculating the error for each neuron as a function of the error of the neurons that its output is connected to. This is the key concept of back propagation - although it is just an application of the chain rule, because we can propagate the error at each point in the network through subsequent neurons, the calculations can be done on the fly by utilising calculations that have already been performed.

Next time we'll describe the architecture of the actual ANN we'll use to classify the make moons example detailted in pt1. To be honest, these concepts didn't fully sink in until the first time I implemented an ANN, going through the details of the calculations, I suggest you work through this and the next blog post by hand in order to ensure you have a handle on how the feedforward and error propagations are affected by different topologies of ANNs.