Question

You want to train a feedforward network to recognize a hand written symbol. The network should have 7 hidden layers with 4 neurons each and categorize the image into one of 8 possible categories. The input is an image with resolutions 5 x 2 and a color depth of 2. Each layer - including the input - must have one additional neuron outside what was allocated, as a bias term. In total, how many trainable individual weight parameters (not matrices) are there within the network?

Answer 1

This answer has been edited as per the assumption to flatten the input image.

Given Information

1. 7 hidden layers with 4 neurons each
2. The output layer has 8 neurons
3. The image's dimensions are (5X2X2), It's a fair assumption to flatten the image ou. This will give us a dimension of 20.
4. Bias has been added to every layer including the input layer.

Solution

The formula for calculating the weighted parameters between the two layers is

Number of neurons in the previous layer + Number of neurons in the current layer + Number of neurons in the current layer (BIAS)

Therefore the entire calculation will be divided into 3 parts,

1. Weighted parameters for the hidden layer next to the input layer
2. Weighted parameters for hidden layers connected to the hidden layers
3. Weighted parameters for the output layers

Part 1:

Note that as the input layer doesn't learn anything there is no calculation for the input layer.

Our image shape is 20. Therefore,

The number of nodes in the input layer is 20,

The number of nodes in the hidden layer is 4.

Substituting the values in the formula mentioned above we get,

20*4+4 = 84 ---(i)

Part 2:

The following calculation is a repetitive calculation. As we have 7 hidden layers. Out of the 7 hidden layers, 6 hidden layers are connected to other hidden layers. As the number of nodes in each hidden layer is constant then all we need to do is to calculate the weights between two hidden layers and multiply it by 6.

The number of nodes in previous hidden layer is 4,

The number of nodes in the current hidden layer is 4,

Substituting the values in the formula mentioned above we get,

4*4+4 = 20

Now we have to repeat this calculation for 6 times, so we simply multiply this number by 6,

Therefore we get, 20*6 = 120 ---(ii)

Part 3:

Now we calculate the weighted parameters between the last hidden layer and the output layer.

The number of nodes in the last hidden layer is 4,

The number of nodes in the output layer (number of categories) is 8,

Substituting the values in the formula mentioned above we get,

4*8+8 = 40 ---(iii)

Now adding (i), (ii), and (iii), we get, 84+120+40 = 244

There are a total of 244 weighted parameters in the above-mentioned feed-forward network.