Question

Training a convolutional neural network for speech recognition, one finds that performance on the training set is very good while the performance on the validation set is unacceptably low. A reasonable fix might be to: (Select the single best answer)

And please give a explanation why they are true or false

(A) Decrease the weight decay
(B) Reduce the training set size
(C) Reduce the number of layers and neurons
(D) Increase the number of layers and neurons

Answer 1

A.True:
Having fewer parameters is only one way of preventing our model from getting overly complex. But it is actually a very limiting strategy. More parameters mean more interactions between various parts of our neural network. And more interactions mean more non-linearities. These non-linearities help us solve complex problems.However, we don’t want these interactions to get out of hand. Hence, what if we penalize complexity. We will still use a lot of parameters, but we will prevent our model from getting too complex. To prevent that from happening, we multiply the sum of squares with another smaller number. This number is called weight decay.

B.False:
The gap in errors between training and test suggests a high variance problem in which the algorithm has overfit the training set. Adding more training data will make the model to learn more accurately alsowith more data will increase the diversity .Hence if you reduce the traning set size the model may not learn properly and increase the variance as for eg if we train a model to classify images of dog and cat and model have only  seen images of larger dog like lobster,Boxer etc will not be able to recognize Pomeranian dog.

C.False.(one can use this method as well to increase the proformance on validation data).
To decrease the complexity, we can simply remove layers or reduce the number of neurons to make the network smaller. While doing this, it is important to calculate the input and output dimensions of the various layers involved in the neural network. There is no general rule on how much to remove or how large your network should be. But, if your neural network is overfitting, try making it smaller.This will lead to make your traning model more genralize and less prone to validation errors.

D.False
Increasing the number of hidden units and/or layers may lead to overfitting because it will make it easier for the neural network to memorize the training set, that is to learn a function that perfectly separates the training set but that does not generalize to unseen data.