Question

Q3# Briefly provide the answers to the following questions?

(a) Define overfitting.

(b) List causes of overfitting in neural networks.

(c) How can overfitting be avoided in neural nertworks?

Answer 1

a)-->Answer::

Overfitting is a modeling error that occurs when a function is too closely fit to a limited set of data points. Overfitting the model generally takes the form of making an overly complex model.Model does not generalize well from observed data to unseen data, which is called overfitting.

In other words, A model that learns the training dataset too well, performing well on the training dataset but does not perform well on a hold out sample is called Overfitted model.

b)-->>Answer::

List of causes of overfitting

• Overfitting occurs when a model tries to predict a trend in data that is too noisy.
• Caused due to an overly complex model with too many parameters.
• Hypothesis complexity

c)Answer::

Avoiding overfitting while training neural networks.

1. Simplifying The Model : The first step when dealing with overfitting is to decrease the complexity of the model. To decrease the complexity, we can simply remove layers or reduce the number of neurons to make the network smaller.So to it important to calculate input ouput dimensions and try to make it smaller.

2. Early Stopping: Early stopping is a form of regularization while training a model with an iterative method, such as gradient descent. All neural nwtworks uses gradient descent and the technique early stopping is used by all the problems.This method update the model so as to make it better fit the training data with each iteration.Improving models perfomance leads to generalization error.Early stopping rules provide guidance as to how many iterations can be run before the model begins to overfit.

3. Use Data Augmentation: In the case of neural networks, data augmentation simply means increasing size of the data that is increasing the number of images present in the dataset. Some of the popular image augmentation techniques are flipping, translation, rotation, scaling, changing brightness, adding noise etc.

4. Use Regularization: Regularization is a technique to reduce the complexity of the model. It does so by adding a penalty term to the loss function.

The most common techniques are known as L1 and L2 regularization:

• The L1 penalty aims to minimize the absolute value of the weights.
• The L2 penalty aims to minimize the squared magnitude of the weights

5. Use Dropouts: Dropout is a regularization technique that prevents neural networks from overfitting.Dropout modify the network by itself  It randomly drops neurons from the neural network during training in each iteration. When we drop different sets of neurons, it’s equivalent to training different neural networks. The different networks will overfit in different ways, so the net effect of dropout will be to reduce overfitting.