Question

Why is sigmoid activation function not recommended for hidden units but is fine for an output unit?

Answer 1

Answer :

• As you can see, the gradient for the sigmoid function will saturate and when using the chain rule, it will contract. By difference, the subsidiary for ReLU is dependably 1 or 0.
• The sigmoid, Gaussian and sinusoidal capacities are chosen because of their autonomous and major space division properties.
• The sigmoid capacity isn't successful for a solitary concealed unit. Despite what might be expected, alternate capacities can give great execution.
• At the point when a few shrouded units are utilized, the sigmoid capacity is helpful. Be that as it may, the union speed is still slower than the others.
• The Gaussian function is sensitive to the additive noise, while the others are rather insensitive. As a result, based on convergence rates, the minimum error and noise sensitivity, the sinusoidal function is most useful for both with and without additive noise.
• The property of each function is discussed based on the internal representation, that is the distribution of the hidden unit inputs and outputs.
• Although this selection depends on the input signals to be classified, the periodic function can be effectively applied to a wide range of application fields.
• Actually,sigmoid function is obsoletely replaced by relu function. As we all know,deep nn is hard to train when the network goes deep.
• Why that happens,you can reference to this article . Neural networks and deep learning.Generally, it is caused by Gradient disappearance.
• So, people create relu function to remission this issue. Relu make it possible to train deeper network .
• Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is h=max(0,a)h=max(0,a)h=max(0,a)where a=Wx+ba=Wx+ba=Wx+b.
• One major benefit is the reduced likelihood of the gradient to vanish. This arises when a>0a>0a>0. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.
• The other benefit of ReLUs is sparsity. Sparsity arises when a?0a?0a?0. The more such units that exist in a layer the more sparse the resulting representation.
• Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.
• Actually,sigmoid function is obsoletely replaced by relu function. As we all know,deep nn is hard to train when the network goes deep.
• Why that happens,you can reference to this article . Neural networks and deep learning.Generally, it is caused by Gradient disappearance.
• So, people create relu function to remission this issue. Relu make it possible to train deeper network .
• Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is h=max(0,a)h=max(0,a)h=max(0,a)where a=Wx+ba=Wx+ba=Wx+b.
• One major benefit is the reduced likelihood of the gradient to vanish. This arises when a>0a>0a>0. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.
• The other benefit of ReLUs is sparsity. Sparsity arises when a?0a?0a?0. The more such units that exist in a layer the more sparse the resulting representation.
• Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.