Tanh Activation Function

The tanh (Hyperbolic Tangent) activation function is the hyperbolic analogue of the tan circular function used throughout trigonometry.

The equation for tanh is:

$\\ f\left ( x \right )=tanh\left ( x \right )=\frac{2}{1+e^{-2x}}-1$

Compared to the Sigmoid function, tanh produces a more rapid rise in result values.

The derivative of tanh is:

$f'\left ( x \right )=1-\left (f\left ( x \right ) ^{2} \right )$

Common negative comments about tanh activation functions include:

Tanh can saturate and kill gradients. Gradients (change) at the tails of -1 and 1 are almost zero.

References