Activation Functions

Activation functions define the output of a graph node given a set of inputs, as illustrated below. Activation functions: 

• provide the capability to introduce non-linearity in order to model non-linear aspects of the real world

• are monotonic - that is, they either constantly increase or decrease - this is important in neural network training to avoid chaotic behavior

Activation Functions commonly used in Machine Learning include:

• Rectifier

• Sigmoid

• Tanh

Activation Function Generalized Equation

An activation function receives the sum of inputs which are modified by weights and a bias, modifies the input sum and produces an output. During neural network training, the weights and bias are repeatedly modified in order to produce a neural network result that has a minimal error (loss) compared to observed training examples.

Using:

the generalized activation function equation is:

Activation Function Data Flow Generalized Equation

In a Machine Learning network, data flows into and out of activation functions.

Using:

the generalized activation function data flow equation is:

In an Artificial Neural Network, Activation Function nodes are positioned between input and output nodes:

and the flow of data through nodes looks like this:

Activation Function Example from Nature

Activation functions are modeled on biological brain and other body cells.

One example is the shift in the action potential of a brain cell. Brain neurons and axons activate and carry electrical signals via neurotransmitter molecules:

The voltage of a signal over time looks like this:

References

• Activation Functions Wikipedia Article

• Video Tutorial on Artificial Neurons

• Video Tutorial on Artificial Neuron Capacity

• Video Tutorial on Activation Functions

• Activation Functions in Convolutional Neural Networks

• Web Search

• YouTube Search