In machine learning, it is common to use activation functions like tanh, sigmoid, or ReLU to introduce non-linearity into a neural network. These non-linearities help the network learn complex relationships between input features and output labels.
Can anyone give a specific example that clearly shows an activation function provides benefits?
The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):
$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$
(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)
Therefore, without activations, a multilayer neural network is equivalent to a linear regression model and therefore only available to learn linear functions over the feature space (lines in 2d, planes in 3D, etc).
While linear models are useful for some types of data (especially with feature engineering), there are many other types of data where the relationship between input and output is non-linear. Some specific examples of non-linear relationship between input and output are the cases where the input is an image or audio signal; in those cases, you can hardly model the relation between the input and output with a linear function.
Activation functions, such as tanh, are used in artificial neural networks to introduce non-linearity into the output of a neuron. Without a non-linear activation function, a neural network would simply be a linear function of its inputs, which would severely limit its ability to model complex relationships between input and output.
Tanh, short for hyperbolic tangent, is one of several activation functions that can be used in neural networks. It is a non-linear function that maps the input to an output in the range of -1 to +1. Tanh is a popular choice of activation function because it is differentiable, which allows for the use of gradient-based optimization algorithms during the training of the neural network. Additionally, it has a nice property of being zero-centered, which can help in reducing the bias in the neural network.
Other activation functions, such as ReLU (rectified linear unit) and sigmoid, are also commonly used in neural networks. Each activation function has its own strengths and weaknesses, and the choice of activation function depends on the specific problem and network architecture.
