The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

Therefore, without activations, a multilayer neural network is equivalent to a linear regression model and therefore only available to learn linear functions over the feature space (lines in 2d, planes in 3D, etc).

While linear models are useful for some types of data, there are many other types of data where the relationship between input and output is non-linear.

The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

Therefore, without activations, a multilayer neural network is equivalent to a linear regression model and therefore only available to learn linear functions over the feature space (lines in 2d, planes in 3D, etc).

While linear models are useful for some types of data (especially with feature engineering), there are many other types of data where the relationship between input and output is non-linear. Some specific examples of non-linear relationship between input and output are the cases where the input is an image or audio signal; in those cases, you can hardly model the relation between the input and output with a linear function.

The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

Therefore, without activations, a multilayer neural network is equivalent to a linear regression model and therefore only available to learn linear functions over the feature space (lines in 2d, planes in 3D, etc).

While linear models are useful for some types of data, there are many other types of data where the relationship between input and output is non-linear.