0
$\begingroup$

In my textbook I read that an MLP and linear activation functions for the hidden layers can be reduced to a simple input-output system, i.e. no hidden layers. This makes sense to me. Later on I read that for regression problems, the linear activation function is commonly used (along with MSE loss function). How does this work together? Is it reasonable to say that no layers are ever needed if using an MLP to solve a regression problem, or is any of the statements wrong (or compatible)?

$\endgroup$

2 Answers 2

0
$\begingroup$

In regression, it makes perfect sense to have multiple hidden layers to model the complex relationships between inputs and outputs. However, those layers need to have non-linear activation functions, otherwise, they would be equivalent to a single layer.

When your textbook said that "linear activation function is commonly used (along with MSE loss function)", it refers only to the last layer, to allow the model to generate unbounded real numbers. The hidden layers must have non-linear activation functions.


P.D.: the equivalence of multiple layers with linear activations and a single layer can be proved easily:

$(xW_1 + b_1)W_2 + b_2 = xW_1W_2 + b_1W_2 + b_2 = x(W_1W_2) + (b_1W_2 + b_2)$).

$\endgroup$
1
  • $\begingroup$ "it refers only to the last layer" this was the key part that I didn't understand (and to be frank, my textbook didn't make very clear. Thank you. $\endgroup$
    – falxman
    Commented Jan 10, 2022 at 14:58
0
$\begingroup$

Yes it does, because if the relation between target and training data is a very complex function we may not be able to capture it in one layer with limited units.

Increasing the number of hidden layers may help you model a highly complex function with limited units on each layer. Number of hidden layers varies from problem to problem and mostly is a hot and trial method

$\endgroup$
2
  • $\begingroup$ Did you read the text body of my question? So I'm assuming I'm misinterpreting my textbook, and that linear activation functions are typically not used for the hidden layers? $\endgroup$
    – falxman
    Commented Jan 9, 2022 at 16:33
  • $\begingroup$ Yes..Linear activation functions are usually avoided as they cant model non linear function..but if you are sure target variable and training variable are linear you dont need non linear activation function $\endgroup$ Commented Jan 9, 2022 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.