Lack of activation function in output layer at regression?

Question

I supposed that the output layer should have certain kind of activation function (preferably linear or tanh) for regression, but I recently read that in case of regression this is not necessary. This seems me reasonable, after all, the node(s) of the output layer produce(s) numeric values themselves. Which solution is better: with or without activation func?

Answer 1

Activation "linear" is identical to "no activation function". The term "linear output layer" also means precisely "the last layer has no activation function". Whether you use one or the other term might be down to how your NN library implements it. You may also see it described either way around in documents, but it is exactly the same thing mathematically:

$$a^{out}_j = b^{out}_j + \sum_{i=1}^{N^{hidden}} W_{ij}a^{hidden}_i$$

Where $a$ values are activation, $b$ are biases, $W$ is weight matrix.

For a regression problem with a mean squared error objective, this is the most common approach.

There is nothing stopping you using other activation functions. They might help you if they match the target variable distribution. About the only rule is that your network output should be able to cover possible values of the target variable. So if the target variable is always between -1.0 and 1.0, with higher density around 0.0, perhaps tanh could also work for you.