Alternatives to linear activation function in regression tasks to limit the output

Question

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not inside the range while training. I've not seen any paper but I have two solutions for that right now but I don't know if there is any standard way or not.

1. My first suggested solution is to use an activation function like the following. It is linear in the specified range and has high slope out of the specified range. Although the gradient will oscillate if I use simple gradient descent, I guess it will perform well if I use second-order optimization algorithms like Adam. Any suggestions for this?

2. My second solution is inspired by L1/L2 regularization. I find the output of the model. If it exceeds the absolute value of the range, suppose it's symmetric, I'll add a constant big value which is far from the real output. Here We can replace it with a big constant or maybe multiply it by a constant. The second one has this property that it will be differentiable with a slope.

Does anyone have suggestions which are applicable or even these are ok or any other thing?

---

In regression tasks, it is customary to use linear activation function as the non-linearity of the final layer in order to estimate a function which outputs real value(s). The reason sigmoid function is used is that its output is limited to the range 0 to 1 which is a good range for specifying probability. Moreover, sigmoid is used more for classification tasks where the classes are mutually exclusive in the input o.w. softmax activation function is used. Tanh is used due to having zero mean which accelerates the training process. These are usually not applied in classification tasks these days because their differentiation saturates in their limits. Relu is a customary choice because its slope does not saturate. Sigmoid-shape activation functions cannot be used as the last layer of the regression tasks. I want to use a kind of activation function like linear as the last layer's activation function in order to have outputs in a special range. While training I want to train the model in a way that it never tries to output a value which is outside a specified range. For doing so, I've put two suggestions that I have tried. I don't know whether there is a paper about that or not. Moreover, about the provided answer, sigmoid does not penalize the outputs, I mean does not increase them in a way that the error value increases, which are outside the range of the domain of the function. What it does is saturating the output to zero or one. What I want to do is to make a huge difference between the output of the network and the real output of the input in order to cause a high error. Consequently, the network will try to find weights which do not yield results out of the range.

Answer 1

Don't use that activation function shown in the question. It doesn't do what you think it does. Instead:

If you want the output to be in the range $[-1,1]$, you can use a sigmoid or tanh activation function in the final layer.

If you want the output to be in the range $[-100,100]$, do the same, then multiply the final output by 100 (a hardcoded constant).

If you want the output to be in the range $[-50,150]$, do the same, then multiply the final output by 100, then add 50.

Hopefully you can now see how to handle any desired range.

---

After some edits, I see some misconceptions in your question:

• Contrary to what you write, there is no reason that the last layer of a regression model has to use a linear activation function. It's not like there are "regression police" who are going to arrest you for using a different activation function.

  You say it is customary to use a linear function at the output of a regression model. That's not really because those models are doing regression; rather, that's more because they are solving a task where you want range of possible outputs to be $[-\infty,+\infty]$, so of course they're not going to use an activation function that restricts the range. Now you have a new setting where you do want to restrict the output range -- so it is reasonable to use a different activation function. You're have a non-customary regression task, so don't be surprised if this means you might benefit from an activation function that is non-customary.

• Sigmoid and tanh have been used in other contexts because they do have other useful properties, but that doesn't mean they're not also useful here. An activation function can have multiple useful properties, some of which are relevant in one setting and some of which are relevant in another.

  • Sigmoid (also called softmax) is often used in classification partly because in classification we want the output to be in the range $[0,1]$, so it is natural to pick an activation function that ensures that range; and partly because there are some statistical reasons to choose sigmoid (aka softmax). The former is relevant here; the latter is not. You're allowed to use the sigmoid function for your task for a different reason than why people use sigmoid for classification.

  • It's true that some sources have used tanh because of its derivatives, but that doesn't mean it's the only allowed reason to use tanh. You're allowed to use tanh because of the way it restricts the output range rather than because of properties of its derivatives.

• Contrary to what you wrote, sigmoid can be used at the last layer of regression tasks.

Ultimately, I think you are prematurely rejecting plausible approaches without a good reason for doing so.

It's also possible to experiment with approaches where you penalize outputs that are outside the range, instead of changing the last-layer activation function. Indeed, as long as all outputs in the training set are in-range, a standard loss function (like squared loss) will already do that. However, it's hard to say how well that will work, or whether it will work better than a clever choice of activation function for the last layer. The only way to find out is to try it and see.

Answer 2

Finally, I find time to answer this question whose answer was found in a well-known online course provided by Pr. Boyd for convex optimisation. In that course, he refers to applications of optimisation. One of its applications is penalty function approximation. As a brief answer, just define your penalty for the parameters you want and add it to the cost function, it will be like multi-objective optimisation which should be optimised using scalarization. For simplicity, consider what we have in l1/l2 regularisation methods for avoiding overfitting. Simply, add this to the current cost function. But, consider your cost function to be smooth. For instance, I've added the example he has provided.

As you can see, the log-barrier limits the range of errors for a particular variable, and values more/less than one will be penalised significantly while inputs in the range $(-1, 1)$ won't.

As you can see, there are other methods too. In today's usual neural networks, people employe l2 norm for tasks like classification to penalise mislabeled items..