I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not inside the range while training. I've not seen any paper but I have two solutions for that right now but I don't know if there is any standard way or not.
- My first suggested solution is to use an activation function like the following. It is linear in the specified range and has high slope out of the specified range. Although the gradient will oscillate if I use simple gradient descent, I guess it will perform well if I use second-order optimization algorithms like
Adam. Any suggestions for this?
- My second solution is inspired by
L1/L2regularization. I find the output of the model. If it exceeds the absolute value of the range, suppose it's symmetric, I'll add a constant big value which is far from the real output. Here We can replace it with a big constant or maybe multiply it by a constant. The second one has this property that it will be differentiable with a slope.
Does anyone have suggestions which are applicable or even these are ok or any other thing?
In regression tasks, it is customary to use linear activation function as the non-linearity of the final layer in order to estimate a function which outputs real value(s). The reason sigmoid function is used is that its output is limited to the range
0 to 1 which is a good range for specifying probability. Moreover, sigmoid is used more for classification tasks where the classes are mutually exclusive in the input o.w. softmax activation function is used.
Tanh is used due to having zero mean which accelerates the training process. These are usually not applied in classification tasks these days because their differentiation saturates in their limits.
Relu is a customary choice because its slope does not saturate.
Sigmoid-shape activation functions cannot be used as the last layer of the regression tasks. I want to use a kind of activation function like linear as the last layer's activation function in order to have outputs in a special range. While training I want to train the model in a way that it never tries to output a value which is outside a specified range. For doing so, I've put two suggestions that I have tried. I don't know whether there is a paper about that or not. Moreover, about the provided answer, sigmoid does not penalize the outputs, I mean does not increase them in a way that the error value increases, which are outside the range of the domain of the function. What it does is saturating the output to zero or one. What I want to do is to make a huge difference between the output of the network and the real output of the input in order to cause a high error. Consequently, the network will try to find weights which do not yield results out of the range.