1
$\begingroup$

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway.

Is it possible to change the activation function of the output layer in an MLPRegressor neural network in scikit-learn?

I would like to use it for function approximation. I.e.

y = f(x)

where x is a vector of no more than 10 variables and y is a single continuous variable.

So I would like to change the output activation to linear or tanh. Right now it looks like sigmoid.

If not, I fail to see how you can use scikit-learn for anything other than classification which would be a shame.

Yes, I realise I could use tensorflow or PyTorch but my application is so basic I think scikit learn would be perfect fit (pardon the pun there).

Is it possible to build a more customized network with MultiLayerPerceptron or perhaps from individual layers (sknn.mlp)?

UPDATE:

In the documentation for MultiLayerPerceptron it does say:

For output layers, you can use the following layer types: Linear or Softmax.

But then further down it says:

When using the multi-layer perceptron, you should initialize a Regressor or a Classifier directly.

And there is no example of how to instantiate a MultiLayerPerceptron object.

$\endgroup$
2
$\begingroup$

I see in the code for the MLPRegressor, that the final activation comes from a general initialisation function in the parent class: BaseMultiLayerPerceptron, and the logic for what you want is shown around Line 271.

# Output for regression
if not is_classifier(self):
    self.out_activation_ = 'identity'
# Output for multi class
...

Then during a foward pass this self.out_activation_ is called (defined here):

# For the last layer
output_activation = ACTIVATIONS[self.out_activation_]
activations[i + 1] = output_activation(activations[i + 1])

That ominous looking variable ACTIVATIONS is simply a dictionary ,with the keywords being the descriptions you can choose as a parameter in your MLP, each mapping an actual function. Here is the dictionary:

ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,
               'relu': relu, 'softmax': softmax}

With all of this information, you might be able to come up with a few ways of putting in your custom function. Off the top of my head, I can't see a quick way to simply provide a function. You could for example:

  1. define your function where all the other activation functions are defined
  2. add it to that ACTIVATIONS dictionary
  3. make self.out_activation_ equal to your custom function (or even a new parameter in MLPRegressor
  4. cross your fingers it doesn't break something somewhere else
  5. run it and solve the inevitable small adaptations that will be necessary in a few places

I'm, afraid I have never looked at the source code of that library before, so cannot give more nuanced advice. Perhaps there is a beautifully elegant way to do it that we have both overlooked.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks very much for this. If I understand this correctly the default activation for an MLPRegressor is set to 'identity'. But MLPRegressor object has no attribute out_activation_ so I guess it isn't exposed as an attribute. Identity (linear activation) may be fine for what I need. Is there an easy way to confirm what activation it is? $\endgroup$ – Bill Apr 27 '18 at 15:59
  • $\begingroup$ I don't know if it's relevant but the last change to this module in Aug 2017 was to line 271 to use this is_classifier method from ..base import. Is out_activation_ not being created for some reason? $\endgroup$ – Bill Apr 27 '18 at 16:21
  • $\begingroup$ I raised an issue in scikit-learn. $\endgroup$ – Bill Apr 27 '18 at 16:38
  • $\begingroup$ It is the norm to have a linear activation for the last layer. This basically means no activation, as a linear-transform doesn't really do anything but scale your output. Keras has a linear activation as the end; it just multiplies by 1 - described here. Here is a related question and here is a related blog post $\endgroup$ – n1k31t4 Apr 27 '18 at 16:38
  • 1
    $\begingroup$ It seems they don't have it straight after instantiation. Perhaps it is buried somewhere or is only created after you have done further setup with the MLPRegressor object. Afraid I don't know anymore! $\endgroup$ – n1k31t4 Apr 27 '18 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.