Let's make an example: I want to build a neural network which should predict if a person is obese. It gets the following features:

  • Weight
  • Height
  • Number of hours the person makes sports
  • superspecial health index
  • foobar

The output of the model is a probability that the given person is obese. The higher, the more likely.

Now I know, if everything else stays the same, that a higher weight should always be the same or a higher output probability. But I don't know the exact relationship, only that it changes monothonically.

How can I include this knowledge into the network and force it to show the desired behaviour?


If you plug this kind of data into a standard network, e.g. an MLP, you will usually hope that the model actually extracts this information itself. You could introduce a dummy variable that encodes this information, but you run the risk that the model learns to just follow the dummy variable and doesn't learn its own powerful abstractions and feastures from the data.

[EDITED: ] An example could be to create a weight based variable that is normalised to other physical characteristics, such as DV = weight / (height + waist circumfrence). This should then, according to your assumptions, scale nicely with the output obesity.

Another way that people might include prior information into a model, is to use a Probabilistic modelling, which incorporates ideas from Bayesian statistics. You can do things such as define a prior distribution for your outputs, conditioned on your inputs - essentially allowing you to provide information to the model (such as weight being correlated with obesity, ceteris paribus) - this then nudges the model to go along these lines.

If you would like to get into it, there are already some great libraries to make it really easy:

  1. Stan - which has interfaces to many languages: Python, R, State etc.
  2. Edward - probabilistic programming with modern GPU accelration and deep learning integration (Tensorflow and Keras).

This seems to be a nice overview of some of the methods and tools, but I haven't read through it all.

Another way that I could suggest is to play with the architecture of your model and the idea of auxilliary models. Have a look at my recent answer on a question talking about the Inception model from Szegedy et al..

The idea is that you have branches coming off the model at train time only, which also make predictions and produce error to be backpropagated through the preceding weights.

You could make a side model that predicts the obesity, based on e.g. the input weights and perhaps some related features extracted from the first layer of your neural network. This would make the idea or importance of this relationship more prominent during training and the weights would subsequently be tuned to take that into account.

At test time, you simply ignore these auxilliary branches.

| improve this answer | |
  • $\begingroup$ I don't understand how you would solve this with a dummy variable. Can you elaborate on this? $\endgroup$ – Martin Thoma May 4 '18 at 14:25
  • $\begingroup$ Please also elaborate on the Probabilistic modeling. How can I do this if I only know one vague information about a single feature and the output (monotonicity)? I always assumed I need the distribution for all features to be of any value. Also, please correct me if I'm wrong, but this does not enforce the desired monotonicity, right? $\endgroup$ – Martin Thoma May 4 '18 at 14:31
  • $\begingroup$ The part with the auxillary models could work ... I need to think about that :-) $\endgroup$ – Martin Thoma May 4 '18 at 14:33
  • $\begingroup$ I have edited the answer with a dummy variable example - just something to try out. I think my first comment is probably, unfortunately, the most accepted answer: the NN should learn that weight is important to predict obesity. As neural networks are not parameterised as with e.g. linear regression, you cannot easily manipulate/interpret the coefficients. As for the probabilistic programming, I agree that the distribution for one variable alone is difficult to enforce on a wide dataset. As far as I know, you may have to build meta models and combine them. Sorry - not an expert in that domain. $\endgroup$ – n1k31t4 May 4 '18 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.