I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?

  • $\begingroup$ Do you know roughly how wrong it is? Without that I’m not sure how much you can do. $\endgroup$
    – kbrose
    Oct 15, 2018 at 23:55
  • $\begingroup$ @kbrose: Yes, I think I should be able to build such a metric. I am looking forward to your suggestion. $\endgroup$
    – Make42
    Oct 18, 2018 at 9:08

1 Answer 1


An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$ J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2 $$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$ J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2 $$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$ J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right) $$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.

  • $\begingroup$ -1, whenever the $type_i$ is -1 the network will learn to send the prediction to +inf or -inf. $\endgroup$
    – kbrose
    Oct 18, 2018 at 14:25
  • $\begingroup$ Can you improve the answer, instead of the -1? It should be possible to bound the -Inf and the +Inf, shouldn't it? Besides: You hinted that you have an idea if the "how wrong it is" is given. What is your idea? $\endgroup$
    – Make42
    Oct 18, 2018 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.