# Train neural network for regression with negative samples

I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?

• Do you know roughly how wrong it is? Without that I’m not sure how much you can do. Oct 15 '18 at 23:55
• @kbrose: Yes, I think I should be able to build such a metric. I am looking forward to your suggestion. Oct 18 '18 at 9:08

An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2$$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2$$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right)$$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.

• -1, whenever the $type_i$ is -1 the network will learn to send the prediction to +inf or -inf. Oct 18 '18 at 14:25
• Can you improve the answer, instead of the -1? It should be possible to bound the -Inf and the +Inf, shouldn't it? Besides: You hinted that you have an idea if the "how wrong it is" is given. What is your idea? Oct 18 '18 at 14:34