Train neural network for regression with negative samples

Question

I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?

Answer 1

An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$ J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2 $$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$ J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2 $$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$ J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right) $$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.