1
$\begingroup$

I am trying to make a neural network to predict some values but I know my training data contains lots of "bad" expected outputs. That is I know some of the data would train it in the wrong direction.

Would it make sense to train the network once, then remove data that the network produces a big error for (supposing that that's the "bad" data) and then retrain it with the rest? Is there some other better way to approach this?

I'm sorry I'm only starting out dealing with NNs and I don't know much of the related jargon to make the question clearer.

$\endgroup$
1
$\begingroup$

If you want to identify and remove problematic data, then this is much better done before training.

You might have some luck with your approach, but you have no guarantees that the neural network will help you isolate the problem entries based on error values, if it has been trained on the "good" and "bad" entries at the same time. It depends on the nature of the problem data. You take a significant risk that the model will fit to the problem data well enough to cause you to reject good data.

You should instead try to think of a way to identify the bad data more directly, and prior to training for the predictive task.

Here's one idea: Are you able to identify enough good and bad data manually - enough to train and test a classifier? Then label some of the data "good" or "bad", train a classifier and test it - the test should be on some held out labeled values, to help you assess accuracy. If your classifier has a good accuracy, you can have some confidence to use it to filter the remainder of your data, and only use the "good" classified data for your original training goal.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Interesting approach, however I don't think I can apply it to my problem since for one the NN would be retrained on intervals and I don't think the new bad data would be similar to the old. So this manual work would have to be done on every retraining making it infeasible. Also I'm not really positive a human would even be able to be sure of what data is problematic. So I'm probably looking for a way to minimise the risk of dropping good data instead of bad as you mention (if possible). $\endgroup$ – eirc Feb 26 '17 at 19:38
  • 1
    $\begingroup$ @eirc: Is it possible to clarify how you know that there is a large amount of problem data, and give some idea of what numerically "lots of bad expected outputs" is e.g. 100 out of 1000 measurements? The nature of how it is going wrong could be useful to identify a strategy to remove it. $\endgroup$ – Neil Slater Feb 26 '17 at 21:20
  • $\begingroup$ Ok I guess I can't be too secretive if I'm looking for help on this. My goal is to analyse data from an online game's marketplace and then have the NN do automatic pricing. Now lots of people have no idea how to price their goods but I can't think of a way to detect this apart from training the NN with all the market data and then when the NN disagrees with some pricing I make the guess that that's a "bad" data point. Does it make sense to retrain the NN without that data? - Embracing the risk you mention in your original answer of course. $\endgroup$ – eirc Feb 27 '17 at 16:35
  • $\begingroup$ @eirc: I think you have a tough problem. Ideally you need to find a way to filter down to "trusted" data, that your prediction task will try to emulate to establish fair prices (as opposed to weirdly low or high ones). Is there any data about the buyers and sellers available to you, such as ratings given by buyers to good sellers. Can you reject all sellers who have not yet made a sale, or only accept those that have > 3 star ratings? $\endgroup$ – Neil Slater Feb 27 '17 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.