0
$\begingroup$

I am trying to train a machine learning model to help me classify some real data. Since the acquisition and labeling of real data can be very expensive, the training data is generated with simulation. However, the trained model doesn't perform very well on real data, my suspicion is that the simulation is not a 100% accurate representation of real data. Therefore, I am wondering will the performance be improved if I train the model with a mixture of simulation and real data (say 20% real data). I would greatly appreciate it if you could either answer the question or point me to the right reference!

$\endgroup$
3
  • 1
    $\begingroup$ Welcome to DataScienceSE. It depends how representative the artificial data is: if it follow exactly the same distribution as the real data, it will probably improve performance. If not, it could cause a bias which decreases performance. $\endgroup$
    – Erwan
    Commented Oct 10, 2022 at 22:29
  • $\begingroup$ @Erwan thanks for your comment. I tested the trained model on other unseen simulation data, it works very well. But the performance is terrible on the real data. Do you think my suspicion that the simulation is not a 100% accurate representation of real data reasonable? Or do you think there could be any other factors affecting the performance? $\endgroup$
    – usr01
    Commented Oct 10, 2022 at 23:10
  • 1
    $\begingroup$ Yes, I would say that your suspicion is even confirmed by these results: if the simulated data was accurately representative, there would be no drop in performance. Except in very simple cases, it's rarely possible to properly generate artificial data. By construction artificial data is easier to represent for the model, so there is probably a strong bias when evaluating on it. Normally evaluation should always be done on real data, simply because the ultimate goal is to apply the model to real data. $\endgroup$
    – Erwan
    Commented Oct 11, 2022 at 9:30

1 Answer 1

4
$\begingroup$

I suggest that you add an extra input binary variable indicating whether the data is simulated or not. For the simulated data, you would set it to 1, while for real data you would set it to 0. This may help the model profit from simulated data while still being able to do well in real data.

This advice is inspired by something we use in machine translation called "tagged back-translation". When we are training a translation system from language A to language B, if we have a small training dataset A → B but we have a lot of monolingual data in language B, we first train a translation system from B to A and then use it with the monolingual B data to obtain a synthetic dataset used to train our final A → B system. The final system, however, performs better if we indicate as part of the input if the data is real or synthetic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.