2
$\begingroup$

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS is making an error of 10 meters and you want to correct it. So I brought another super GPS which is very accurate and measured 40k data while driving so in my dataset I have some vehicle informations which are speed, Acceleration, yaw rate, timestamp and wheel angle and I have position Informations which are the ground truth longitudes latitudes and the false longitudes and latitudes from my normal GPS. I'm transofrming those latitudes and longitudes to an x and y just to know how much should I shift my false longitudes and latitudes so that my position can be more accurate and similar to the Ground truth values. Can my data be bad in this case? I'm trying to predict the error in longs and lats that the GPS make so that I can later correct it so it's a regression problem and I'm using those features above to do that which I think they are informative since speed, acceleration, yaw rate and wheel angle are related somehow to a position(am I wrong?)

I'm asking this generally, I read some articles in the internet, that say that data is sometimes bad or the quality of the data is bad but I don't know what the mysterious sentence really mean.

I also had the problem when training neural networks that my loss start to decrease for the first 10-20 epochs and then it stuck on some high value and the network stops learning like if it were struggling to go out of that loss value but it can't. I tried to use only 100 data points instead of all the 40k and I noticed that it worked good, the NN achieved to fit those but as I increase the number of data points the performance become worse(do you have any ideas on this?)

some people suggest that I don't have many data and many features and in this case it would be better to use some machine learning approach since it outperforms NNs in case of small datasets or if I have few features like in my case so I also tried using random forest and I noticed that it gives better results than neural networks but it also doesn't generalize well, even if it gave me good results on train and validation sets, when I try it on test data(data that the random forest have never seen), it perform really bad.

so I was reading in the internet what can cause those problems and I noticed that I sometimes saw people or articles that claim that maybe the quality of the data is bad! but what does this really mean? I thought neural networks can map any kind of data, if I have one feature and one target then neural networks can map those two together, at least it can overfit the data right?

so can someone please tell me what is bad data or better how do I know if my data are bad? well if there is a way to know that then I would probably save time and not start working on a project that will take me a month to complete and then figure out my data is bad. Also can you tell me whether my case make sense ? I mean I find it weird that NNs gives very bad performance way worse than random forest. at least my NN should overfit the data or am I wrong?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

NN is not a magic bullet

Not every model works well on every dataset, is the ethos.

Even tough NN is very powefull for certain tasks you could find tasks where linear regression would be a better choice.

Quality of data could mean 2 things:

a) data is unstructured and not processed. Think in terms of some super ugly text logs

b) data is uninformative. There are no patterns that can be extracted no matter what you try.

To conclude, experiment with different approaches and always have a holdout dataset to do sanity checks

Improve this answer
$\endgroup$
9
  • $\begingroup$ thanks, since my case is not text data, I just updated the first paragraph to give you more informations about my project. since it is project specific I hope you read that and tell me what you think about it and how or rather when can my data be bad? how can my data be unstructed or not processed? can you please explain more? $\endgroup$
    – basilisk
    Dec 18, 2019 at 13:09
  • $\begingroup$ Ok, given what you wrote I would forgethttps://en.wikipedia.org/wiki/Unstructured_data and leave the possibility that your data is not informative enough to make predictions. You can not know that in advance. You have to test,test and again test (experiment) to see if you gain desired values on the holdout set. As I said dont focus on the model, focus on understanding the data, feature engineering, (pre-processing in general) there is enough info out there on this topic. When you are done with this try different models, NN, Treebased, NaiveBayes etc... As a last step you can ensamble model $\endgroup$
    – Noah Weber
    Dec 18, 2019 at 13:32
  • $\begingroup$ thanks again for your answer. Ok I understand I already made some researches about that. I cleaned the data, removed outliers, standardize all the features, analyzed correlation etc.. I already done that I don't know where to go further. what confused me more is that random forest did well on the train and validation but did bad on test data. Neural networks didn't even achieved to overfit the data how can this be possible? If I tried with only 100 data rows, yes it overfits it but when I take the whole 40k then it fails even to overfit while random forest did a good job at it $\endgroup$
    – basilisk
    Dec 18, 2019 at 13:42
  • $\begingroup$ I also noticed something interesting but it is bad for my case. Random Forest achieved good results on validation set because I shuffled the data but when I want to make a predictions in production, the data would be sequential so no shuffle is needed and that's why it give poor results on test data. this also confuses me since Random Forest is a good model but I can't use it in production obviously $\endgroup$
    – basilisk
    Dec 18, 2019 at 13:44
  • $\begingroup$ Why cant you put RF in production :) ? Hot tip: simple is better and putting neural networks is extremely expensive. If you are having different results on the validation one it could be that you are experiencing covariate shift ( totaly different distribution of validation set) while you also over-fitted on the train. (PS remember to accept answers that answer your question, not mine necessarily ) $\endgroup$
    – Noah Weber
    Dec 18, 2019 at 13:47
2
$\begingroup$

Adding to the previous answer, you should know that using a reasonable number of features should give a score that is somewhat close to what it can give with optimal settings.

If your data is uninformative and there are no patterns to capture in them, it should render a score way off what's asked from you by your supervisor no matter what algorithm or model you would use.

I have had a similar issue and I concluded it was a lack of data quality ( as in either the content of the data is bad and it contains a lot of randomness and also lack of predictors that could explain the target better ).

EDIT : Setup a proper validation scheme to eliminate the threat of overfitting, it's probably the reason you're getting high scores on validation and low ones on test. If even with that , your test score is still far off your expectations, consider asking yourself if you're using the right training data for that test set.

Hope this helps!

Improve this answer
$\endgroup$
10
  • $\begingroup$ thanks for your answer. I find the answer a little bit ambiguous, can you try to clarify to me. how do I know whether my data is uninformative or not? and by score do you mean r2 score or which score? and how can I conclude that my data lack quality as you concluded? $\endgroup$
    – basilisk
    Dec 18, 2019 at 15:33
  • $\begingroup$ Use every available feature in the simplest way, make sure you didn't do any mistakes while creating these simple features. Create a simple model and make sure you don't create leaks. Then if it's a regression, sure use R² combined with RMSE ( R² is computed using MSE so they are related ). Model choice given the size of your data set won't be of much effect. The score you'll get if you do not make mistakes in modelling will be close to what you'll get after optimizing your solution. $\endgroup$
    – Blenz
    Dec 18, 2019 at 15:36
  • $\begingroup$ RF gives me 0.94 r2 score on the train data and 0.91 on validation data but on test data it gives -2. now the trick is for the train and validation data I'm using a random split but for the test data I'm not using any random split I have sequential data for testing so let's say for example that I ll drive 8 rounds with the vehicle to taka a dataset. I use the first two rounds and keep it away for later testing and the rest of the 6 rounds I shuffle and split it randomly and use it for train and validation. Notice that I tried not to shuffle them and the RF gives bad score on validation $\endgroup$
    – basilisk
    Dec 18, 2019 at 15:49
  • $\begingroup$ when I shuffle the data randomly, RF gives a good score on train and validation both. but if I didn't shuffle and made the same as I did to the test data then RF fail to work $\endgroup$
    – basilisk
    Dec 18, 2019 at 15:50
  • $\begingroup$ When you're doing validation try to replicate the test set behavior. If your test data is sequential,make your validation sequential. Very good scores on validation and bad ones on test mean 1 of 2 things or both : your model is overfitting the validation set, or the test data is different from the train data in terms of target distribution. How was your test created compared to your train data? A more probable cause for this in your case is overfitting. Try to setup a proper validation scheme to eliminate the overfitting option, if it's still bad, then the option of difference is plausible. $\endgroup$
    – Blenz
    Dec 18, 2019 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.