I have a DataFrame of users, some of them are "bots" and they are identified with a bit equal to 1 in the "is_bot" column, if the bit is 0, the user is considered as "human".

The problem is that some users may be misclassified as "humans" instead of "bots" since the "bots" have been identified on the basis on an incomplete list in the gathering data phase.

I will train and test my model on this partially correct data, but when I test it, I will find that my model correctly predicts some users as "bots" even if in the original dataset they are "humans".

Correctly predicts means that, in reality, the users are bots because I checked some of them manually, but I can't do this for my entire dataset of 1 mln users.

This would result in a model with low accuracy, even if the predictions may be right.

How do I handle this problem?


You are dealing with noisy labels. I would not switch the labelling according to a trained model that learned on that particular data set, since probably you don't know which patterns lead to your models decision.

Otherwise if you know the reason for the wrong labelling, you could try to build methods yourself that run a sanity check on your data.

Nonetheless, models that don't overfit often can deal with a bit of noise in the labels. But you can also try methods that incorporate the noise aspect. Maybe check out https://stats.stackexchange.com/questions/218656/classification-with-noisy-labels

  • $\begingroup$ What if I try to handle the problem by another perspective, that is, by just using t-sne or clustering algorithms, since in the dataset I have information about the number of post and comments, votes, etc.? In this way I may discover some common behaviour among the bots and the "humans". Is this a good idea? Thanks for helping me! $\endgroup$ – Carmine Jun 1 '20 at 8:39
  • $\begingroup$ Sure, often many ML methods are combined to an ensemble. Ensemble methods might be a good approach for you since they naturally regularize, as in they decrease the variance of your model. However, I did not quite understand what you want to do. Do you want to clean your data? Or just train a successful model? Or first clean and then train? $\endgroup$ – bonfab Jun 1 '20 at 9:09
  • $\begingroup$ I would like to train a model that is able to identify users that were misinterpreted as humans but are in reality bots. My model already does so but has a low accuracy since this incorrect base classification of my training data. I think to shift to other kinds of machine learning tools, such as the t-SNE or clustering algorithms, because of the low accuracy and the uncorrected data. In this way, I may discover clusters of users that may be very close to the known bots. $\endgroup$ – Carmine Jun 1 '20 at 10:11
  • $\begingroup$ Of course you can use other methods to get to know your data better. If your data is high dimensional you should be careful with clustering. Maybe first reduce the dimensionality with autoencoder. The accuracy of your model is always relative to your lossfunction and your data. But if your data is bad, that doesn't mean your model does not do what you intend it to do, but just that the higher in error of your model comes from the noise in your labels. No model will be able to learn the noise and thus increase the accuracy. Maybe use some explainability to understand how your model is working. $\endgroup$ – bonfab Jun 1 '20 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.