2
$\begingroup$

I'm working on a project to predict bots from legit users from social medias. The data that I collected has about 5% of bots for 95% of legit users. The problem is as I labelled my data, I was more looking to label bots rather than legit users as it's easier to find bots (they mostly have the same messages, bio, photos, bio URL domain, etc). Labelling real people his very hard though, and I didn't find a good way to label them with certainty except manually, one by one.

Totally, there are 140k rows of data. I labelled about 35k, 20% are bots, not the same as 5%. Is that a big issue?

I used Randomforest to make a model that got me .87+ for accuracy, precision, recall, auc and MCC. Is it okay to not have the same distribution? What should I do?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

The difference between 5% and 20% is huge.

Normally, if you take 35k random rows, you should have around 5%, not 20%.

I recommend checking first if the data is really random.

If it is the case, you can extract a small sample (ex: 100 or 200) from the 35k rows and confirm manually that there is around 20% and hence there is no mistake from the model.

Once confirmed, you can confirm that the real percentage of bots is around 20% and not 5%.

It is not impossible: There are lots of projects that start with a low percentage of bots and it increases with time.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The data is not random. When I collected it at first, it was not labelled so I had to do it myself. I labelled about 25% of the overall data (35k rows) and now have a distribution of 20/80 as I mainly labelled bots rather than legit users as they were easy to spot. Where when randomly selecting users among the 140k I get 5/95. $\endgroup$
    – Marc
    Commented Nov 25, 2022 at 18:21
  • $\begingroup$ Why didn't you randomize the dataset first? $\endgroup$
    – Nicolas Martin
    Commented Nov 25, 2022 at 18:47
  • $\begingroup$ Little precision, I didnt label the users one by one. Many had the same photos, same comment, same URL domain in their bio so when I labelled one bot, as multiple others were very similar, I actually labelled more than one. As I was mainly focus on finding bots, the proportion shifted. $\endgroup$
    – Marc
    Commented Nov 25, 2022 at 19:01
  • $\begingroup$ That's why you've found 20% instead of 5%, right? $\endgroup$
    – Nicolas Martin
    Commented Nov 26, 2022 at 8:19
  • $\begingroup$ yes! Just because I was more focus on finding bots that legit users, it makes sense that the proportion is not the same. Labelling legit users is very time consuming, what are my options? $\endgroup$
    – Marc
    Commented Nov 26, 2022 at 19:01
1
$\begingroup$

It appears your labels are not consistent with actual labels. The results from traditional machine learning algorithms can not be trusted when using low-quality labels (e.g., Random Forest requires high-quality labels).

You have two choices:

  1. Label all the data with high-quality labels, then use traditional machine learning algorithms.

  2. Use machine learning algorithms robust to low-quality labels. The field is called weak supervision. Bayesian machine learning methods tend to be useful.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It's not that the labels are not consistent with the actual labels but that when labelling the data, I was more trying to find bots that legit users. For example I did a value_count() of the bio of all users. The top 20 were bots bio only because they often use the same, where real human have more unique ones. Same for photos, I used imagededupe to find all bots sharing same nude photos which got me more bot and not more legit users, again, because bots use the same content. The labels are corrects, it's just that the proportion of bots/legit users that I labelled is not the same as my sample. $\endgroup$
    – Marc
    Commented Nov 26, 2022 at 1:31
  • $\begingroup$ And I know that my sample is 5/95 because of one of the labelling technique where I was randomly selecting a user, looked at their information (follow count, post count, link in bio, etc), their photos, their content and then label 1/0. I did that ~6k times. $\endgroup$
    – Marc
    Commented Nov 26, 2022 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.