1
$\begingroup$

I have this project I'm working on where I scraped users' data from social media to predict if they are bots, fake accounts or legit users based on their comments, likes, posts, public data only.

I'm at this point where I have data about more than 80k users. The data is not labeled, so what are my options here? I can, by manually looking at each account flag them as fake/not fake so I can do it with supervised ML and get an accuracy score but this will take forever as I'll have to accurately label my data. I could also do unsupervised learning by clustering those users but then how can I measure if my model is doing a good job? How do I convey my results? I haven't worked a lot with unsupervised learning.

I'm at this point where I've been scraping the data, and I'm now doing the EDA but I'm not sure about the next steps and would love some help and guidance.

$\endgroup$

1 Answer 1

0
$\begingroup$

this will take forever as I'll have to accurately label my data.

This is why one should plan the full experimental setup before collecting the data, ideally ;)

I could also do unsupervised learning by clustering those users but then how can I measure if my model is doing a good job?

This is the right question, and the answer is: you can't.

To be clear, this task requires some labelled data at least for the purpose of evaluation, and likely also for training. Thus at least a sample needs to be labelled manually, because no automatic method can provide you with a dataset which is guaranteed to be labelled accurately for this task. And anyway you would have to evaluate this automatic method itself with... labelled data.

Before doing any manual labelling, you should define a clear methodology. Typically annotators study a sample of data in order to establish precise annotation criteria, before even starting the actual annotation process. Even with clearly defined criteria, this task is likely to involve subjective decisions, since you're going to find cases where the answer is not clear. Ideally one would use multiple different annotators for such tasks, in order to measure discrepancies between them.

$\endgroup$
3
  • $\begingroup$ I believe I thought about doing something similar but I want to be sure I understand you right. Could I start by labeling a sample, cluster a bigger sample, look in which cluster my labeled data leaves, and then, based on the results, label non-labbelled data more easily until I come to a point where I have enough labeled data to use supervised learning? $\endgroup$
    – Marc
    Jan 23, 2022 at 16:59
  • $\begingroup$ @Marc ideally the labeled data should be random sample. Your method is a kind of bootstrapping, it's sometimes used for efficiency reasons so it's not wrong, but there's a risk to introduce a bias in the distribution of the labeled data. Also it's possible that the positive instances are scattered across all the clusters, in this case the method won't help. $\endgroup$
    – Erwan
    Jan 23, 2022 at 17:35
  • $\begingroup$ That makes a lot of sense. Thank you for your help! $\endgroup$
    – Marc
    Jan 23, 2022 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.