0
$\begingroup$

I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed that even though the text (around 1-5 %) mentioned some covid-related keywords, the context of those texts are not actually about the pandemic, they are telling a life story (from 5-year-old to 27-year-old) instead of how covid affects their lives.
The data I want to use and am looking for is some texts that tell people how covid makes depression worse and what not.
Is there a general way to clean those irrelevant data whose contexts are not covid-related (or outliers)?
Or is it ok to keep them in the dataset since they only count for 1-5% ?

$\endgroup$
3
  • $\begingroup$ do you have labelled data? how do your labels look like? $\endgroup$ Feb 1 at 7:51
  • $\begingroup$ @DavidMasip, no, not at all. All data is unlabelled unfortunately. $\endgroup$
    – zxcisnoias
    Feb 1 at 14:45
  • $\begingroup$ Welcome to DataScienceSE. These social media users are so selfish, they mention covid terms without giving you exactly what you need for your study ;) Seriously, it's not surprising that text extracted from social media is very messy. The main problem you have is to define "irrelevant": I'd say there are two approaches: either you annotate a sample of documents and train a model, or you measure the similarity of the documents to a reference text which represents what is relevant. $\endgroup$
    – Erwan
    Feb 1 at 19:32
0
$\begingroup$

You can use BERT to create vectors that will capture the context of the whole tweet. Once, you do that, try clustering (K-Means or GMM). You can then look at the clusters found and separate out this unwanted data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.