I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed that even though the text (around 1-5 %) mentioned some covid-related keywords, the context of those texts are not actually about the pandemic, they are telling a life story (from 5-year-old to 27-year-old) instead of how covid affects their lives.
The data I want to use and am looking for is some texts that tell people how covid makes depression worse and what not.
Is there a general way to clean those irrelevant data whose contexts are not covid-related (or outliers)?
Or is it ok to keep them in the dataset since they only count for 1-5% ?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ do you have labelled data? how do your labels look like? $\endgroup$– David MasipFeb 1, 2021 at 7:51
-
$\begingroup$ @DavidMasip, no, not at all. All data is unlabelled unfortunately. $\endgroup$– zxcisnoiasFeb 1, 2021 at 14:45
-
$\begingroup$ Welcome to DataScienceSE. These social media users are so selfish, they mention covid terms without giving you exactly what you need for your study ;) Seriously, it's not surprising that text extracted from social media is very messy. The main problem you have is to define "irrelevant": I'd say there are two approaches: either you annotate a sample of documents and train a model, or you measure the similarity of the documents to a reference text which represents what is relevant. $\endgroup$– ErwanFeb 1, 2021 at 19:32
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You can use BERT to create vectors that will capture the context of the whole tweet. Once, you do that, try clustering (K-Means or GMM). You can then look at the clusters found and separate out this unwanted data.
