0
$\begingroup$

I have two textual datasets collected from different domains (Twitter and Reddit).

I extracted a set of features in the same way from these two datasets, one of these features as an example called X_positive.

How could I know if the feature X_positive is more representative (exists more) in one of the datasets?

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you please share how did you extract these features? $\endgroup$
    – yoav_aaa
    Commented Dec 23, 2018 at 13:09
  • $\begingroup$ @DaFanat .. frequency of specific cue words $\endgroup$
    – Minions
    Commented Dec 23, 2018 at 13:17
  • $\begingroup$ So your feature is a count on how many times specific words appear in each of the data sets? $\endgroup$
    – yoav_aaa
    Commented Dec 23, 2018 at 16:51
  • $\begingroup$ @DaFanat ,, yes .. how many times specific words appear in the tweets as an example .. each of the datasets consist of N records of tweets/reddits posts $\endgroup$
    – Minions
    Commented Dec 23, 2018 at 16:54

1 Answer 1

Reset to default
0
$\begingroup$

My two cents:
I can think of several ways calculating specific terms repressiveness:

1) Calculating rate of documents(tweets/posts) where these terms appear more than 1(or X > 1) times. You can decide on X by looking at the distribution of frequency per document.
2) Calculating terms TF-IDF score per each document and aggregating across all documents. Aggregating across all document depends on your need and what you mean by representative.
3) Calculating terms appearance rate - how many times these terms appear divided by total number of words in the corpus(possibly after subtracting stop words).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.