2
$\begingroup$

I am working on a text classification problem on tweets. At the moment I was only considering the content of the tweets as a source of information, and I was using a simple bag of words approach using term frequencies as features, using Random Forests (this is something I cannot change).

Now my idea is to try to incorporate information present in the URLs used in tweets. Now, not all the tweets have URLs, and if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs. For this reason, I suppose that having a single set of features containing both the tweet term frequencies and the URL term frequencies could be bad. Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

Do you have any suggestions regarding this issue?

$\endgroup$
1
$\begingroup$

Are you using raw term frequencies, or TF-IDF?

Perhaps you could simply combine the terms in the tweet with the terms in the URL-linked pages (if any) into a single bag of words, calculate TF-IDF, and normalize to avoid bias towards longer documents (i.e., those tweets containing URL links).

if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs

I don't understand what you mean here. Aren't your features the terms in your bag of words? So the number of features will be the size of your vocabulary, which I imagine won't change much whether you include URLs or not.

Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

I don't understand this either. Term-document matrices are virtually always a sparse matrix, since most of the terms in your vocabulary won't appear in most of your documents. So, the vast majority of values in your TDM will be 0. I don't know where you're getting -1 from.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Actually I would expect the size of the vocabulary to increase greatly when considering URLs, because the tweets vocabulary is quite limited. Regarding the second misunderstanding, one of the ideas was to simply append the features for URLs to the features for the tweet. But, if a tweet does not have URLs, I will have to put some "impossible" values in the URL features (like -1), because the lack of a certain feature (-1) must be different from the absence of a word (0 if considering term frequencies). This would not happen if considering your initial proposed approach to have one bag of words $\endgroup$ – papafe Sep 23 '14 at 14:09
  • $\begingroup$ I am not sure about your initial approach to have one big bag of words though, because, apart from the eventual different vocabulary between tweets and webpages, this would put on the same level the words from the tweet and the words from the URL, when actually they could have different importance in the classification $\endgroup$ – papafe Sep 23 '14 at 14:15
  • $\begingroup$ @markusian re: "the tweets vocabulary is quite limited" -- that's not what I've seen in the real world. What kind of Twitter corpus are you dealing with? re: "simply append the features for URLs to the features for the tweet" What kind of features are you talking about? Something other than bag-of-words? re: "this would put on the same level the words from the tweet and the words from the URL" So just boost the score of the words in the tweets. $\endgroup$ – Charlie Greenbacker Sep 29 '14 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.