I am trying to perform sentiment classification task where I have some text and some information about whether the user smiled or not. Now when I use count-vectorizer to convert my text to feature set (Bag of Words) approach it results in more than 5000 features. When I add smile to these 5000 features, total number of features become 5001. However, I do not see any improvement in the classification accuracy. I think the 5000 sparse features dominate the one and only smile feature. Hence no effect on the classification. Can anyone recommend, what is the good way to merge textual feature with smile, prosody or other features?
If you are using a RNN and the 5000 features refers to size of your dictionary, I would recommend you consider an embedding of your words, eg word2vec. Each word would then result in a maybe 200-300 - sized vector, to which you can concatenate the single smile feature, and then run that through your RNN.
If that is not doing much, you can run the embedded text through an RNN, and then take the result of that (lets say 10-20 features per time-step) and concatenate the smile feature, and run those through a second RNN. The idea is the first one is a further compression over time, and the second is your 'final' RNN which works at a higher level of abstraction.
In general, this is a classic case of combining a high-dimensional feature set with a low dimensional one, to which there are many approaches, the question being, when do you join the features. Take a look at this paper, which covers a few ideas on multi-modal features aka when and how to merge them (diagram 3.2), though I think it is more instructive than useful since their features are more rich than what you have with the single additional feature.
Another thing to try is to soften the 'smile' feature if it is not so already, aka if it is a 0 or 1, make it a random (0, 0.1) and (0.9, 1) respectively.