How to select features from text data?

Question

I have a data set of questions belonging to ten different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation).

The categories are briefly described as follows:

1. Definition – A question that contains a definition of the answer.
2. Category Relation – The answer has a semantic relation to the question where the relation is specified in the category.
3. FITB – These are generic fill in the blank questions – some of them ask for the completion of a phrase.
4. Abbreviation – The answer is an expansion of an abbreviation in the question.
5. Puzzle – These require derivation or synthesis for the answer.
6. Etymology – The answer is an English word derived from a foreign word.
7. Verb – The answer is a verb.
8. Number – The answer is a numeral.
9. Date – The question asks for a date or a year.
10. Factoid – A question is a factoid if its answer can be found on Wikipedia.

I used the Stanford core NLP package called shiftreducer to find out the Part-Of-Speech (POS) values for each question in a category. I thought of using this POS pattern as a discriminant among the classes but it turned out to be generalized since:

1. All the classes follow a similar pattern
2. Nouns top the POS count followed by Determinants, Prepositions, Adjectives, Plural nouns and finally verbs.

What could be the other ways in which I could differentiate among the question categories? Or as my question was in its first place, "What kind of features do I select for efficient categorization?"

Answer 1

If I understand you correctly, you're looking to take the text of these questions and train classifiers to identify which of 10 categories they belong to. And you'd like to come up with a decent feature representation in order to do this.

I think your finding about part-of-speech is intuitive. It makes sense that in grammatical English (assuming your question data is written in English), most questions would follow similar part-of-speech sequences since grammatically correct questions follow a particular syntactic form (at least when posed interrogatively as in the case of "When was George Washington born?")

So, you've ruled something out - which you should actually view as progress. If you haven't tried it already, one simple thing you might do is use the actual words within the questions as features. You could use any order n-gram you like, but unigrams stick out as an immediate linguistic feature to try. It seems likely to me that while the POS-tags are similar across classes, making them difficult to distinguish between, the actual words being used in the questions may vary from class to class, giving your model a better shot of differentiating between classes.

That is, maybe words like "time", "year", and "when" co-occur more highly with the Date class while words like "numerical" and "quantity" co-occur more with the Number class (obviously, this is speculation - I haven't seen your data). You might also look at bigrams, trigrams, or any other number n-gram for this feature set as well.

Finally, there may be other features you could generate using NLP methods that may be useful. I'm not familiar with the Shiftreducer software, but Named-Entity Recognition could be helpful in generating features for the Factoid class if there are many questions about proper nouns. Other really simple features such as length of the question (counted in number of tokens). A final thought would be to use only the tagged verbs from your POS-tagger, tab these up and to see whether they differ between classes. This may be a useful feature for identifying questions present in your Verb class. Hopefully, those are some ideas to get you started.