5
$\begingroup$

I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like "stronger leadership", "better customer service", "programs", and etc...

My question is, can I use this as a training data set in order to develop a model that I can use in the future as we get more survey responses? We would like to be able to tag, label, or classify each survey response into (up to) 3 of the 70 categories.

Is this even possible? Or do I have to use a NB with simple words? Can you please guide me to tutorials, examples, etc.?

Using R in this exercise.

|improve this question
$\endgroup$
3
$\begingroup$

Assigning ~3 of 70 categories means you would be performing multi-label classification.

In the end, it doesn't make much difference if you use Naive Bayes or SVM; they are both families of algorithms that translate provided independent variables (your feature space) into hopefully correct dependent variables (target classes).

The question is how to construct a good feature space. The state of the art approaches in text mining are (or were) first tokenizing words, stripping punctuation and stop words, stemming or lemmatizing them, creating a bag-of-words model of those words' relative frequencies and perhaps the frequencies of those words' bigrams or trigrams.

Then run your classification learners on that. Assume the resulting feature space table might get really wide (lots of words and combinations of words), so you might want to consider some form of dimensionality reduction.

Of course, you will have to repeat the same filtering process with exact same parameters for each new survey you want to classify.

Here's another good batch of answers on multi-label text classification.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the reply @K3--rnc. What package in R do you recommend for the classification? Do give you an idea of how much I know about these things, I know and understand all the concepts in your post and I've done some sentiment analysis before. Thank you $\endgroup$ – econstat Jan 25 '16 at 14:06
3
$\begingroup$

Can you clarify what you're trying to predict with these responses?

My initial reaction is that with open-ended surveys you'll have a tough time implementing classification algorithms. Open-ended-ness means you don't have a finite feature space and thus you can't do the usual transformation of responses into a feature matrix.

However, there may be other ways to make this work. If you have 5 questions, for instance, you may be able to use sentiment analysis or other methods to come up with metadata about the open ended response that can help you design a classification scheme.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your reply. It's a two part question. The question is of this nature: Part 1: are you happy with the service? Part 2: if you answered no, what can we do to improve our service? And for Part 2 the answer is open-ended. We are interested in sorting the part 2 answers into the 60 categories. Doing so, we would like to match each answer to the top 3 most likely categories. Does that make sense? $\endgroup$ – econstat Jan 21 '16 at 18:29
  • $\begingroup$ That does make sense. Could you provide examples of the categories you'd like to sort them into? $\endgroup$ – Derek Janni Jan 21 '16 at 18:30
  • $\begingroup$ You mean aside from the 3 I mentioned in the original question? $\endgroup$ – econstat Jan 21 '16 at 18:37
  • 1
    $\begingroup$ Oops, my bad. Ok so I think what may be more instructive for you is to perform some NLP algorithms (LDA & K-means strike me as good ideas) to cluster the responses to this question. Then, you can examine the qualities of these clusters of text to come up with some understanding of what similarities they share. You can also take new data points and assign them to clusters, which is similar to classification. In either case, if you have 60 labels your prospects are very dim in any classification scheme. It is hard to coerce data into classifying well in 2 dimensions, much less 60. $\endgroup$ – Derek Janni Jan 21 '16 at 18:48
  • $\begingroup$ The main problem I'm having with this is how you'd accurately turn open-ended text into a finite space of features without falling back on an identity function. For instance, you could just search the response for "customer service" and if you find it, label it as such, but that's not really machine learning. $\endgroup$ – Derek Janni Jan 21 '16 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.