2
$\begingroup$

I have a general methodological question. I have two columns of data, with one a column a numeric variable for age and another column a short character variable for text responses to a question.

My goal is to group the age variable (that is, create cut points for the age variable), based on the text responses. I'm unfamiliar with any general approaches for doing this sort of analysis. What general approaches would you recommend? Ideally I'd like to categorize the age variable based on linguistic similarity of the text responses.

$\endgroup$
  • $\begingroup$ How do text responses look like? And what are you trying to achieve with the groping? $\endgroup$ – ffriend Sep 13 '14 at 21:19
  • $\begingroup$ The text responses are sentences or phrases of 50-200 characters. The goal is just to group the numeric variable (that is, create cut points) based on the text responses. $\endgroup$ – statsRus Sep 13 '14 at 23:02
  • 1
    $\begingroup$ Then you can group them by first letter of texts. It will satisfy your current description, but most probably you want something different. So what you actually want to achieve? Do you want to split people in age categories based on words they use? Or you are looking for specific signs in responses? Even assuming you want clustering, there are literally dozens of features that may extracted from text and same number of distance measures to use. But in current wording I can't even be sure we are talking about clustering. So please provide example and/or context of what you want to achieve. $\endgroup$ – ffriend Sep 14 '14 at 0:18
  • $\begingroup$ What do you mean by "linguistic similarity"? Grammar/syntax? reading comprehension level? semantics (meaning)? $\endgroup$ – MrMeritology Sep 15 '14 at 22:25
  • $\begingroup$ @ffriend: It's a general methodological question, so I'll pick one at random: splitting the age categories based on the words respondents use. $\endgroup$ – statsRus Sep 17 '14 at 0:58
2
$\begingroup$

Since it is general methodological question, let's assume we have only one text-based variable - total number of words in a sentence. First of all, it's worth to visualize your data. I will pretend I have following data:

number of words vs. age

Here we see slight dependency between age and number of words in responses. We may assume that young people (approx. between 12 and 25) tend to use 1-4 words, while people of age 25-35 try to give longer answers. But how do we split these points? I would do it something like this:

enter image description here

In 2D plot it looks pretty straightforward, and this is how it works most of the time in practise. However, you asked for splitting data by a single variable - age. That is, something like this:

enter image description here

Is it a good split? I don't know. In fact, it depends on your actual needs and interpretation of the "cut points". That's why I asked about concrete task. Anyway, this interpretation is up to you.

In practise, you will have much more text-based variables. E.g. you can use every word as a feature (don't forget to stem or lemmatize it first) with values from zero to a number of occurrences in the response. Visualizing high-dimensional data is not an easy task, so you need a way to discover groups of data without plotting them. Clustering is a general approach for this. Though clustering algorithms may work with data of arbitrary dimensionality, we still have only 2D to plot it, so let's come back to our example.

With algorithm like k-means you can obtain 2 groups like this:

enter image description here

Two dots - red and blue - show cluster centres, calculated by k-means. You can use coordinates of these points to split your data by any subset of axes, even if you have 10k dimensions. But again, the most important question here is: what linguistic features will provide reasonable grouping of ages.

$\endgroup$
1
$\begingroup$

If I understand you correctly, I would try a few featurization methods to transform the text column to a numeric value. Then you can proceed with analysis as usual. There is a great book on NLP called Taming Text that would give numerous ways to think about your text variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.