12
$\begingroup$

I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary is diverse and data become too sparse. I also tried to classify the types of attributes separately and combine the results through meta-learning techniques, but it did not work well.

$\endgroup$
11
$\begingroup$

Christopher's answers seem very reasonable. In particular tree based methods do well with this sort of data because they branch on discriminative features. It's a little hard to say without knowing your specific application, but in general if you think that some of your features might be significantly more discriminative than others, you could try some dimensionality reduction techniques to clean this up a bit.

Also if you use a dimensionality reduction technique you end up getting a slightly more robust format for your feature vector (they generally end up being straight numerical vectors instead of mixed data types), which might let you leverage different methods. You could also look into hand engineering features. With properly hand engineered features Random Forest will get you very close to state of the art on most tasks.

$\endgroup$
  • 2
    $\begingroup$ Another vote for dimensionality reduction. Just some additions: Principal Component Analysis or Non-Negative Matrix Factorization will reduce number of variables, enrich sparse data, and transform all variables to quantitative. Moreover, evaluating quality of dimensionality reduction model, question author can estimate usefulness of textual variables. $\endgroup$ – sobach Jun 20 '14 at 16:13
  • $\begingroup$ @indico great answer. Could you give us a reference in a book or a paper, that tree-based algorithms perform better with mixed-type (categorical and quantitative data) data? $\endgroup$ – ZelelB Sep 21 '18 at 15:16
6
$\begingroup$

It is hard to answer this question without knowing more about the data. That said, I would offer the following advice:

Most machine learning techniques can handle mixed-type data. Tree based methods (such as AdaBoost and Random Forests) do well with this type of data. The more important issue is actually the dimensionality, about which you are correct to be concerned.

I would suggest that you do something to reduce that dimensionality. For example, look for the words or phrases that separate the data the best and discard the other words (note: tree based methods do this automatically).

$\endgroup$
2
$\begingroup$

With the little information you have provided regarding the nature of your data, I would advise you to follow the following approach:

  1. Convert text data into categories. You can try different alternatives for how much information the categories should contain, but specific categories have to exist for each variable. As an example, I will assume a variable that came from a text field of a survey questionnaire regarding preferable way of people to get to work.

    At first, we need to make sure that answers with similar meaning are written on the same way and belong to the same category (e.g. "by bike", "cycling", "by bicycle" all have the same meaning). Then you can try further merging into less detailed categories (e.g. merge "tram", "metro" and "bus" into "Means of public transport") or even more (e.g. "Walking", "Jogging", "Cycling" into "Physical activity") depending on what you are trying to find out.

    You can even put some different combinations in your dataset and then the next steps will determine which ones will be used for the analysis. In cases where the text data can be "translated" in ordered variables make sure you do this (e.g. if you have "small, medium, high" transform it to "1,2,3").

  2. Turn your categorical variables (not the ordinal ones) into dummy (binary) variables. Most of classification/feature selection algorithms do this automatically, but make sure this is the case with the ones you select. I realise that the dimensionality of the data will become quite big at this point, but this will be handled in the next step.

  3. Apply a feature selection/dimensionality reduction technique on your data. You can find a useful review of such techniques here. If you are using Python, sklearn tools give you a lot of options (see more details here). Make sure you use a technique that also considers multicollinearity. I would try Principal Component Analysis or a tree-based algorithm.

  4. For classifying the data, I would go with Decision Tree Classifier (also available via sklearn). It also performs feature selection setting importance weights to the features. You can set the level of detail on the generated tree depending on your options (e.g. max_depth, min_samples_split) Make sure to adjust the level of detail based on cross-validation to avoid overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.