Best way to classify datasets with mixed types of attributes

Question

I would like to know what is the best way to classify a data set composed of mixed types of attributes, for example, textual and numerical. I know I can convert textual to boolean, but the vocabulary is diverse and data become too sparse. I also tried to classify the types of attributes separately and combine the results through meta-learning techniques, but it did not work well.

Answer 1

Christopher's answers seem very reasonable. In particular tree based methods do well with this sort of data because they branch on discriminative features. It's a little hard to say without knowing your specific application, but in general if you think that some of your features might be significantly more discriminative than others, you could try some dimensionality reduction techniques to clean this up a bit.

Also if you use a dimensionality reduction technique you end up getting a slightly more robust format for your feature vector (they generally end up being straight numerical vectors instead of mixed data types), which might let you leverage different methods. You could also look into hand engineering features. With properly hand engineered features Random Forest will get you very close to state of the art on most tasks.

Answer 2

It is hard to answer this question without knowing more about the data. That said, I would offer the following advice:

Most machine learning techniques can handle mixed-type data. Tree based methods (such as AdaBoost and Random Forests) do well with this type of data. The more important issue is actually the dimensionality, about which you are correct to be concerned.

I would suggest that you do something to reduce that dimensionality. For example, look for the words or phrases that separate the data the best and discard the other words (note: tree based methods do this automatically).

Answer 3

With the little information you have provided regarding the nature of your data, I would advise you to follow the following approach:

1. Convert text data into categories. You can try different alternatives for how much information the categories should contain, but specific categories have to exist for each variable. As an example, I will assume a variable that came from a text field of a survey questionnaire regarding preferable way of people to get to work.

   At first, we need to make sure that answers with similar meaning are written on the same way and belong to the same category (e.g. "by bike", "cycling", "by bicycle" all have the same meaning). Then you can try further merging into less detailed categories (e.g. merge "tram", "metro" and "bus" into "Means of public transport") or even more (e.g. "Walking", "Jogging", "Cycling" into "Physical activity") depending on what you are trying to find out.

   You can even put some different combinations in your dataset and then the next steps will determine which ones will be used for the analysis. In cases where the text data can be "translated" in ordered variables make sure you do this (e.g. if you have "small, medium, high" transform it to "1,2,3").

2. Turn your categorical variables (not the ordinal ones) into dummy (binary) variables. Most of classification/feature selection algorithms do this automatically, but make sure this is the case with the ones you select. I realise that the dimensionality of the data will become quite big at this point, but this will be handled in the next step.

3. Apply a feature selection/dimensionality reduction technique on your data. You can find a useful review of such techniques here. If you are using Python, sklearn tools give you a lot of options (see more details here). Make sure you use a technique that also considers multicollinearity. I would try Principal Component Analysis or a tree-based algorithm.

4. For classifying the data, I would go with Decision Tree Classifier (also available via sklearn). It also performs feature selection setting importance weights to the features. You can set the level of detail on the generated tree depending on your options (e.g. max_depth, min_samples_split) Make sure to adjust the level of detail based on cross-validation to avoid overfitting.

Answer 4

I recently started trying to solve a similar problem, using both categorical and acoustic/time series features for classification. Here are a few suggested approaches in order of difficulty. Some of them overlap with previous answers, as some previous answers were really good.

1. Similar to what others have suggested, convert the text/categorical/non-numerical values into numerical values. Then simply append the newly transformed features with the original numerical features. This has it's obvious benefits in that it is super easy to implement; however, it is severely flawed in that it combines completely different features together. This can be particularly bad if the original numerical features are long sequences of values, at which point the categorical features will be deprioritized by the most classifiers.
2. Use dimensionality reduction techniques (Principle Component Analysis, Clustering approaches like Gaussian Mixture Models; Kmeans; etc., Linear Discriminant Analysis, etc) on the numerical features and then append with the text features that you have transformed into numerical features. The dimensionality reduction may help to prevent the problem with the first approach by ensuring the numerical features don't overpower the text features.
3. Train separate models on the text features and numerical features. One (or several) on the text features, and another (or others) on the numerical features. Use whatever models you want, as long as they generate probability scores. Then you can use an ensemble technique to combine the predicted probabilities. This approach is flexible and you may find that, depending on the nature of the text data and/or the numerical data, it may be useful to have several different models for either. Or not. Who knows! Like the other answers have said, with little information on the nature of the data, these approaches must be left vague.
4. Use a mixed data clustering approach. This paper is a bit advanced, but if you can this off, you may get some really good results.
5. Deep learning approaches that allow for multiple input modalities. Here is a paper that used fusion neural networks.