0
$\begingroup$

I have a dataset with 261 predictors scraped from a larger set of survey questions. 224 have values which are in a range of scale (some 1-10, some 1-4, some simply binary, all using 0 where no value is given), and the rest are unordered categories.

I'm trying to perform classification using these predictors and identify the top n predictors. Am thinking of the following approach:

  1. convert the 224 ordered predictors into numeric, centered, and scaled.
  2. Run separate modeling (I use caret from R): one for using the numeric predictors, another using the remaining 37 categorical predictors (both cross-validated within each modeling exercise).
  3. Choose the respective best-fitting models modelN and modelC for the numeric and categorical predictors.
  4. Choose top n (say 10) predictors from model N and model C.
  5. Combine them in an ensemble model that can handle both numeric and categorical data (say, random forest).
  6. Choose top n predictors in the ensemble model.

I am going through this a roundabout way rather than directly fitting all predictors into an ensemble model to try and reduce the complexity of the problem first (and because in R, I'm having a problem with too many levels from the predictors).

Would this be a valid approach to identifying the n most salient predictors? Any possible issues to mitigate?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Ricky,

Loose thoughts:

  • Depending on the algorithm you intend to use, centering might not be a good idea (e.g. if you go for SVM, centering will destroy sparsity)
  • I would suggest not to handle ordered / unordered separately, as you are likely to miss interactions that way. If the categorical ones don't have too many possible values, randomForest in R can handle factors.
  • if that is an issue (as you seem to hint), I think you have two possibilities: binary indicators or response rates
  • if it's feasible in terms of computational cost, i would convert all factors to binaries (use sparse matrices if necessary) and then try a greedy feature selection. caret, if memory serves, has rfe or somesuch.
  • if that's too much trouble, try calculating response rates / average values per factor level (I don't see any info whether your problem is classification or regression): you split your set into folds, and then for each fold fit a mixed effects model (e.g. via lme4) on the remainder, using the factor of interest as the main variable. It's a bit of a pain to setup all the cv correctly, but it's the only way to avoid leaking information.

Hope this helps, K

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks @kpb. My problem is classification. And you are right, randomForest broke because I had too many possible values over the range of the categorical predictors. Would converting the categorical predictors to binary sparse matrix myself, rather than letting caret do the grunt work behind the scene on a data frame, allow randomForest to handle more possible levels? If so, I can give it a try. And I agree about issue with centering; I may resort to normalizing the numerics into a 0-1 range instead. $\endgroup$
    – Ricky
    Commented Jul 28, 2015 at 15:47
  • $\begingroup$ 1. converting to sparse format (yourself or caret, doesn't matter really) will help, but randomForest is not super efficient - you want tree-based methods, i would suggest xgboost 2. normalizing to 0-1 (i am guessing by min/max scaling and subtraction) has exact same problem. if you insist on normalizing, just divide by standard deviation. BTW, this is only important for logistic regression, svm and the like - trees are invariant under monotone transforms, so for xgboost / randomForest normalization is not necessary $\endgroup$
    – kpb
    Commented Jul 29, 2015 at 9:07
0
$\begingroup$

Just some random thoughts

Do you have a mathematical model to base on? For example, you want to predict how pressure is changing against temperature. You won't drop any predictor that is related to 'temperature', no matter how remote it is. If so, that should govern your choice of predictors and you should start with that as that would lend more credibility to your final model.

If not and you just want an algorithm to pick the best predictors, have you thought of running a regression model with L1 norm on? This will drive out insignificant factors and you can start with that set as a basis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ No mathematical model unfortunately. The data is from a survey, so a lot have to do with subjective perspective. I am considering regression-based feature eliminations for the numeric models. But I'm not sure how well it will work combined with the other pure categorical predictors. I believe in a regression, each level in a category will be split into its own yes/no variable, so lasso may eliminate one level but retain others. My aim is to identify the category as a whole to retain or not. Hence my thought of doing those I can represent numerically separately from the pure categories. $\endgroup$
    – Ricky
    Commented Jul 28, 2015 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.