Skip to main content
Data Science

Return to Question

added 8 characters in body; edited tags
Source Link
Green Falcon
Green Falcon
  • 14.2k
  • 10
  • 58
  • 98

I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case, it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognizerecognized the numeric/categorical nature of some variables (and as a consequence, wrongly try to encondeencode or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course, I could turn to models which are capable of handlehandling non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on the other hand. What is the way to get over this issue?

I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognize the numeric/categorical nature of some variables (and as a consequence, wrongly try to enconde or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course I could turn to models which are capable of handle non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on other hand. What is the way to get over this issue?

I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case, it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognized the numeric/categorical nature of some variables (and as a consequence, wrongly try to encode or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course, I could turn to models which are capable of handling non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on the other hand. What is the way to get over this issue?

Source Link
Fredrik
Fredrik
  • 1k
  • 3
  • 10
  • 12

How to handle large number of features in machine learning?

I try to do normal classification on high dimensional traditional columnar data (several hundred columns). The features are of different type. In this case it's clearly out of question to examine each features one by one to figure out what are they exactly and what optimization or feature engineering could be done with them.

Still, I have to do all the necessary preprocessing steps like imputation, standardization etc. But even such basic steps like categorical feature encoding or imputation are problematic because R/Python-pandas are sometimes wrongly recognize the numeric/categorical nature of some variables (and as a consequence, wrongly try to enconde or mean-impute the NAs), not to mention other very problematic issues that could be handled if one could oversee the features one by one.

Of course I could turn to models which are capable of handle non-standardized features with NAs but this limits the number of possible models on one hand and seems me very unprofessional on other hand. What is the way to get over this issue?