I'm currently working on the Titanic dataset from Kaggle. The features consist of both numerical and categorical variables and I've also engineered a few categorical variables using original features. I want to know what might be the best practice in terms of encoding the categorical variables and scaling features.
I know there are a few ways to encode categorical variables including:
- Ordinal Encoding
- One Hot Encoding
- Binary Encoding
- Hash Encoding
I know that Ordinal Encoding is undesirable sometimes as it makes the assumes that there's an ordered relationship between feature classes. I haven't gotten around to trying Binary and Hash encoding yet, and have done most of my work using One Hot Encoding.
Now I run into another issue while using OHE. I have a pipeline that handles data preparation, feature selection/extraction and prediction that I run grid search on. So what happens is for different folds, the number of features generated through OHE are different as in certain folds one or more features' classes are likely being missed. I wanted to ask how this impacts performance and is there any other way I can go about OHE to avoid this? One way I was thinking is to handle it before data is split into a train and test split- would this work? Since in preprocessing like imputation the fitting is to be done using only train data and then used to transform the validation and test data, I've constructed the pipeline as such (train test split -> fit using train data -> transform test/val data when predicting).
cat_pipeline = Pipeline([ ('Selector', DataFrameSelector(cat_attribs)), ('Imputer', DataFrameImputer()), ('CategoryEncoder', cat_enc), ]) num_pipeline = Pipeline([ ('Selector', DataFrameSelector(num_attribs)), ('Imputer', Imputer(strategy='median')), ('Log', log_apply()), ('MinMaxScaler', MinMaxScaler()), ]) full_pipeline = FeatureUnion([ ('NumPipeline', num_pipeline), ('CatPipeline', cat_pipeline), ]) Prepare = Pipeline([ ('AttributesAdder', CustomAttributesAdder()), ('Scaler', full_pipeline), ]) PrepSelPred = Pipeline([ ("Prepare", Prepare), ("TopFeatureSelector", KernelPCA(n_components=3)), ("Classifier", RandomForestClassifier()), ])
As you can see from above I use a MinMaxScaler and normalize my numerical attributes rather than standardize them. This is so that they match the scale of the OH features. I've read that standardization is preferable for ML algorithms but also that it is inadvisable on OH vectors. What would be the best approach here?
I've tried both Feature Selection (using SelectFromKBest() and SelectFromModel()) as well as dimensionality reduction using PCA, though not together. Is one preferable to another, and do they work well in succession?
Given the nature of the problem and the kind of features I am dealing with, are there any algorithms that are better suited to the task. If I stick with OHE for example, are there algorithms that perform better when the majority of features are sparse?