New answers tagged

1

If you have the whole population and do not want to predict anything, as stated in the question link you gave, then it is fine to use the whole population for preprocessing as it will give better results during training (which is our only goal). But if you do not have the whole population and want to do predictions, it is best to keep the test set untouched ...


0

I had the same problem in one of the datasets I was using, and the answer is focus more on feature transformation. If you simply include all the features of your dataset for encoding, you would probably end up with more no of columns than you rows! I am positive there might be many features in you dataset that can be grouped in one column, some features can ...


0

Scikit-learn has an impute module that supports many of those imputation methods.


0

Answer Take two step approach. What should be imputed at missing data points? Can we add a new column to indicate missing second transaction? E.g. First, impute the missing values with a negative value. Second, create additional feature say Col1_flag which will have binary value yes and No. Yes indicating missing second transaction- the reason for NaN, and ...


0

An interesting solution is to measure uncertainty of your model in order to quantify the quality of each result, because some cases would have a low uncertainty, some other cases wouldn't. Here is a paper that explains how to do it: https://arxiv.org/abs/1506.02142


4

The best technique to handle missing data comes from understanding your data better Step 1: Do a exploratory data analysis along with your missing data. Python package missingno helps to Visualize data with all missing values. Here's the python package link Click here for missingno youtube demo Step 2: Do analyze the nature of missingness to help you better ...


1

If 80% of the values for a feature are missing, you probably should drop that feature. There is just not that much signal in the remaining 20% of values. Also, the reason that the data is missing will most likely impact the modeling. Many of the other features also have more data missing than the present. Even at a 40-36% missing rate, the feature is suspect....


1

Columns 1 to 6: if the data is missing because it does not exist, does that tell you something about the variable/target/customer? If so, you want to preserve that information in your imputation. For instance, if Column X is the average number of days since the last transaction, does a missing value mean that this is a new customer? If that's the case, then ...


0

It's a really easy problem. Suppose you want to fit an imputer to your X_train and test data and keep the column names of X in the imputed results: imp = SimpleImputer() X_train = pd.DataFrame(imp.fit_transform(X_train), columns = X.columns) X_test = pd.DataFrame(imp.transform(X_test), columns = X.columns)


Top 50 recent answers are included