15

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak. Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially ...


13

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart. A common step I think you have missed is imputation of missing values. I would put that before feature engineering. Overall I ...


7

There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR). Your case is the second, where according to wikipedia it: occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information This means that the ...


5

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often better than any imputation done in preprocessing. The way they achieve this is by using the splitting of the decision trees they are built on. The missing values ...


5

I agree with Simon's advice. I find that the gains that you obtain from using any external method of imputation is often inferior to an internal method, and on top of this, exposes you to even more potential of severely screwing up with respect to data leakage. That being said, besides using an algorithm that automatically handles missing data for you (...


5

Generally speaking you have two options: impute the missing data discard the missing data Due to the fact that ML models perform better with more data, the former is usually preferred. However, you should keep in mind that the imputed data should not affect the distribution of the feature. This is especially the case with features with a high percentage ...


5

I've got the same issue today, and it's a shame your post got no answers. I think this question is not well addressed in the sklearn documentation. I can show you my workaround to this issue: headers = X.columns.values X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) empty_train_columns = [] for col in X_train.columns.values: # ...


3

For the missing data problem, one thing to be aware of, is the missingness mechanism. Depending of the dataset, the NA's (Missing Values) you have could be a result of a condition of the phenomenon and you shouldn't impute then using mean but maybe. Besides, for the dependent variable, if you want to train a model with the independent ones to predict this, ...


3

As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are: Mean/ Mode/ Median Imputation: Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the ...


3

If you impute/standardize before splitting and then split into train/test you are leaking data from your test set (that is supposed to be completely withheld) into your training set. This will yield extremely biased results on model performance. The correct way is to split your data first, and to then use imputation/standardization (the order will depend on ...


3

It depends on your knowledge of the problem. First, you should classify why is it missing?? Structurally missing data Structurally missing data is data that is missing for a logical reason. In other words, it is data that is missing because it should not exist.check this Structurally missing data is data that is missing for a logical reason. In other words,...


3

Yes, these are the basics step. Then in each step there is a lot more. If you want to get a bit deeper you can follow this book of Andriy Burkov of Machine Learning engineering A couple notes in your process: Before get data I Will put, define the question to resolve or something similar, but maybe this parted is granted. Feature Engineering is one of the ...


3

After 12 "Choose the model with highest scores." Maybe add "create ensemble of models" and try to improve accuracy further.


2

Certain models are able to deal with missing values 'naturally', like certain tree based models. Most models however are just a mathematical function which is shaped after the training data. A very easy example would be: $$f(x) = \alpha x_1 + log(2x_2)$$ What do you do if one of them is NaN? The function is undefined and no prediction can be made. By ...


2

It may be usefull to put what you are doing in a formal context, that way you can look at some standard solutions to your problem. You have obsvations of a (random) variable X (price), together with some explanatory variables L, C, P (L="location" = city, C=company, P=price). So your data consists of cuadruples $(C_i, L_i, P_i, X_i)$. You are assuming a $...


2

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does ...


2

I am not able to understand how and what modelling techniques do we use? Every data science workflow has the follwing steps: Pre-processing (data cleaning and wrangling) Exploratory analytics Model selection Prediction and testing. (And re-iteration) (Optional) Reporting the workflow Does it depend on the data type? Yes, the entire workflow is ...


2

You can treat the missing feature as the target variable of a sub-problem and create a classifier (e.g., a linear model, SVM, etc) for it.


2

Definitions of missingness process are tricky. Missing completely at random occurs when the missingness is really at random (MCAR; e.g. when conducting a survey there are error in the data entry process). Missing at random (MAR) occurs when the missingness is not really at random, but when it could be considered at random conditioning on what is observed in ...


2

If you plan to use LightGBM or XGBoost, the advise is "Do not do anything". These methods treat NA in a specific way, different in each decision tree and the results obtained are usually much better than using imputing.


2

So if you want to impute some missing values, based on the group that they belong to (in your case A, B, ...), you can use the groupby method of a Pandas DataFrame. So make sure your data is in one of those first. import pandas as pd df = pd.DataFrame(your_data) # read documentation to achieve this Then, it is just a case of chaining a few ...


2

I think like @El Burro suggested, you I believe you should focus on feature transformation mainly. Use different techniques for different features. For straightforward features, such as occupation or gender for example, use one-hot encoding, while for others you can use some kind of hierarchical mapping-clustering (e.g. map values to groups defined by you, ...


2

Welcome to the site! If I understand your question correctly you mean to say that you are facing issue with replacing the missing values. Firstly, we cannot use the Same Technique to replace missing values for all the features. It depends on the Feature and Business Understanding. Let us consider a scenario, where you are having missing vales in your ...


2

There is no globally - one could say even locally - ideal way to deal with missing data. This aspect points to incompleteness in the data you're feeding your algorithms, and imputing is simply a technique meant to fill gaps. Data imputation's motivation is to make feature distribution in your sets the closest possible to whatever real-world distribution it ...


2

I usually use mice for missing data imputation. It relies on chained equations and performs very well. It also has a random forest method, but typing methods(mice) you will find a list of available options, that can overcome the issues encountered by standard approaches such as logistic regression techniques. Another option is using the na.impute option ...


2

You should try all of: Using a classifier that can handle missing data. Decision trees can handle missing features both in input and in output. Try xgboost, which does great on kaggle competition winners. See this answer. Off the shelf imputation routines Writing your own custom imputation routines ( this option will probably get you the best performance) ...


2

Welcome to the site! The usual approach to missing values is to handle them manually. There are a few algorithms which can do this automatically, such as LightGBM and XGBoost, but in most cases it's better for model performance to decide on how you should indicate that a value is missing in your data. For example in with a Pandas dataframe in Python, I ...


2

If you want to do TargetEncoder you have to impute the missing values first. First of all you should convert your categorical features into int, using LabelEncoder or OrdinalEncoder. I used a huge numeric value (my choice : 8888) in order to fill the NaN values, before running OrdinalEncoder. Then transform your matrix to int, it will be more efficient. For ...


2

If you are planning to go with imputing values as a pre-processing step, I just want to add that it is better to impute missing values of multiple variables jointly because you will be able to better preserve relationships between those variables. The MICE algorithm should automatically take care of that for you. Here is an example of how MICE works when ...


2

Imputing missing data (that is, filling in missing values with some other value) is not appropriate for analysis or regression. It would only be valuable if you were going to try to train a learning model for predictions. Putting in random or inferred values would mess with your analysis


Only top voted, non community-wiki answers of a minimum length are eligible