0
$\begingroup$

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN values in 6-7 columns. I want to fill those NaN with mode() / most_frequent() of each column. But it should be filled with two different values depending on the target. So, NaN for all those with Target=1; mode() of values with Target=1 should be filled and similarly, for all those with Target=0; mode() for of values with Target=0.

I've tried the following code:

train_df[train_df.TARGET==1].fillna(train_df[train_df.TARGET==0].mode(),inplace=True)

and

train_df[train_df.TARGET==0]=train_df[train_df.TARGET==0].fillna(train_df[train_df.TARGET==0].mode())

Both of these codes didn't worked. I was still left with the same amount of NaN values.

But, when I run this- train_df[train_df.TARGET==0].fillna(train_df[train_df.TARGET==0].mode()).isnull().sum()

I observed all the missing values do get filled where Target=0.

Sometimes, I get this Warning:

/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py:6287: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._update_inplace(new_data)

I'm stuck at this for a long time and don't know how to proceed. Please suggest some solution to this.

Thanks

$\endgroup$
  • $\begingroup$ what's wrong with this ? train_df.fillna(train_df[train_df.TARGET==0].mode(),inplace=True) $\endgroup$ – quester Oct 12 '19 at 19:33
  • $\begingroup$ I don't know. Even after running this, I have the same amount of missing values. $\endgroup$ – Ammar Akhtar Oct 12 '19 at 19:36
  • $\begingroup$ train_df.fillna(train_df[train_df.TARGET==0].mode()).isnull().sum() ? $\endgroup$ – quester Oct 12 '19 at 19:40
  • $\begingroup$ yeah...when I run this...the count decreases...so the missing values did get replaced but it just happens in the copy and not on the dataset. I tried equating it to the variable to the dataset variable, but no good. $\endgroup$ – Ammar Akhtar Oct 13 '19 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.