0
$\begingroup$

I have a python pandas dataframe representing a superset. The data contains a lot of nulls which I want to overwrite with real values.

the superset has:

  • both numerical and categorical data
  • some nulls for most attributes
  • multi class attributes (attributes can have multiple values)
  • It is not time dependent
  • each row is a unique person

It would be neat to use machine learning to fill in the nulls, any recommendations on how I can do this?

(I guess that I can tranform the categorical data to numerical if required)

$\endgroup$
0
$\begingroup$

Firstly, filling a column which has most values as null might not make sense. You'll have to look at the nature of the business problem you face. You can try dropping those columns. You can drop the rows which contain null values if the number of null values for columns in that row fall below a particular threshold (based on your data, of course). Use: dataset.dropna(inplace=True) to achieve this

That being said, there are multiple ways you can fill (impute) null values:

  1. You can try filling values based on the mean (or median or mode): dataset.fillna(dataset.mean(), inplace=True). However this approach has it's limitations if your column is non-numeric.

If you want to use an sklearn implementation, try using the Imputer method.

from sklearn.preprocessing import Imputer

impute = Imputer(missing_values='NaN', strategy='mean', axis=0)

impute.fit(dataset)

dataset = impute.transform(dataset)

  1. For categorical columns, try using fillna with the most frequent value in a column:

dataset[column].fillna(value=df['column1'].value_counts().index[0], inplace =True)

more information on this method can be found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

  1. Try using a KNN to fill in missing values. There's a package called fancyimpute which will allow you to do this:

from fancyimpute import KNN

dataset = pd.DataFrame(KNN(number_of_neighbors).complete(dataset))

You'll need to choose the number of neighbors appropriately. Also, fancyimpute needs a numpy array as input.

There are many other methods, but these should do the trick.

$\endgroup$
  • $\begingroup$ First of all thank you for a good and well explained answer. Most of the values are not null for any column, then I would as you say drop them. I belive that both solution 1 and 2 are too simple. Solution 3 sounds however very interesting. I will try it out. $\endgroup$ – Filip Eriksson Sep 4 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.