Firstly, filling a column that has most values as null might not make sense. You'll have to look at the nature of the business problem you face. You can try dropping those columns. You can drop the rows which contain null values if the number of null values for columns in that row falls below a particular threshold (based on your data, of course).
Use: dataset.dropna(inplace=True) to achieve this
That being said, there are multiple ways you can fill (impute) null values:
- You can try filling values based on the mean (or median or mode): dataset.fillna(dataset.mean(), inplace=True). However, this approach has it's limitations if your column is non-numeric.
If you want to use a sklearn implementation, try using the Imputer method.
from sklearn.preprocessing import Imputer
impute = Imputer(missing_values='NaN', strategy='mean', axis=0)
impute.fit(dataset)
dataset = impute.transform(dataset)
- For categorical columns, try using fillna with the most frequent value in a column:
dataset[column].fillna(value=df['column1'].value_counts().index[0], inplace =True)
more information on this method can be found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html
- Try using a KNN to fill in missing values. There's a package called fancyimpute that will allow you to do this:
from fancyimpute import KNN
dataset = pd.DataFrame(KNN(number_of_neighbors).complete(dataset))
You'll need to choose the number of neighbors appropriately. Also, fancyimpute needs a numpy array as input.
There are many other methods, but these should do the trick.
