I have a dataset of around 10 million rows and around 10 columns. I have missing data that occurs sporadically across 4 of these variables. What technique would you recommend I use in sklearn to fill these gaps?

I see most of the techniques in sklearn are for imputing 1 specific variable that you have missing data in. In this case there are 4 variables that are missing, sometimes individually, sometimes a number of them at the same time.


Welcome to the site!

If I understand your question correctly you mean to say that you are facing issue with replacing the missing values.

Firstly, we cannot use the Same Technique to replace missing values for all the features. It depends on the Feature and Business Understanding. Let us consider a scenario, where you are having missing vales in your Income feature. Now to replace all the missing values with mean but that might work in your scenario or not. It should be your Understanding on Data and if it works for your business.

Another scenario, here this is a categorical variable. The field is room category in reservation data. Here if the data is missing then you can replace them with others/other rooms so on. All these depend on your business.

Below links would give you a better idea about different methods and how to choose the one suits your business:

  1. Link-1
  2. Link-2
  3. Link-3,(in this link go for second segment, under the table of content)

Do let me if you have any issues.

| improve this answer | |
  • $\begingroup$ Hi, thanks for your welcome. You do understand correctly that I wish to replace these missing values. I wish to replace them based on the nearest neighbor from the complete dataset with no missing variables. Would Knn be a suitable solution for this? In practice how would I go about doing this in sklearn. $\endgroup$ – Tim Nov 22 '17 at 12:45
  • $\begingroup$ yeah you can use that too. That would be helpful. I would say you should try and see if it is fitting well in the data. If yes, it is a good approach else you need to try the above methods mentioned. Yes, replacing with with KNN is a good method. $\endgroup$ – Toros91 Nov 23 '17 at 2:43
  • $\begingroup$ If my answer was helpful, +1 is appreciated. $\endgroup$ – Toros91 Mar 7 '18 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.