1
$\begingroup$

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer.

Trying this off the bat I hit memory problems, but I think I can solve this by chunking my dataset... I was hoping someone could confirm my method is sound and I haven't hit any gotchas.

The sklearn KNNImputer has a fit method and a transform method so I believe if I fit the imputer instance on the entire dataset, I could then in theory just go through the dataset in chunks of even, row by row, imputing all the missing values using the transform method and then reconstructing a newly imputed dataset.

I'm wondering if there's an issue with this method regarding the chunksize or is the transformation on each new row independent?

50% of the dataset rows are fully populated... would it be better in terms of computation to fit the imputer object on only this portion of the dataset?

$\endgroup$
2
  • $\begingroup$ Hi! Could you please tell me how you solved your problem? Right now I'm in a similar situation. $\endgroup$ May 19 at 4:27
  • $\begingroup$ Accordig to the doc KNN is recommended for less than 100k rows... scikit-learn.org/stable/tutorial/machine_learning_map/…. Did you still managed to fit the model on the whole data set ? $\endgroup$
    – Malo
    Sep 5 at 8:29
0
$\begingroup$

You could use a memmap

import numpy as np
from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'newfile.dat') # or you could use another dat file that already constains your dataset

# supposing your data is loaded in a variable named "data"

fp = np.memmap(filename, dtype='float32', mode='w+', shape=data.shape)
fp[:] = data[:]

You can check full documentation on this page, the code above is based on the documentation.

This way you:

  • only change the declaration of your data matrices, keeping you code as clean as possible
  • uses numpy built-in ndarray subclass without having to explicitly manage data retrieval from the disk
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.