1
$\begingroup$

I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string objects into categories, then using these new defined types directly when reading the data. I would even be interested in going further (typically rounding some floats to downcast them into int).

However I suspect that it may not have so much impact on the rest of my sklearn Pipeline (pre-processing of my data types ints, floats and categories, and calibration of a small ANN). More specifically :

  • I can't find anything in the documentation, but there seems to be a tremendous amount of discussion on GitHub on which transformer can handle float32. This might limit the usefullness of downcasting my float, but also for some transformation (min/max scaler, logarithmic transformation) might 'upcast' my ints into floats.

  • Category seems handled like objects in the pre-processing step. Those features mainly go trough categorical encoding. So as far as the labels are handled correctly, I don't think it should have much impact.

  • It's unclear to me how those preprocessed data are handled trough the Main model fitting sequence. From litterature, it seems that the float precision may have a significant impact overall (reducing float precision, without getting rid of too much information, can improve speed of calibration and overall statistical performance of the model). But it's unclear how to ensure that enough, but not too much, float precision is used trough the whole Sklearn process.

So, quite generally speaking, what are the steps to ensure that Sklearn is efficient in terms of memory usage and precisions of operations ?

$\endgroup$
1
$\begingroup$

Overall, scikit-learn is designed to be not inefficient but it's goal is not hyperefficiency. If your goal is hyperefficiency then you should switch to a different machine learning package.

The goal of scikit-learn is to provide a high level interface with machine learning and handle the implementation details "under the hood". That is why there is little documentation about how different datatypes are handled. If you are interested in how different algorithms handle different datatypes, you have to look directly at the source code. All of the scikit-learn source code is on GitHub.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.