I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string objects into categories, then using these new defined types directly when reading the data. I would even be interested in going further (typically rounding some floats to downcast them into int).

However I suspect that it may not have so much impact on the rest of my sklearn Pipeline (pre-processing of my data types ints, floats and categories, and calibration of a small ANN). More specifically :

  • I can't find anything in the documentation, but there seems to be a tremendous amount of discussion on GitHub on which transformer can handle float32. This might limit the usefullness of downcasting my float, but also for some transformation (min/max scaler, logarithmic transformation) might 'upcast' my ints into floats.

  • Category seems handled like objects in the pre-processing step. Those features mainly go trough categorical encoding. So as far as the labels are handled correctly, I don't think it should have much impact.

  • It's unclear to me how those preprocessed data are handled trough the Main model fitting sequence. From litterature, it seems that the float precision may have a significant impact overall (reducing float precision, without getting rid of too much information, can improve speed of calibration and overall statistical performance of the model). But it's unclear how to ensure that enough, but not too much, float precision is used trough the whole Sklearn process.

So, quite generally speaking, what are the steps to ensure that Sklearn is efficient in terms of memory usage and precisions of operations ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.