With today's standard amount of RAM for a personal desktop or laptop computer (usually 8Gb or more), there should be little problem in handling 4Gb in memory. If your hardware setup is short in memory and your ML algorithms make your SO start paging, the best hint is to use algorithms and settings that do not copy the data around, but do everything in place.
In pandas, many operations provide a boolean parameter precisely called inplace
that controls this behaviour, e.g. DataFrame.reset_index. Numpy also has support for in-place operations, e.g. sort.
Take into account that:
inplace=True
does not guarantee that the operation is actually performed in place, it is just a hint that may or may not be honored.
- for some operations.
inplace=True
can worsen performance, like dropna
.
All that said, in my experience inplace=True
actually makes a difference in memory consumption and can be used to avoid paging and therefore enabling computations to take place completely in RAM for datasets like yours.
Finally, be careful with inplace
default values. As noted in this warning in the pandas documentation:
Warning: For backwards compatability, inplace defaults to True if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True