Working with large datasets python

Question

My data file is of 4 GB (json), I need to apply everything on this dataset (from applying clustering/ML algorithm to wrangling it) like the way people do with pandas/scikit. But it is difficult to work with them on my local machine. I tried using dask and blaze but could not find them efficient for my case because of their limited functionalities. I don't wanna use hadoop because it is not meant for such small datasets.

I wanted to know approaches for working with not so large datasets with python and applying ML algos on them. Just like the way you work on small datasets using pandas (if any exists).

Answer 1

With today's standard amount of RAM for a personal desktop or laptop computer (usually 8Gb or more), there should be little problem in handling 4Gb in memory. If your hardware setup is short in memory and your ML algorithms make your SO start paging, the best hint is to use algorithms and settings that do not copy the data around, but do everything in place.

In pandas, many operations provide a boolean parameter precisely called inplace that controls this behaviour, e.g. DataFrame.reset_index. Numpy also has support for in-place operations, e.g. sort.

Take into account that:

• inplace=True does not guarantee that the operation is actually performed in place, it is just a hint that may or may not be honored.
• for some operations. inplace=True can worsen performance, like dropna.

All that said, in my experience inplace=True actually makes a difference in memory consumption and can be used to avoid paging and therefore enabling computations to take place completely in RAM for datasets like yours.

Finally, be careful with inplace default values. As noted in this warning in the pandas documentation:

Warning: For backwards compatability, inplace defaults to True if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True

Answer 2

There are at least two uncomplicated ways to deal with this issue. If your data has a lot of zeros etc. it might be worth trying to load it as a sparse dataframe.

One thing that always works is to process your data in an iterative fashion. For most datatypes pandas can let you iterate through your file, e.g. with pandas.read_csv(data, iterator=True). This is not available for JSON data, but you can use the ijson package instead.

Many scikit-learn algorithms can also be trained iteratively on one or a few samples (mini-batches) at a time. In particular, the flexible linear models with stochastic gradient descent (SGDClassifier and SGDRegressor). For clustering, you can use MiniBatchKMeans. You can read more about this in the documentation.

If your data fits almost into memory, you may just read in as much as possible. If most of it doesn't fit, there is probably no way that lets you use pandas/sklearn without some adjustments.

Answer 3

One thing you can do is explicitly set types when loading the dataset using the dtypes option i.e. (... dtype={'ORDER NO': np.int32, 'CUSTOMER': str} ...) - that could potentially save memory when you load the file.