totally new to larger than ram datasets but I have csv files that are about 100 gb each with around 300 million rows each (just two of them).

What I am looking for is something like: don't go analyzing more than 1 terabyte of data on a laptop. Or don't analyze data that is more than 10x your ram or the wait times will prove frustrating. This is what I mean by rules of thumb.

I have a mac laptop with an i5, 8gb ram, ssd is it reasonable to process this data (100gb per file and I have 2 files) with either dask or blaze in python?

I have tried and could read in the csv but when doing simple trials like dropping one column or finding the length of the data frame it takes at least an hour For the drop (I gave up waiting) and half an hour for the length() (that did finish). Way too slow to be productive. I changed the data format to parquet and amazingly the data is down to just 10 gb. This is the only way they the length() command ever even finished.

So my question is: are these response times normal experience in general given my hardware and csv size? Are my expectations for what dask can do way too high? Any rough guidelines on how to troubleshoot or is this clearly underpowered hardware?

  • 1
    $\begingroup$ It depends on what you want to do, and what your performance requirements are. ETL can be done in a database like postgres (relational), MapD (analytics on GPU), or Druid (analytics). Try using one If that's not fast enough then you can spin up a Spark cluster in the cloud. If you want to build a model, you might not need all that data for a prototype. $\endgroup$
    – Emre
    Commented Aug 4, 2017 at 6:04
  • $\begingroup$ the problem with a relational database is that there is a 1024 column limit. With biological data, it's real common to have > 1000 columns. I was using a RDBMS and realized that I'm going to routinely exceed 1000 columns. I was hoping that doing everything in python would be possible. $\endgroup$
    – user798719
    Commented Aug 4, 2017 at 6:11
  • $\begingroup$ Adding to @Emre's suggestion, you can store the data in a DB which is suitable to your use case (columnar databases like HBase), wherein you can store the related columns in the table within the same column family. Chances are that you would not use a majority of the total columns at the same time. If so, a columnar DB can be quite useful. Additionally, pull the data from HBase into Spark (pySpark / Scala) and proceed. ^may not be the exact solution you sought, but one alternative $\endgroup$
    – vsdaking
    Commented Aug 4, 2017 at 11:58
  • 1
    $\begingroup$ Did you try any databases in the meantime? $\endgroup$
    – Emre
    Commented Aug 22, 2017 at 2:15
  • $\begingroup$ yes, I tried SQL Server. Problem is that I can not exceed 1024 columns for a table. So that's going to be a big issue when there can easily be 10,000 features to feed into a neural network. $\endgroup$
    – user798719
    Commented Aug 22, 2017 at 2:48

1 Answer 1


I am not an expert but this is how I see things: use spark. Pyspark seems to be the flavor to use since it's python and that's the language that data science has settled on.

Dataset sizes are expanding way faster than RAM is on laptops. What "they" don't tell you is that scikit learn, pandas, R, etc all require your dataset and all intermediate steps to fit inside RAM. If you have an 8gb dataset, and you pivot it, your intermediate step could easily be say 30gb. Even if it's temporary and your final cleaned dataset to be fed into. A neural network shrinks back down to say 6gb, too bad. All of your steps must fit inside RAM.

In the real world, and even the limitation that datasets be limited to RAM sizes is quite limiting. This is what I wished I had known earlier. It's like saying here is a grocery store which is all yours (scikit learn has tons of features ) but you can only leave the store with a single shopping bag full of items). I would much rather have a grocery store with the standard 100 items and let me have a buffet on just those 100.

Yes, there are some libraries here and there for scikit learn that read from files in chunks, and then you add the chunks together, etc. But you will quickly see that that is reinventing the wheel and distracts from doing data science. The fact remains that these frameworks were built for in RAM usage. It's not a bad thing at all per se, but quickly becomes limiting.

What you want is to write the code once and not have it change whether the dataset is 1gb or 100gb. This is what pyspark is for. You can just learn their API and pyspark will handle chunking and distributing the workload such that it fits in the available RAM that you have. Of course if you are RAm limited computations take longer but this is ok. At least you will get an answer.

The downside is that if you want the very much appreciated ability to not have to worry about "is this dataset too large for my ram" question then you need to stay in pyspark's API. If they don't have a feature that say scikit learn has, or keras has, then too bad. Write your own or work around it.

Thankfully the API looks very complete.

Where does it lack? Well for say neural networks you aren't going to have all kinds of special activation functions and customization that you can have with Keras.

But in my experience and it really is very little, more data and better data beats the specific algorithm much of the time. I only mean to say that I would rather be able to process tons of data and use a vanilla classifier than be limited to in RAM dataset sizes and while being able to optimize 1000 hyperparameters.

You also don't want to be switching between different aPIs all the time as a beginner.

So if there is any chance that your dataset will be larger than RAM (I think that likelihood is a near certainty in the years ahead) I would most definitely learn pyspark. Even before scikit learn.

  • $\begingroup$ You should talk about the storage; Spark is just the processing engine. $\endgroup$
    – Emre
    Commented Aug 31, 2017 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.