I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose. For smaller data sets (MB to GB), I use modules in Python, R, or just good ole scripting with pipes for stdin/stdout, often with a relational database for storage.

My question is: what tools are available on Linux to handle terabyte+ sized data sets in other fields, such as data science, finance, web analytics, etc? Are there any standard references on them, such as K&R for learning C?

  • $\begingroup$ Distributed databases, cluster computing. What are your specific problems, bottlenecks? Welcome to DataScience.SE! $\endgroup$ – Emre Jul 6 '16 at 17:46

Browse other questions tagged or ask your own question.