I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose. For smaller data sets (MB to GB), I use modules in Python, R, or just good ole scripting with pipes for stdin/stdout, often with a relational database for storage.

My question is: what tools are available on Linux to handle terabyte+ sized data sets in other fields, such as data science, finance, web analytics, etc? Are there any standard references on them, such as K&R for learning C?


closed as too broad by Spacedman, Sean Owen Jul 6 '16 at 22:23

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Distributed databases, cluster computing. What are your specific problems, bottlenecks? Welcome to DataScience.SE! $\endgroup$ – Emre Jul 6 '16 at 17:46

Browse other questions tagged or ask your own question.